danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Refresh benchmarks doc and remove unused eval datasets

Browse source
This commit is contained in:
Yiorgis Gozadinos 2026-05-29 14:23:00 +03:00
parent 6ac7d5a4be
commit 00f2a40a60
No known key found for this signature in database
7 changed files with 22 additions and 404 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

95
docs/benchmarks.md
View file

@ -36,13 +36,6 @@ Active datasets:
| `orb_text` — OpenRAG Bench, text embedder (`qwen3-embedding:4b`) with VLM picture descriptions baked into chunk content | ~18 GB |
| `orb_multimodal` — OpenRAG Bench, multimodal embedder (`qwen3-vl-embedding-8b`); picture vectors live in the same space as text for cross-modal retrieval | ~16 GB |
Inactive (kept downloadable, not currently maintained):
| Dataset | Size |
|---------|------|
| `repliqa` | ~30MB |
| `hotpotqa` | ~331MB |
After downloading, run benchmarks with `--skip-db` to use the pre-built database:
```bash
@ -116,18 +109,6 @@ This is computed alongside QA accuracy from the same skill run, no extra invocat
Numbers measured under the current pinned judge (`ollama:qwen3.6`) on a recent `haiku.rag` version.
### Wix
[WixQA](https://huggingface.co/datasets/Wix/WixQA) is real customer support questions paired with curated answers. 200 cases.
`evaluations run wix --target rag-skill` runs the RAG skill end-to-end and produces both QA accuracy and a citation retrieval metric (`cited_map`) computed from the URIs the skill registered via the `cite` tool against the gold `expected_uris`.
| Skill model | Reranker | QA accuracy | Mean `cited_map` |
|------------------------------|------------------------|-------------|------------------|
| `vllm:Gemma-4-26B-A4B-NVFP4` | `mxbai-rerank-base-v2` | 0.87 | 0.38 |
*Measured on haiku.rag v0.48.0 with `qwen3-embedding:4b` (vLLM, dim 2560), `chunk_size=256`, `search.limit=5`. Judged by `vllm:Qwen3.6-35B-A3B-NVFP4` (qwen3.6 family, NVFP4 quant served via vLLM rather than the default Ollama). 172 / 198 completed cases (2 errored).*
### OpenRAG Bench (ORB)
[OpenRAG Bench](https://huggingface.co/datasets/vectara/open_ragbench) contains ArXiv research papers with multimodal question-answering pairs. Queries include both text-based and image-based questions, testing retrieval and reasoning over visual content like figures, charts, and diagrams. Each query maps to one relevant document.
@ -173,76 +154,14 @@ Two approaches are benchmarked separately:
*qwen3 measured on haiku.rag v0.48.0 over all 3045 cases; nemotron on v0.50.0 over 2836 / 3045 cases (run stopped early; numbers stable from ~12% onward). Both with `mxbai-rerank-base-v2`, judged by `vllm:Qwen3.6-35B-A3B-NVFP4`.*
## Inactive datasets
### Wix
The benchmarks below are not currently maintained. Numbers were measured against earlier `haiku.rag` versions and an older pinned judge (`ollama:gpt-oss`), before the skill workflow became the only path. Retrieval tables don't depend on the judge, but the QA tables aren't reproducible against the current skill-only setup. We may revive them.
[WixQA](https://huggingface.co/datasets/Wix/WixQA) is real customer support questions paired with curated answers. 200 cases.
### RepliQA
`evaluations run wix --target rag-skill` runs the RAG skill end-to-end and produces both QA accuracy and a citation retrieval metric (`cited_map`) computed from the URIs the skill registered via the `cite` tool against the gold `expected_uris`.
[RepliQA](https://huggingface.co/datasets/ServiceNow/repliqa) contains synthetic news stories with question-answer pairs. We use `News Stories` from `repliqa_3` (1035 documents). Each question has exactly one relevant document, so we use MRR for retrieval evaluation.
| Skill model | Reranker | QA accuracy | Mean `cited_map` |
|------------------------------|------------------------|-------------|------------------|
| `vllm:Gemma-4-26B-A4B-NVFP4` | `mxbai-rerank-base-v2` | 0.87 | 0.38 |
#### Retrieval (MRR)
| Embedding Model | MRR | Reranker |
|-------------------------------|------|----------|
| Ollama / `qwen3-embedding:8b` | 0.91 | - |
*Measured on haiku.rag v0.19.6.*
#### QA Accuracy
| Embedding Model | QA Model | Accuracy | Reranker |
|------------------------------|----------------------------------|----------|------------------------|
| Ollama / `qwen3-embedding:4b` | Ollama / `gpt-oss` - no thinking | 0.82 | None |
| Ollama / `qwen3-embedding:8b` | Ollama / `gpt-oss` - thinking | 0.89 | None |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` - thinking | 0.85 | None |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` - thinking | 0.87 | `mxbai-rerank-base-v2` |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3:0.6b` | 0.28 | None |
*Measured on haiku.rag v0.19.6, judged by `ollama:gpt-oss`.*
Note the significant degradation when very small models are used such as `qwen3:0.6b`.
### HotpotQA
[HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) is a multi-hop question answering dataset requiring reasoning over multiple Wikipedia paragraphs. Each question requires evidence from 2+ documents, making it ideal for testing retrieval and reasoning capabilities. We use MAP for retrieval evaluation since queries have multiple relevant documents.
#### Retrieval (MAP)
| Embedding Model | MAP | Reranker |
|----------------------|------|----------|
| `qwen3-embedding:4b` | 0.69 | none |
*Measured on haiku.rag v0.20.2.*
#### QA Accuracy
| Embedding Model | QA Model | Accuracy |
|----------------------|--------------------------|----------|
| `qwen3-embedding:4b` | `gpt-oss:20b` - thinking | 0.86 |
*Measured on haiku.rag v0.20.2, judged by `ollama:gpt-oss`.*
### Wix (historical, plain text and HTML)
Earlier Wix runs measured under different chunk settings and reranker combinations, against the older `gpt-oss` judge.
#### Retrieval (MAP)
| Embedding Model | Chunk size | MAP | Reranker | Notes |
|------------------------|------------|------|------------------------|------------------------------|
| `qwen3-embedding:4b` | 256 | 0.34 | None | html, `chunk-radius=2` |
| `qwen3-embedding:4b` | 256 | 0.39 | `mxbai-rerank-base-v2` | html, `chunk-radius=2` |
| `qwen3-embedding:4b` | 256 | 0.43 | None | plain text, `chunk-radius=0` |
| `qwen3-embedding:4b` | 512 | 0.45 | None | plain text, `chunk-radius=0` |
*Measured on haiku.rag v0.27.2.*
#### QA Accuracy
| Embedding Model | Chunk size | QA Model | Accuracy | Notes |
|----------------------|------------|-----------------------------|----------|------------------------------|
| `qwen3-embedding:4b` | 256 | `gpt-oss:20b` - no thinking | 0.80 | html, `chunk-radius=2` |
| `qwen3-embedding:4b` | 256 | `gpt-oss:20b` - no thinking | 0.83 | html, `chunk-radius=2`, `jinaai/jina-reranker-v3` |
*Measured on haiku.rag v0.27.2, judged by `ollama:gpt-oss`.*
*Measured on haiku.rag v0.48.0 with `qwen3-embedding:4b` (vLLM, dim 2560), `chunk_size=256`, `search.limit=5`. Judged by `vllm:Qwen3.6-35B-A3B-NVFP4` (qwen3.6 family, NVFP4 quant served via vLLM rather than the default Ollama). 172 / 198 completed cases (2 errored).*

22
evaluations/README.md
View file

@ -8,9 +8,7 @@ This package is not published to PyPI and is only used for development and testi
Contains evaluation scripts for benchmarking RAG retrieval and QA performance. Available datasets:
- RepliQA (`repliqa`)
- WiX (`wix`)
- HotpotQA (`hotpotqa`)
- OpenRAG Bench, two variants:
- `orb_text` — text embedder (`qwen3-embedding:4b`, 2560-dim) with VLM picture descriptions baked into chunk content at ingest. Use for text-only retrieval/QA against figure-rich corpora.
- `orb_multimodal` — multimodal embedder (`qwen3-vl-embedding-8b`, 4096-dim) with picture vectors in the same space as text. Use for cross-modal retrieval (text-as-query → figure hits, image-as-query) and vision QA where the figure itself is the answer.
@ -21,24 +19,24 @@ After installing the package, you can run evaluations using the `evaluations` co
```bash
# Run retrieval + QA benchmarks
evaluations run repliqa
evaluations run wix
evaluations run orb_text
# Use a custom config file
evaluations run repliqa --config /path/to/haiku.rag.yaml
evaluations run wix --config /path/to/haiku.rag.yaml
# Override the database path
evaluations run repliqa --db /path/to/custom.lancedb
evaluations run wix --db /path/to/custom.lancedb
# Skip database population and run only benchmarks
evaluations run repliqa --skip-db
evaluations run wix --skip-db
# Skip specific benchmarks
evaluations run repliqa --skip-retrieval
evaluations run repliqa --skip-qa
evaluations run wix --skip-retrieval
evaluations run wix --skip-qa
# Limit the number of test cases
evaluations run repliqa --limit 100
evaluations run wix --limit 100
```
### Choosing the target
@ -63,15 +61,15 @@ is computed alongside QA accuracy from the URIs the skill registered via the
Download pre-built evaluation databases from HuggingFace:
```bash
evaluations download repliqa
evaluations download wix
evaluations download all
evaluations download repliqa --force
evaluations download wix --force
```
Upload databases (maintainer only):
```bash
evaluations upload repliqa
evaluations upload wix
evaluations upload all
```

4
evaluations/evaluations/datasets/__init__.py
View file

@ -1,16 +1,12 @@
from evaluations.config import DatasetSpec
from .hotpotqa import HOTPOTQA_SPEC
from .open_rag_bench import ORB_MULTIMODAL_SPEC, ORB_TEXT_SPEC
from .repliqa import REPLIQA_SPEC
from .wix import WIX_SPEC
DATASETS: dict[str, DatasetSpec] = {
spec.key: spec
for spec in (
REPLIQA_SPEC,
WIX_SPEC,
HOTPOTQA_SPEC,
ORB_TEXT_SPEC,
ORB_MULTIMODAL_SPEC,
)

108
evaluations/evaluations/datasets/hotpotqa.py
View file

@ -1,108 +0,0 @@
from collections.abc import Mapping
from typing import Any, cast
from datasets import Dataset, load_dataset
from pydantic_evals import Case
from evaluations.config import DatasetSpec, DocumentPayload, RetrievalSample
from evaluations.evaluators import MAPEvaluator
def load_hotpotqa_validation() -> Dataset:
dataset_dict = load_dataset("hotpotqa/hotpot_qa", "distractor")
return dataset_dict["validation"]
def extract_unique_documents(dataset: Dataset) -> list[dict[str, Any]]:
"""Extract unique documents from all context paragraphs, deduplicated by title."""
seen_titles: set[str] = set()
documents: list[dict[str, Any]] = []
for sample in dataset:
sample = cast(Mapping[str, Any], sample)
context = sample["context"]
titles = context["title"]
sentences_list = context["sentences"]
for title, sentences in zip(titles, sentences_list):
if title in seen_titles:
continue
seen_titles.add(title)
content = " ".join(sentences)
documents.append({"title": title, "content": content})
return documents
_cached_documents: list[dict[str, Any]] | None = None
def load_hotpotqa_documents() -> list[dict[str, Any]]:
"""Load and cache unique documents from HotpotQA."""
global _cached_documents
if _cached_documents is None:
dataset = load_hotpotqa_validation()
_cached_documents = extract_unique_documents(dataset)
return _cached_documents
def document_loader() -> Dataset:
"""Return documents as a Dataset-like iterable."""
docs = load_hotpotqa_documents()
return Dataset.from_list(docs)
def map_hotpotqa_document(doc: Mapping[str, Any]) -> DocumentPayload:
return DocumentPayload(
uri=doc["title"],
content=doc["content"],
title=doc["title"],
)
def map_hotpotqa_retrieval(doc: Mapping[str, Any]) -> RetrievalSample | None:
supporting_facts = doc["supporting_facts"]
titles = supporting_facts["title"]
if not titles:
return None
unique_titles = tuple(dict.fromkeys(titles))
return RetrievalSample(
question=doc["question"],
expected_uris=unique_titles,
)
def build_hotpotqa_case(
index: int, doc: Mapping[str, Any]
) -> Case[str, str, dict[str, str]]:
question_id = doc["id"]
question_type = doc["type"]
level = doc["level"]
case_name = f"{index}_{question_id}"
return Case(
name=case_name,
inputs=doc["question"],
expected_output=doc["answer"],
metadata={
"question_id": str(question_id),
"type": str(question_type),
"level": str(level),
"case_index": str(index),
},
)
HOTPOTQA_SPEC = DatasetSpec(
key="hotpotqa",
db_filename="hotpotqa.lancedb",
document_loader=document_loader,
document_mapper=map_hotpotqa_document,
qa_loader=load_hotpotqa_validation,
qa_case_builder=build_hotpotqa_case,
retrieval_loader=load_hotpotqa_validation,
retrieval_mapper=map_hotpotqa_retrieval,
retrieval_evaluator=MAPEvaluator(),
)

60
evaluations/evaluations/datasets/repliqa.py
View file

@ -1,60 +0,0 @@
from collections.abc import Mapping
from typing import Any
from datasets import Dataset, load_dataset
from pydantic_evals import Case
from evaluations.config import DatasetSpec, DocumentPayload, RetrievalSample
from evaluations.evaluators import MRREvaluator
def load_repliqa_corpus() -> Dataset:
dataset_dict = load_dataset("ServiceNow/repliqa")
dataset = dataset_dict["repliqa_3"]
return dataset.filter(lambda doc: doc["document_topic"] == "News Stories")
def map_repliqa_document(doc: Mapping[str, Any]) -> DocumentPayload:
return DocumentPayload(
uri=str(doc["document_id"]),
content=doc["document_extracted"],
)
def map_repliqa_retrieval(doc: Mapping[str, Any]) -> RetrievalSample | None:
expected_answer = doc["answer"]
if expected_answer == "The answer is not found in the document.":
return None
return RetrievalSample(
question=doc["question"],
expected_uris=(str(doc["document_id"]),),
)
def build_repliqa_case(
index: int, doc: Mapping[str, Any]
) -> Case[str, str, dict[str, str]]:
document_id = doc["document_id"]
case_name = f"{index}_{document_id}" if document_id is not None else f"case_{index}"
return Case(
name=case_name,
inputs=doc["question"],
expected_output=doc["answer"],
metadata={
"document_id": str(document_id),
"case_index": str(index),
},
)
REPLIQA_SPEC = DatasetSpec(
key="repliqa",
db_filename="repliqa.lancedb",
document_loader=load_repliqa_corpus,
document_mapper=map_repliqa_document,
qa_loader=load_repliqa_corpus,
qa_case_builder=build_repliqa_case,
retrieval_loader=load_repliqa_corpus,
retrieval_mapper=map_repliqa_retrieval,
retrieval_evaluator=MRREvaluator(),
)

10
evaluations/tests/test_benchmark.py
View file

@ -74,19 +74,19 @@ class TestBuildExperimentMetadata:
class TestResolveDataset:
def test_valid_dataset(self) -> None:
spec = _resolve_dataset("repliqa")
assert spec.key == "repliqa"
spec = _resolve_dataset("wix")
assert spec.key == "wix"
def test_case_insensitive(self) -> None:
spec = _resolve_dataset("REPLIQA")
assert spec.key == "repliqa"
spec = _resolve_dataset("WIX")
assert spec.key == "wix"
def test_unknown_dataset_raises(self) -> None:
with pytest.raises(typer.BadParameter, match="Unknown dataset 'nonexistent'"):
_resolve_dataset("nonexistent")
def test_error_lists_valid_datasets(self) -> None:
with pytest.raises(typer.BadParameter, match="repliqa"):
with pytest.raises(typer.BadParameter, match="wix"):
_resolve_dataset("nonexistent")

127
evaluations/tests/test_datasets.py
View file

@ -1,11 +1,5 @@
from pathlib import Path
from evaluations.datasets.hotpotqa import (
build_hotpotqa_case,
extract_unique_documents,
map_hotpotqa_document,
map_hotpotqa_retrieval,
)
from evaluations.datasets.open_rag_bench import (
build_orb_case,
download_pdf,
@ -13,11 +7,6 @@ from evaluations.datasets.open_rag_bench import (
map_orb_document,
map_orb_retrieval,
)
from evaluations.datasets.repliqa import (
build_repliqa_case,
map_repliqa_document,
map_repliqa_retrieval,
)
from evaluations.datasets.wix import (
build_wix_case,
map_wix_document,
@ -25,50 +14,6 @@ from evaluations.datasets.wix import (
)
class TestRepliqa:
def test_map_document(self) -> None:
doc = {"document_id": "doc-42", "document_extracted": "Some content here."}
payload = map_repliqa_document(doc)
assert payload.uri == "doc-42"
assert payload.content == "Some content here."
def test_map_retrieval(self) -> None:
doc = {
"question": "What happened?",
"answer": "Something happened.",
"document_id": "doc-42",
}
sample = map_repliqa_retrieval(doc)
assert sample is not None
assert sample.question == "What happened?"
assert sample.expected_uris == ("doc-42",)
def test_map_retrieval_skips_unanswerable(self) -> None:
doc = {
"question": "What?",
"answer": "The answer is not found in the document.",
"document_id": "doc-1",
}
assert map_repliqa_retrieval(doc) is None
def test_build_case(self) -> None:
doc = {
"document_id": "doc-7",
"question": "Why?",
"answer": "Because.",
}
case = build_repliqa_case(3, doc)
assert case.name == "3_doc-7"
assert case.inputs == "Why?"
assert case.expected_output == "Because."
assert case.metadata == {"document_id": "doc-7", "case_index": "3"}
def test_build_case_none_document_id(self) -> None:
doc = {"document_id": None, "question": "Q?", "answer": "A."}
case = build_repliqa_case(1, doc)
assert case.name == "case_1"
class TestWix:
def test_map_document_with_all_fields(self) -> None:
doc = {
@ -136,78 +81,6 @@ class TestWix:
assert case.name == "case_1"
class TestHotpotQA:
def test_map_document(self) -> None:
doc = {"title": "Albert Einstein", "content": "Was a physicist."}
payload = map_hotpotqa_document(doc)
assert payload.uri == "Albert Einstein"
assert payload.content == "Was a physicist."
assert payload.title == "Albert Einstein"
def test_map_retrieval(self) -> None:
doc = {
"question": "Who was Einstein?",
"supporting_facts": {"title": ["Albert Einstein", "Physics"]},
}
sample = map_hotpotqa_retrieval(doc)
assert sample is not None
assert sample.expected_uris == ("Albert Einstein", "Physics")
def test_map_retrieval_deduplicates_titles(self) -> None:
doc = {
"question": "Q?",
"supporting_facts": {"title": ["A", "B", "A"]},
}
sample = map_hotpotqa_retrieval(doc)
assert sample is not None
assert sample.expected_uris == ("A", "B")
def test_map_retrieval_no_titles(self) -> None:
doc = {"question": "Q?", "supporting_facts": {"title": []}}
assert map_hotpotqa_retrieval(doc) is None
def test_build_case(self) -> None:
doc = {
"id": "abc123",
"question": "What is X?",
"answer": "X is Y.",
"type": "comparison",
"level": "hard",
}
case = build_hotpotqa_case(5, doc)
assert case.name == "5_abc123"
assert case.inputs == "What is X?"
assert case.expected_output == "X is Y."
assert case.metadata == {
"question_id": "abc123",
"type": "comparison",
"level": "hard",
"case_index": "5",
}
def test_extract_unique_documents(self) -> None:
# Simulate a minimal dataset with context
dataset = [
{
"context": {
"title": ["Doc A", "Doc B"],
"sentences": [["Sentence 1."], ["Sentence 2.", " More."]],
}
},
{
"context": {
"title": ["Doc A", "Doc C"],
"sentences": [["Dupe."], ["Sentence 3."]],
}
},
]
docs = extract_unique_documents(dataset) # type: ignore[arg-type] # ty: ignore[invalid-argument-type]
assert len(docs) == 3
titles = [d["title"] for d in docs]
assert titles == ["Doc A", "Doc B", "Doc C"]
assert docs[1]["content"] == "Sentence 2. More."
class TestOpenRAGBench:
def test_map_document(self, tmp_path: Path) -> None:
# Pre-create a cached PDF