Remove the wix evaluation dataset
This commit is contained in:
Yiorgis Gozadinos
parent
d864cf8008
commit
e522d8cfa8
10 changed files with 31 additions and 241 deletions
|
|
@ -1,6 +1,10 @@
|
|||
# Changelog
|
||||
## [Unreleased]
|
||||
|
||||
### Removed
|
||||
|
||||
- `wix` evaluation dataset and its reference config `evaluations/configs/wix.yaml`.
|
||||
|
||||
## [0.74.0] - 2026-08-13
|
||||
|
||||
### Added
|
||||
|
|
|
|||
|
|
@ -1,13 +1,13 @@
|
|||
# Benchmarks
|
||||
|
||||
We evaluate `haiku.rag` on a small set of datasets that exercise different parts of the pipeline. OpenRAG Bench (ORB), T²-RAGBench, HotpotQA, and Wix are the datasets we currently track. Retrieval, QA accuracy, and citation retrieval are scored end-to-end through the RAG and analysis capabilities.
|
||||
We evaluate `haiku.rag` on a small set of datasets that exercise different parts of the pipeline. OpenRAG Bench (ORB), T²-RAGBench, and HotpotQA are the datasets we currently track. Retrieval, QA accuracy, and citation retrieval are scored end-to-end through the RAG and analysis capabilities.
|
||||
|
||||
## Running Evaluations
|
||||
|
||||
You can run evaluations with the `evaluations` CLI:
|
||||
|
||||
```bash
|
||||
evaluations run wix
|
||||
evaluations run hotpotqa
|
||||
evaluations run orb_text
|
||||
```
|
||||
|
||||
|
|
@ -19,20 +19,19 @@ Building evaluation databases from scratch can take a long time, especially for
|
|||
|
||||
```bash
|
||||
# Download a specific dataset
|
||||
evaluations download wix
|
||||
evaluations download hotpotqa
|
||||
|
||||
# Download all datasets
|
||||
evaluations download all
|
||||
|
||||
# Force re-download (overwrite existing)
|
||||
evaluations download wix --force
|
||||
evaluations download hotpotqa --force
|
||||
```
|
||||
|
||||
Active datasets:
|
||||
|
||||
| Dataset | Size |
|
||||
|---------|------|
|
||||
| `wix` | ~511MB |
|
||||
| `orb_text` — OpenRAG Bench, text embedder (`qwen3-embedding:4b`) with VLM picture descriptions baked into chunk content | ~18 GB |
|
||||
| `orb_multimodal` — OpenRAG Bench, multimodal embedder (`qwen3-vl-embedding-8b`); picture vectors live in the same space as text for cross-modal retrieval | ~16 GB |
|
||||
| `orb_multimodal_nemotron` — OpenRAG Bench, multimodal embedder (`nvidia/llama-nemotron-embed-vl-1b-v2`), the embedder behind the published headline results | ~16 GB |
|
||||
|
|
@ -52,7 +51,7 @@ The configs use `vllm` as the model host. Point `base_url` at your own OpenAI-co
|
|||
The benchmark script accepts several options:
|
||||
|
||||
```bash
|
||||
evaluations run wix --config /path/to/haiku.rag.yaml --db /path/to/custom.lancedb
|
||||
evaluations run hotpotqa --config /path/to/haiku.rag.yaml --db /path/to/custom.lancedb
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
|
@ -202,15 +201,3 @@ The reranker's contribution is larger here than on the single-doc datasets: hybr
|
|||
| `vllm:Gemma-4-26B-A4B-NVFP4` | none | 0.83 | 0.75 |
|
||||
|
||||
*Measured on haiku.rag v0.66.0 with `qwen3-embedding:4b` (vLLM, dim 2560), judged by `vllm:Qwen3.6-35B-A3B-NVFP4`, 7,405 cases. The reranker lifts QA accuracy +2.7pts and `cited_map` +4.6pts. Without a reranker, `cited_map` (0.75) still exceeds the no-reranker retrieval MAP (0.70): the skill reformulates queries across search calls, partially recovering second-hop documents that a single query misses.*
|
||||
|
||||
### Wix
|
||||
|
||||
[WixQA](https://huggingface.co/datasets/Wix/WixQA) is real customer support questions paired with curated answers. 200 cases.
|
||||
|
||||
`evaluations run wix --target rag-capability` runs the RAG capability end-to-end and produces both QA accuracy and a citation retrieval metric (`cited_map`) computed from the URIs the capability registered via the `cite` tool against the gold `expected_uris`.
|
||||
|
||||
| Capability model | Reranker | QA accuracy | Mean `cited_map` |
|
||||
|------------------------------|------------------------|-------------|------------------|
|
||||
| `vllm:Gemma-4-26B-A4B-NVFP4` | `mxbai-rerank-base-v2` | 0.87 | 0.38 |
|
||||
|
||||
*Measured on haiku.rag v0.48.0 with `qwen3-embedding:4b` (vLLM, dim 2560), `chunk_size=256`, `search.limit=5`. Judged by `vllm:Qwen3.6-35B-A3B-NVFP4` (qwen3.6 family, NVFP4 quant served via vLLM rather than the default Ollama). 172 / 198 completed cases (2 errored).*
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ Documents flow through: **chunking → embedding → hybrid search (vector + FTS
|
|||
|
||||
### Chunking
|
||||
|
||||
`chunk_size` controls the granularity of retrieval. Smaller chunks match queries more precisely but carry less context each. Larger chunks provide more surrounding information but dilute relevance signals. On the Wix benchmark, increasing from 256 to 512 tokens raised MAP from 0.43 to 0.45 on plain text, a modest gain that also increases token cost per result. See [Processing](configuration/processing.md#chunk-size) for configuration.
|
||||
`chunk_size` controls the granularity of retrieval. Smaller chunks match queries more precisely but carry less context each. Larger chunks provide more surrounding information but dilute relevance signals. See [Processing](configuration/processing.md#chunk-size) for configuration.
|
||||
|
||||
`chunker_type` selects between `hybrid` (default) and `hierarchical` chunking. Hierarchical chunking preserves the document's heading structure and works better for deeply nested or structured content. See [Chunking Strategies](configuration/processing.md#chunking-strategies).
|
||||
|
||||
|
|
@ -22,7 +22,7 @@ Larger embedding models produce better representations at the cost of slower ind
|
|||
|
||||
### Reranking
|
||||
|
||||
When configured, a cross-encoder reranker re-scores 10x the requested candidates and returns the top results. This adds latency but improves precision. On the Wix benchmark, adding `mxbai-rerank-base-v2` raised MAP from 0.34 to 0.39 on HTML content. See [Search Settings](configuration/qa.md#search-settings) for how reranking integrates with search.
|
||||
When configured, a cross-encoder reranker re-scores 10x the requested candidates and returns the top results. This adds latency but improves precision. See [Search Settings](configuration/qa.md#search-settings) for how reranking integrates with search.
|
||||
|
||||
### Search Settings
|
||||
|
||||
|
|
|
|||
|
|
@ -8,7 +8,6 @@ This package is not published to PyPI and is only used for development and testi
|
|||
|
||||
Contains evaluation scripts for benchmarking RAG retrieval and QA performance. Available datasets:
|
||||
|
||||
- WiX (`wix`)
|
||||
- HotpotQA (`hotpotqa`) — multi-hop QA over Wikipedia paragraphs (distractor validation split, 7,405 questions, two gold documents per question)
|
||||
- OpenRAG Bench, two variants:
|
||||
- `orb_text` — text embedder (`qwen3-embedding:4b`, 2560-dim) with VLM picture descriptions baked into chunk content at ingest. Use for text-only retrieval/QA against figure-rich corpora.
|
||||
|
|
@ -20,24 +19,24 @@ After installing the package, you can run evaluations using the `evaluations` co
|
|||
|
||||
```bash
|
||||
# Run retrieval + QA benchmarks
|
||||
evaluations run wix
|
||||
evaluations run hotpotqa
|
||||
evaluations run orb_text
|
||||
|
||||
# Use a custom config file
|
||||
evaluations run wix --config /path/to/haiku.rag.yaml
|
||||
evaluations run hotpotqa --config /path/to/haiku.rag.yaml
|
||||
|
||||
# Override the database path
|
||||
evaluations run wix --db /path/to/custom.lancedb
|
||||
evaluations run hotpotqa --db /path/to/custom.lancedb
|
||||
|
||||
# Skip database population and run only benchmarks
|
||||
evaluations run wix --skip-db
|
||||
evaluations run hotpotqa --skip-db
|
||||
|
||||
# Skip specific benchmarks
|
||||
evaluations run wix --skip-retrieval
|
||||
evaluations run wix --skip-qa
|
||||
evaluations run hotpotqa --skip-retrieval
|
||||
evaluations run hotpotqa --skip-qa
|
||||
|
||||
# Limit the number of test cases
|
||||
evaluations run wix --limit 100
|
||||
evaluations run hotpotqa --limit 100
|
||||
```
|
||||
|
||||
### Choosing the target
|
||||
|
|
@ -47,8 +46,8 @@ evaluations run wix --limit 100
|
|||
datasets and judge:
|
||||
|
||||
```bash
|
||||
evaluations run wix --target rag-capability
|
||||
evaluations run wix --target analysis-capability --capability-model ollama:gpt-oss
|
||||
evaluations run hotpotqa --target rag-capability
|
||||
evaluations run hotpotqa --target analysis-capability --capability-model ollama:gpt-oss
|
||||
```
|
||||
|
||||
`--capability-model "provider:name"` overrides the capability model independently from
|
||||
|
|
@ -68,15 +67,15 @@ cases) for use from Claude Code.
|
|||
Download pre-built evaluation databases from HuggingFace:
|
||||
|
||||
```bash
|
||||
evaluations download wix
|
||||
evaluations download hotpotqa
|
||||
evaluations download all
|
||||
evaluations download wix --force
|
||||
evaluations download hotpotqa --force
|
||||
```
|
||||
|
||||
Upload databases (maintainer only):
|
||||
|
||||
```bash
|
||||
evaluations upload wix
|
||||
evaluations upload hotpotqa
|
||||
evaluations upload all
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -1,42 +0,0 @@
|
|||
# Reference config for the `wix` pre-built evaluation database.
|
||||
# Run: evaluations run wix --skip-db --config configs/wix.yaml
|
||||
# base_url uses the `vllm` host serving each model over an OpenAI-compatible API.
|
||||
|
||||
environment: development
|
||||
|
||||
storage:
|
||||
auto_vacuum: false
|
||||
|
||||
embeddings:
|
||||
model:
|
||||
provider: openai
|
||||
name: qwen3-embedding-4b
|
||||
vector_dim: 2560
|
||||
base_url: http://vllm:11431/v1
|
||||
|
||||
reranking:
|
||||
model:
|
||||
provider: vllm
|
||||
name: Qwen/Qwen3-Reranker-4B
|
||||
base_url: http://vllm:11455
|
||||
|
||||
qa:
|
||||
model:
|
||||
provider: openai
|
||||
name: gemma4-26b
|
||||
base_url: http://vllm:11432/v1
|
||||
vision: true
|
||||
|
||||
evaluations:
|
||||
judge:
|
||||
provider: openai
|
||||
name: RedHatAI/Qwen3.6-35B-A3B-NVFP4
|
||||
base_url: http://vllm:11430/v1
|
||||
temperature: 0.6
|
||||
max_tokens: 16384
|
||||
extra_body:
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
min_p: 0
|
||||
chat_template_kwargs:
|
||||
enable_thinking: true
|
||||
|
|
@ -7,12 +7,10 @@ from .open_rag_bench import (
|
|||
ORB_TEXT_SPEC,
|
||||
)
|
||||
from .t2_ragbench import T2_FINQA_SPEC, T2_TATDQA_SPEC
|
||||
from .wix import WIX_SPEC
|
||||
|
||||
DATASETS: dict[str, DatasetSpec] = {
|
||||
spec.key: spec
|
||||
for spec in (
|
||||
WIX_SPEC,
|
||||
HOTPOTQA_SPEC,
|
||||
ORB_TEXT_SPEC,
|
||||
ORB_MULTIMODAL_SPEC,
|
||||
|
|
|
|||
|
|
@ -1,84 +0,0 @@
|
|||
import json
|
||||
from collections.abc import Iterable, Mapping
|
||||
from typing import Any
|
||||
|
||||
from datasets import Dataset, load_dataset
|
||||
from pydantic_evals import Case
|
||||
|
||||
from evaluations.config import DatasetSpec, DocumentPayload, RetrievalSample
|
||||
from evaluations.evaluators import MAPEvaluator
|
||||
|
||||
|
||||
def load_wix_corpus() -> Dataset:
|
||||
dataset_dict = load_dataset("Wix/WixQA", "wix_kb_corpus")
|
||||
return dataset_dict["train"]
|
||||
|
||||
|
||||
def map_wix_document(doc: Mapping[str, Any]) -> DocumentPayload:
|
||||
article_id = doc.get("id")
|
||||
url = doc.get("url")
|
||||
uri = str(article_id) if article_id is not None else str(url)
|
||||
|
||||
metadata: dict[str, str] = {}
|
||||
if article_id is not None:
|
||||
metadata["article_id"] = str(article_id)
|
||||
if url:
|
||||
metadata["url"] = str(url)
|
||||
|
||||
return DocumentPayload(
|
||||
uri=uri,
|
||||
content=doc["html_content"],
|
||||
title=doc.get("title"),
|
||||
metadata=metadata or None,
|
||||
format="html",
|
||||
)
|
||||
|
||||
|
||||
def load_wix_qa() -> Dataset:
|
||||
dataset_dict = load_dataset("Wix/WixQA", "wixqa_expertwritten")
|
||||
return dataset_dict["train"]
|
||||
|
||||
|
||||
def map_wix_retrieval(doc: Mapping[str, Any]) -> RetrievalSample | None:
|
||||
article_ids: Iterable[int | str] | None = doc.get("article_ids")
|
||||
if not article_ids:
|
||||
return None
|
||||
|
||||
expected_uris = tuple(str(article_id) for article_id in article_ids)
|
||||
return RetrievalSample(
|
||||
question=doc["question"],
|
||||
expected_uris=expected_uris,
|
||||
)
|
||||
|
||||
|
||||
def build_wix_case(
|
||||
index: int, doc: Mapping[str, Any]
|
||||
) -> Case[str, str, dict[str, str]]:
|
||||
article_ids = tuple(str(article_id) for article_id in doc.get("article_ids") or [])
|
||||
joined_ids = "-".join(article_ids)
|
||||
case_name = f"{index}_{joined_ids}" if joined_ids else f"case_{index}"
|
||||
|
||||
metadata = {
|
||||
"case_index": str(index),
|
||||
"document_ids": json.dumps(article_ids),
|
||||
}
|
||||
|
||||
return Case(
|
||||
name=case_name,
|
||||
inputs=doc["question"],
|
||||
expected_output=doc["answer"],
|
||||
metadata=metadata,
|
||||
)
|
||||
|
||||
|
||||
WIX_SPEC = DatasetSpec(
|
||||
key="wix",
|
||||
db_filename="wix.lancedb",
|
||||
document_loader=load_wix_corpus,
|
||||
document_mapper=map_wix_document,
|
||||
qa_loader=load_wix_qa,
|
||||
qa_case_builder=build_wix_case,
|
||||
retrieval_loader=load_wix_qa,
|
||||
retrieval_mapper=map_wix_retrieval,
|
||||
retrieval_evaluator=MAPEvaluator(),
|
||||
)
|
||||
|
|
@ -113,19 +113,19 @@ class TestBuildExperimentMetadata:
|
|||
|
||||
class TestResolveDataset:
|
||||
def test_valid_dataset(self) -> None:
|
||||
spec = _resolve_dataset("wix")
|
||||
assert spec.key == "wix"
|
||||
spec = _resolve_dataset("hotpotqa")
|
||||
assert spec.key == "hotpotqa"
|
||||
|
||||
def test_case_insensitive(self) -> None:
|
||||
spec = _resolve_dataset("WIX")
|
||||
assert spec.key == "wix"
|
||||
spec = _resolve_dataset("HOTPOTQA")
|
||||
assert spec.key == "hotpotqa"
|
||||
|
||||
def test_unknown_dataset_raises(self) -> None:
|
||||
with pytest.raises(typer.BadParameter, match="Unknown dataset 'nonexistent'"):
|
||||
_resolve_dataset("nonexistent")
|
||||
|
||||
def test_error_lists_valid_datasets(self) -> None:
|
||||
with pytest.raises(typer.BadParameter, match="wix"):
|
||||
with pytest.raises(typer.BadParameter, match="hotpotqa"):
|
||||
_resolve_dataset("nonexistent")
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -20,78 +20,6 @@ from evaluations.datasets.t2_ragbench import (
|
|||
map_t2_document,
|
||||
map_t2_retrieval,
|
||||
)
|
||||
from evaluations.datasets.wix import (
|
||||
build_wix_case,
|
||||
map_wix_document,
|
||||
map_wix_retrieval,
|
||||
)
|
||||
|
||||
|
||||
class TestWix:
|
||||
def test_map_document_with_all_fields(self) -> None:
|
||||
doc = {
|
||||
"id": 123,
|
||||
"url": "https://wix.com/article",
|
||||
"html_content": "<p>Content</p>",
|
||||
"title": "My Article",
|
||||
}
|
||||
payload = map_wix_document(doc)
|
||||
assert payload.uri == "123"
|
||||
assert payload.content == "<p>Content</p>"
|
||||
assert payload.title == "My Article"
|
||||
assert payload.format == "html"
|
||||
assert payload.metadata == {
|
||||
"article_id": "123",
|
||||
"url": "https://wix.com/article",
|
||||
}
|
||||
|
||||
def test_map_document_no_id(self) -> None:
|
||||
doc = {
|
||||
"id": None,
|
||||
"url": "https://wix.com/page",
|
||||
"html_content": "<p>Text</p>",
|
||||
"title": None,
|
||||
}
|
||||
payload = map_wix_document(doc)
|
||||
assert payload.uri == "https://wix.com/page"
|
||||
|
||||
def test_map_document_no_metadata(self) -> None:
|
||||
doc = {"id": None, "url": None, "html_content": "<p>X</p>", "title": None}
|
||||
payload = map_wix_document(doc)
|
||||
assert payload.metadata is None
|
||||
|
||||
def test_map_retrieval(self) -> None:
|
||||
doc = {"question": "How to add a page?", "article_ids": [10, 20]}
|
||||
sample = map_wix_retrieval(doc)
|
||||
assert sample is not None
|
||||
assert sample.question == "How to add a page?"
|
||||
assert sample.expected_uris == ("10", "20")
|
||||
|
||||
def test_map_retrieval_no_article_ids(self) -> None:
|
||||
doc = {"question": "Q?", "article_ids": None}
|
||||
assert map_wix_retrieval(doc) is None
|
||||
|
||||
def test_map_retrieval_empty_article_ids(self) -> None:
|
||||
doc = {"question": "Q?", "article_ids": []}
|
||||
assert map_wix_retrieval(doc) is None
|
||||
|
||||
def test_build_case(self) -> None:
|
||||
doc = {
|
||||
"question": "How?",
|
||||
"answer": "Like this.",
|
||||
"article_ids": [5, 10],
|
||||
}
|
||||
case = build_wix_case(2, doc)
|
||||
assert case.name == "2_5-10"
|
||||
assert case.inputs == "How?"
|
||||
assert case.expected_output == "Like this."
|
||||
assert case.metadata is not None
|
||||
assert case.metadata["case_index"] == "2"
|
||||
|
||||
def test_build_case_no_article_ids(self) -> None:
|
||||
doc = {"question": "Q?", "answer": "A.", "article_ids": None}
|
||||
case = build_wix_case(1, doc)
|
||||
assert case.name == "case_1"
|
||||
|
||||
|
||||
class TestHotpotQA:
|
||||
|
|
|
|||
|
|
@ -658,9 +658,9 @@ class TestDoclingLocalConverter:
|
|||
async def test_convert_text_html_mixed_img_sources(self, config, monkeypatch):
|
||||
"""End-to-end: HTML with a mix of remote http, data:, broken http, and
|
||||
file:// `<img>` sources. Remote and data: URIs land as picture bytes;
|
||||
broken URLs and file:// stay as placeholder pictures. Models the
|
||||
wix-style ingest where most images are remote URLs with a handful of
|
||||
broken or local-only references mixed in."""
|
||||
broken URLs and file:// stay as placeholder pictures. Models a
|
||||
knowledge-base ingest where most images are remote URLs with a handful
|
||||
of broken or local-only references mixed in."""
|
||||
import base64
|
||||
|
||||
from docling.backend import html_backend as html_backend_module
|
||||
|
|
|
|||
Loading…
Reference in a new issue