diff --git a/docs/benchmarks.md b/docs/benchmarks.md index bfff426e..2ef0fc81 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -36,13 +36,6 @@ Active datasets: | `orb_text` — OpenRAG Bench, text embedder (`qwen3-embedding:4b`) with VLM picture descriptions baked into chunk content | ~18 GB | | `orb_multimodal` — OpenRAG Bench, multimodal embedder (`qwen3-vl-embedding-8b`); picture vectors live in the same space as text for cross-modal retrieval | ~16 GB | -Inactive (kept downloadable, not currently maintained): - -| Dataset | Size | -|---------|------| -| `repliqa` | ~30MB | -| `hotpotqa` | ~331MB | - After downloading, run benchmarks with `--skip-db` to use the pre-built database: ```bash @@ -116,18 +109,6 @@ This is computed alongside QA accuracy from the same skill run, no extra invocat Numbers measured under the current pinned judge (`ollama:qwen3.6`) on a recent `haiku.rag` version. -### Wix - -[WixQA](https://huggingface.co/datasets/Wix/WixQA) is real customer support questions paired with curated answers. 200 cases. - -`evaluations run wix --target rag-skill` runs the RAG skill end-to-end and produces both QA accuracy and a citation retrieval metric (`cited_map`) computed from the URIs the skill registered via the `cite` tool against the gold `expected_uris`. - -| Skill model | Reranker | QA accuracy | Mean `cited_map` | -|------------------------------|------------------------|-------------|------------------| -| `vllm:Gemma-4-26B-A4B-NVFP4` | `mxbai-rerank-base-v2` | 0.87 | 0.38 | - -*Measured on haiku.rag v0.48.0 with `qwen3-embedding:4b` (vLLM, dim 2560), `chunk_size=256`, `search.limit=5`. Judged by `vllm:Qwen3.6-35B-A3B-NVFP4` (qwen3.6 family, NVFP4 quant served via vLLM rather than the default Ollama). 172 / 198 completed cases (2 errored).* - ### OpenRAG Bench (ORB) [OpenRAG Bench](https://huggingface.co/datasets/vectara/open_ragbench) contains ArXiv research papers with multimodal question-answering pairs. Queries include both text-based and image-based questions, testing retrieval and reasoning over visual content like figures, charts, and diagrams. Each query maps to one relevant document. @@ -173,76 +154,14 @@ Two approaches are benchmarked separately: *qwen3 measured on haiku.rag v0.48.0 over all 3045 cases; nemotron on v0.50.0 over 2836 / 3045 cases (run stopped early; numbers stable from ~12% onward). Both with `mxbai-rerank-base-v2`, judged by `vllm:Qwen3.6-35B-A3B-NVFP4`.* -## Inactive datasets +### Wix -The benchmarks below are not currently maintained. Numbers were measured against earlier `haiku.rag` versions and an older pinned judge (`ollama:gpt-oss`), before the skill workflow became the only path. Retrieval tables don't depend on the judge, but the QA tables aren't reproducible against the current skill-only setup. We may revive them. +[WixQA](https://huggingface.co/datasets/Wix/WixQA) is real customer support questions paired with curated answers. 200 cases. -### RepliQA +`evaluations run wix --target rag-skill` runs the RAG skill end-to-end and produces both QA accuracy and a citation retrieval metric (`cited_map`) computed from the URIs the skill registered via the `cite` tool against the gold `expected_uris`. -[RepliQA](https://huggingface.co/datasets/ServiceNow/repliqa) contains synthetic news stories with question-answer pairs. We use `News Stories` from `repliqa_3` (1035 documents). Each question has exactly one relevant document, so we use MRR for retrieval evaluation. +| Skill model | Reranker | QA accuracy | Mean `cited_map` | +|------------------------------|------------------------|-------------|------------------| +| `vllm:Gemma-4-26B-A4B-NVFP4` | `mxbai-rerank-base-v2` | 0.87 | 0.38 | -#### Retrieval (MRR) - -| Embedding Model | MRR | Reranker | -|-------------------------------|------|----------| -| Ollama / `qwen3-embedding:8b` | 0.91 | - | - -*Measured on haiku.rag v0.19.6.* - -#### QA Accuracy - -| Embedding Model | QA Model | Accuracy | Reranker | -|------------------------------|----------------------------------|----------|------------------------| -| Ollama / `qwen3-embedding:4b` | Ollama / `gpt-oss` - no thinking | 0.82 | None | -| Ollama / `qwen3-embedding:8b` | Ollama / `gpt-oss` - thinking | 0.89 | None | -| Ollama / `mxbai-embed-large` | Ollama / `qwen3` - thinking | 0.85 | None | -| Ollama / `mxbai-embed-large` | Ollama / `qwen3` - thinking | 0.87 | `mxbai-rerank-base-v2` | -| Ollama / `mxbai-embed-large` | Ollama / `qwen3:0.6b` | 0.28 | None | - -*Measured on haiku.rag v0.19.6, judged by `ollama:gpt-oss`.* - -Note the significant degradation when very small models are used such as `qwen3:0.6b`. - -### HotpotQA - -[HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) is a multi-hop question answering dataset requiring reasoning over multiple Wikipedia paragraphs. Each question requires evidence from 2+ documents, making it ideal for testing retrieval and reasoning capabilities. We use MAP for retrieval evaluation since queries have multiple relevant documents. - -#### Retrieval (MAP) - -| Embedding Model | MAP | Reranker | -|----------------------|------|----------| -| `qwen3-embedding:4b` | 0.69 | none | - -*Measured on haiku.rag v0.20.2.* - -#### QA Accuracy - -| Embedding Model | QA Model | Accuracy | -|----------------------|--------------------------|----------| -| `qwen3-embedding:4b` | `gpt-oss:20b` - thinking | 0.86 | - -*Measured on haiku.rag v0.20.2, judged by `ollama:gpt-oss`.* - -### Wix (historical, plain text and HTML) - -Earlier Wix runs measured under different chunk settings and reranker combinations, against the older `gpt-oss` judge. - -#### Retrieval (MAP) - -| Embedding Model | Chunk size | MAP | Reranker | Notes | -|------------------------|------------|------|------------------------|------------------------------| -| `qwen3-embedding:4b` | 256 | 0.34 | None | html, `chunk-radius=2` | -| `qwen3-embedding:4b` | 256 | 0.39 | `mxbai-rerank-base-v2` | html, `chunk-radius=2` | -| `qwen3-embedding:4b` | 256 | 0.43 | None | plain text, `chunk-radius=0` | -| `qwen3-embedding:4b` | 512 | 0.45 | None | plain text, `chunk-radius=0` | - -*Measured on haiku.rag v0.27.2.* - -#### QA Accuracy - -| Embedding Model | Chunk size | QA Model | Accuracy | Notes | -|----------------------|------------|-----------------------------|----------|------------------------------| -| `qwen3-embedding:4b` | 256 | `gpt-oss:20b` - no thinking | 0.80 | html, `chunk-radius=2` | -| `qwen3-embedding:4b` | 256 | `gpt-oss:20b` - no thinking | 0.83 | html, `chunk-radius=2`, `jinaai/jina-reranker-v3` | - -*Measured on haiku.rag v0.27.2, judged by `ollama:gpt-oss`.* +*Measured on haiku.rag v0.48.0 with `qwen3-embedding:4b` (vLLM, dim 2560), `chunk_size=256`, `search.limit=5`. Judged by `vllm:Qwen3.6-35B-A3B-NVFP4` (qwen3.6 family, NVFP4 quant served via vLLM rather than the default Ollama). 172 / 198 completed cases (2 errored).* diff --git a/evaluations/README.md b/evaluations/README.md index b573721d..112bf9cf 100644 --- a/evaluations/README.md +++ b/evaluations/README.md @@ -8,9 +8,7 @@ This package is not published to PyPI and is only used for development and testi Contains evaluation scripts for benchmarking RAG retrieval and QA performance. Available datasets: -- RepliQA (`repliqa`) - WiX (`wix`) -- HotpotQA (`hotpotqa`) - OpenRAG Bench, two variants: - `orb_text` — text embedder (`qwen3-embedding:4b`, 2560-dim) with VLM picture descriptions baked into chunk content at ingest. Use for text-only retrieval/QA against figure-rich corpora. - `orb_multimodal` — multimodal embedder (`qwen3-vl-embedding-8b`, 4096-dim) with picture vectors in the same space as text. Use for cross-modal retrieval (text-as-query → figure hits, image-as-query) and vision QA where the figure itself is the answer. @@ -21,24 +19,24 @@ After installing the package, you can run evaluations using the `evaluations` co ```bash # Run retrieval + QA benchmarks -evaluations run repliqa evaluations run wix +evaluations run orb_text # Use a custom config file -evaluations run repliqa --config /path/to/haiku.rag.yaml +evaluations run wix --config /path/to/haiku.rag.yaml # Override the database path -evaluations run repliqa --db /path/to/custom.lancedb +evaluations run wix --db /path/to/custom.lancedb # Skip database population and run only benchmarks -evaluations run repliqa --skip-db +evaluations run wix --skip-db # Skip specific benchmarks -evaluations run repliqa --skip-retrieval -evaluations run repliqa --skip-qa +evaluations run wix --skip-retrieval +evaluations run wix --skip-qa # Limit the number of test cases -evaluations run repliqa --limit 100 +evaluations run wix --limit 100 ``` ### Choosing the target @@ -63,15 +61,15 @@ is computed alongside QA accuracy from the URIs the skill registered via the Download pre-built evaluation databases from HuggingFace: ```bash -evaluations download repliqa +evaluations download wix evaluations download all -evaluations download repliqa --force +evaluations download wix --force ``` Upload databases (maintainer only): ```bash -evaluations upload repliqa +evaluations upload wix evaluations upload all ``` diff --git a/evaluations/evaluations/datasets/__init__.py b/evaluations/evaluations/datasets/__init__.py index ca344dc4..c973aa5f 100644 --- a/evaluations/evaluations/datasets/__init__.py +++ b/evaluations/evaluations/datasets/__init__.py @@ -1,16 +1,12 @@ from evaluations.config import DatasetSpec -from .hotpotqa import HOTPOTQA_SPEC from .open_rag_bench import ORB_MULTIMODAL_SPEC, ORB_TEXT_SPEC -from .repliqa import REPLIQA_SPEC from .wix import WIX_SPEC DATASETS: dict[str, DatasetSpec] = { spec.key: spec for spec in ( - REPLIQA_SPEC, WIX_SPEC, - HOTPOTQA_SPEC, ORB_TEXT_SPEC, ORB_MULTIMODAL_SPEC, ) diff --git a/evaluations/evaluations/datasets/hotpotqa.py b/evaluations/evaluations/datasets/hotpotqa.py deleted file mode 100644 index d47f939e..00000000 --- a/evaluations/evaluations/datasets/hotpotqa.py +++ /dev/null @@ -1,108 +0,0 @@ -from collections.abc import Mapping -from typing import Any, cast - -from datasets import Dataset, load_dataset -from pydantic_evals import Case - -from evaluations.config import DatasetSpec, DocumentPayload, RetrievalSample -from evaluations.evaluators import MAPEvaluator - - -def load_hotpotqa_validation() -> Dataset: - dataset_dict = load_dataset("hotpotqa/hotpot_qa", "distractor") - return dataset_dict["validation"] - - -def extract_unique_documents(dataset: Dataset) -> list[dict[str, Any]]: - """Extract unique documents from all context paragraphs, deduplicated by title.""" - seen_titles: set[str] = set() - documents: list[dict[str, Any]] = [] - - for sample in dataset: - sample = cast(Mapping[str, Any], sample) - context = sample["context"] - titles = context["title"] - sentences_list = context["sentences"] - - for title, sentences in zip(titles, sentences_list): - if title in seen_titles: - continue - seen_titles.add(title) - content = " ".join(sentences) - documents.append({"title": title, "content": content}) - - return documents - - -_cached_documents: list[dict[str, Any]] | None = None - - -def load_hotpotqa_documents() -> list[dict[str, Any]]: - """Load and cache unique documents from HotpotQA.""" - global _cached_documents - if _cached_documents is None: - dataset = load_hotpotqa_validation() - _cached_documents = extract_unique_documents(dataset) - return _cached_documents - - -def document_loader() -> Dataset: - """Return documents as a Dataset-like iterable.""" - docs = load_hotpotqa_documents() - return Dataset.from_list(docs) - - -def map_hotpotqa_document(doc: Mapping[str, Any]) -> DocumentPayload: - return DocumentPayload( - uri=doc["title"], - content=doc["content"], - title=doc["title"], - ) - - -def map_hotpotqa_retrieval(doc: Mapping[str, Any]) -> RetrievalSample | None: - supporting_facts = doc["supporting_facts"] - titles = supporting_facts["title"] - if not titles: - return None - - unique_titles = tuple(dict.fromkeys(titles)) - return RetrievalSample( - question=doc["question"], - expected_uris=unique_titles, - ) - - -def build_hotpotqa_case( - index: int, doc: Mapping[str, Any] -) -> Case[str, str, dict[str, str]]: - question_id = doc["id"] - question_type = doc["type"] - level = doc["level"] - - case_name = f"{index}_{question_id}" - - return Case( - name=case_name, - inputs=doc["question"], - expected_output=doc["answer"], - metadata={ - "question_id": str(question_id), - "type": str(question_type), - "level": str(level), - "case_index": str(index), - }, - ) - - -HOTPOTQA_SPEC = DatasetSpec( - key="hotpotqa", - db_filename="hotpotqa.lancedb", - document_loader=document_loader, - document_mapper=map_hotpotqa_document, - qa_loader=load_hotpotqa_validation, - qa_case_builder=build_hotpotqa_case, - retrieval_loader=load_hotpotqa_validation, - retrieval_mapper=map_hotpotqa_retrieval, - retrieval_evaluator=MAPEvaluator(), -) diff --git a/evaluations/evaluations/datasets/repliqa.py b/evaluations/evaluations/datasets/repliqa.py deleted file mode 100644 index 981750fb..00000000 --- a/evaluations/evaluations/datasets/repliqa.py +++ /dev/null @@ -1,60 +0,0 @@ -from collections.abc import Mapping -from typing import Any - -from datasets import Dataset, load_dataset -from pydantic_evals import Case - -from evaluations.config import DatasetSpec, DocumentPayload, RetrievalSample -from evaluations.evaluators import MRREvaluator - - -def load_repliqa_corpus() -> Dataset: - dataset_dict = load_dataset("ServiceNow/repliqa") - dataset = dataset_dict["repliqa_3"] - return dataset.filter(lambda doc: doc["document_topic"] == "News Stories") - - -def map_repliqa_document(doc: Mapping[str, Any]) -> DocumentPayload: - return DocumentPayload( - uri=str(doc["document_id"]), - content=doc["document_extracted"], - ) - - -def map_repliqa_retrieval(doc: Mapping[str, Any]) -> RetrievalSample | None: - expected_answer = doc["answer"] - if expected_answer == "The answer is not found in the document.": - return None - return RetrievalSample( - question=doc["question"], - expected_uris=(str(doc["document_id"]),), - ) - - -def build_repliqa_case( - index: int, doc: Mapping[str, Any] -) -> Case[str, str, dict[str, str]]: - document_id = doc["document_id"] - case_name = f"{index}_{document_id}" if document_id is not None else f"case_{index}" - return Case( - name=case_name, - inputs=doc["question"], - expected_output=doc["answer"], - metadata={ - "document_id": str(document_id), - "case_index": str(index), - }, - ) - - -REPLIQA_SPEC = DatasetSpec( - key="repliqa", - db_filename="repliqa.lancedb", - document_loader=load_repliqa_corpus, - document_mapper=map_repliqa_document, - qa_loader=load_repliqa_corpus, - qa_case_builder=build_repliqa_case, - retrieval_loader=load_repliqa_corpus, - retrieval_mapper=map_repliqa_retrieval, - retrieval_evaluator=MRREvaluator(), -) diff --git a/evaluations/tests/test_benchmark.py b/evaluations/tests/test_benchmark.py index 43686262..030ff492 100644 --- a/evaluations/tests/test_benchmark.py +++ b/evaluations/tests/test_benchmark.py @@ -74,19 +74,19 @@ class TestBuildExperimentMetadata: class TestResolveDataset: def test_valid_dataset(self) -> None: - spec = _resolve_dataset("repliqa") - assert spec.key == "repliqa" + spec = _resolve_dataset("wix") + assert spec.key == "wix" def test_case_insensitive(self) -> None: - spec = _resolve_dataset("REPLIQA") - assert spec.key == "repliqa" + spec = _resolve_dataset("WIX") + assert spec.key == "wix" def test_unknown_dataset_raises(self) -> None: with pytest.raises(typer.BadParameter, match="Unknown dataset 'nonexistent'"): _resolve_dataset("nonexistent") def test_error_lists_valid_datasets(self) -> None: - with pytest.raises(typer.BadParameter, match="repliqa"): + with pytest.raises(typer.BadParameter, match="wix"): _resolve_dataset("nonexistent") diff --git a/evaluations/tests/test_datasets.py b/evaluations/tests/test_datasets.py index 1b3feffc..07a516ee 100644 --- a/evaluations/tests/test_datasets.py +++ b/evaluations/tests/test_datasets.py @@ -1,11 +1,5 @@ from pathlib import Path -from evaluations.datasets.hotpotqa import ( - build_hotpotqa_case, - extract_unique_documents, - map_hotpotqa_document, - map_hotpotqa_retrieval, -) from evaluations.datasets.open_rag_bench import ( build_orb_case, download_pdf, @@ -13,11 +7,6 @@ from evaluations.datasets.open_rag_bench import ( map_orb_document, map_orb_retrieval, ) -from evaluations.datasets.repliqa import ( - build_repliqa_case, - map_repliqa_document, - map_repliqa_retrieval, -) from evaluations.datasets.wix import ( build_wix_case, map_wix_document, @@ -25,50 +14,6 @@ from evaluations.datasets.wix import ( ) -class TestRepliqa: - def test_map_document(self) -> None: - doc = {"document_id": "doc-42", "document_extracted": "Some content here."} - payload = map_repliqa_document(doc) - assert payload.uri == "doc-42" - assert payload.content == "Some content here." - - def test_map_retrieval(self) -> None: - doc = { - "question": "What happened?", - "answer": "Something happened.", - "document_id": "doc-42", - } - sample = map_repliqa_retrieval(doc) - assert sample is not None - assert sample.question == "What happened?" - assert sample.expected_uris == ("doc-42",) - - def test_map_retrieval_skips_unanswerable(self) -> None: - doc = { - "question": "What?", - "answer": "The answer is not found in the document.", - "document_id": "doc-1", - } - assert map_repliqa_retrieval(doc) is None - - def test_build_case(self) -> None: - doc = { - "document_id": "doc-7", - "question": "Why?", - "answer": "Because.", - } - case = build_repliqa_case(3, doc) - assert case.name == "3_doc-7" - assert case.inputs == "Why?" - assert case.expected_output == "Because." - assert case.metadata == {"document_id": "doc-7", "case_index": "3"} - - def test_build_case_none_document_id(self) -> None: - doc = {"document_id": None, "question": "Q?", "answer": "A."} - case = build_repliqa_case(1, doc) - assert case.name == "case_1" - - class TestWix: def test_map_document_with_all_fields(self) -> None: doc = { @@ -136,78 +81,6 @@ class TestWix: assert case.name == "case_1" -class TestHotpotQA: - def test_map_document(self) -> None: - doc = {"title": "Albert Einstein", "content": "Was a physicist."} - payload = map_hotpotqa_document(doc) - assert payload.uri == "Albert Einstein" - assert payload.content == "Was a physicist." - assert payload.title == "Albert Einstein" - - def test_map_retrieval(self) -> None: - doc = { - "question": "Who was Einstein?", - "supporting_facts": {"title": ["Albert Einstein", "Physics"]}, - } - sample = map_hotpotqa_retrieval(doc) - assert sample is not None - assert sample.expected_uris == ("Albert Einstein", "Physics") - - def test_map_retrieval_deduplicates_titles(self) -> None: - doc = { - "question": "Q?", - "supporting_facts": {"title": ["A", "B", "A"]}, - } - sample = map_hotpotqa_retrieval(doc) - assert sample is not None - assert sample.expected_uris == ("A", "B") - - def test_map_retrieval_no_titles(self) -> None: - doc = {"question": "Q?", "supporting_facts": {"title": []}} - assert map_hotpotqa_retrieval(doc) is None - - def test_build_case(self) -> None: - doc = { - "id": "abc123", - "question": "What is X?", - "answer": "X is Y.", - "type": "comparison", - "level": "hard", - } - case = build_hotpotqa_case(5, doc) - assert case.name == "5_abc123" - assert case.inputs == "What is X?" - assert case.expected_output == "X is Y." - assert case.metadata == { - "question_id": "abc123", - "type": "comparison", - "level": "hard", - "case_index": "5", - } - - def test_extract_unique_documents(self) -> None: - # Simulate a minimal dataset with context - dataset = [ - { - "context": { - "title": ["Doc A", "Doc B"], - "sentences": [["Sentence 1."], ["Sentence 2.", " More."]], - } - }, - { - "context": { - "title": ["Doc A", "Doc C"], - "sentences": [["Dupe."], ["Sentence 3."]], - } - }, - ] - docs = extract_unique_documents(dataset) # type: ignore[arg-type] # ty: ignore[invalid-argument-type] - assert len(docs) == 3 - titles = [d["title"] for d in docs] - assert titles == ["Doc A", "Doc B", "Doc C"] - assert docs[1]["content"] == "Sentence 2. More." - - class TestOpenRAGBench: def test_map_document(self, tmp_path: Path) -> None: # Pre-create a cached PDF