Exclude FRAMES questions whose articles were deleted from Wikipedia

evaluations/README.md

@@ -10,7 +10,7 @@ Contains evaluation scripts for benchmarking RAG retrieval and QA performance. A
 
 - HotpotQA (`hotpotqa`) — multi-hop QA over Wikipedia paragraphs (distractor validation split, 7,405 questions, two gold documents per question)
 - MTRAG ClapNQ (`mtrag_clapnq`, `mtrag_clapnq_rewrite`) — IBM's multi-turn RAG benchmark, ClapNQ (Wikipedia) domain: 183,408 passages, 208 retrieval queries with binary qrels, 224 generation tasks. The base key retrieves with the raw last user turn; the `_rewrite` variant uses the human standalone rewrites (both share one database). Retrieval reports Recall@5/@10, nDCG@5/@10, and MAP against IBM's published setup. QA replays each task's reference conversation prefix as message history and answers the final turn; the judge sees the conversation as a transcript, citation MAP is scored only on turns with gold passages, and refusal precision/recall is reported against the answerability labels. Generation scores are internal (our judge and rubric), not comparable with IBM's published generation numbers. The `mtrag_clapnq_live` key replays whole conversations (one case per conversation, `--limit` counts conversations) through a single capability session, carrying the model's own answers and tool history across turns; it reports the same outcomes per turn plus micro (per-turn) and macro (per-conversation) aggregates.
-- FRAMES (`frames`) — multi-hop QA (824 questions, 2-23 gold Wikipedia articles per question). The corpus is the union of the ~2.5k linked articles, fetched from the Wikipedia REST API at current revision (revision id and fetch date recorded in the article cache) with navigation chrome stripped. There is no official FRAMES evaluation setup; numbers here correspond to the paper's multi-step retrieval setting (fixed corpus, agentic retrieval, judged accuracy) and are not comparable to its closed-book, oracle-prompt, or web-search settings. Answers were authored against ~2024 revisions and may have drifted with article content.
+- FRAMES (`frames`) — multi-hop QA (822 questions, 2-23 gold Wikipedia articles per question; 2 of the original 824 questions are excluded because a linked article has been deleted from Wikipedia). The corpus is the union of the 2,521 linked articles, fetched from the Wikipedia REST API at current revision (revision id and fetch date recorded in the article cache) with navigation chrome stripped. There is no official FRAMES evaluation setup; numbers here correspond to the paper's multi-step retrieval setting (fixed corpus, agentic retrieval, judged accuracy) and are not comparable to its closed-book, oracle-prompt, or web-search settings. Answers were authored against ~2024 revisions and may have drifted with article content.
 - OpenRAG Bench, two variants:
   - `orb_text` — text embedder (`qwen3-embedding:4b`, 2560-dim) with VLM picture descriptions baked into chunk content at ingest. Use for text-only retrieval/QA against figure-rich corpora.
   - `orb_multimodal` — multimodal embedder (`qwen3-vl-embedding-8b`, 4096-dim) with picture vectors in the same space as text. Use for cross-modal retrieval (text-as-query → figure hits, image-as-query) and vision QA where the figure itself is the answer.

evaluations/evaluations/datasets/frames.py

@@ -33,10 +33,28 @@ THROTTLE_SECONDS = 1.0
 RATE_LIMIT_BACKOFF_SECONDS = 60.0
 
 
+# Articles deleted from Wikipedia since FRAMES was authored; the questions
+# linking them have lost their evidence and are excluded from the benchmark.
+_DELETED_ARTICLES = frozenset(
+    {
+        "https://en.wikipedia.org/wiki/Nemanja_Marković",
+        "https://en.wikipedia.org/wiki/Jack_Vance_(tennis)",
+    }
+)
+
+
 def load_frames_test() -> Dataset:
     return load_dataset("google/frames-benchmark")["test"]
 
 
+def question_is_answerable(doc: Mapping[str, Any]) -> bool:
+    return not _DELETED_ARTICLES & set(question_expected_uris(doc))
+
+
+def load_frames_questions() -> Dataset:
+    return load_frames_test().filter(question_is_answerable)
+
+
 def parse_wiki_links(raw: str) -> list[str]:
     """Extract URLs from a `wiki_links` value.
 
@@ -231,7 +249,7 @@ def load_frames_corpus() -> list[dict[str, Any]]:
     global _cached_corpus
     if _cached_corpus is None:
         uris: dict[str, None] = {}
-        for doc in load_frames_test():
+        for doc in load_frames_questions():
             for uri in question_expected_uris(doc):
                 uris.setdefault(uri)
         cache_dir = get_cache_dir()
@@ -301,9 +319,9 @@ FRAMES_SPEC = DatasetSpec(
     db_filename="frames.lancedb",
     document_loader=document_loader,
     document_mapper=map_frames_document,
-    qa_loader=load_frames_test,
+    qa_loader=load_frames_questions,
     qa_case_builder=build_frames_case,
-    retrieval_loader=load_frames_test,
+    retrieval_loader=load_frames_questions,
     retrieval_mapper=map_frames_retrieval,
     retrieval_evaluators=[MAPEvaluator()],
     citation_evaluator=CitationMAPEvaluator(),

evaluations/tests/test_datasets.py

@@ -11,6 +11,7 @@ from evaluations.datasets.frames import (
     normalize_wiki_url,
     parse_revid,
     parse_wiki_links,
+    question_is_answerable,
     strip_navigation,
 )
 from evaluations.datasets.hotpotqa import (
@@ -675,7 +676,7 @@ class TestFrames:
         monkeypatch.setattr(frames, "_cached_corpus", None)
         monkeypatch.setattr(
             frames,
-            "load_frames_test",
+            "load_frames_questions",
             lambda: [
                 {
                     "wiki_links": "['https://en.wikipedia.org/wiki/A', "
@@ -688,3 +689,12 @@ class TestFrames:
         )
         with pytest.raises(RuntimeError, match="0/2"):
             frames.load_frames_corpus()
+
+    def test_question_with_deleted_article_is_excluded(self) -> None:
+        gone = {
+            "wiki_links": "['https://en.wikipedia.org/wiki/Jack_Vance_(tennis)', "
+            "'https://en.wikipedia.org/wiki/Capybara']"
+        }
+        kept = {"wiki_links": "['https://en.wikipedia.org/wiki/Capybara']"}
+        assert question_is_answerable(gone) is False
+        assert question_is_answerable(kept) is True