Stop failing a correct answer for naming the other document

Both S6 cases were scored wrong while answering correctly: the model gave
the right uri and database, then added a note distinguishing the near-name
document, and the forbidden-distractor rule rejected it for mentioning the
twin's uri at all.

Naming the other document as a different document is correct behaviour, and
better than the minimum. Forbidding a distractor belongs on B3, where two
elevations cannot both be the answer; here it punished disambiguation. S6
now requires the uri and then the database that holds it.
This commit is contained in:
Yiorgis Gozadinos 2026-08-26 15:40:29 +03:00
parent d2b6b78d16
commit ad06d6a497
No known key found for this signature in database

View file

@ -707,18 +707,21 @@ def surface_rows() -> list[dict]:
)
)
# S6 — per-document metadata, with the twin's uri forbidden.
for north_name, south_name in NEAR_NAME_PAIRS[:2]:
# S6 — per-document metadata: the uri and then the database that holds it.
#
# The twin's uri is deliberately NOT forbidden here. Naming the other
# document as a different document is correct behaviour, and better than the
# minimum; forbidding it failed two answers that were right. Forbidding a
# distractor belongs on B3, where two elevations cannot both be the answer.
for north_name, _south_name in NEAR_NAME_PAIRS[:2]:
s = station(north_name, NORTHERN)
twin = station(south_name, SOUTHERN)
rows.append(
_row(
f"s6-{s.slug}",
"S6",
f"What is the uri of the document titled {s.title!r}, and which "
"database holds it?",
expected_ordered=[s.uri],
forbidden_text=[twin.uri],
expected_ordered=[s.uri, s.database],
)
)
return rows