Stop failing a correct answer for naming the other document
Both S6 cases were scored wrong while answering correctly: the model gave the right uri and database, then added a note distinguishing the near-name document, and the forbidden-distractor rule rejected it for mentioning the twin's uri at all. Naming the other document as a different document is correct behaviour, and better than the minimum. Forbidding a distractor belongs on B3, where two elevations cannot both be the answer; here it punished disambiguation. S6 now requires the uri and then the database that holds it.
This commit is contained in:
parent
d2b6b78d16
commit
ad06d6a497
1 changed files with 8 additions and 5 deletions
|
|
@ -707,18 +707,21 @@ def surface_rows() -> list[dict]:
|
|||
)
|
||||
)
|
||||
|
||||
# S6 — per-document metadata, with the twin's uri forbidden.
|
||||
for north_name, south_name in NEAR_NAME_PAIRS[:2]:
|
||||
# S6 — per-document metadata: the uri and then the database that holds it.
|
||||
#
|
||||
# The twin's uri is deliberately NOT forbidden here. Naming the other
|
||||
# document as a different document is correct behaviour, and better than the
|
||||
# minimum; forbidding it failed two answers that were right. Forbidding a
|
||||
# distractor belongs on B3, where two elevations cannot both be the answer.
|
||||
for north_name, _south_name in NEAR_NAME_PAIRS[:2]:
|
||||
s = station(north_name, NORTHERN)
|
||||
twin = station(south_name, SOUTHERN)
|
||||
rows.append(
|
||||
_row(
|
||||
f"s6-{s.slug}",
|
||||
"S6",
|
||||
f"What is the uri of the document titled {s.title!r}, and which "
|
||||
"database holds it?",
|
||||
expected_ordered=[s.uri],
|
||||
forbidden_text=[twin.uri],
|
||||
expected_ordered=[s.uri, s.database],
|
||||
)
|
||||
)
|
||||
return rows
|
||||
|
|
|
|||
Loading…
Reference in a new issue