Stop failing a correct answer for naming the other document
Both S6 cases were scored wrong while answering correctly: the model gave the right uri and database, then added a note distinguishing the near-name document, and the forbidden-distractor rule rejected it for mentioning the twin's uri at all. Naming the other document as a different document is correct behaviour, and better than the minimum. Forbidding a distractor belongs on B3, where two elevations cannot both be the answer; here it punished disambiguation. S6 now requires the uri and then the database that holds it.
This commit is contained in:
parent
d2b6b78d16
commit
ad06d6a497
1 changed files with 8 additions and 5 deletions
|
|
@ -707,18 +707,21 @@ def surface_rows() -> list[dict]:
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
# S6 — per-document metadata, with the twin's uri forbidden.
|
# S6 — per-document metadata: the uri and then the database that holds it.
|
||||||
for north_name, south_name in NEAR_NAME_PAIRS[:2]:
|
#
|
||||||
|
# The twin's uri is deliberately NOT forbidden here. Naming the other
|
||||||
|
# document as a different document is correct behaviour, and better than the
|
||||||
|
# minimum; forbidding it failed two answers that were right. Forbidding a
|
||||||
|
# distractor belongs on B3, where two elevations cannot both be the answer.
|
||||||
|
for north_name, _south_name in NEAR_NAME_PAIRS[:2]:
|
||||||
s = station(north_name, NORTHERN)
|
s = station(north_name, NORTHERN)
|
||||||
twin = station(south_name, SOUTHERN)
|
|
||||||
rows.append(
|
rows.append(
|
||||||
_row(
|
_row(
|
||||||
f"s6-{s.slug}",
|
f"s6-{s.slug}",
|
||||||
"S6",
|
"S6",
|
||||||
f"What is the uri of the document titled {s.title!r}, and which "
|
f"What is the uri of the document titled {s.title!r}, and which "
|
||||||
"database holds it?",
|
"database holds it?",
|
||||||
expected_ordered=[s.uri],
|
expected_ordered=[s.uri, s.database],
|
||||||
forbidden_text=[twin.uri],
|
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
return rows
|
return rows
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue