The two hard gates are pass/fail, not rates: any scope leak or attribution
error is a bug. The report names the offending cases and prints a greppable
GATES: PASSED / FAILED line, since this dataset is an acceptance gate rather
than a number to watch drift on.
The attribution gate covers only families where exactly one database is
correct. B5 is answered by two, so citing both is right there and policing
it would report a bug that is not one.
Surface coverage comes from the model's own Python, now carried on the run
result: counting executions said how often code ran but never which surface
it reached. Coverage is measured, not required, so a surface nothing reaches
is a finding about whether it earns its place in the VFS.
answer_correct is recorded as a score rather than a boolean, since booleans
become assertions and the run reads its accuracy headline from scores. The
two gates stay boolean assertions, which is what they are.
DatasetSpec grows a report_hook so this lives with the dataset instead of
becoming a key check in the shared QA reporting.
The reference-config invariant said a dataset with a deterministic evaluator
must declare no judge. That is not true here: RefusalJudge runs on any case
carrying an answerability label, which B6 and B7 do, so its sampling has to
be pinned. The assertion now requires a pinned block wherever a judge runs
and allows its absence only as the claim that none does.
Two dataset keys, because an S-family surface question cannot pass under
the RAG target and scoring it there would be noise: multidb carries the
behaviour families, multidb_surfaces the sandbox surfaces.
Scoring is deterministic. The answers are known numbers, names and counts,
so a red gate means the code is broken rather than that a judge spiralled.
Numbers are extracted and compared, since "1,240 metres" and "1240 m" are
both legitimate. B1 and B3 require the gold value present AND the twin's
absent: a hedge naming both elevations passes a presence check while
demonstrating the confusion those families exist to provoke. Refusals stay
judged, via the answerability labels the existing RefusalJudge reads, since
a phrase matcher keys on wording the model may never use.
B3 and B4 carry 9 and 10 instances because their gates are pass/fail and a
handful of cases is not evidence of absence. Half the B4 instances ask about
a near-name pair member with its twin excluded, so honouring scope costs the
model the other strong match instead of being free.
B2 rotates the order it lists the databases. RRF ties resolve to insertion
order, which is the configured order, so a fixed order would measure
ordering rather than fusion.
A scope travels in the case inputs via ScopedQuestion, since the task
function receives inputs and never metadata.
