The build-time assertion fired on the first real build: the chunker keeps the
whole twelve-row table in one chunk at chunk_size 256 and at 128, which would
have let a single search hit answer the whole-document question. 64 splits it.
The cost is a corpus chunked more finely than a real one. The alternative is a
longer table at a realistic chunk size, which would change what the question
asks.
The two hard gates are pass/fail, not rates: any scope leak or attribution
error is a bug. The report names the offending cases and prints a greppable
GATES: PASSED / FAILED line, since this dataset is an acceptance gate rather
than a number to watch drift on.
The attribution gate covers only families where exactly one database is
correct. B5 is answered by two, so citing both is right there and policing
it would report a bug that is not one.
Surface coverage comes from the model's own Python, now carried on the run
result: counting executions said how often code ran but never which surface
it reached. Coverage is measured, not required, so a surface nothing reaches
is a finding about whether it earns its place in the VFS.
answer_correct is recorded as a score rather than a boolean, since booleans
become assertions and the run reads its accuracy headline from scores. The
two gates stay boolean assertions, which is what they are.
DatasetSpec grows a report_hook so this lives with the dataset instead of
becoming a key check in the shared QA reporting.
The reference-config invariant said a dataset with a deterministic evaluator
must declare no judge. That is not true here: RefusalJudge runs on any case
carrying an answerability label, which B6 and B7 do, so its sampling has to
be pinned. The assertion now requires a pinned block wherever a judge runs
and allows its absence only as the claim that none does.
Port 11433 serves nothing; Qwen3-Reranker-4B is on 11455. Qwen3.6 spends
its budget reasoning before it answers, and its 131072 window leaves ample
input space at 32768.
IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.
Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
