Say precisely which conversations go unenforced
The documentation claimed a conversation that has never cited anything is not enforced, which reads as though a first question could answer from fresh evidence without declaring it. Enforcement needs neither condition to hold: no evidence outcome in this question, and nothing cited earlier. The structured-output test accepted a redirect or a recorded violation, so a break in ending detection would still have passed on the backstop alone. The cite tool is available in that scenario, so it asserts the redirect.
This commit is contained in:
parent
eb9934a6e5
commit
cee0824744
3 changed files with 6 additions and 6 deletions
|
|
@ -3,7 +3,7 @@
|
|||
|
||||
### Added
|
||||
|
||||
- `CitationPolicyCapability` (`haiku.rag.capabilities.policy.create_capability`): registering it requires every answer to declare its grounding, in any conversation that has something to declare — this question retrieved evidence, or something was cited earlier. A question that ends undeclared is sent back once to record what grounded the answer already given, and is recorded in `CitationPolicyState.violations` if it finishes undeclared regardless. A conversation that has never cited anything is not enforced.
|
||||
- `CitationPolicyCapability` (`haiku.rag.capabilities.policy.create_capability`): registering it requires every answer to declare its grounding, in any conversation that has something to declare — this question retrieved evidence, or something was cited earlier. A question that ends undeclared is sent back once to record what grounded the answer already given, and is recorded in `CitationPolicyState.violations` if it finishes undeclared regardless. A conversation with neither a current-question evidence outcome nor any earlier citation is not enforced.
|
||||
- `haiku.rag.capabilities.evidence.discover_evidence()` and `DiscoveredEvidence`, moved out of `compaction` so both optional capabilities share them. `RAGCapabilityBase.cite_available`.
|
||||
- `EvidenceCompactionCapability` (`haiku.rag.capabilities.compaction.create_capability`): registering it replaces earlier questions' evidence on the model request with the evidence that was cited, grouped by the question that cited it, cited page images re-attached, other earlier evidence returns reduced to a receipt. Requests only; `all_messages()` is untouched. No configuration.
|
||||
- `RAGState.evidence` / `AnalysisState.evidence` (`CapabilityEvidenceRecord`): which evidence a capability retrieved and cited, per question, keyed by message-count question identities and epochs. `haiku.rag.capabilities.ledger.citation_status(records, question=...)` derives `missing` / `grounded` / `ungrounded` across capabilities.
|
||||
|
|
|
|||
|
|
@ -100,8 +100,8 @@ something, which stays available to later answers. So a follow-up about evidence
|
|||
earlier is enforced even though it searched nothing — that case is the reason the
|
||||
capability exists. It also means that once anything has been cited, later turns are
|
||||
enforced too, a greeting included; the model satisfies the policy by citing an empty
|
||||
list, at the cost of one extra request. A conversation that has never cited anything is
|
||||
not enforced at all.
|
||||
list, at the cost of one extra request. A conversation with neither a current-question
|
||||
evidence outcome nor any earlier citation is not enforced.
|
||||
|
||||
Exactly one policy capability makes the decision, however many evidence capabilities
|
||||
are registered, so two of them cannot each demand a citation for one answer.
|
||||
|
|
|
|||
|
|
@ -461,9 +461,9 @@ async def test_a_structured_output_answer_does_not_escape_enforcement(temp_db_pa
|
|||
with patch.object(RAGCapability, "_search", stub_search):
|
||||
await agent.run("what does the supervisor do?", deps=deps)
|
||||
|
||||
redirected = [p for p in prompts_of(sent[-1]) if CITATION_REDIRECT_TAG in p]
|
||||
violations = deps.state.get("citation_policy", {}).get("violations", [])
|
||||
assert redirected or violations
|
||||
# The cite tool is available here, so the redirect is what must happen; the
|
||||
# backstop recording a violation would pass an `or` even with detection broken.
|
||||
assert [p for p in prompts_of(sent[-1]) if CITATION_REDIRECT_TAG in p]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
|
|
|||
Loading…
Reference in a new issue