Say precisely which conversations go unenforced
The documentation claimed a conversation that has never cited anything is not enforced, which reads as though a first question could answer from fresh evidence without declaring it. Enforcement needs neither condition to hold: no evidence outcome in this question, and nothing cited earlier. The structured-output test accepted a redirect or a recorded violation, so a break in ending detection would still have passed on the backstop alone. The cite tool is available in that scenario, so it asserts the redirect.
This commit is contained in:
parent
eb9934a6e5
commit
cee0824744
3 changed files with 6 additions and 6 deletions
|
|
@ -3,7 +3,7 @@
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
- `CitationPolicyCapability` (`haiku.rag.capabilities.policy.create_capability`): registering it requires every answer to declare its grounding, in any conversation that has something to declare — this question retrieved evidence, or something was cited earlier. A question that ends undeclared is sent back once to record what grounded the answer already given, and is recorded in `CitationPolicyState.violations` if it finishes undeclared regardless. A conversation that has never cited anything is not enforced.
|
- `CitationPolicyCapability` (`haiku.rag.capabilities.policy.create_capability`): registering it requires every answer to declare its grounding, in any conversation that has something to declare — this question retrieved evidence, or something was cited earlier. A question that ends undeclared is sent back once to record what grounded the answer already given, and is recorded in `CitationPolicyState.violations` if it finishes undeclared regardless. A conversation with neither a current-question evidence outcome nor any earlier citation is not enforced.
|
||||||
- `haiku.rag.capabilities.evidence.discover_evidence()` and `DiscoveredEvidence`, moved out of `compaction` so both optional capabilities share them. `RAGCapabilityBase.cite_available`.
|
- `haiku.rag.capabilities.evidence.discover_evidence()` and `DiscoveredEvidence`, moved out of `compaction` so both optional capabilities share them. `RAGCapabilityBase.cite_available`.
|
||||||
- `EvidenceCompactionCapability` (`haiku.rag.capabilities.compaction.create_capability`): registering it replaces earlier questions' evidence on the model request with the evidence that was cited, grouped by the question that cited it, cited page images re-attached, other earlier evidence returns reduced to a receipt. Requests only; `all_messages()` is untouched. No configuration.
|
- `EvidenceCompactionCapability` (`haiku.rag.capabilities.compaction.create_capability`): registering it replaces earlier questions' evidence on the model request with the evidence that was cited, grouped by the question that cited it, cited page images re-attached, other earlier evidence returns reduced to a receipt. Requests only; `all_messages()` is untouched. No configuration.
|
||||||
- `RAGState.evidence` / `AnalysisState.evidence` (`CapabilityEvidenceRecord`): which evidence a capability retrieved and cited, per question, keyed by message-count question identities and epochs. `haiku.rag.capabilities.ledger.citation_status(records, question=...)` derives `missing` / `grounded` / `ungrounded` across capabilities.
|
- `RAGState.evidence` / `AnalysisState.evidence` (`CapabilityEvidenceRecord`): which evidence a capability retrieved and cited, per question, keyed by message-count question identities and epochs. `haiku.rag.capabilities.ledger.citation_status(records, question=...)` derives `missing` / `grounded` / `ungrounded` across capabilities.
|
||||||
|
|
|
||||||
|
|
@ -100,8 +100,8 @@ something, which stays available to later answers. So a follow-up about evidence
|
||||||
earlier is enforced even though it searched nothing — that case is the reason the
|
earlier is enforced even though it searched nothing — that case is the reason the
|
||||||
capability exists. It also means that once anything has been cited, later turns are
|
capability exists. It also means that once anything has been cited, later turns are
|
||||||
enforced too, a greeting included; the model satisfies the policy by citing an empty
|
enforced too, a greeting included; the model satisfies the policy by citing an empty
|
||||||
list, at the cost of one extra request. A conversation that has never cited anything is
|
list, at the cost of one extra request. A conversation with neither a current-question
|
||||||
not enforced at all.
|
evidence outcome nor any earlier citation is not enforced.
|
||||||
|
|
||||||
Exactly one policy capability makes the decision, however many evidence capabilities
|
Exactly one policy capability makes the decision, however many evidence capabilities
|
||||||
are registered, so two of them cannot each demand a citation for one answer.
|
are registered, so two of them cannot each demand a citation for one answer.
|
||||||
|
|
|
||||||
|
|
@ -461,9 +461,9 @@ async def test_a_structured_output_answer_does_not_escape_enforcement(temp_db_pa
|
||||||
with patch.object(RAGCapability, "_search", stub_search):
|
with patch.object(RAGCapability, "_search", stub_search):
|
||||||
await agent.run("what does the supervisor do?", deps=deps)
|
await agent.run("what does the supervisor do?", deps=deps)
|
||||||
|
|
||||||
redirected = [p for p in prompts_of(sent[-1]) if CITATION_REDIRECT_TAG in p]
|
# The cite tool is available here, so the redirect is what must happen; the
|
||||||
violations = deps.state.get("citation_policy", {}).get("violations", [])
|
# backstop recording a violation would pass an `or` even with detection broken.
|
||||||
assert redirected or violations
|
assert [p for p in prompts_of(sent[-1]) if CITATION_REDIRECT_TAG in p]
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue