urlparse().path keeps the leading slash in front of a Windows drive, so
file:///C:/docs/a.pdf read as \C:\docs\a.pdf and the ingester reported
"File does not exist" for every file it discovered. url2pathname is the
stdlib conversion that strips it, per platform.
Four sites each decided both "is this local" and "what path is this":
FSSource._uri_to_path and supports, resolve_adhoc_fetcher,
create_document_from_source and check_source_accessible, and convert.
is_local_uri and uri_to_path in haiku.rag.uri own those two decisions now,
which closes two more cases of the same root cause. A bare C:\docs\a.pdf
parses with scheme "c", so add-src raised "No source adapter for URI scheme
'c'" and convert silently treated the path as raw text. And convert and
check_source_accessible never percent-decoded at all, so a file named
a[b] c.md read as missing on Linux and macOS too.
A file URI's host is reattached after conversion rather than passed to
url2pathname, which as of 3.14 rejects a non-local authority off Windows.
file:////server/share is the empty-authority spelling of a UNC path, its
host being the first path segment, so that host is normalised into the
authority before conversion. Output is identical on 3.12, 3.13 and 3.14.
The ad-hoc FS fetcher roots at the path's own anchor rather than "/", which
on Windows is only the current drive.
test_uri.py runs on ubuntu, macos and windows across 3.13 and 3.14 without
the project installed: --noconftest because the repo conftest imports
dependencies that job does not need, and -o addopts= to drop the
repository's -n auto. The Windows legs are what cover the drive conversion.
Fixes#574.
cli.py carried 40 pragmas over whole command bodies and app.py a
class-level one over all 412 statements, while tests/test_cli.py already
drove 29 commands through CliRunner. The pragmas hid lines the suite
executed, so the 100% gate understated real coverage and gave new CLI code
no scrutiny.
Both are measured now. 38 CLI tests stub HaikuRAGApp and assert the parsed
arguments reach the right method; 60 app tests stub the client and record
the console, pinning what each command asks for and what it prints. The only
pragma left in either file is cli() under __main__. The omit list is back to
the two Textual TUIs.
Three defects the coverage surfaced:
haiku-rag settings masked only top-level secret-named fields, so nested ones
printed in full — lancedb.api_key, providers.docling_serve.api_key, WebDAV
source passwords. It uses redact_secrets, which walks the dump.
chat guarded the wrong thing: haiku.rag.chat imports without Textual, and
run_chat raises when it imports ChatApp, so the missing extra escaped as an
ImportError. The guard is on the call. inspector raises at module import
instead, so inspect keeps its guard on the import; each has a test that
fails the way the real installation fails.
search --limit/--search-type and history --limit default to None so the
config resolves the default. Now pinned.
CI passed --cov=haiku while pyproject declares source = ["haiku_rag_slim"];
pass --cov and let the config decide. build-docs.yml only ran on push to
main, so a broken docs build merged and failed at deploy: build on pull
requests, with configure-pages, upload-pages-artifact and deploy gated to
push, and a per-ref concurrency group.
engine.py held four unrelated things: what the tables are, how to open a
connection, how to read a database's state, and the Store that coordinates
writes. At 1240 lines the Store's own concerns — locks, migrations, vacuum,
tags — were hard to find among them.
Table records, Arrow schemas, index_specs, ensure_indexes, REQUIRED_TABLES
and query_to_pydantic move to store/schema.py, which imports nothing from
haiku.rag: it describes the tables and never opens or mutates one.
gather_database_info, get_database_stats, DatabaseInfo and its result models
move to store/info.py. Nothing in Store calls them — they are read paths for
the CLI, doctor, inspector and ingester API — so info depends on engine and
not the reverse.
engine.py keeps the Store, ConnectionMode, connect_lancedb, the tag helpers
and the restore-order and retention constants. No re-exports: importers
point at the new modules.
test_app_info_uses_connect_lancedb_for_remote patched
haiku.rag.store.engine.connect_lancedb; gather_database_info now binds that
name in info.py, so the patch targets where the call is looked up.
haiku.rag.ingester.sources was never ingester-only: one-shot client
ingestion resolves adapters through it (create_document_from_source), and
convert() now fetches through HTTPSource, so the core client imported into
the ingester package to reach them.
Move the package to haiku.rag.sources and update every import. No shims:
haiku.rag.ingester.sources is gone.
The haiku.rag.sources plugin entry-point group is unchanged, so third-party
source packages need no edit — the group name now matches the module path it
always implied.
Source unit tests move to tests/sources/. test_source_plugins.py stays in
tests/ingester/: it drives a PeriodicPoller against the job repo, so it is
plugin wiring through ingester machinery rather than a source test.
Five call sites repeated the same post-conversion preparation: store the
Docling representation and resolve a title when none was supplied.
_prepare_and_title now owns that sequence. update_document continues to
call _prepare_document_from_docling directly because an explicit update
must preserve an existing empty title.
create_document, both content-replacement branches of update_document,
and source ingestion embedded eagerly before passing chunks to a
persistence funnel that checked them again. The funnels now own
embedding, including the checks required by import_document and
import_documents for caller-supplied chunks.
Move the document.embed span into ensure_chunks_embedded after its early
return. Every path that performs embedding is now instrumented, while
operations whose chunks are already embedded emit no span.
convert() previously used its own HTTP client and temporary-file path.
Route URL conversion through HTTPSource, matching source ingestion, and
move _write_fetch_body to processing.py so both paths share temporary
file handling without an import cycle.
Add walk_files for filesystem enumeration and use it from both
FSSource.discover and one-shot directory ingestion. Symlink escape
filtering now has one implementation.
RAGState and AnalysisState each declared the same five fields, so the generic
base could not name them: StateT was bound to BaseModel, and every access
went through cast(Any, state), a getattr by string, or a loop clearing fields
by name so it could skip the one only AnalysisState has.
EvidenceState declares them once. RAGState adds nothing, AnalysisState adds
executions and overrides begin_invocation to clear them. StateT binds to
EvidenceState, which removes all ten casts and both state-shape getattrs; the
three getattr(ctx.deps, "state") probes stay, since those check a
host-supplied object rather than our own state.
discover_evidence reached into capability.state for two fields. It now asks
through evidence_record() and citation_index(), alongside the
evidence_tool_names() and cite_available accessors it already used. The eval
runner's _RagLikeState protocol and the chat app's getattr reads described
this shape from outside and are gone.
Compatibility is semantic JSON-object equivalence, not bytes: field names and
nesting are unchanged, so a dict stored by 0.75.0 loads and re-dumps equal,
but deriving from a shared base reorders AnalysisState's keys. Nothing
serializes, hashes or string-compares this state — every carry point
re-validates by key.
search.limit was documented as 10 in three places while the default is 5.
The documented way to disable reranking, provider: "", is a valid
ModelConfig, so it raised "Unknown reranking provider" — disabling means
omitting reranking.model or setting it to null. The inline provider list
named four of the six rerankers. prompts.picture_description: null fails
validation, since the field is a non-optional str.
storage.data_dir: "" coerced to Path("") — the working directory — while two
doc pages promise the platform default and soliplex's example config relies
on it. Empty or whitespace now resolves to the platform directory; an
explicit "." is still honoured, so a config that wants the working directory
says so.
Three tests keep this from drifting again: every fenced yaml block in the
docs validates against AppConfig, every value in the complete example either
equals its default or is listed as a deliberate deviation, and empty
data_dir resolves to the platform default.
init-config's test reimplemented the command body instead of invoking it,
which is why the command carried a coverage pragma. It now goes through
CliRunner, with the refuse-to-overwrite guard covered too.
Every section inherited plain BaseModel, so unknown keys were dropped
silently: providers.docling_serve.timeout was documented for months while
being ignored, and a typo in any setting took the default. Sections now
derive from ConfigModel, which forbids extras, so a stale or misspelled key
fails with its path. This already found search.context_radius in a live app
config and providers.vllm in soliplex's example.
converter, chunker and chunker_type are Literals. Sizes, limits,
dimensions, token budgets, attempt counts and breaker thresholds must be
positive; retention, delays, intervals and cooldowns non-negative;
similarity_threshold within 0-1; port within 0-65535. port 0 keeps its
OS-assigned meaning and worker_count allows 0 for an API-and-reaper-only
process.
get_reranker caught ImportError and returned None, so a configured reranker
whose extra was missing silently disappeared. It now propagates.
raise_missing_extra names the install command and re-raises when the failure
came from inside an installed package, so a broken transitive import is not
reported as a missing one. zeroentropy imported bare and now guards like the
others.
The haiku.rag package declares the jina extra. jina-local already worked
there through cross-encoder's transitive transformers and torch; the
resolved package set is unchanged, but the support is now promised rather
than inherited.
Provider fields stay unconstrained: get_model ends in a pass-through to
pydantic-ai for any provider it supports, so a Literal there would reject
valid configurations.
haiku.rag.config exported two configuration instances: the lazy _config
behind get_config/set_config, and Config, loaded at import time. Nothing
linked them, and eleven signatures captured Config as a default argument,
so set_config could not reach the factories, the client, the store or the
MCP server. reranking/base.py went further and snapshotted the configured
reranker name into a class attribute at import.
Config is removed. Internal defaults are config: AppConfig | None = None,
resolved through get_config() per call. RerankerBase._model is None and
CohereReranker takes its model name as an argument, like every other
reranker.
The suite patched attributes on Config while production read the instance
get_config() returns, a different object, so those patches were no-ops
waiting to happen. They now go through get_config().
Four call sites repeated lock, snapshot, try/except, restore. Each used
restore_table_versions, which restores in _tables() order — documents
first, contradicting RESTORE_TABLE_ORDER — and each caught Exception, so
a cancellation mid-write skipped rollback and left the earlier table
writes committed.
Store.write_transaction() holds the lock, snapshots under it, and rolls
back through _rollback_to_snapshot: RESTORE_TABLE_ORDER, shielded from
cancellation, absorbed cancellation re-delivered, rollback failure raised
with the original as cause. The two single-table update_meta sites keep
the bare lock.
The batch documents write moves inside the guarded body; it was outside
the try, so a failure there was never rolled back. Auto-vacuum is
scheduled after the transaction rather than inside it.
restore_table_versions is removed; those four sites were its only callers.
The setting was documented but did not exist on DoclingServeConfig, so it
was silently dropped, and DoclingServeClient.from_config never forwarded
the timeout parameter it already accepted. The per-request timeout was
therefore pinned at the constructor default of 300s with no way to change
it. Add the field, forward it, and reject a non-positive value.
Also parametrize over the checked-in *.yaml.example files and validate
each through AppConfig, so an example that no longer loads fails a test
rather than a user's first run.
create_document_from_source resolves a fetcher per call and never closed
it, so every one-shot URL or WebDAV ingest leaked an httpx connection
pool. Close it, but only when we built it: resolve_adhoc_fetcher returns
a caller-supplied source when one matches the URI, and the ingester keeps
those open across jobs.
Directory ingestion also yielded symlinked files resolving outside the
directory it was given. rglob does not recurse into symlinked
directories, so a symlinked file was the only way out of the tree; skip
those, as FSSource.discover already does.
The chunk repository docstring named client._ensure_chunks_embedded,
which does not exist.
FULL rebuild deleted a document before re-ingesting it from its URI, and
the handler around that logged and continued. A 404, a timeout or any
conversion error therefore removed the document permanently.
Deleting after a successful create is not an alternative:
create_document_from_source resolves the same URI to the existing
document and updates it in place, so a trailing delete would remove the
freshly rebuilt row.
Refresh in place instead. create_document_from_source takes an internal
force flag that skips the revision and MD5 short-circuits, so an
unchanged source is still re-converted, re-chunked and re-embedded into
the existing document, and the document id survives a rebuild.
A failed refresh now falls through to the stored-content path rather
than skipping the document: FULL recreates the chunks table before the
loop, so skipping left the document present but unsearchable until the
next rebuild.
The pending-batch flush moves out of the try. A failed flush is a lost
write and should abort the rebuild, not be logged and skipped.
Collapsing the caption text into `get_pictures_grouped` served the enrichment
path, which uses it, but the multimodal reranker discards the second return value
while still paying to read the column. That is the widest fan-out in the codebase,
`limit * 10` candidates, and it previously projected self_ref and picture_data
alone.
`with_text` is opt-in and off by default, so the cheap projection is what a caller
gets unless it asks for more. The reranker test asserts the projection as well as
the query count, since a count alone would not notice the column coming back.
Splitting the assembly into a passthrough pass and an expandable pass reordered
equal-scored results: the score sort that follows is stable, so the order results
arrive in is the tiebreak. Results are assembled in document_groups order again,
after the batched fetch rather than around it.
Also ports the caption negative cases the removed single-document test carried: a
table's caption and an ordinary text reference map to no picture.
`resolve_refs`, `get_items_in_range`, `get_caption_picture_refs` and
`get_all_items_grouped` have no callers left: the grouped equivalents serve every
path that used them. `get_all_items_grouped` had none even before this branch.
Tests whose subject was a removed method go with it. Tests that only used one to
fetch a fixture now use the grouped call, so what they assert is unchanged.
`_attach_picture_data` fetched picture bytes once per document, over the
`limit * 10` candidates reranking asks for, so it was the per-document fetch with
the most candidates behind it. It now issues one query however many documents the
candidates span: one for ten documents, as for one.
Removes `get_text_for_refs`, whose only caller now gets the text back with the
bytes from `get_pictures_grouped`.
`test_client_search_include_images_false_skips_lookup` returned no search
results, so asserting the picture accessor went uncalled held whatever the code
did. It now returns a picture-carrying result, making "did not fetch" the
assertion rather than "had nothing to fetch".
`expand_with_items` fetched its own inputs per document: one query to resolve
refs to positions, one for the window of items around them. A result set spanning
N documents cost 2N queries, which was 10 of the 18 measured for a limit=5 search
on a remote object-store corpus.
`expand_context` now does both fetches once for every document it is expanding,
and `expand_with_items` takes the positions and items it needs. Two queries for
one document, and two for five.
Each document keeps its own inclusive window in `get_items_in_ranges`. Positions
repeat across documents, so a shared range would splice one document's items into
another's context.
`_populate_image_data` ran its stages once per result document, so a result set
spanning N documents cost 4N `document_items` queries. Measured on a remote
object-store corpus, a limit=5 search with expansion was 18 queries, 16 of them
against `document_items`.
The stages now run once each across every document, and flat in document count:
two queries for the dependent caption-to-picture mapping when results ranked on a
caption, one for the picture bytes. Two queries for a picture-ref result set,
three at most.
Picture text comes back with the bytes rather than from a second query, since it
is on the same rows.
Predicates are per document, `(document_id = 'a' AND self_ref IN (…)) OR (…)`,
rather than `self_ref IN (union)`. self_ref and position values repeat across
documents, so a union predicate would return other documents' rows: for
picture_data that fetches blobs nobody asked for, and it can hand one document
another document's picture.
`client.ask` and `client.analyze` built their capability from a db_path, so the
capability opened a second connection to the database the client already had
open, once per call.
Ownership is now explicit rather than inferred. `rag` stays the connection the
capability opened and must close; `borrowed_rag` is a caller's, which
`_ensure_rag` prefers and `_close` never touches. Two fields rather than a flag,
so closing a borrowed connection is not expressible.
`for_run` still clears `rag` per run, since a run owns what it opens. It leaves
`borrowed_rag` alone: that connection belongs to the caller and outlives the run.
All ten tool bodies opened their own `HaikuRAG`, so every tool call paid a
connection open and, on object storage, refetched the index the previous call had
just cached. The client is now opened once, lazily so that calling a tool
function directly still works, and eagerly from the lifespan so an unopenable
database fails startup instead of every call. Teardown clears the cached client
in a finally, since `_lifespan_manager` can be re-entered and would otherwise
hand out a closed connection, including when the close itself fails.
`delete_document` no longer opens its own connection with `skip_validation=True`.
Keeping it separate broke consistency once connections became long-lived: the
delete committed on one connection while reads served from another, which with a
30s consistency interval showed the deleted document as still present. A
connection always sees its own writes, so sharing one is what makes delete
visible to the next read.
So the server no longer opts out of embedding-config validation. Drift that
validation rejects now fails MCP startup, where before the server started and
only `delete_document` worked while every read returned empty. Same-dimension
identity drift still starts a read-only server, matching every other read verb.
Delete under drift is now a CLI operation; CLAUDE.md and the CHANGELOG record it.
Opening a database ran `list_tables` three times, opened the settings table
twice, and read and parsed the same settings row three times: once for the stored
vector dimension, once for the version behind the migration check, and once for
config validation. On object storage each of those is a round trip.
`_initialize` now reads both once and threads them down. `_init_tables` and
`_check_migrations` take what it read instead of fetching their own copy, and
`validate_config_compatibility` accepts the settings it should compare against,
still reading for itself when called directly.
Passing the pre-init read to validation is equivalent: nothing between the read
and the validation rewrites `embeddings`, which is all it compares.
The settings read no longer swallows every exception. It did before, when the
only consequence was falling back to the configured vector dimension; now the
same empty result feeds the migration check, where it would read as version
0.0.0 and declare every migration pending. Only decode failures are tolerated,
and a decoded non-object normalizes to {} rather than reaching callers that
expect a mapping.
Every `Store` built its own connection with its own caches and discarded them on
close, so the index a vector query loads was refetched by the next connection.
On object storage that first fetch dominates: measured on a ~500k-chunk 2560-dim
corpus over a ~200ms link, the first query cost ~41s and the second ~3s, and a
new connection reusing the session cost ~7s instead of ~47s.
`connect_lancedb` now passes a process-wide session, keyed on the configured
cache sizes so a caller asking for different sizes gets its own.
Also sets `read_consistency_interval`, defaulting to 30s. It was None, meaning a
connection never re-checked for other processes' writes. Per-call connections hid
that; a shared session makes connections long-lived enough for a reader to go
stale against the ingester.
All three settings reject negatives at the config boundary. A negative cache size
raises OverflowError and a negative interval panics inside Lance, so neither is
catchable further in. Zero stays valid for both: no cache, and check on every
read.
The routing tests now assert the kwargs they care about rather than the full call
signature, since every connection carries the two new kwargs.
Also drops two things that were stale rather than merely verbose: the CLI docs
note for 0.75.0, which was the only release-tagged note in the docs tree while
the CHANGELOG already records that existing databases need `haiku-rag migrate`;
and "(created or corrected)" from the migration log line, left over from the
earlier behaviour that replaced wrong-typed indexes.
Adds the 0.75.0 upgrade, which brings a pre-existing database up to the index
set `_init_tables` now creates. It rewrites no table data, so unlike the earlier
data migrations its cost is the index builds alone, each of which reads the
column it indexes.
`ensure_indexes` ensures an index of the declared *type* covers each declared
column, rather than checking that the column is indexed at all. The distinction
is what makes it safe to run against a database of unknown provenance:
- A wrong-typed index no longer satisfies the check. A BTree on `label` covers
the column while losing the low-cardinality equality lookup the Bitmap is for.
- Nothing is dropped or converted away from. Two index types over one column can
be deliberate, serving different query shapes, so an index this function did
not declare survives even on a column it does. The one thing it overwrites is
an index at LanceDB's default name, `{column}_idx`, which is the name it
creates itself.
- A column already carrying the declared type is skipped, so a database with the
full set migrates instantly rather than re-sorting every indexed column.
- Undeclared columns are untouched, so a vector index on `chunks` survives.
It returns the columns it acted on, because a change is not always visible from
outside: adding a Bitmap beside an existing BTree leaves the column indexed
before and after.
The version bump to 0.75.0 is required, not incidental: `_set_initial_version`
stamps a new database with the installed package version, so a migration
numbered above it would be pending the moment the database was created.
`test_client_update_document_replaces_rows_with_bounded_versions` turns
auto_vacuum off. Indexing `documents` means a background vacuum now has an index
to maintain on that table, so `optimize()` writes a version where it previously
had nothing to do, and it landed inside the window the test measures. The
document update itself is still one version, so the bound stays exact.
`_init_tables` left `chunks.id`, `chunks.document_id` and `documents.id`
unindexed, so those lookups scanned the column. On object storage that is
network I/O per query, on paths that run per document: citation lookup,
delete-by-document, the re-ingest merge, and every dedup probe.
Declare the index set per table in `index_specs()` and apply it through
`ensure_indexes()`, which skips a column only when it is already indexed with
the declared type. Both halves of that are load-bearing. Skipping is required
because `create_index(replace=True)` rebuilds an identical index, writing a new
index and a new table version and orphaning the old files until the next vacuum.
Comparing the type is required because column coverage alone would let a
wrong-typed index stand, and a BTree on `label` silently loses the
low-cardinality equality lookup the Bitmap is there for.
Columns not declared for a table are left alone, so an externally created index
such as a vector index on `chunks` survives.
`_init_tables`, `recreate_embeddings_table`, `ChunkRepository.delete_all` and
`DocumentRepository.delete_all` now all route through it instead of repeating
their own subsets.
Also recreate `document_items` from `get_document_items_arrow_schema()` in
`DocumentRepository.delete_all`, which was using the LanceModel and so returned
`picture_data` as 32-bit `binary`.
Existing databases are unchanged; the migration follows separately.
_store_documents_with_chunks embedded each document's chunks in its own
embed_chunks call; chunks missing embeddings are now flattened across the
whole batch, embedded in one pass honoring embeddings.batch_size, and
assigned back positionally.
The guard accepted one carried record as covering the whole request, so a host
retaining only the RAG namespace had its earlier analysis evidence replaced by
receipts and lost, with the RAG record making the loss look accounted for.
Each capability is now judged on its own, and only when its own evidence is at
stake: another capability's record says nothing about this one's, and requiring
every record would stop a host that registers both capabilities and only ever
uses one.
Both optional capabilities read what earlier questions retrieved and cited from
the capability's state, so a host that carries only the message history hands
every run an empty record. Compaction then replaced the earlier evidence with
receipts and retained nothing, and the loss was invisible: the citations the host
already displayed were still there. It now refuses when it finds evidence from an
earlier question and no record of what that question cited.
`state_carried` reaches the optional capabilities through discovery, so the
refusal distinguishes a host that never carries state from a question that simply
cited nothing.
The documentation taught the pattern that breaks: the compose example is now
stateful and the requirement is stated where each capability is introduced.
The app's browser storage was doing exactly this, keeping only the fields the UI
reads. It now persists the whole namespace map, so the citation policy's
violations survive a reload as well as the evidence record.
A cite call naming one good id and one mangled one registered the good one and
then asked the model to cite again. A model that mangles ids obeys, mangles
again, and the run dies on output retries with the answer lost: observed at 22
consecutive cite calls against gemma4-26b, with the citation policy registered
to press for a declaration. The branch is only reachable once something has
registered, so the answer already has grounding and there is nothing to ask for.
Resumption was inferred from the transcript, and no shape says it. A missing
prompt is how UI adapters ask their first question as much as how pydantic-ai
resumes one, so every AG-UI host failed on its first message. Reading the tail
instead moved the error rather than fixing it: a settled structured answer ends
with an output tool's return, indistinguishable from results delivered to a
question still in progress, so following questions inherited the first one's
identity.
`CapabilityEvidenceRecord.in_progress` is now the authority. `begin_question`
sets it, `after_run` clears it unless the run is only pausing for deferred work,
and a run that raised never reaches `after_run`, which is what leaves an
interrupted question claimable. The history is consulted only to catch a host
that dropped the state of a question the model is unmistakably still owed.
The documentation claimed a conversation that has never cited anything is not enforced,
which reads as though a first question could answer from fresh evidence without
declaring it. Enforcement needs neither condition to hold: no evidence outcome in this
question, and nothing cited earlier.
The structured-output test accepted a redirect or a recorded violation, so a break in
ending detection would still have passed on the backstop alone. The cite tool is
available in that scenario, so it asserts the redirect.
An output tool call is a `ToolCallPart` like any other, and treating every tool call as
intermediate meant a model could search, skip citing, emit its structured answer and
finish with neither a redirect nor a record. A response ends the question when it
carries no tool calls, or when one of its calls names an output tool.
Some endings are not visible from a single response — a host running
`end_strategy="early"` can finish on text beside a function call — so `after_run` is
the backstop: it cannot ask the model for anything by then, but it records a question
that reached the end of its run undeclared. That also covers a question that was asked
once, ignored, and finished anyway, which previously returned early on the redirect
marker and went unrecorded.
The capability documentation and the changelog said a question that gathered no
evidence is left alone. That describes neither the code nor the intent: enforcement
applies wherever there is something to declare, which includes a follow-up that reuses
evidence cited earlier without searching again.
`_already_asked` matched the redirect's phrasing, so a question that merely contained
it read as a redirect we had sent and switched enforcement off for that question. The
redirect carries `[haiku.rag/citation-redirect]` and detection matches that, the same
way retrieved pictures are identified by a tag rather than by the prose beside them.
Requiring an evidence outcome from the current question exempted the case enforcement
exists for: a follow-up about evidence already cited needs no new search, since that
evidence is still on the wire — in a capsule when a compactor is registered, in full
when not. The condition is now that the conversation has something to declare, either
an outcome in this question or evidence it has already cited, which is independent of
whether anything compacts. A conversation that has neither is still left alone.
Citing again cannot narrow a question at any epoch. Declarations merged only within one
epoch, so an empty second thought a request later replaced the refs with nothing and
reported a grounded question ungrounded. They merge while no evidence outcome has
followed the standing declaration, and only genuinely newer evidence starts one afresh.
Whether a question has already been asked to declare is read from the message history
rather than remembered on the run instance, which a resumption's `for_run` discarded —
the same question was asked twice. Reading the history also makes the right call when a
redirect was enqueued but the run ended before it reached the model: nothing is in the
history, so it is asked again. Violations are recorded once per question for the same
reason.
`CitationPolicyCapability` makes the single enforcement decision, whatever mix of
evidence capabilities is registered: two of them must not each demand a citation for
one answer. It decides in `after_model_request`, when a response carries no tool calls
and the question can still be redirected. An explicitly ungrounded answer is a
declaration and is left alone; a question that gathered no evidence is left alone too,
read from the ledger rather than from a searches dict that a new question clears. When
the cite tool is already withdrawn the question is recorded in
`CitationPolicyState.violations` instead of pointing the model at a tool that is gone.
Registering it is the only switch. `DiscoveredEvidence` and discovery move to
`capabilities.evidence` so both optional capabilities share them, and `cite_available`
joins `evidence_tool_names` as public for the same reason.
Measured on Qwen3.6-35B over two arms of 37 questions, 29 of them unanswerable from
the corpus: explicit ungrounded declarations rose from 23 to 26, grounded answers to
unanswerable questions fell from 4 to 2, answerable questions stayed at 8 of 8, and
one redirect fired in the whole arm. Reading every answer found no invented grounding.
`rag_cite` and `analysis_cite` accepted only a non-empty `chunk_ids`, so a model with
nothing to cite could comply only by staying silent — indistinguishable from
forgetting. An empty list is now a valid answer to "what grounds this?", recorded as a
declaration with no refs, which derives `ungrounded` rather than leaving the question
undeclared. Citing again cannot narrow it: an empty call after a grounded one leaves
it grounded.
The instructions lose their carve-outs. Refusing for lack of information no longer
exempts the call, and a corpus-level computation cites an empty list instead of
skipping.
A question's evidence epoch and declaration describe that question and end with
it, so `begin_question` clears them. Held past it, they let the last question's
citations ground the next one and hold a horizon its own declarations cannot
pass, and they force every question's message count to be comparable to counts
taken over a history that a host may since have rebuilt: a thread reconstructed
without a run that never finished shifts every later position, and the next
question was refused where answering it is correct.
Identities still separate one question from the next. Occurrences outlive the
question that recorded them and carry the identities that cited them, which the
capsule is grouped and ordered by, so reuse merges two questions into one group
and a lower identity renders later evidence as earlier.
A request can carry several of this capability's returns — a model can call search
twice in one response — and the carrier was identified by message alone, so each of
them received the whole capsule. It is selected by message and part now, so exactly
one carries it however many share the request.
The chat TUI passed its persisted state into the run, so tool synchronisation mutated
it in place while the message history was promoted only on success. A cancelled or
failed run therefore kept the evidence the tools had recorded and discarded the
messages that justified it, and the next question derived its identity from the
shorter history: behind the recorded epoch, refused as non-append-only, the
conversation unusable until cleared. The run gets a copy, promoted with the messages
or not at all.
Five decorators had been left attached to a helper by an earlier extraction, which
pytest does not collect, so the resume case they carried was silently untested. The
wire test covers both resume shapes again, no prompt and deferred results.
`_compact_old_tool_returns`, `PRIOR_TURN_NOTICE` and `turn_start` leave
`RAGCapabilityBase`, along with its `wrap_model_request` hook. The evidence
capabilities now retrieve and validate, and nothing else. Registering the compaction
capability is what rewrites a request; leaving it out sends the transcript untouched,
which was never a choice a host could make before.
The boundary is the recorded question identity rather than message shape, so a
resumption compacts what lies below the question in progress instead of switching
compaction off for the whole run. The newest earlier evidence return carries the
capsule and every other becomes a receipt, so one capsule exists by construction and
every return stays paired with its call.
Pictures of cited evidence are fetched through the capability that retrieved them and
re-attached beside the capsule with fresh labels. Ownership of a picture on the wire
requires the machine tag we write and an image directly after it, since neither
position nor prose is proof: several tools' results can arrive in one request, and a
user quoting our wording above their own picture had it removed. A picture that cannot
be fetched or decoded is emitted with neither its image nor its label.
The chat TUI and the example backend register the compactor, being multi-turn.
`client.ask`, `client.analyze` and the MCP tools do not: a single-shot question has
nothing earlier to compact.
`EvidenceCompactionCapability` reads what the evidence capabilities recorded out of
the run registry, and `build_capsule` renders it: every cited item, grouped by the
question that last cited it, newest group first, each rendered once, with the
pictures of cited evidence and the labels that must accompany them. Discovery runs
one way and reads only, so no capability holds a reference to another and a host
running one, both or neither needs no wiring change.
Everything cited is kept whole and everything else is dropped. There is no character
budget, no picture cap and nothing to configure: a cap would only half-rescue models
that fail on long conversations regardless, and a host that needs earlier evidence
pruned can compact its own requests further.
A capability reports which of its tools produce evidence, so a cite acknowledgement
is never mistaken for one. Pictures are identified by owner, document and reference,
so one figure cited through overlapping chunks is attached once while the same
reference in another document stays a different picture.
The builder does no I/O and never sees the message history, so a picture travels with
its label and the caller fetches the bytes. Nothing reaches the wire yet.
A question identity is unset until a run establishes it. Testing whether the
record exists cannot stand in for that: a host with no state to send seeds a
default record, and a default record is truthy, so a resumption missing its real
state would have proceeded as question zero.
Every way of recording a message count now refuses one behind what is already
stored, through one shared check: a new identity, an evidence outcome and a
declaration, each against the newest identity, evidence epoch and declaration
epoch. Identities and epochs are only comparable while the conversation grows,
and `before_model_request` results are assigned back onto history, so that is a
constraint on the host rather than a guarantee of the framework. Left unchecked,
an evidence outcome moving backwards freezes every later declaration as stale,
and a declaration moving backwards replaces a newer one with an older one and
revives the answer it grounded.
The searches, citations and executions of a question in progress survive a
resumption. Clearing them cost the results the model was still answering from: a
citation afterwards recorded no provenance and could not resolve against the
expanded result it had seen, falling through to a database lookup.
A code execution counts as evidence when it succeeded or printed something. A
raised error with an empty stdout grounds nothing.
`CapabilityEvidenceRecord` holds the relationships a transcript cannot express:
which chunks a capability retrieved, which it cited, in which questions, and at
which point in the conversation. RAG and analysis each own one in their own state
namespace. Nothing is co-written: the host's state is JSON storage, so a shared
record would be overwritten by whichever capability synced last, and merging
happens in transient per-request views instead.
Both clocks are derived from the conversation rather than counted locally, so
every participant computes the same values without sharing a counter. Question
identity is the message count when the question arrived; epoch is the message
count at an outcome. Epochs are therefore globally comparable, which is what
lets `citation_status` require a declaration to follow the newest evidence of
every capability, and what makes equal epochs mean one request.
A declaration is written only after `resolve_citations` succeeds, so a call
naming only unresolvable ids is not a citation. Status is derived, never stored,
so refs and status cannot contradict.
Resuming a question requires the host to carry the capability state from the run
being resumed. Without it the identity of the question in progress is unknowable,
and adopting the current message count would relabel that question as a new one
and judge every declaration in it against the wrong identity.
Nothing reads the records yet and no wire behaviour changes.
deferred_tool_results may arrive with a non-empty prompt, so the absence of a
prompt cannot be the only test for a resumption. Reproduced: resuming a live
question that way replaced its search result with the earlier-question notice
while the deferred result arrived alongside it.
_is_resumption accepts either signal — no prompt, or a history ending with a
request the model has not answered or a response whose tool calls have no
returns. A settled history ends with the previous answer, so a genuinely new
question is unaffected.
A new prompt on top of an unanswered tail is ambiguous and now counts as a
continuation: compacting costs the answer if it is one, while not compacting
only costs a larger request.
A run carrying no prompt is continuing a question rather than asking one:
pydantic-ai resumes that way for deferred tool results, interruptions and
suspended responses. len(ctx.messages) then counts the live question's own
messages, so its search result was replaced by the earlier-question notice and
the model was asked to answer with the evidence removed. Reproduced: resuming
with an in-flight history left a notice where the only evidence was.
Switch compaction off for the whole run when ctx.prompt is None. The absence of
a prompt is the signal rather than the message layout — the resume shapes differ
from each other, and deriving the boundary from layout is what broke this to
begin with.
ToolReturn.content reaches the model as a user-role message and the pictures
arrive bare, so nothing connects a figure to the chunk it came from:
BinaryContent.identifier does not survive serialization to the vision API, and
the captions in the result text correlate only by position.
Precede each picture with its position, source chunk id and self_ref.
build_binary_parts_from_results becomes build_image_content_from_results and
returns the labels interleaved with the pictures, so both attachment sites emit
them the same way.
This does not stop a model narrating retrieved pictures as user-supplied.
Measured on gemma4-26b with a single note ahead of the batch, and again with
per-image labels: it quotes the label and still says the user provided them.
The message role wins over its text.
_compact_old_tool_returns ran in before_model_request, whose result core
assigns back onto ctx.state.message_history, so the trim reached
all_messages() and every host that persists a thread. It now runs in
wrap_model_request, which operates on a detached list: the model sees the
trimmed history, the host keeps what it retrieved.
Content attached to a ToolReturn arrives as its own UserPromptPart in the
same ModelRequest as the ToolReturnPart, so the turn-boundary scan read an
image-bearing search result as a new user turn and discarded evidence
retrieved earlier in the same turn. _is_user_turn() now requires a request
with no tool returns.
Page images on a replaced return stay. Dropping them with their text bounds
context growth, but a follow-up about a figure already shown ("what colour
is that box?") carries no terms that could retrieve it again: measured on
gemma4-26b against two ORB figures, removing the image turned both answers
into "I cannot find enough information", and keeping it answers correctly.
Bounding that growth needs to preserve cited figures, which is a separate
change.
The replacement notice no longer claims citations remain in state;
_clear_invocation_state has cleared them by then.
Import Store from haiku.rag.store.engine directly instead of re-exporting it
through the package __init__, and defer HaikuRAGApp's import in cli.py behind
TYPE_CHECKING/lazy imports, avoiding an eager import at CLI startup.
mxbai-rerank-base-v2 ships a Sigmoid activation and evaluates it in bf16, so
every strongly-relevant candidate rounds to exactly 1.0. Ties then leave the
order to the stable sort, which preserves the incoming hybrid ranking: on 100
t2_finqa retrieval cases the reranker scored MAP 0.661 against 0.659 with no
reranker at all, and 0.742 once the scores separate.
Ask the model for logits and apply the sigmoid here, where it runs in float64.
Scores stay 0-1, matching the cohere, vllm and zeroentropy rerankers.
Also drop the remaining pyright references; the project type-checks with ty.
The request-limit notice said only the cite tool remained available, but
chat registers rag and analysis in one agent, so exhausting analysis
claimed rag_search was gone too. Scoped to the capability's own tools.
The cite window was counted over every model request once loaded, so
turns spent on another capability expired it before the model was ever
placed where citing was the obvious move. Count only requests whose
preceding response called one of this capability's tools; engagement is
also the only thing that can loop, which is all the bound guards against.
Also: _count_tool_traffic returns a named tuple rather than four bare
ints, and counts failures only for this capability's tools, so host-tool
retries and output-validation retries no longer read as its failures.
The notice told the model to answer from what it had the moment
qa.max_searches ran out, while up to 15 code executions remained and
in-code search() does not count against that budget. It now names the
spent tool and points at whichever evidence tool still has budget,
falling back to answer-and-cite only when none do.
Also count RetryPromptPart in n_failed_tools: _cite rejects with
ModelRetry, so a run whose every cite attempt was refused reported zero
failures. And note that n_requests is the run's request count, which
tracks a capability's own budget only while it stays loaded.
- _budget_notice no longer names the cite tool after prepare_tools has
withdrawn it; the post-grace state gets the plain no-tools text back.
- Split search-budget rejections from any failed tool call: the code tool
raises ToolFailed for every error in model-written Python, so
budget_spent was true for a ZeroDivisionError.
- docs/capabilities/rag.md described the old single-turn removal.
- Drop the rationale clause from the CHANGELOG entry.
50 executions across 40 cases in an 822-case run died on
'_io.TextIOWrapper' object is not iterable, and those cases scored 37.5%
judged against 60.1% and cited 12.5% against 55.2%.
_session_limits still called the session budget the backstop for code that
computes, which stopped being true when the pool gained a request_timeout in the
same commit. The watchdog kills such a call at code_timeout, so it can never
spend a budget of code_timeout * max_executions. The docstring now names the two
per-call limits and claims neither.
The crash test said the discard prevents a RuntimeError escaping execute(). The
handler catches RuntimeError too, so without the discard every later call
returns a failed result instead.
Name MemoryFile's missing write hook as the reason metadata.json takes a reader
and a deny pair. Drop the clause in the CHANGELOG that narrated how the old
overrun accumulated.
The pool watchdog counts only the time a worker spends running code, so a read
that blocks the worker never trips it. The margin above code_timeout therefore
guarded a race that cannot happen, and only bought a runaway call 90s where 60s
was configured. Pass code_timeout straight through. The two limits are disjoint:
the watchdog bounds a call that computes, and the read deadline bounds a call
that reads.
Drop the ordering test, which was true for any positive margin. The containment
test no longer sets code_timeout to zero, because that value now also disables
the watchdog and kills the worker before the read guard can refuse. It patches
the guard instead.
Monty 0.0.19 runs a plain class and a class with __enter__ and __exit__. Only
inheritance and metaclasses raise. Say that in the instructions.
MemoryFile is no longer constructed, so drop the import, the annotation, and the
stale references in the sandbox docstring and CLAUDE.md. That CLAUDE.md line
also still claimed a ThreadPoolExecutor, _run_async, and a fresh interpreter per
call.
The read deadline only gets control at a read, so code that computes without
reading escaped it. Give the pool a request_timeout above code_timeout. The
watchdog kills the worker, and execute() already replaces a dead session, so a
runaway call fails and the next call recovers. A read refusal keeps the
variables, so it has to win the race whenever code does read.
Restore test_open_file_objects_are_not_iterable. Replacing the neighbouring
write test by text range deleted it, which left the instructions carrying a
prohibition with nothing to signal when monty lifts it.
Reuse the VFS and the pool when a session is replaced, so recovery skips the
document scan. Mount metadata.json with a lambda rather than a factory. Cover
open() in write mode. Fold the crash entry into the pydantic-monty bullet,
because no release shipped the worker without it.
A crashed worker used to poison the rest of the run. execute() reported the
crash as a failed result, but kept the dead session, and every later call then
raised RuntimeError out of the tool. Clear the session so the next call checks
out a replacement. Keep the session for a syntax or runtime error, which leaves
the worker healthy, and say in the failure text that a restart loses the
variables.
A MemoryFile accepts writes, so metadata.json took them while the other three
document files refused. Mount it through the same read and deny pair. The
write-denial test now covers all four files.
The VFS bridge suspends the Monty worker for the length of a read. Monty checks
its duration budget between interpreter steps, so it cannot check while a read
is in flight. Code that reads in a loop overran a 60s budget by minutes. A read
takes about 20ms on a 2789-document corpus, so a full scan spends about 55s in
reads alone.
Check the deadline before each read. Raising from inside the callback answers the
worker's suspension, which keeps the session usable.
Monty also spends max_duration_secs across the session rather than per call, and
the sandbox reuses the session so that variables persist. Budget it for
code_timeout * max_executions. At the old per-call value the first slow call
starved every later one.
Do not wrap feed_run in asyncio.wait_for. Cancelling during pure compute is
clean, but cancelling while a read waits for an answer wedges the session with a
protocol RuntimeError that escapes execute(). A call that computes without
reading stays bounded by the session budget alone.
Monty 0.0.19 supports open() and with blocks, but a file object is still not
iterable. See pydantic/monty#490, which is still open. The instructions now
state the limitation in three places and give the alternative next to each
one: readlines() or read().split("\n").
Replace chr(10) with "\n" in the prose and in the example. Both work on
0.0.19, and chr(10) implies that the escape is broken.
Add a test that pins the limitation. The test fails when Monty gains
iteration support, which is the signal to relax the instructions.
Document files can be read with open() and with-blocks (.read(),
.readline(), .readlines()); writes raise PermissionError. File objects
remain non-iterable and the collections module is still unavailable.
0.0.19 runs sandboxed code in a subprocess worker pool (AsyncMonty /
AsyncMontySession) and drops MontyRepl. The sandbox checks out a session,
drives it with feed_run, and closes it to return the worker; close() is
now async. Worker crashes surface as a failed SandboxResult. The document
VFS (OSAccess/CallbackFile/MemoryFile) is unchanged.
get_config() builds its instance on first use, so the branch was covered
only when a worker happened to call it before set_config(). Under xdist that
depends on how cases shard across workers, which varies with the core count:
covered locally, uncovered on the two-core runner. Assert it directly.
setup-uv v4 bundles a cache client the GitHub cache service now rejects with
400, so the uv cache never restored and every wheel was redownloaded. Bump
to v9 across all four workflows (build-docs was already drifting at v5).
setup-python read requires-python (">=3.12") and installed 3.14, which uv
then ignored in favour of .python-version (3.13) and downloaded itself.
Point it at .python-version so the interpreter it installs is the one used.
Rewriting the document through client.update_document re-chunks and
re-embeds, so the test needed an embedding endpoint that CI does not have.
Write the row through the repository instead, matching how the rest of the
file avoids the embedder.
check_source_accessible narrowed its handler to ValueError, but Path.exists
re-raises errno values outside its ignored set (EACCES, ENAMETOOLONG). Those
were swallowed before and now escaped into the rebuild sweep the guard exists
to protect. Catch OSError too.
Restore the arity guard in _common_path_prefix: without it an empty list
raises from min() and a single label yields a prefix covering the whole path.
Two tests would have hung rather than failed on regression (the vacuum skip
and the protected-wait cancellation); both are now bounded. The import
vacuum test raced against the done-callback that discards the task, and now
spies on the call instead, with a negative control.
Replace assertions that could not fail: blank-query search against an empty
corpus, a batch flush counted against an empty table, a picture description
asserting its own input state, and an FS scheme check with nothing on disk to
resolve. The get_model matrix asserted only the returned type across 26
cases and now pins the per-provider settings. The three batching tests now
count flushes, which revealed embed-only writes through chunks_table.add
rather than _flush_rebuild_batch.
Cover the remaining paths in the client, context, downloads, title
generation, document tools and store models, and add fail_under=100 so
uncovered lines fail CI.
Six lines that no test can reach get a pragma with its reason: the docling
import guard, the nameless PDF attachment, the FS symlink OSError guard that
resolve(strict=False) absorbs, the docling bbox and LanceDB document-id
shape guards, the tag-retention branch vacuum makes unreachable, and Monty's
Rust-thread print callback.
Fix test_find_config_file_user_config, which wrote its config into the cwd it
had chdir'd to, so the cwd branch answered first and the user-directory
lookup it names was never exercised.
Add vector-index creation tests including the warned failure, chunk
repository get_by_id, list_all pagination, blank-query and precomputed-vector
search, the unknown-score-column guard, settings row recreation, and
replace_for_document with no items.
Add tests for the vacuum-failure warning, documents deleted mid-rebuild,
chunkless documents, batch flushes in the embed-only, descriptions and full
paths, a missing docling blob under rechunk, missing and recoverable picture
bytes, and the source-missing fallback. Direct-call tests cover the staging
helpers and the idempotent phase-1 marker.
Parametrize the get_model tests whose only assertion was the returned model
type, extending the table to the reasoning-off, groq-parsed and Bedrock
o-series/qwen/unmapped branches. Fold format_citations_no_title into the
superset test that already covered its scenario.
Add tests for cosine_similarity on parallel vectors, multi-citation Rich
rendering, picture rendering with missing and undecodable bytes, a missing
docling distribution, and the visualize_chunk short-circuits for absent
documents, absent rasters and refless chunks.
Add tests for the MCP tools' degradation contracts, malformed WebDAV
multistatus bodies, dry-run poller sweeps including the circuit-open and
discover-failure paths, FS source scheme and symlink handling, docling-serve
zip parsing, and the remaining embedding and reranker helpers. Parametrize
_strip_etag.
Drop the misplaced pragma on the analyze handler, which sat on the return and
left the except uncovered. Add one on the FS symlink OSError guard, which
resolve(strict=False) absorbs for every real link.
Delete repository methods with no callers (SettingsRepository CRUD,
ChunkRepository.update/delete/get_chunks_in_range), the DataFrame branch of
_process_search_results whose only caller always passes a query, and guards
that cannot be reached from their call sites: the rebuild mode=None default,
the staging drop already performed by _resolve_rebuild_recovery, the empty
batch skip, two context fast paths, the doctor prefix guard, the poller
_task attribute that is never assigned, and a docling caption fallback for
a field name no item class defines.
set_haiku_version built a recreated settings row from the process-global
Config rather than the store's own, so a store opened with a custom config
stamped global settings into the database.
check_source_accessible called urlparse outside its try block, so a stored
URI with a malformed IPv6 host raised ValueError instead of reporting the
source as inaccessible, aborting the whole rebuild sweep.
Parametrize sibling tests that differed only in a literal value, and fold
two strict-subset tests into the survivors that already covered their
scenario. Every case that ran before still runs; the union of assertions
is applied to each case, strengthening list_all, get_pages_data and
resolve_doc_items.
Replace four hand-rolled log-capture handlers with a shared
capture_logs() contextmanager in conftest.
13 fewer test functions, 348 fewer lines.
ChatApp and InspectorApp assigned self.client before __aenter__
completed, so a failed open was masked by an AttributeError from
on_unmount tearing down the never-opened client.
Drive analysis-only chat with analysis.model (falling back to qa.model) so
the running model matches the one the analysis capability configures,
including its vision flag; RAG-bearing chats still run on qa.model.
Give the AG-UI example state-bearing deps and a final STATE_SNAPSHOT so
registered citations reach the client, mirroring the app backend.
Update chat docs: haiku-rag chat uses --capability/-c, and drop the removed
"View state" command-palette entry.
reranking.multimodal (vllm provider only) attaches picture bytes to
synthetic picture chunks before rerank; VLLMReranker sends them as
content-parts documents (base64 data URI + description text) in the
same /v1/rerank request as plain text documents.
Docling guesses a DocumentStream's format from its first bytes before
considering the extension, so markdown/HTML content starting with a binary
magic signature ("BM" = BMP, "ID3" = MP3) was routed to an image or audio
backend and fell back to plain-text conversion. Prefix the encoded text
with a newline so the sniff finds nothing and the extension decides.
visualize_chunk re-expanded chunks from scratch to recover their refs,
which could not faithfully reproduce the original merge, scores, and
clip — so a visualization could highlight different pages than the
citation covered. Carry the cited items on Citation.doc_item_refs and
resolve bounding boxes from them directly; re-expansion remains only as
the fallback for callers with no stored context (CLI, inspector). Chat,
inspector, the app endpoint, and the frontend pass the refs through.
A merged search result took its chunk_id from whichever constituent
sorted earliest in the document, while its score was the max across the
group — so the citation's identity could point at a different, less
relevant chunk. Anchor chunk_id and the content/refs fallbacks on the
max-score constituent, clip the budget window around that same chunk so
its evidence is never trimmed away, and narrow page_numbers, doc_item_refs,
and attached image bytes to the items that survive the clip.
