Search results carry text and image blocks only. Claude Code and the
Agent SDK do not forward text blocks when structuredContent is present
and Desktop forwards both, so sending both either hid the rendering or
doubled it. The invalid-filter error keeps the engine's diagnosis and
lists our columns instead of lance's internals. format_citations no
longer repeats the URI of an untitled document.
Refs #599
Both search tools pass their results through HaikuRAG.expand_context, as
every other consumer of search results already did, so a client reads
the hit in its section rather than the chunk that matched. The rendering
gains an opt-in include_chunk_meta that shows the metadata stored with
the matched chunk beyond haiku.rag's structural keys, labelled as the
matched chunk's because an expanded passage spans several chunks and
only the anchor's metadata survives expansion. The capabilities' rendering
is unchanged.
Refs #599
fastmcp>=4.0.2,<5.0.0. Protocol types are snake_case (read_only_hint,
mime_type, input_schema, server_info); the camelCase names warn and go
in fastmcp 5, so the next major is an explicit upgrade. The client
defaults to the sessionless protocol, where initialize_result is None,
so the tests read client.instructions and client.server_info, which both
protocol modes populate. The server answers either mode.
claude-plugin/ holds the plugin manifest, the server configuration
(haiku-rag mcp --stdio, the configuration decides the database) and a
skill that says when to reach for the knowledge base and how to move
from a search result to a document, a section, an answer or a
computation. A repo-root marketplace manifest makes
`claude plugin marketplace add ggozad/haiku.rag` work. The skill
pre-approves every tool the server registers, and a test keeps the two
in step.
The manifest carries the package version, which bump_version.py now
rewrites: a versioned plugin updates only on a bump, so the installed
skill stays in step with the haiku-rag release the user has.
Refs #599
haiku-rag mcp --no-agents reaches create_mcp_server(agents=False):
ask_question and analyze are not registered and the instructions drop
the line describing them. qa.model always has a default, so a server
without a usable model cannot be detected from configuration; the flag
is how an operator says so.
Refs #599
search_documents and search_documents_by_image return a ToolResult: the
format_for_agent rendering with rank, Document ID and Collection so the
text alone drives the document tools; one ImageContent per distinct
picture, labelled with its result; and the SearchResult list without
image_data as structured content. format_for_agent gains an opt-in
include_document_id, so the capabilities' rendering is unchanged.
collect_pictures is the one place pictures are deduplicated and
validated for both wire formats.
Refs #599
build_toc moves from the sandbox into haiku.rag.context; the sandbox
keeps its toc.json unchanged. get_document_outline returns the heading
tree with page numbers and get_document_section one section's text,
subsections included, both resolved in the database holding the
document. Chunk ids never leave the server. ask_question drops `cite`
and always appends its citations.
Refs #599
FastMCP masks unexpected exceptions and logs the traceback server-side,
so paths and provider URLs never cross the transport. Expected failures
raise ToolError with a message: unknown document, unknown collection,
invalid filter, invalid base64, and agent failures naming only the
exception type. No tool returns an empty value or an error string on
failure any more.
A filter is validated on its own before the read that would use it, with
a filtered count on one of the selected databases: that is the query
engine rejecting the filter and nothing else, so its message (columns and
the statement) can be forwarded, while a ValueError raised later in the
read stays masked and no database outside the selection is opened.
Refs #599
FastMCP gets instructions, the haiku.rag-slim version and a lifespan.
Every tool carries read-only ToolAnnotations with a title, a description
that says when to use it, and a description on every parameter. The
filter description lists the document columns from DocumentMetaRecord
and how to match metadata; filter is accepted by both search tools.
One strict base64 decoder serves every image parameter.
DocumentInfo carries metadata.
Refs #599
`haiku-rag mcp` passes covers_set=True and _covering no longer refuses a
scope over several databases. search_documents, search_documents_by_image,
ask_question and analyze take `sources`; get_document takes `source`;
DocumentInfo carries `source`. format_citations gains include_source, which
ask_question sets from covers_multiple so citations name their database
only when the server covers several.
Refs #599
add_document_from_file, add_document_from_url, add_document_from_text
and delete_document are gone. create_mcp_server and _covering lose
read_only; the client always opens with read_only=True, so the
--read-only flag before `mcp` is redundant and the docs, Dockerfiles and
compose example drop it.
Ingestion is `haiku-rag add`/`add-src`/`delete` and haiku-ingester. No
known consumer used the write tools; one stdio server per client window
made multi-writer the accidental default, and streamable HTTP has no auth.
The test_app stub drops a comment and __init__ that described run_mcp
constructing the client positionally; every path goes through _covering.
Refs #599
telemetry.configure() passes send_to_logfire="if-token-present", so on a
machine with a logfire token the CLI tests configured a live exporter:
spans and the periodic metrics export went to the developer's project, and
where the export fired while a cassette was open, VCR rejected it with
CannotOverwriteExistingCassetteException in record mode none.
A token resolves from LOGFIRE_TOKEN or from logfire_credentials.json under
LOGFIRE_CREDENTIALS_DIR (default .logfire, relative to the working
directory), so an authenticated developer uploads with the variable unset.
conftest drops the variable and points credentials discovery at an empty
temporary directory, alongside the existing HAIKU_RAG_CONFIG_PATH setup and
before any haiku.rag import. The explicit send_to_logfire argument beats
LOGFIRE_SEND_TO_LOGFIRE, so denying the token is what disables the exporter.
test_split_and_merge_matches_single_pass resolved tests/data/doclaynet.pdf
relative to the working directory, so it failed with FileNotFoundError for
any invocation outside the repository root. conftest.py resolves the same
file as Path(__file__).parent / "data"; this was the only site that did not.
Replaces gpt-oss on ModelConfig, qa.model and processing.title_model, and
ministral-3 on the picture-description model. qa.model.vision follows the
model and is now true.
enable_thinking was gated on the gpt-oss name, so it did nothing for
qwen3.8. With title_model's max_tokens of 100 the reasoning consumed the
whole budget and title generation returned an empty string. The mapping
now applies to any ollama model via reasoning_effort(): false sends
"none", true sends "high". Measured on qwen3.8:27b-mlx, "low" does not
disable thinking and "none" does; gpt-oss is the inverse, its template
has no "none" level, so it keeps "low".
Picture description bypasses get_model -- docling posts the request
itself from a params dict -- so the flag was inert on that path too.
vlm_api_params() carries reasoning_effort into both converters' request
bodies. At max_tokens 200 the description survived either way, but the
switch cut completion tokens from 141 to 45.
test_search_tool_skips_binary_content_when_qa_model_is_text_only asserted
the vision default rather than setting it; it now configures vision=False
itself.
docs/benchmarks.md keeps ministral-3: those are recorded measurements.
A configured or default database that does not exist raises
SourceUnavailableError with the way to create it and without its location.
An unknown --db-name lists the databases there are, configured or default.
The CHANGELOG records Store.db_path, SingleDatabaseSession(ref, config),
the empty lancedb.uri migration and the error-type change.
An empty lancedb.uri, which init-config used to emit, is told to remove the
key; only a non-empty value is told the databases spelling. haiku-rag --db
and haiku-ingester --db turn a path with no stem into typer.BadParameter.
HAIKU_RAG_DB and DB_PATH are gone: a capability covers what the
configuration places or the db_path it is given, and the app backend and
the AG-UI example load their configuration as the CLI does. The compose
files point HAIKU_RAG_CONFIG_PATH at the mounted haiku.rag.yaml, which
places the database at /data where DB_VOLUME is mounted; the backend
refuses a configured set since it serves one database. The chat scopes a
selection by source only over a set and names databases on filter rows
only across several. Docstrings, docs and test fixtures stop describing an
unnamed database; every database a search, listing or citation reports
carries a name.
DatabaseRef is a name and a location. The configuration places databases
through lancedb.databases alone; with none configured the default is the
entry haiku.rag under storage.data_dir, selectable like any other.
lancedb.uri is removed, and a config carrying it fails to load with the
replacement spelled out. A path passed from Python is valid where the
configuration places nothing and raises AmbiguousDatabaseError beside
lancedb.databases; haiku-rag --db and haiku-ingester --db construct the
scope directly, so a human's override keeps working. Every database
answers to a name, and a database given as a path keeps its own errors.
Store, connect_lancedb, gather_database_info and run_doctor take a
location, a path or a URI, and classify it with ConnectionMode.of.
SingleDatabaseSession owns the resolved DatabaseRef and passes its
location down. This removes DatabaseRef.connection(), default_db_path,
the placeholder path for URI-backed databases and the per-database
config copies, so the configuration a client holds is the one the caller
gave it. The chat hands its capabilities the scope it opened along with
the client it lends, and the v0.58.0 migration no longer checks local
free disk for a database behind a URI.
A picture deduplicated within a burst must still reach the model once in
its own question and, when cited, again via the capsule re-fetch after
compaction; uncited it is dropped. Two wire tests pin both paths. The
harness builds the rag capability with defer_loading=False: a deferred
evidence capability the model has not loaded presents an empty record,
so compaction computes boundary 0 and rewrites nothing.
qa.md describes max_searches in units.
Sibling searches emitted in one response overlap heavily (40.6% of
returned chunk slots on Glimmer ORB fan-out cases). A result whose
rendered evidence a sibling already showed keeps its rank slot but
collapses to a reference line, and a picture attaches once per response
keyed on (source, document_id, self_ref). Equivalence is the
format_for_agent rendering at neutral rank/total plus picture keys,
bucketed under the qualified chunk id, so another database's copy or a
different expansion of the same anchor formats in full.
Search state now commits only after formatting and image construction
succeed: a raising image build no longer leaves results citable that
the model never saw, notes evidence for them, or suppresses a later
sibling.
Searches a model emits in one response share a budget unit, up to
FREE_SIBLINGS_PER_ROUND (3) per unit; sequential searches pay one unit
each, as before. Grouping keys on RunContext.run_step, which pydantic-ai
increments once per model request. A budget-rejected round fails all its
remaining siblings, and tracking resets per run.
Glimmer opens most questions with a burst of ~3 rephrasings in a single
response (95.8% of its three-search ORB cases are one-response bursts),
spending 3 of 5 searches before reading anything. Pass rate at 3 calls
equals 1 call, so a burst is priced as one probe.
The IVF probe count was the one vector search parameter with no
setting, and it is what bounds recall on a large indexed corpus:
partitions grow with the corpus, so a fixed probe count covers less of
it, and vector_refine_factor can only re-score what the probes
returned. Defaults to 20, matching lance, so search is unchanged.
Also corrects the create-index docs, which predate the measurement:
optimize() covers new chunks, so a rebuild is about retraining
centroids rather than reaching unindexed rows.
Retrieval scores are each database's own rank arithmetic; the databases
in a selection share an embedder, so similarity in that one space is the
signal comparable across databases by construction. Measured product to
product against score ordering: +8.3 to +16.6pp recall@5 across five
cells on two corpora, flat in collection count and corpus shape where
score ordering dips with both, closing roughly 60% of the gap to a
reranker; order-sensitivity residual 0.00pp in every cell. Exact ties
collapse from 51-81% of candidates to under 1%. Full-text-only searches
keep retrieval-score order, having no query vector. The vector column
already travels with every search result, so the similarity costs no
additional transfer; per-chunk embeddings are materialized only for the
federated path that reads them.
Rank interleaving guarantees every database slots regardless of content;
on domain-split collections it allocates no better than chance and costs
4.7pp recall@5 at four collections against score ordering (7.1pp at
eight). Hybrid scores are each database's own vector/FTS rank agreement,
which carries across databases; equal scores resolve by within-database
rank, and only a tie on both falls to configured order, leaving
permutation sensitivity at 0.02-0.26pp. Fused results carry the
candidate's own retrieval score, so the context-expansion re-sort
preserves fused order.
Disjoint corpora give every database's rank-r candidate the same RRF
score, and the stable sort resolved those ties to lancedb.databases
declaration order, discarding the retrieval scores entirely. Ties now
break on the raw retrieval score, which is uncalibrated across indexes
but only ever orders candidates within one rank tier: the databases in
a fusion share an embedder and ran the same search type, and it can
never lift a candidate above another rank. Hybrid per-database scores
are themselves rank-derived, so exact agreement still ties and keeps
configured order, deterministically. The n > limit depth quota is
unchanged, pending the retrieval eval.
The newest release with wheels for every supported platform: 0.38.0
publishes none for x86_64 Linux or Windows. The zero-coverage FTS scan
path is still broken at 0.37.1, so the guard, prevention and repair
carry unchanged. ensure_indexes names the index it declares, keeping
replace deterministic on 0.38+, where an unnamed create_index builds a
suffixed sibling instead of replacing a different-typed index. An
invalid search filter raises ValueError.
Deleting or replacing every indexed row while unindexed rows remain
returns lance to the zero-coverage scan path. ensure_indexes rebuilds
a declared FTS index that covers none of a populated table's rows,
and delete_by_document_id runs index maintenance like the other chunk
writes. A legacy database in that state is repaired by its first write.
An FTS index built over an empty table indexes nothing and lance never
catches it up on add. ensure_indexes skips FTS while the table is
empty; create, replace_for_document and embed-only rebuild ensure
indexes after writing, so the index always covers at least its first
rows. The first write into a fresh table writes one extra chunks table
version for the index build; a failed build fails the write.
lance serves unsorted results with matching rows dropped when a
declared FTS index has indexed nothing, or when a populated table has
no FTS index at all. doctor fails on both: vacuum remediation for an
existing index, rebuild --embed-only for a missing one, since optimize
never creates an index. The chunk repository warns once per repository
on the first FTS or hybrid search against either state; a failing
coverage check is logged and never fails the search. Removes
_ensure_fts_index, which had no callers.
Both were exempt from coverage, which is how the 0.38.0 blob-encoding bug
reached a release: nothing exercised the chain that produced it.
Move the historical `documents` shapes into tests/store/legacy_documents.py
so each migration's tests can seed the table as its predecessor left it.
v0.25.0 compresses through compress_json, which switched from gzip to
zstd in 0.38.0 — the same release that added this migration. v0.38.0
decompressed with gzip and fell back to a raw UTF-8 decode, and both
fail on a zstd frame, so upgrading any database older than 0.25.0 has
died with UnicodeDecodeError since the migration shipped.
Reported with a fix by @omaer0 in #586.
`_rich_print_document` escapes uri, title and metadata, the sibling of
the escaped search-result renderer. The remaining comments and
docstrings that narrated rejected alternatives, consequences or history
now state the current invariant. The Sandbox class docstring names the
held connection close() releases, and wrapped docs paragraphs join to
one line.
Comments and docstrings across the branch narrated rejected
alternatives, consequences and history; each now states the current
contract. Renames test_a_legacy_uri_client_keeps_its_error to
test_an_unnamed_database_keeps_its_error. Documents the Sandbox
connection paths, the citation header's database segment, both
AmbiguousDatabaseError conditions on create_app, and run_inspector's
scope parameter. Doc paragraphs added by the branch in python.md,
storage.md and cli.md are one physical line each.
Sandbox.close released the pool and the held federated client only when
the monty session's __aexit__ returned. Each release now runs under
suppress(Exception), matching _discard_session and aclose_quietly, so a
raising step neither masks an unwinding error nor leaks the databases a
federated `_opened` holds.
The chat filter modal keys a selection by (database, id): copies of a
database share document ids, and checking one copy left the other
reading as selected. Applying the filter narrows the question's
`sources` to the databases the selection names. A twin id inside
another selected database still matches there: a serialized id filter
cannot carry a source.
A database name, document title, uri or heading containing Rich markup
crashed search output, chat citations and the chat document filter with
MarkupError. Every interpolation into markup-parsed text now escapes.
`_fetch_limit` had no test: a text query over-fetches `limit * 10` only with a
reranker, and an image query keeps its vector ranking either way. The sandbox
test asserted `KeyError`, which `UnknownDatabaseError` subclasses, so it could
not tell the contract from a bare one.
`uses_configured_databases` documents a mapping of one as covered; the test
named for it passed no mapping at all. Its `config` parameter is an `AppConfig`.
`test_an_analysis_capability_mounts_the_configured_set` carried a VCR marker and
no cassette, making no HTTP calls.
`tag_restore` reached into `HaikuRAGApp._display_path` and `_path`, and seven
places spelled out `self._is_local and not self._path.exists()`. `display_path`
and `database_missing` say both, and `database_missing` is False for a database
behind a URI, which has no path to check.
`init` keeps its own check: it asks the opposite question.
A database name, a location and an exception message all reach the info modal
as configuration-derived text, and Rich reads `[...]` in any of them as markup:
a name like `beta [prod]` disappeared, and a message carrying `[Errno 2]` or a
stray closing tag could break the line it sat in.
Two collections can hold documents with the same title, and the collapsed
citation carried only that title, so the two read as one source repeated. The
name is appended when the client covers a set, the decision already made for
search results and retrieved images.
`resolve_document` and `find_document` selected a document through a listing,
then dropped its source and looked the id up across the set. Ids repeat between
copies of a database, so a title that matched in one could be answered by
another's document.
`get_document_by_id` and `get_chunk_by_id` join `get_picture_bytes` in taking an
optional `source`, and all three route it through `clients_covering`, so a name
the client does not cover raises `UnknownDatabaseError` rather than being
answered by the database it does cover. Without a source the reads are as they
were, answering from the first database in configured order that holds the id.
`_resolve_scope` returned a scope without keeping it, so a client re-entered
after its configuration was edited covered whatever the configuration then said.
Resolving once is what the rest of the design rests on: the scope is what names
results, citations and errors.
A configuration is still free to change before first entry.
Search results and capsule entries name the collection they came from; the
images attached beside them carried only the chunk id and reference. Two
collections can return the same picture of the same document, so the two
labels were identical and the model could place neither.
The decision is the one already made for the text: `covers_multiple` at the
generic search tool, and the flag `search_corpus` computed for the capability
tools, which it now returns.
