Replaces gpt-oss on ModelConfig, qa.model and processing.title_model, and
ministral-3 on the picture-description model. qa.model.vision follows the
model and is now true.
enable_thinking was gated on the gpt-oss name, so it did nothing for
qwen3.8. With title_model's max_tokens of 100 the reasoning consumed the
whole budget and title generation returned an empty string. The mapping
now applies to any ollama model via reasoning_effort(): false sends
"none", true sends "high". Measured on qwen3.8:27b-mlx, "low" does not
disable thinking and "none" does; gpt-oss is the inverse, its template
has no "none" level, so it keeps "low".
Picture description bypasses get_model -- docling posts the request
itself from a params dict -- so the flag was inert on that path too.
vlm_api_params() carries reasoning_effort into both converters' request
bodies. At max_tokens 200 the description survived either way, but the
switch cut completion tokens from 141 to 45.
test_search_tool_skips_binary_content_when_qa_model_is_text_only asserted
the vision default rather than setting it; it now configures vision=False
itself.
docs/benchmarks.md keeps ministral-3: those are recorded measurements.
HAIKU_RAG_DB and DB_PATH are gone: a capability covers what the
configuration places or the db_path it is given, and the app backend and
the AG-UI example load their configuration as the CLI does. The compose
files point HAIKU_RAG_CONFIG_PATH at the mounted haiku.rag.yaml, which
places the database at /data where DB_VOLUME is mounted; the backend
refuses a configured set since it serves one database. The chat scopes a
selection by source only over a set and names databases on filter rows
only across several. Docstrings, docs and test fixtures stop describing an
unnamed database; every database a search, listing or citation reports
carries a name.
DatabaseRef is a name and a location. The configuration places databases
through lancedb.databases alone; with none configured the default is the
entry haiku.rag under storage.data_dir, selectable like any other.
lancedb.uri is removed, and a config carrying it fails to load with the
replacement spelled out. A path passed from Python is valid where the
configuration places nothing and raises AmbiguousDatabaseError beside
lancedb.databases; haiku-rag --db and haiku-ingester --db construct the
scope directly, so a human's override keeps working. Every database
answers to a name, and a database given as a path keeps its own errors.
`_rich_print_document` escapes uri, title and metadata, the sibling of
the escaped search-result renderer. The remaining comments and
docstrings that narrated rejected alternatives, consequences or history
now state the current invariant. The Sandbox class docstring names the
held connection close() releases, and wrapped docs paragraphs join to
one line.
Comments and docstrings across the branch narrated rejected
alternatives, consequences and history; each now states the current
contract. Renames test_a_legacy_uri_client_keeps_its_error to
test_an_unnamed_database_keeps_its_error. Documents the Sandbox
connection paths, the citation header's database segment, both
AmbiguousDatabaseError conditions on create_app, and run_inspector's
scope parameter. Doc paragraphs added by the branch in python.md,
storage.md and cli.md are one physical line each.
`_fetch_limit` had no test: a text query over-fetches `limit * 10` only with a
reranker, and an image query keeps its vector ranking either way. The sandbox
test asserted `KeyError`, which `UnknownDatabaseError` subclasses, so it could
not tell the contract from a bare one.
`uses_configured_databases` documents a mapping of one as covered; the test
named for it passed no mapping at all. Its `config` parameter is an `AppConfig`.
`test_an_analysis_capability_mounts_the_configured_set` carried a VCR marker and
no cassette, making no HTTP calls.
Restructure the storage page into search and provenance, duplicate ids,
ranking, Python operations and CLI commands, and state each in reference
voice. `Several Databases` becomes `Multiple Databases`, with the anchor
carried through every referrer, and the instruction files follow the same
name. Example databases are `papers`, `wiki` and `notes`.
`AmbiguousCitationError` is raised for a cited chunk id held by more than one
selected database, not for any shared id.
`covers_a_set` is true for a mapping of one, which is a configured database
like any other; `uses_configured_databases` says that. Its population guard
said "several" for the same reason. Document evaluating the configured set
with `--skip-db`, against population, which writes one database and needs a
path.
0.78.0 released `api_key` and two authorization fixes out of the section
this branch was still adding to, so the automatic merge filed every
multi-database entry under it and dropped the `0.77.0` heading, which both
sides had de-duplicated. Everything from `0.78.0` down is main's record
verbatim; the multi-database entries stay under Unreleased.
`get_document_by_id`, `get_document_by_uri` and `get_chunk_by_id` read
through repositories a client covering a set does not have, so a lookup by
identifier raised AttributeError and `resolve_document` with it. An
identifier says nothing about which database holds it, so every database is
asked at once and the first that has it, in configured order, answers.
On the evaluation side, `--db` overrides the configured set as the CLI
documents, and population refuses a set rather than ingesting into a
database the run would not read. A case filter matching nothing raises
instead of reporting 0.0000 as though it were a score.
A run over several databases could report which documents were cited but
not which database grounded the answer: `_result_from_run` walked the
citation index for `document_uri` and dropped `Citation.source`. The
distribution is not recoverable from the report afterwards, so a sharded
run would have measured everything except attribution.
`cited_sources` is one entry per cited chunk, in citation order, empty
where the database is unnamed.
The runner passed the dataset's path to every arm, which opens one database
and is what makes `--db` meaningful. A run over `lancedb.databases` has to
pass none instead, so the client resolves the set, and `DatasetSpec.covers_a_set`
is the one place that decides which of the two a run is.
Port 11433 serves nothing; Qwen3-Reranker-4B is on 11455. Qwen3.6 spends
its budget reasoning before it answers, and its 131072 window leaves ample
input space at 32768.
benchmark.py was 1220 lines holding six unrelated jobs: populating a
database, running retrieval, running QA, resolving datasets, moving
databases to and from HuggingFace, and wiring the Typer CLI.
qa.py takes both QA runners with their live summary, refusal metrics and
target resolution. population.py takes populate_db and the batched ingest.
retrieval.py takes run_retrieval_benchmark. artifacts.py takes HF_REPO_ID and
the download/upload bodies. experiment.py takes DEFAULT_JUDGE_MODEL and
build_experiment_metadata, which retrieval and QA both record.
benchmark.py keeps the CLI at 258 lines: the Typer app, config and case-id
loading, dataset resolution, evaluate_dataset, and three commands whose
bodies are now a loop over specs. The module-level side effects stay with it
— load_dotenv before configure_telemetry, so credentials and LOGFIRE_TOKEN
are in the environment before telemetry and model setup read them — so no
importable module carries one.
Test patch targets follow the code. get_model, run_capability_question,
run_capability_conversation, set_eval_attribute and HaikuRAG are patched
inside moved code, so they move with it; run_qa_benchmark,
run_retrieval_benchmark and find_config_file stay patchable on
evaluations.benchmark because evaluate_dataset and _load_config still look
them up there.
One assertion got stronger: a QA test patched benchmark.HaikuRAG to prove the
QA path does not open its own client. qa.py has no HaikuRAG reference at all
now, so the test asserts that instead.
RAGState and AnalysisState each declared the same five fields, so the generic
base could not name them: StateT was bound to BaseModel, and every access
went through cast(Any, state), a getattr by string, or a loop clearing fields
by name so it could skip the one only AnalysisState has.
EvidenceState declares them once. RAGState adds nothing, AnalysisState adds
executions and overrides begin_invocation to clear them. StateT binds to
EvidenceState, which removes all ten casts and both state-shape getattrs; the
three getattr(ctx.deps, "state") probes stay, since those check a
host-supplied object rather than our own state.
discover_evidence reached into capability.state for two fields. It now asks
through evidence_record() and citation_index(), alongside the
evidence_tool_names() and cite_available accessors it already used. The eval
runner's _RagLikeState protocol and the chat app's getattr reads described
this shape from outside and are gone.
Compatibility is semantic JSON-object equivalence, not bytes: field names and
nesting are unchanged, so a dict stored by 0.75.0 loads and re-dumps equal,
but deriving from a shared base reorders AnalysisState's keys. Nothing
serializes, hashes or string-compares this state — every carry point
re-validates by key.
Adds the 0.75.0 upgrade, which brings a pre-existing database up to the index
set `_init_tables` now creates. It rewrites no table data, so unlike the earlier
data migrations its cost is the index builds alone, each of which reads the
column it indexes.
`ensure_indexes` ensures an index of the declared *type* covers each declared
column, rather than checking that the column is indexed at all. The distinction
is what makes it safe to run against a database of unknown provenance:
- A wrong-typed index no longer satisfies the check. A BTree on `label` covers
the column while losing the low-cardinality equality lookup the Bitmap is for.
- Nothing is dropped or converted away from. Two index types over one column can
be deliberate, serving different query shapes, so an index this function did
not declare survives even on a column it does. The one thing it overwrites is
an index at LanceDB's default name, `{column}_idx`, which is the name it
creates itself.
- A column already carrying the declared type is skipped, so a database with the
full set migrates instantly rather than re-sorting every indexed column.
- Undeclared columns are untouched, so a vector index on `chunks` survives.
It returns the columns it acted on, because a change is not always visible from
outside: adding a Bitmap beside an existing BTree leaves the column indexed
before and after.
The version bump to 0.75.0 is required, not incidental: `_set_initial_version`
stamps a new database with the installed package version, so a migration
numbered above it would be pending the moment the database was created.
`test_client_update_document_replaces_rows_with_bounded_versions` turns
auto_vacuum off. Indexing `documents` means a background vacuum now has an index
to maintain on that table, so `optimize()` writes a version where it previously
had nothing to do, and it landed inside the window the test measures. The
document update itself is still one version, so the bound stays exact.
IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.
Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
No `DatasetSpec` declared `search_filter`, so `resolve_search_filter`
and the `--filter ""` clearing rule reconciled the flag against a
default that never existed. The flag alone covers the case. An empty
clause reaches `ChunkRepository.search`, which already treats it as
unfiltered.
Rename to `document_filter` throughout, matching
`run_capability_question`'s parameter and the metadata key that lands
in Logfire.
`_stub_spec` merges its overrides, so a test can override a loader
instead of rebuilding the whole spec.
The `--filter` examples used `uri LIKE '%arxiv%'`, which matches no
`orb_text` document: its URIs are bare arXiv ids such as `2407.01528v3`.
A clause that matches nothing scores MAP 0 rather than erroring, so the
example failed silently.
`await_args` is typed `_Call | None`, so subscripting it fails
`ty check`; `call_args` carries the same call for an AsyncMock.
