Import Store from haiku.rag.store.engine directly instead of re-exporting it
through the package __init__, and defer HaikuRAGApp's import in cli.py behind
TYPE_CHECKING/lazy imports, avoiding an eager import at CLI startup.
mxbai-rerank-base-v2 ships a Sigmoid activation and evaluates it in bf16, so
every strongly-relevant candidate rounds to exactly 1.0. Ties then leave the
order to the stable sort, which preserves the incoming hybrid ranking: on 100
t2_finqa retrieval cases the reranker scored MAP 0.661 against 0.659 with no
reranker at all, and 0.742 once the scores separate.
Ask the model for logits and apply the sigmoid here, where it runs in float64.
Scores stay 0-1, matching the cohere, vllm and zeroentropy rerankers.
Also drop the remaining pyright references; the project type-checks with ty.
The request-limit notice said only the cite tool remained available, but
chat registers rag and analysis in one agent, so exhausting analysis
claimed rag_search was gone too. Scoped to the capability's own tools.
The cite window was counted over every model request once loaded, so
turns spent on another capability expired it before the model was ever
placed where citing was the obvious move. Count only requests whose
preceding response called one of this capability's tools; engagement is
also the only thing that can loop, which is all the bound guards against.
Also: _count_tool_traffic returns a named tuple rather than four bare
ints, and counts failures only for this capability's tools, so host-tool
retries and output-validation retries no longer read as its failures.
The notice told the model to answer from what it had the moment
qa.max_searches ran out, while up to 15 code executions remained and
in-code search() does not count against that budget. It now names the
spent tool and points at whichever evidence tool still has budget,
falling back to answer-and-cite only when none do.
Also count RetryPromptPart in n_failed_tools: _cite rejects with
ModelRetry, so a run whose every cite attempt was refused reported zero
failures. And note that n_requests is the run's request count, which
tracks a capability's own budget only while it stays loaded.
- _budget_notice no longer names the cite tool after prepare_tools has
withdrawn it; the post-grace state gets the plain no-tools text back.
- Split search-budget rejections from any failed tool call: the code tool
raises ToolFailed for every error in model-written Python, so
budget_spent was true for a ZeroDivisionError.
- docs/capabilities/rag.md described the old single-turn removal.
- Drop the rationale clause from the CHANGELOG entry.
50 executions across 40 cases in an 822-case run died on
'_io.TextIOWrapper' object is not iterable, and those cases scored 37.5%
judged against 60.1% and cited 12.5% against 55.2%.
_session_limits still called the session budget the backstop for code that
computes, which stopped being true when the pool gained a request_timeout in the
same commit. The watchdog kills such a call at code_timeout, so it can never
spend a budget of code_timeout * max_executions. The docstring now names the two
per-call limits and claims neither.
The crash test said the discard prevents a RuntimeError escaping execute(). The
handler catches RuntimeError too, so without the discard every later call
returns a failed result instead.
Name MemoryFile's missing write hook as the reason metadata.json takes a reader
and a deny pair. Drop the clause in the CHANGELOG that narrated how the old
overrun accumulated.
The pool watchdog counts only the time a worker spends running code, so a read
that blocks the worker never trips it. The margin above code_timeout therefore
guarded a race that cannot happen, and only bought a runaway call 90s where 60s
was configured. Pass code_timeout straight through. The two limits are disjoint:
the watchdog bounds a call that computes, and the read deadline bounds a call
that reads.
Drop the ordering test, which was true for any positive margin. The containment
test no longer sets code_timeout to zero, because that value now also disables
the watchdog and kills the worker before the read guard can refuse. It patches
the guard instead.
Monty 0.0.19 runs a plain class and a class with __enter__ and __exit__. Only
inheritance and metaclasses raise. Say that in the instructions.
MemoryFile is no longer constructed, so drop the import, the annotation, and the
stale references in the sandbox docstring and CLAUDE.md. That CLAUDE.md line
also still claimed a ThreadPoolExecutor, _run_async, and a fresh interpreter per
call.
The read deadline only gets control at a read, so code that computes without
reading escaped it. Give the pool a request_timeout above code_timeout. The
watchdog kills the worker, and execute() already replaces a dead session, so a
runaway call fails and the next call recovers. A read refusal keeps the
variables, so it has to win the race whenever code does read.
Restore test_open_file_objects_are_not_iterable. Replacing the neighbouring
write test by text range deleted it, which left the instructions carrying a
prohibition with nothing to signal when monty lifts it.
Reuse the VFS and the pool when a session is replaced, so recovery skips the
document scan. Mount metadata.json with a lambda rather than a factory. Cover
open() in write mode. Fold the crash entry into the pydantic-monty bullet,
because no release shipped the worker without it.
A crashed worker used to poison the rest of the run. execute() reported the
crash as a failed result, but kept the dead session, and every later call then
raised RuntimeError out of the tool. Clear the session so the next call checks
out a replacement. Keep the session for a syntax or runtime error, which leaves
the worker healthy, and say in the failure text that a restart loses the
variables.
A MemoryFile accepts writes, so metadata.json took them while the other three
document files refused. Mount it through the same read and deny pair. The
write-denial test now covers all four files.
The VFS bridge suspends the Monty worker for the length of a read. Monty checks
its duration budget between interpreter steps, so it cannot check while a read
is in flight. Code that reads in a loop overran a 60s budget by minutes. A read
takes about 20ms on a 2789-document corpus, so a full scan spends about 55s in
reads alone.
Check the deadline before each read. Raising from inside the callback answers the
worker's suspension, which keeps the session usable.
Monty also spends max_duration_secs across the session rather than per call, and
the sandbox reuses the session so that variables persist. Budget it for
code_timeout * max_executions. At the old per-call value the first slow call
starved every later one.
Do not wrap feed_run in asyncio.wait_for. Cancelling during pure compute is
clean, but cancelling while a read waits for an answer wedges the session with a
protocol RuntimeError that escapes execute(). A call that computes without
reading stays bounded by the session budget alone.
Monty 0.0.19 supports open() and with blocks, but a file object is still not
iterable. See pydantic/monty#490, which is still open. The instructions now
state the limitation in three places and give the alternative next to each
one: readlines() or read().split("\n").
Replace chr(10) with "\n" in the prose and in the example. Both work on
0.0.19, and chr(10) implies that the escape is broken.
Add a test that pins the limitation. The test fails when Monty gains
iteration support, which is the signal to relax the instructions.
Document files can be read with open() and with-blocks (.read(),
.readline(), .readlines()); writes raise PermissionError. File objects
remain non-iterable and the collections module is still unavailable.
0.0.19 runs sandboxed code in a subprocess worker pool (AsyncMonty /
AsyncMontySession) and drops MontyRepl. The sandbox checks out a session,
drives it with feed_run, and closes it to return the worker; close() is
now async. Worker crashes surface as a failed SandboxResult. The document
VFS (OSAccess/CallbackFile/MemoryFile) is unchanged.
get_config() builds its instance on first use, so the branch was covered
only when a worker happened to call it before set_config(). Under xdist that
depends on how cases shard across workers, which varies with the core count:
covered locally, uncovered on the two-core runner. Assert it directly.
setup-uv v4 bundles a cache client the GitHub cache service now rejects with
400, so the uv cache never restored and every wheel was redownloaded. Bump
to v9 across all four workflows (build-docs was already drifting at v5).
setup-python read requires-python (">=3.12") and installed 3.14, which uv
then ignored in favour of .python-version (3.13) and downloaded itself.
Point it at .python-version so the interpreter it installs is the one used.
Rewriting the document through client.update_document re-chunks and
re-embeds, so the test needed an embedding endpoint that CI does not have.
Write the row through the repository instead, matching how the rest of the
file avoids the embedder.
check_source_accessible narrowed its handler to ValueError, but Path.exists
re-raises errno values outside its ignored set (EACCES, ENAMETOOLONG). Those
were swallowed before and now escaped into the rebuild sweep the guard exists
to protect. Catch OSError too.
Restore the arity guard in _common_path_prefix: without it an empty list
raises from min() and a single label yields a prefix covering the whole path.
Two tests would have hung rather than failed on regression (the vacuum skip
and the protected-wait cancellation); both are now bounded. The import
vacuum test raced against the done-callback that discards the task, and now
spies on the call instead, with a negative control.
Replace assertions that could not fail: blank-query search against an empty
corpus, a batch flush counted against an empty table, a picture description
asserting its own input state, and an FS scheme check with nothing on disk to
resolve. The get_model matrix asserted only the returned type across 26
cases and now pins the per-provider settings. The three batching tests now
count flushes, which revealed embed-only writes through chunks_table.add
rather than _flush_rebuild_batch.
Cover the remaining paths in the client, context, downloads, title
generation, document tools and store models, and add fail_under=100 so
uncovered lines fail CI.
Six lines that no test can reach get a pragma with its reason: the docling
import guard, the nameless PDF attachment, the FS symlink OSError guard that
resolve(strict=False) absorbs, the docling bbox and LanceDB document-id
shape guards, the tag-retention branch vacuum makes unreachable, and Monty's
Rust-thread print callback.
Fix test_find_config_file_user_config, which wrote its config into the cwd it
had chdir'd to, so the cwd branch answered first and the user-directory
lookup it names was never exercised.
Add vector-index creation tests including the warned failure, chunk
repository get_by_id, list_all pagination, blank-query and precomputed-vector
search, the unknown-score-column guard, settings row recreation, and
replace_for_document with no items.
Add tests for the vacuum-failure warning, documents deleted mid-rebuild,
chunkless documents, batch flushes in the embed-only, descriptions and full
paths, a missing docling blob under rechunk, missing and recoverable picture
bytes, and the source-missing fallback. Direct-call tests cover the staging
helpers and the idempotent phase-1 marker.
Parametrize the get_model tests whose only assertion was the returned model
type, extending the table to the reasoning-off, groq-parsed and Bedrock
o-series/qwen/unmapped branches. Fold format_citations_no_title into the
superset test that already covered its scenario.
Add tests for cosine_similarity on parallel vectors, multi-citation Rich
rendering, picture rendering with missing and undecodable bytes, a missing
docling distribution, and the visualize_chunk short-circuits for absent
documents, absent rasters and refless chunks.
Add tests for the MCP tools' degradation contracts, malformed WebDAV
multistatus bodies, dry-run poller sweeps including the circuit-open and
discover-failure paths, FS source scheme and symlink handling, docling-serve
zip parsing, and the remaining embedding and reranker helpers. Parametrize
_strip_etag.
Drop the misplaced pragma on the analyze handler, which sat on the return and
left the except uncovered. Add one on the FS symlink OSError guard, which
resolve(strict=False) absorbs for every real link.
Delete repository methods with no callers (SettingsRepository CRUD,
ChunkRepository.update/delete/get_chunks_in_range), the DataFrame branch of
_process_search_results whose only caller always passes a query, and guards
that cannot be reached from their call sites: the rebuild mode=None default,
the staging drop already performed by _resolve_rebuild_recovery, the empty
batch skip, two context fast paths, the doctor prefix guard, the poller
_task attribute that is never assigned, and a docling caption fallback for
a field name no item class defines.
set_haiku_version built a recreated settings row from the process-global
Config rather than the store's own, so a store opened with a custom config
stamped global settings into the database.
check_source_accessible called urlparse outside its try block, so a stored
URI with a malformed IPv6 host raised ValueError instead of reporting the
source as inaccessible, aborting the whole rebuild sweep.
Parametrize sibling tests that differed only in a literal value, and fold
two strict-subset tests into the survivors that already covered their
scenario. Every case that ran before still runs; the union of assertions
is applied to each case, strengthening list_all, get_pages_data and
resolve_doc_items.
Replace four hand-rolled log-capture handlers with a shared
capture_logs() contextmanager in conftest.
13 fewer test functions, 348 fewer lines.
ChatApp and InspectorApp assigned self.client before __aenter__
completed, so a failed open was masked by an AttributeError from
on_unmount tearing down the never-opened client.
Drive analysis-only chat with analysis.model (falling back to qa.model) so
the running model matches the one the analysis capability configures,
including its vision flag; RAG-bearing chats still run on qa.model.
Give the AG-UI example state-bearing deps and a final STATE_SNAPSHOT so
registered citations reach the client, mirroring the app backend.
Update chat docs: haiku-rag chat uses --capability/-c, and drop the removed
"View state" command-palette entry.
reranking.multimodal (vllm provider only) attaches picture bytes to
synthetic picture chunks before rerank; VLLMReranker sends them as
content-parts documents (base64 data URI + description text) in the
same /v1/rerank request as plain text documents.
Docling guesses a DocumentStream's format from its first bytes before
considering the extension, so markdown/HTML content starting with a binary
magic signature ("BM" = BMP, "ID3" = MP3) was routed to an image or audio
backend and fell back to plain-text conversion. Prefix the encoded text
with a newline so the sniff finds nothing and the extension decides.
visualize_chunk re-expanded chunks from scratch to recover their refs,
which could not faithfully reproduce the original merge, scores, and
clip — so a visualization could highlight different pages than the
citation covered. Carry the cited items on Citation.doc_item_refs and
resolve bounding boxes from them directly; re-expansion remains only as
the fallback for callers with no stored context (CLI, inspector). Chat,
inspector, the app endpoint, and the frontend pass the refs through.
A merged search result took its chunk_id from whichever constituent
sorted earliest in the document, while its score was the max across the
group — so the citation's identity could point at a different, less
relevant chunk. Anchor chunk_id and the content/refs fallbacks on the
max-score constituent, clip the budget window around that same chunk so
its evidence is never trimmed away, and narrow page_numbers, doc_item_refs,
and attached image bytes to the items that survive the clip.
When a docling-serve instance crashes or returns 5xx, DoclingServeClient now
retries the request on a different instance (up to max_attempts) and trips a
per-instance circuit breaker so subsequent jobs skip a dead instance until its
cooldown elapses. Reuses the shared CircuitBreaker; adds max_attempts and a
nested circuit_breaker to providers.docling_serve.
Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Relocate CircuitBreaker from ingester/pollers to haiku/rag/circuit_breaker
so non-ingester callers (docling-serve provider) can reuse it without
depending on the ingester package.
Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A single oversized document_items row (e.g. a spreadsheet converted to one
table) expanded far past search.max_context_chars and could overflow the
model context window. _expand_outward only used the budget as a soft
accumulation threshold and expand_with_items never capped the joined result.
Add _clip_to_budget to clip each expanded result to max_context_chars,
returning a window centered on the matched chunk (via _evidence_anchors) so
the retrieved evidence survives the cut.
Add `ingester.api.root_path` so the HTTP control plane (dashboard + API)
can be reverse-proxied behind a sub-path (e.g. /ingester/) on a shared
origin, instead of needing nginx sub_filter URL-rewriting.
- APIConfig.root_path: normalized ('', or single leading slash, no trailing
slash) via a field_validator; validate_assignment so CLI overrides
normalize the same way as config-file values.
- Forwarded to FastAPI(root_path=) and uvicorn.Config(root_path=) so
OpenAPI/docs links are prefix-aware.
- Dashboard route injects a <base href> matching root_path; all dashboard
fetches are now base-relative, so they resolve under the prefix while
staying identical at the root.
- `serve --root-path` CLI flag.
- Docs: "Behind a reverse proxy" section with an nginx example.
Closes#431
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate the ingester queue storage from raw aiosqlite to SQLAlchemy Core
async. The backend is chosen by ingester.queue.dburi: a SQLAlchemy async
URL points the queue at a database server, and SQLite remains the default
when unset. The Postgres path claims jobs with FOR UPDATE SKIP LOCKED so
multiple ingester processes can share one queue; SQLite caps the pool to a
single connection to keep the select-then-update claim atomic.
Large files buffered entirely in RAM can OOM workers. Add
max_file_size to source config (default None = no limit).
FS checks stat().st_size before read_bytes(). HTTP and WebDAV issue
a HEAD request before GET when a limit is configured. S3 checks the
size from the existing head_async() call before get_async().
FileTooLargeError is classified as PermanentError so oversized files
go straight to the DLQ instead of retrying.
Each discovered file previously triggered a separate sync.upsert()
call with its own lock acquire + SQLite commit (fsync). On a sweep
finding 1,000 files this meant 1,000 individual commits.
Collect sync_state rows into a list during the sweep and flush them
in a single SyncStateRepo.batch_upsert() call at the end — one lock
acquisition, one commit, one fsync.
HTTP, S3, and WebDAV sources all check `revision is not None and
snapshot.get(uri) == revision` to decide UPSERT vs UNCHANGED. When
a server returns no ETag or Last-Modified, revision is None and the
condition always fails — every sweep emits UPSERT even though the
content hasn't changed.
Now emit UNCHANGED when revision is None and the URI is already
known (has been ingested before). A first-time discovery with no
revision still correctly emits UPSERT.
These files were not touched by the recent performance and
correctness PRs but had coverage gaps. Adds tests for:
- CLI: serve, queue init/migrate, config loading, cli() entry point
including MigrationRequiredError exit path
- filter: _default_supported_extensions, __call__ watchfiles callback,
FileFilter with supported_extensions=None
- registry: resolve_adhoc_fetcher with bucket-less S3 URI
- migrations: pragma no-cover on unreachable schema upgrade path
(no diff migrations exist until SCHEMA_VERSION > 1)
Both are OSError subclasses caught by the broad timeout/io handler
and classified as transient. Pointing at a directory instead of a
file or a broken path component will never succeed on retry.
The bare `except Exception` in HTTPSource.discover() silently
swallowed all errors from HEAD requests — including configuration
errors (bad auth, invalid headers) and programming errors (TypeError,
AttributeError) — treating them identically to network failures by
emitting UPSERT with no revision.
Narrow the catch to httpx.TransportError (the umbrella for
ConnectError, TimeoutException, etc.) and add a debug log. Other
exceptions now propagate to the poller's circuit breaker where they
surface as failures instead of being silently retried forever.
Workers share the pollers' Source instances for fetch(), so the httpx
clients must be closed only after the pool has stopped. Guards both
run_batch() and serve() against reintroducing the shutdown-order bug.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Workers share the same Source instances as pollers and use them for
fetch(). PollerManager.stop() was closing httpx clients before the
worker pool drained, so in-flight fetches during the shutdown grace
hit a closed client.
- Move source closing out of stop() into a separate close_sources()
- Call close_sources() after _stop_pool() in both serve() and
run_batch()
- Promote aclose() to the Source protocol with no-op defaults for
FS and S3, removing the hasattr duck-typing
The condition only checked claimed jobs, but queued jobs with no
live workers also hang forever. Check live_workers == 0 regardless
of whether outstanding work is queued or claimed.
Rewrite the test to actually crash workers: patch _process to raise
a bare Exception (which _worker_loop doesn't catch), use
worker_count=1 so the single crash leaves live_workers == 0, and
assert the abort log message fires.
The jitter-before-first-sleep block was duplicated verbatim in
PeriodicPoller.run() and FSPoller._sweep_loop(). Move it to
BasePoller._stagger_start() with a named _STAGGER_FRACTION constant.
This also gives a testable seam outside the pragma-no-cover
event-loop glue methods.
The previous test deleted a file mid-iteration, but is_file()
caught it before stat() ran — so the new try/except never executed.
Monkeypatch Path.stat to raise FileNotFoundError on the third call
for the victim path (after is_symlink and is_file pass), simulating
the exact TOCTOU window between is_file() and stat().
Workers parked on job_available.wait() were not woken by stop(),
causing them to sleep out the full poll_idle_interval_s before
noticing _stop. With the default 1.0s interval, stop() took ~0.8s
instead of ~0.007s.
Notify all waiters on the condition in stop() so idle workers exit
immediately. Add tests for fast job pickup via notification and
fast shutdown with idle workers.
PermissionError is a subclass of OSError, so it was caught by the
broad timeout/io handler and classified as transient. An unreadable
file would retry 5 times then DLQ — permissions don't fix themselves
without operator intervention.
Add an explicit PermissionError check before the OSError catch so
unreadable files go straight to the DLQ.
If sync.upsert() or sync.delete() raises after a job is already
marked succeeded (e.g. disk full, DB locked), the unhandled
exception crashes the worker. The job stays succeeded but sync_state
is stale, and the crashed worker stops processing other jobs.
Wrap the post-success sync_state writes in a try/except. On failure,
log the error and continue. The worst case is a redundant re-ingest
on the next sweep — better than killing the worker.
The drain loop in run_batch() polls counts_by_status() waiting for
queued and claimed counts to reach zero. If all worker tasks crash
(unhandled exception, OOM), claimed jobs stay claimed forever and
the loop never exits — the CLI command hangs.
Check live_workers during the drain loop. If claimed jobs exist but
no workers are alive to process them, log an error and break out.
The stranded jobs will be reaped on the next start.
The expression `str(path.stat().st_mtime_ns) if path.exists() else None`
has a TOCTOU race: the file can be deleted between exists() and stat().
The resulting FileNotFoundError propagates up to _watch_loop's except
handler, which records a breaker failure and terminates the loop — no
more push events are processed until restart.
Replace with a try/except around stat() and return early on
FileNotFoundError. The deletion event from watchfiles will handle
cleanup.
FileNotFoundError is a subclass of OSError, so it was caught by the
broad timeout/io handler and classified as transient. A file deleted
between discovery and fetch would retry 5 times on a file that's
permanently gone, then DLQ with a confusing error message.
Add an explicit FileNotFoundError check before the OSError catch so
deleted files go straight to the DLQ.
A file deleted between os.walk() and path.stat() raises
FileNotFoundError, which propagated uncaught and failed the entire
discover() sweep. With enough failures this trips the circuit
breaker, silencing the poller.
Catch FileNotFoundError around the stat() call and skip the file.
The next sweep (or watchfiles) will emit the DELETE event.
