Commit graph

943 commits

Author SHA1 Message Date
Yiorgis Gozadinos
53084d6fdd
Bump pydantic-monty to 0.0.19
0.0.19 runs sandboxed code in a subprocess worker pool (AsyncMonty /
AsyncMontySession) and drops MontyRepl. The sandbox checks out a session,
drives it with feed_run, and closes it to return the worker; close() is
now async. Worker crashes surface as a failed SandboxResult. The document
VFS (OSAccess/CallbackFile/MemoryFile) is unchanged.
2026-07-28 19:10:53 +03:00
Yiorgis Gozadinos
f13a3fb677
Report tool failures with ToolFailed 2026-07-27 18:26:42 +03:00
Yiorgis Gozadinos
ae345cc39f
Map thinking onto Pydantic AI's unified setting 2026-07-27 18:26:42 +03:00
Yiorgis Gozadinos
04ad3823b9
Cover the lazy config init and restore the uv cache in CI
get_config() builds its instance on first use, so the branch was covered
only when a worker happened to call it before set_config(). Under xdist that
depends on how cases shard across workers, which varies with the core count:
covered locally, uncovered on the two-core runner. Assert it directly.

setup-uv v4 bundles a cache client the GitHub cache service now rejects with
400, so the uv cache never restored and every wheel was redownloaded. Bump
to v9 across all four workflows (build-docs was already drifting at v5).

setup-python read requires-python (">=3.12") and installed 3.14, which uv
then ignored in favour of .python-version (3.13) and downloaded itself.
Point it at .python-version so the interpreter it installs is the one used.
2026-07-27 13:35:50 +03:00
Yiorgis Gozadinos
ab88188529
Keep the lazy-read test off the embedder
Rewriting the document through client.update_document re-chunks and
re-embeds, so the test needed an embedding endpoint that CI does not have.
Write the row through the repository instead, matching how the rest of the
file avoids the embedder.
2026-07-27 13:17:05 +03:00
Yiorgis Gozadinos
a602e0fbf5
Test vacuum against real table state 2026-07-27 12:11:01 +03:00
Yiorgis Gozadinos
f96a428ef1
Fix defects found reviewing the coverage work
check_source_accessible narrowed its handler to ValueError, but Path.exists
re-raises errno values outside its ignored set (EACCES, ENAMETOOLONG). Those
were swallowed before and now escaped into the rebuild sweep the guard exists
to protect. Catch OSError too.

Restore the arity guard in _common_path_prefix: without it an empty list
raises from min() and a single label yields a prefix covering the whole path.

Two tests would have hung rather than failed on regression (the vacuum skip
and the protected-wait cancellation); both are now bounded. The import
vacuum test raced against the done-callback that discards the task, and now
spies on the call instead, with a negative control.

Replace assertions that could not fail: blank-query search against an empty
corpus, a batch flush counted against an empty table, a picture description
asserting its own input state, and an FS scheme check with nothing on disk to
resolve. The get_model matrix asserted only the returned type across 26
cases and now pins the per-provider settings. The three batching tests now
count flushes, which revealed embed-only writes through chunks_table.add
rather than _flush_rebuild_batch.
2026-07-27 10:44:32 +03:00
Yiorgis Gozadinos
7ba78fdde3
Normalize pragma comments to the codebase's single-line form 2026-07-26 20:18:19 +03:00
Yiorgis Gozadinos
f6acb65e95
Reach and enforce 100% coverage
Cover the remaining paths in the client, context, downloads, title
generation, document tools and store models, and add fail_under=100 so
uncovered lines fail CI.

Six lines that no test can reach get a pragma with its reason: the docling
import guard, the nameless PDF attachment, the FS symlink OSError guard that
resolve(strict=False) absorbs, the docling bbox and LanceDB document-id
shape guards, the tag-retention branch vacuum makes unreachable, and Monty's
Rust-thread print callback.

Fix test_find_config_file_user_config, which wrote its config into the cwd it
had chdir'd to, so the cwd branch answered first and the user-directory
lookup it names was never exercised.
2026-07-26 20:11:02 +03:00
Yiorgis Gozadinos
7c120587f0
Cover sandbox VFS reads, binary part dedup and picture spans 2026-07-26 19:41:09 +03:00
Yiorgis Gozadinos
2afe1bd28c
Cover store engine and repository paths
Add vector-index creation tests including the warned failure, chunk
repository get_by_id, list_all pagination, blank-query and precomputed-vector
search, the unknown-score-column guard, settings row recreation, and
replace_for_document with no items.
2026-07-26 19:36:37 +03:00
Yiorgis Gozadinos
dc5ad8f699
Cover the rebuild paths
Add tests for the vacuum-failure warning, documents deleted mid-rebuild,
chunkless documents, batch flushes in the embed-only, descriptions and full
paths, a missing docling blob under rechunk, missing and recoverable picture
bytes, and the source-missing fallback. Direct-call tests cover the staging
helpers and the idempotent phase-1 marker.
2026-07-26 19:32:09 +03:00
Yiorgis Gozadinos
7c76c3399e
Cover get_model provider branches and visualize_chunk edge paths
Parametrize the get_model tests whose only assertion was the returned model
type, extending the table to the reasoning-off, groq-parsed and Bedrock
o-series/qwen/unmapped branches. Fold format_citations_no_title into the
superset test that already covered its scenario.

Add tests for cosine_similarity on parallel vectors, multi-citation Rich
rendering, picture rendering with missing and undecodable bytes, a missing
docling distribution, and the visualize_chunk short-circuits for absent
documents, absent rasters and refless chunks.
2026-07-26 19:23:54 +03:00
Yiorgis Gozadinos
1e8e5e9f6f
Cover MCP, ingester and converter error paths
Add tests for the MCP tools' degradation contracts, malformed WebDAV
multistatus bodies, dry-run poller sweeps including the circuit-open and
discover-failure paths, FS source scheme and symlink handling, docling-serve
zip parsing, and the remaining embedding and reranker helpers. Parametrize
_strip_etag.

Drop the misplaced pragma on the analyze handler, which sat on the return and
left the except uncovered. Add one on the FS symlink OSError guard, which
resolve(strict=False) absorbs for every real link.
2026-07-26 19:14:26 +03:00
Yiorgis Gozadinos
dddaf0f84c
Remove unreachable code and fix two defects it surfaced
Delete repository methods with no callers (SettingsRepository CRUD,
ChunkRepository.update/delete/get_chunks_in_range), the DataFrame branch of
_process_search_results whose only caller always passes a query, and guards
that cannot be reached from their call sites: the rebuild mode=None default,
the staging drop already performed by _resolve_rebuild_recovery, the empty
batch skip, two context fast paths, the doctor prefix guard, the poller
_task attribute that is never assigned, and a docling caption fallback for
a field name no item class defines.

set_haiku_version built a recreated settings row from the process-global
Config rather than the store's own, so a store opened with a custom config
stamped global settings into the database.

check_source_accessible called urlparse outside its try block, so a stored
URI with a malformed IPv6 host raised ValueError instead of reporting the
source as inaccessible, aborting the whole rebuild sweep.
2026-07-26 18:45:48 +03:00
Yiorgis Gozadinos
e2b273ee2a
Consolidate duplicated client-side tests
Parametrize sibling tests that differed only in a literal value, and fold
two strict-subset tests into the survivors that already covered their
scenario. Every case that ran before still runs; the union of assertions
is applied to each case, strengthening list_all, get_pages_data and
resolve_doc_items.

Replace four hand-rolled log-capture handlers with a shared
capture_logs() contextmanager in conftest.

13 fewer test functions, 348 fewer lines.
2026-07-26 13:30:59 +03:00
Yiorgis Gozadinos
c716be9640
cover the mcp path with no images 2026-07-25 15:00:36 +03:00
Yiorgis Gozadinos
cdaeaa94e6
Constrain the chat prompt container height 2026-07-25 14:40:40 +03:00
Yiorgis Gozadinos
93824cdee5
Surface database-open errors in the TUIs
ChatApp and InspectorApp assigned self.client before __aenter__
completed, so a failed open was masked by an AttributeError from
on_unmount tearing down the never-opened client.
2026-07-25 11:27:29 +03:00
Yiorgis Gozadinos
5f4c73f89f
Add image attachment to the chat TUI 2026-07-25 10:58:08 +03:00
Yiorgis Gozadinos
c62fd78c7a
Accept images on CLI ask/analyze and MCP tools 2026-07-25 10:35:25 +03:00
Yiorgis Gozadinos
4c5050d161
Accept images on ask/analyze 2026-07-25 10:23:55 +03:00
Yiorgis Gozadinos
55351a8829
Merge leading system messages for OpenAI-compatible endpoints 2026-07-24 16:57:55 +03:00
Yiorgis Gozadinos
f87fabe556
Make the capability image-attachment gate a vision bool 2026-07-24 16:00:01 +03:00
Yiorgis Gozadinos
6c5bc0aae1
Gate combined-chat capability vision on the driving model 2026-07-24 15:27:20 +03:00
Yiorgis Gozadinos
597808c56e
Fix chat analysis-model selection, AG-UI example state, and chat docs
Drive analysis-only chat with analysis.model (falling back to qa.model) so
the running model matches the one the analysis capability configures,
including its vision flag; RAG-bearing chats still run on qa.model.

Give the AG-UI example state-bearing deps and a final STATE_SNAPSHOT so
registered citations reach the client, mirroring the app backend.

Update chat docs: haiku-rag chat uses --capability/-c, and drop the removed
"View state" command-palette entry.
2026-07-24 15:27:20 +03:00
Yiorgis Gozadinos
b8dcb066dc
Port cite partial-success feedback to capabilities 2026-07-24 15:27:19 +03:00
Yiorgis Gozadinos
175929f23a
Improve coverage 2026-07-24 15:26:18 +03:00
Yiorgis Gozadinos
77585ef26a
fix per-question capability limits and tool isolation 2026-07-24 15:26:18 +03:00
Yiorgis Gozadinos
43c17a6777
fix capability execution limits and chat loading 2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
22ffd5e92e
Add max retries=3 back 2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
9deb1f2bd4
replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
35b8b046ce
Cover the detached-chunk guard in the multimodal rerank search test 2026-07-24 13:07:39 +03:00
Yiorgis Gozadinos
543aba7547
Multimodal reranking: send picture chunks to vllm rerankers as images
reranking.multimodal (vllm provider only) attaches picture bytes to
synthetic picture chunks before rerank; VLLMReranker sends them as
content-parts documents (base64 data URI + description text) in the
same /v1/rerank request as plain text documents.
2026-07-24 12:29:20 +03:00
Yiorgis Gozadinos
0308220769
Read verbs skip the embeddings config-compatibility check 2026-07-24 12:14:31 +03:00
Yiorgis Gozadinos
d9cc9a4eea
Narrow AutoTokenizer type for transformers 5 2026-07-23 12:01:28 +03:00
Yiorgis Gozadinos
75a21c82c2
Search results and citations carry document metadata 2026-07-23 10:23:48 +03:00
Yiorgis Gozadinos
8bf569ef89
cite reports unresolvable chunk ids on partial success 2026-07-22 11:56:56 +03:00
Yiorgis Gozadinos
3034280c78
Skip embedding validation on document deletion 2026-07-22 10:48:00 +03:00
Yiorgis Gozadinos
1ed424b50d
Serialize metadata-only document updates with the write lock 2026-07-16 13:11:35 +03:00
Yiorgis Gozadinos
f02cde5ddf
Fix same-tick cancellation losing recovery results in _wait_protected 2026-07-16 13:11:35 +03:00
Yiorgis Gozadinos
dd2817ff6d
Protect tag-creation cleanup from cancellation 2026-07-16 13:11:35 +03:00
Yiorgis Gozadinos
1f88944ada
Protect restore rollback from cancellation; report delete_tag listing failures 2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
ce1b9ac88e
Harden history against tag annotation failures; document tag restore 2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
5453c00c95
Test restore and migration separation 2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
4522fbdf1b
Add tag restore CLI 2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
f41ec379df
Add Store.restore_tag 2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
a1f3435df3
Vacuum suppresses OSError only 2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
f997d7adc9
Consolidate tag primitives 2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
0e271eaf4b
Remove --before/--at time travel 2026-07-16 13:11:33 +03:00
Yiorgis Gozadinos
0813a1c980
Add tag CLI commands and history tag annotations 2026-07-16 13:11:27 +03:00
Yiorgis Gozadinos
0cbde6b7a2
Serialize tag operations, vacuum, and metadata refresh with writes 2026-07-16 12:32:57 +03:00
Yiorgis Gozadinos
69319c9390
Grow vacuum retention to protect tagged versions 2026-07-16 12:32:57 +03:00
Yiorgis Gozadinos
ae603d9b4d
Add database-level tag primitives to Store 2026-07-16 12:32:56 +03:00
Yiorgis Gozadinos
aa620aeb60
Bump lancedb to 0.34.0 2026-07-16 12:32:56 +03:00
Yiorgis Gozadinos
d2f5ecfc58
Prevent magic-byte sniffing from misrouting text conversion in docling-local
Docling guesses a DocumentStream's format from its first bytes before
considering the extension, so markdown/HTML content starting with a binary
magic signature ("BM" = BMP, "ID3" = MP3) was routed to an image or audio
backend and fell back to plain-text conversion. Prefix the encoded text
with a newline so the sniff finds nothing and the extension decides.
2026-07-15 19:20:54 +03:00
Yiorgis Gozadinos
144900d385
Remove the mxbai reranking provider 2026-07-14 11:09:55 +03:00
Yiorgis Gozadinos
5e2a928013
Allow transformers 5.x in the mxbai extra 2026-07-14 11:09:55 +03:00
Yiorgis Gozadinos
82db15cf3c
Fill missing item-table pages from fully-visible inputs in context expansion 2026-07-14 10:32:45 +03:00
Yiorgis Gozadinos
ee5b7b7313
Split merged context-expansion groups that cannot afford every constituent 2026-07-14 10:15:28 +03:00
Yiorgis Gozadinos
271fdf9b5a
Add Logfire debugging skills and worker-breaker event 2026-07-10 13:23:17 +03:00
Yiorgis Gozadinos
a82673a900
Route eval scripts through the shared telemetry configure 2026-07-10 11:38:33 +03:00
Yiorgis Gozadinos
73ba764922
Emit a docling_serve.request span per instance attempt 2026-07-10 11:16:27 +03:00
Yiorgis Gozadinos
da79f92af1
Honor OTEL_SERVICE_NAME and set service name/version for all processes 2026-07-10 11:11:46 +03:00
Yiorgis Gozadinos
18c0f6c5e8
Add --no-expand to the visualize command for chunk-only grounding 2026-07-09 12:49:42 +03:00
Yiorgis Gozadinos
995d081aa1
Visualize the exact context the model saw via Citation.doc_item_refs
visualize_chunk re-expanded chunks from scratch to recover their refs,
which could not faithfully reproduce the original merge, scores, and
clip — so a visualization could highlight different pages than the
citation covered. Carry the cited items on Citation.doc_item_refs and
resolve bounding boxes from them directly; re-expansion remains only as
the fallback for callers with no stored context (CLI, inspector). Chat,
inspector, the app endpoint, and the frontend pass the refs through.
2026-07-09 11:51:37 +03:00
Yiorgis Gozadinos
c424f51056
Anchor and clip merged citations on the highest-scoring chunk
A merged search result took its chunk_id from whichever constituent
sorted earliest in the document, while its score was the max across the
group — so the citation's identity could point at a different, less
relevant chunk. Anchor chunk_id and the content/refs fallbacks on the
max-score constituent, clip the budget window around that same chunk so
its evidence is never trimmed away, and narrow page_numbers, doc_item_refs,
and attached image bytes to the items that survive the clip.
2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
6d86237dd6
Draw matched content stronger than expanded context in visualizations 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
494f774473
Visualize all constituent chunks of a merged citation 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
0bcf34363a
Carry merged chunk ids on SearchResult and Citation 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
b14152a45b
Keep picture and table hits section-bounded during context expansion 2026-07-09 11:05:40 +03:00
Yiorgis Gozadinos
c02bcd5dc9
Pool HTTP clients for vLLM embedding and vLLM/Jina reranking 2026-07-09 10:32:44 +03:00
Yiorgis Gozadinos
43a580afb3
Dedupe picture chunks and skip small pictures at chunking 2026-07-09 10:24:32 +03:00
Yiorgis Gozadinos
a1cf405cae
Rename document_meta identity column document_id to id 2026-07-08 16:32:55 +03:00
Yiorgis Gozadinos
23bbca5998
Re-check URI under the write lock to prevent duplicate ingestion 2026-07-08 11:26:45 +03:00
Yiorgis Gozadinos
3c7699f156
Fail docling-serve requests over to another instance with per-instance breaking
When a docling-serve instance crashes or returns 5xx, DoclingServeClient now
retries the request on a different instance (up to max_attempts) and trips a
per-instance circuit breaker so subsequent jobs skip a dead instance until its
cooldown elapses. Reuses the shared CircuitBreaker; adds max_attempts and a
nested circuit_breaker to providers.docling_serve.

Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 11:03:44 +03:00
Yiorgis Gozadinos
d781335868
Move CircuitBreaker to a shared module
Relocate CircuitBreaker from ingester/pollers to haiku/rag/circuit_breaker
so non-ingester callers (docling-serve provider) can reuse it without
depending on the ingester package.

Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 11:03:30 +03:00
Yiorgis Gozadinos
c2c8c24951
Allow updating a document's uri via update_document 2026-07-08 10:32:14 +03:00
Yiorgis Gozadinos
d2593304c5
Attach figure bytes when a result matches its caption 2026-07-03 13:22:02 +03:00
Yiorgis Gozadinos
ac64aa9c17
Build the skills rag_db fixture once per session 2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
e4b1f16f97
Shrink the docling-serve context-expansion cassette 2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
afd6e3ae00
Share the VLM URL builder and centralize the embedder empty guard 2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
38079ff89a
Hoist reranker empty-input guard into base; stop factory tests loading models 2026-06-29 14:20:23 +03:00
Yiorgis Gozadinos
4f8cc4eb30
Show a progress spinner while doctor runs 2026-06-28 10:44:30 +03:00
Yiorgis Gozadinos
e7b4988d82
Compute document centroids during the vector scan 2026-06-28 10:23:16 +03:00
Yiorgis Gozadinos
15aae0f242
Simplify duplicate detection to whole-document centroid similarity 2026-06-28 10:23:16 +03:00
Yiorgis Gozadinos
b94e13083f
Hard-cap context expansion at max_context_chars
A single oversized document_items row (e.g. a spreadsheet converted to one
table) expanded far past search.max_context_chars and could overflow the
model context window. _expand_outward only used the budget as a soft
accumulation threshold and expand_with_items never capped the joined result.

Add _clip_to_budget to clip each expanded result to max_context_chars,
returning a window centered on the matched chunk (via _evidence_anchors) so
the retrieved evidence survives the cut.
2026-06-27 09:46:36 +03:00
Yiorgis Gozadinos
73f62ab290
Enforce the duplicate candidate cap during row collection 2026-06-26 16:57:51 +03:00
Yiorgis Gozadinos
f1e1a16f9b
Add doctor --duplicates-out YAML export; summarize terminal report 2026-06-26 16:52:22 +03:00
Yiorgis Gozadinos
044ac62e49
Drop boilerplate handling; make duplicate report readable 2026-06-26 16:52:22 +03:00
Yiorgis Gozadinos
961913dde4
Ignore boilerplate chunks in duplicate-document detection 2026-06-26 16:52:21 +03:00
Yiorgis Gozadinos
43f2130b66
Add near-duplicate document detection to doctor 2026-06-26 16:52:21 +03:00
Yiorgis Gozadinos
ce27a7aae5
Clean up S3 test data per-test to keep SeaweedFS writable 2026-06-26 16:41:20 +03:00
Yiorgis Gozadinos
87f0233bcd
Cover heartbeat health, lease-renewal failure, and breaker recovery 2026-06-25 16:32:56 +03:00
Yiorgis Gozadinos
4707db780f
Renew job leases from the worker pool; lease-based reaping 2026-06-25 13:26:09 +03:00
Yiorgis Gozadinos
fd73f57649
Add last_heartbeat_at lease column to the ingester queue 2026-06-25 13:25:37 +03:00
Yiorgis Gozadinos
ba6b318ece
Make ingester worker ids globally unique 2026-06-25 13:25:37 +03:00
Yiorgis Gozadinos
b61134b747
Log FTS index build failures at WARNING 2026-06-25 13:00:29 +03:00
Yiorgis Gozadinos
2fd025951c
Treat empty env vars as unset in config expansion 2026-06-25 12:41:04 +03:00
Yiorgis Gozadinos
9f4dda9254
Reuse one MarkdownDocSerializer per document in item extraction 2026-06-24 12:02:13 +03:00