Yiorgis Gozadinos
94e03d777a
vb
2026-07-25 15:11:54 +03:00
Yiorgis Gozadinos
081856c6b1
Instruct capabilities to judge user-attached images
2026-07-25 14:40:51 +03:00
Yiorgis Gozadinos
cdaeaa94e6
Constrain the chat prompt container height
2026-07-25 14:40:40 +03:00
Yiorgis Gozadinos
93824cdee5
Surface database-open errors in the TUIs
...
ChatApp and InspectorApp assigned self.client before __aenter__
completed, so a failed open was masked by an AttributeError from
on_unmount tearing down the never-opened client.
2026-07-25 11:27:29 +03:00
Yiorgis Gozadinos
5f4c73f89f
Add image attachment to the chat TUI
2026-07-25 10:58:08 +03:00
Yiorgis Gozadinos
c62fd78c7a
Accept images on CLI ask/analyze and MCP tools
2026-07-25 10:35:25 +03:00
Yiorgis Gozadinos
4c5050d161
Accept images on ask/analyze
2026-07-25 10:23:55 +03:00
Yiorgis Gozadinos
acbd66afbd
vb
2026-07-24 17:32:20 +03:00
Yiorgis Gozadinos
55351a8829
Merge leading system messages for OpenAI-compatible endpoints
2026-07-24 16:57:55 +03:00
Yiorgis Gozadinos
f87fabe556
Make the capability image-attachment gate a vision bool
2026-07-24 16:00:01 +03:00
Yiorgis Gozadinos
6c5bc0aae1
Gate combined-chat capability vision on the driving model
2026-07-24 15:27:20 +03:00
Yiorgis Gozadinos
597808c56e
Fix chat analysis-model selection, AG-UI example state, and chat docs
...
Drive analysis-only chat with analysis.model (falling back to qa.model) so
the running model matches the one the analysis capability configures,
including its vision flag; RAG-bearing chats still run on qa.model.
Give the AG-UI example state-bearing deps and a final STATE_SNAPSHOT so
registered citations reach the client, mirroring the app backend.
Update chat docs: haiku-rag chat uses --capability/-c, and drop the removed
"View state" command-palette entry.
2026-07-24 15:27:20 +03:00
Yiorgis Gozadinos
b8dcb066dc
Port cite partial-success feedback to capabilities
2026-07-24 15:27:19 +03:00
Yiorgis Gozadinos
175929f23a
Improve coverage
2026-07-24 15:26:18 +03:00
Yiorgis Gozadinos
77585ef26a
fix per-question capability limits and tool isolation
2026-07-24 15:26:18 +03:00
Yiorgis Gozadinos
43c17a6777
fix capability execution limits and chat loading
2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
22ffd5e92e
Add max retries=3 back
2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
9e7db72997
Load capabilities eagerly for dedicated single-agent consumers
2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
9deb1f2bd4
replace haiku.skills with native Pydantic AI capabilities
2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
ec78641cc4
vb
2026-07-24 15:15:20 +03:00
Yiorgis Gozadinos
f31060e741
Set explicit 120s timeout on the vllm reranker HTTP client
2026-07-24 12:34:31 +03:00
Yiorgis Gozadinos
543aba7547
Multimodal reranking: send picture chunks to vllm rerankers as images
...
reranking.multimodal (vllm provider only) attaches picture bytes to
synthetic picture chunks before rerank; VLLMReranker sends them as
content-parts documents (base64 data URI + description text) in the
same /v1/rerank request as plain text documents.
2026-07-24 12:29:20 +03:00
Yiorgis Gozadinos
0308220769
Read verbs skip the embeddings config-compatibility check
2026-07-24 12:14:31 +03:00
Yiorgis Gozadinos
9f49bdde6e
vb
2026-07-23 14:28:23 +03:00
Yiorgis Gozadinos
294ced57cd
vb
2026-07-23 10:45:48 +03:00
Yiorgis Gozadinos
75a21c82c2
Search results and citations carry document metadata
2026-07-23 10:23:48 +03:00
Yiorgis Gozadinos
4f0fb7322e
vb
2026-07-22 13:27:35 +03:00
Yiorgis Gozadinos
8bf569ef89
cite reports unresolvable chunk ids on partial success
2026-07-22 11:56:56 +03:00
Yiorgis Gozadinos
3034280c78
Skip embedding validation on document deletion
2026-07-22 10:48:00 +03:00
Yiorgis Gozadinos
b5c0ea630c
vb
2026-07-16 13:50:40 +03:00
Yiorgis Gozadinos
1ed424b50d
Serialize metadata-only document updates with the write lock
2026-07-16 13:11:35 +03:00
Yiorgis Gozadinos
f02cde5ddf
Fix same-tick cancellation losing recovery results in _wait_protected
2026-07-16 13:11:35 +03:00
Yiorgis Gozadinos
dd2817ff6d
Protect tag-creation cleanup from cancellation
2026-07-16 13:11:35 +03:00
Yiorgis Gozadinos
1f88944ada
Protect restore rollback from cancellation; report delete_tag listing failures
2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
ce1b9ac88e
Harden history against tag annotation failures; document tag restore
2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
4522fbdf1b
Add tag restore CLI
2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
f41ec379df
Add Store.restore_tag
2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
a1f3435df3
Vacuum suppresses OSError only
2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
f997d7adc9
Consolidate tag primitives
2026-07-16 13:11:34 +03:00
Yiorgis Gozadinos
0e271eaf4b
Remove --before/--at time travel
2026-07-16 13:11:33 +03:00
Yiorgis Gozadinos
0813a1c980
Add tag CLI commands and history tag annotations
2026-07-16 13:11:27 +03:00
Yiorgis Gozadinos
0cbde6b7a2
Serialize tag operations, vacuum, and metadata refresh with writes
2026-07-16 12:32:57 +03:00
Yiorgis Gozadinos
69319c9390
Grow vacuum retention to protect tagged versions
2026-07-16 12:32:57 +03:00
Yiorgis Gozadinos
ae603d9b4d
Add database-level tag primitives to Store
2026-07-16 12:32:56 +03:00
Yiorgis Gozadinos
aa620aeb60
Bump lancedb to 0.34.0
2026-07-16 12:32:56 +03:00
Yiorgis Gozadinos
d2f5ecfc58
Prevent magic-byte sniffing from misrouting text conversion in docling-local
...
Docling guesses a DocumentStream's format from its first bytes before
considering the extension, so markdown/HTML content starting with a binary
magic signature ("BM" = BMP, "ID3" = MP3) was routed to an image or audio
backend and fell back to plain-text conversion. Prefix the encoded text
with a newline so the sniff finds nothing and the extension decides.
2026-07-15 19:20:54 +03:00
Yiorgis Gozadinos
5dd310a9b7
vb
2026-07-14 16:06:50 +03:00
Yiorgis Gozadinos
c1ec13f081
Lower search.max_context_chars default to 5000
2026-07-14 11:58:42 +03:00
Yiorgis Gozadinos
144900d385
Remove the mxbai reranking provider
2026-07-14 11:09:55 +03:00
Yiorgis Gozadinos
5e2a928013
Allow transformers 5.x in the mxbai extra
2026-07-14 11:09:55 +03:00
Yiorgis Gozadinos
82db15cf3c
Fill missing item-table pages from fully-visible inputs in context expansion
2026-07-14 10:32:45 +03:00
Yiorgis Gozadinos
ee5b7b7313
Split merged context-expansion groups that cannot afford every constituent
2026-07-14 10:15:28 +03:00
Yiorgis Gozadinos
5c41f8a0ed
vb
2026-07-10 13:38:24 +03:00
Yiorgis Gozadinos
271fdf9b5a
Add Logfire debugging skills and worker-breaker event
2026-07-10 13:23:17 +03:00
Yiorgis Gozadinos
a82673a900
Route eval scripts through the shared telemetry configure
2026-07-10 11:38:33 +03:00
Yiorgis Gozadinos
9b3b21b1a4
Document ingestion observability; drop stale logfire install extra
2026-07-10 11:22:23 +03:00
Yiorgis Gozadinos
73ba764922
Emit a docling_serve.request span per instance attempt
2026-07-10 11:16:27 +03:00
Yiorgis Gozadinos
da79f92af1
Honor OTEL_SERVICE_NAME and set service name/version for all processes
2026-07-10 11:11:46 +03:00
Yiorgis Gozadinos
da0748916b
vb
2026-07-09 15:58:58 +03:00
Yiorgis Gozadinos
e7d5a0440d
Tune two-tone visualization colors
2026-07-09 15:37:25 +03:00
Yiorgis Gozadinos
18c0f6c5e8
Add --no-expand to the visualize command for chunk-only grounding
2026-07-09 12:49:42 +03:00
Yiorgis Gozadinos
995d081aa1
Visualize the exact context the model saw via Citation.doc_item_refs
...
visualize_chunk re-expanded chunks from scratch to recover their refs,
which could not faithfully reproduce the original merge, scores, and
clip — so a visualization could highlight different pages than the
citation covered. Carry the cited items on Citation.doc_item_refs and
resolve bounding boxes from them directly; re-expansion remains only as
the fallback for callers with no stored context (CLI, inspector). Chat,
inspector, the app endpoint, and the frontend pass the refs through.
2026-07-09 11:51:37 +03:00
Yiorgis Gozadinos
c424f51056
Anchor and clip merged citations on the highest-scoring chunk
...
A merged search result took its chunk_id from whichever constituent
sorted earliest in the document, while its score was the max across the
group — so the citation's identity could point at a different, less
relevant chunk. Anchor chunk_id and the content/refs fallbacks on the
max-score constituent, clip the budget window around that same chunk so
its evidence is never trimmed away, and narrow page_numbers, doc_item_refs,
and attached image bytes to the items that survive the clip.
2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
6d86237dd6
Draw matched content stronger than expanded context in visualizations
2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
494f774473
Visualize all constituent chunks of a merged citation
2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
0bcf34363a
Carry merged chunk ids on SearchResult and Citation
2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
b14152a45b
Keep picture and table hits section-bounded during context expansion
2026-07-09 11:05:40 +03:00
Yiorgis Gozadinos
c02bcd5dc9
Pool HTTP clients for vLLM embedding and vLLM/Jina reranking
2026-07-09 10:32:44 +03:00
Yiorgis Gozadinos
43a580afb3
Dedupe picture chunks and skip small pictures at chunking
2026-07-09 10:24:32 +03:00
Yiorgis Gozadinos
a1cf405cae
Rename document_meta identity column document_id to id
2026-07-08 16:32:55 +03:00
Yiorgis Gozadinos
23bbca5998
Re-check URI under the write lock to prevent duplicate ingestion
2026-07-08 11:26:45 +03:00
Yiorgis Gozadinos
3c7699f156
Fail docling-serve requests over to another instance with per-instance breaking
...
When a docling-serve instance crashes or returns 5xx, DoclingServeClient now
retries the request on a different instance (up to max_attempts) and trips a
per-instance circuit breaker so subsequent jobs skip a dead instance until its
cooldown elapses. Reuses the shared CircuitBreaker; adds max_attempts and a
nested circuit_breaker to providers.docling_serve.
Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 11:03:44 +03:00
Yiorgis Gozadinos
d781335868
Move CircuitBreaker to a shared module
...
Relocate CircuitBreaker from ingester/pollers to haiku/rag/circuit_breaker
so non-ingester callers (docling-serve provider) can reuse it without
depending on the ingester package.
Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 11:03:30 +03:00
Yiorgis Gozadinos
c2c8c24951
Allow updating a document's uri via update_document
2026-07-08 10:32:14 +03:00
Yiorgis Gozadinos
9a9bea86d3
vb
2026-07-03 13:35:10 +03:00
Yiorgis Gozadinos
d2593304c5
Attach figure bytes when a result matches its caption
2026-07-03 13:22:02 +03:00
Yiorgis Gozadinos
7f50d3698f
Validate reranker config before importing the provider package
2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
afd6e3ae00
Share the VLM URL builder and centralize the embedder empty guard
2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
654cb2b94c
Collapse get_reranker into a single guard and import-guarded dispatch
...
Replace the seven repeated `config.reranking.model and ... == provider`
checks and six per-branch ImportError handlers with one None guard and one
try/except around the provider dispatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
38079ff89a
Hoist reranker empty-input guard into base; stop factory tests loading models
2026-06-29 14:20:23 +03:00
Yiorgis Gozadinos
9a85632cf3
vb
2026-06-29 12:36:36 +03:00
Yiorgis Gozadinos
5fc4056dcd
Require haiku.skills>=0.18.0
2026-06-29 11:44:51 +03:00
Yiorgis Gozadinos
b2abdfaffb
vb
2026-06-28 10:57:26 +03:00
Yiorgis Gozadinos
4f8cc4eb30
Show a progress spinner while doctor runs
2026-06-28 10:44:30 +03:00
Yiorgis Gozadinos
e7b4988d82
Compute document centroids during the vector scan
2026-06-28 10:23:16 +03:00
Yiorgis Gozadinos
15aae0f242
Simplify duplicate detection to whole-document centroid similarity
2026-06-28 10:23:16 +03:00
Yiorgis Gozadinos
c7c3f75508
vb
2026-06-27 09:59:21 +03:00
Yiorgis Gozadinos
b94e13083f
Hard-cap context expansion at max_context_chars
...
A single oversized document_items row (e.g. a spreadsheet converted to one
table) expanded far past search.max_context_chars and could overflow the
model context window. _expand_outward only used the budget as a soft
accumulation threshold and expand_with_items never capped the joined result.
Add _clip_to_budget to clip each expanded result to max_context_chars,
returning a window centered on the matched chunk (via _evidence_anchors) so
the retrieved evidence survives the cut.
2026-06-27 09:46:36 +03:00
Yiorgis Gozadinos
cd1a73b389
vb
2026-06-26 17:13:03 +03:00
Yiorgis Gozadinos
73f62ab290
Enforce the duplicate candidate cap during row collection
2026-06-26 16:57:51 +03:00
Yiorgis Gozadinos
f1e1a16f9b
Add doctor --duplicates-out YAML export; summarize terminal report
2026-06-26 16:52:22 +03:00
Yiorgis Gozadinos
044ac62e49
Drop boilerplate handling; make duplicate report readable
2026-06-26 16:52:22 +03:00
Yiorgis Gozadinos
961913dde4
Ignore boilerplate chunks in duplicate-document detection
2026-06-26 16:52:21 +03:00
Yiorgis Gozadinos
43f2130b66
Add near-duplicate document detection to doctor
2026-06-26 16:52:21 +03:00
Yiorgis Gozadinos
4707db780f
Renew job leases from the worker pool; lease-based reaping
2026-06-25 13:26:09 +03:00
Yiorgis Gozadinos
fd73f57649
Add last_heartbeat_at lease column to the ingester queue
2026-06-25 13:25:37 +03:00
Yiorgis Gozadinos
ba6b318ece
Make ingester worker ids globally unique
2026-06-25 13:25:37 +03:00
Yiorgis Gozadinos
b61134b747
Log FTS index build failures at WARNING
2026-06-25 13:00:29 +03:00
Yiorgis Gozadinos
2fd025951c
Treat empty env vars as unset in config expansion
2026-06-25 12:41:04 +03:00
Yiorgis Gozadinos
197d58a602
vb
2026-06-24 18:41:05 +03:00
Yiorgis Gozadinos
b343f3687d
Depend on fastmcp directly for the MCP server
2026-06-24 18:35:00 +03:00
Yiorgis Gozadinos
ab1775dbc6
vb
2026-06-24 16:45:19 +03:00
Yiorgis Gozadinos
9f4dda9254
Reuse one MarkdownDocSerializer per document in item extraction
2026-06-24 12:02:13 +03:00
Yiorgis Gozadinos
2b2b475279
Collapse docling compression to a single function
2026-06-24 10:43:27 +03:00
bryan davis
e95ac2e25d
remove redundant serialization
2026-06-24 10:42:58 +03:00
Yiorgis Gozadinos
62d2326933
Use single-threaded zstd compressor
2026-06-24 10:08:25 +03:00
bryan davis
2ca12c0096
adjust zstd handling to avoid possible core dumps with concurrency
2026-06-23 15:26:33 -05:00
Yiorgis Gozadinos
65f1339aa0
vb
2026-06-23 16:59:00 +03:00
Yiorgis Gozadinos
3cc3c98986
Store docling-serve chunk bodies via raw_text
2026-06-23 16:35:21 +03:00
Yiorgis Gozadinos
68715c9657
Request page images explicitly from docling-serve
2026-06-23 16:15:02 +03:00
Yiorgis Gozadinos
f22341f003
Fix doctor's vector-index check to match documented guidance
2026-06-23 15:39:44 +03:00
Yiorgis Gozadinos
3586ac30a7
Add Cohere multimodal embedder
2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
a1ec310bf4
Add VoyageAI multimodal embedder
2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
0ea251219c
Add explicit multimodal embedding flag, decoupled from provider name
2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
2c906b16cd
Merge pull request #461 from ggozad/feat/doctor
...
Add `haiku-rag doctor` health check
2026-06-23 14:43:22 +03:00
Yiorgis Gozadinos
4fae733a50
Fix doctor provider checks: custom endpoints and processing models
2026-06-23 11:48:21 +03:00
Yiorgis Gozadinos
eb855f827a
Refine doctor's content and coverage checks
2026-06-23 11:31:20 +03:00
Yiorgis Gozadinos
cc1d8d1e4c
Add provider connectivity probes to haiku-rag doctor
2026-06-23 10:34:27 +03:00
Yiorgis Gozadinos
c2cb3cedf3
Add haiku-rag doctor database health check
2026-06-23 10:03:27 +03:00
Yiorgis Gozadinos
8cb52ca1ed
Merge pull request #459 from mcdonc/thread-rebuild-compress
...
Thread compress_docling_split in rebuild.py off the asyncio event loop
2026-06-23 09:23:53 +03:00
Yiorgis Gozadinos
775dc58b53
Drop redundant _apply_descriptions_sync unit test, annotate the helper
2026-06-23 09:14:36 +03:00
Yiorgis Gozadinos
0ee36a269d
Merge pull request #460 from mcdonc/thread-processing-iterate
...
Thread picture chunk merging off the asyncio event loop
2026-06-23 09:05:45 +03:00
Yiorgis Gozadinos
141e288b83
Tighten _merge_picture_chunks type annotations
...
Restore the typing the merge logic carried inline before extraction:
text_chunks: list[Chunk], existing_picture_data: dict[str, bytes] | None,
-> list[Chunk], and first_pos(c: Chunk) -> int.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 08:56:51 +03:00
Yiorgis Gozadinos
fbb63f7e99
Merge pull request #458 from mcdonc/thread-chunker-chunk
...
Thread chunker.chunk() off the asyncio event loop
2026-06-23 08:50:13 +03:00
Yiorgis Gozadinos
4436673f10
Tighten extract_items hoist annotation in _update_document_with_chunks
2026-06-23 08:47:55 +03:00
Chris McDonough
633517bf04
Widen chunk() type to accept None, add test for None guard
2026-06-22 14:24:12 -04:00
Chris McDonough
99a200c9d2
Restore None guard, add thread-safety test for _chunk_sync
2026-06-22 14:05:56 -04:00
Chris McDonough
25b0a155b0
Remove None guard and test — type system guarantees document is not None
2026-06-22 14:04:58 -04:00
Chris McDonough
ccab012747
Thread picture chunk merging off the asyncio event loop
...
Extracts build_picture_chunks + iterate_items position mapping into a
sync helper and wraps it with asyncio.to_thread so image-heavy
documents don't block the event loop during chunking.
Fixes #456 .
2026-06-22 14:01:35 -04:00
Chris McDonough
cee9484f35
Fix import sort order
2026-06-22 13:49:59 -04:00
Chris McDonough
d160080ff8
Move lazy imports to module scope
2026-06-22 13:48:09 -04:00
Chris McDonough
97748c2fee
Fix import sorting lint error
2026-06-22 13:46:21 -04:00
Chris McDonough
22b4397148
Thread compress_docling_split in rebuild.py off the asyncio event loop
...
Extracts pydantic serialization and zstd compression into a sync
helper and wraps it with asyncio.to_thread so picture description
rebuilds don't block the event loop.
Fixes #457 .
2026-06-22 13:42:40 -04:00
Chris McDonough
8fe8b6d291
Thread chunker.chunk() off the asyncio event loop
...
Extracts the CPU-bound HybridChunker/HierarchicalChunker work into
a sync helper and wraps it with asyncio.to_thread so large documents
don't block the event loop during chunking.
Fixes #455 .
2026-06-22 13:37:22 -04:00
Chris McDonough
09e78ef46a
Thread extract_items off the asyncio event loop
...
Moves CPU-bound docling document item extraction into worker threads
via asyncio.to_thread so the event loop stays responsive during
ingestion. The extract_items calls are hoisted out of the write lock
(they are pure computation with no DB I/O) and run with placeholder
document IDs that are patched after the DB create returns.
Fixes #453 .
2026-06-22 13:19:38 -04:00
Yiorgis Gozadinos
0fcf91fcf7
vb
2026-06-22 15:18:35 +03:00
Yiorgis Gozadinos
463c55673e
improve coverage
2026-06-22 12:59:44 +03:00
Yiorgis Gozadinos
baf8decb27
Show progress for run-batch drains
2026-06-22 12:51:38 +03:00
Yiorgis Gozadinos
6e85bbbafe
Make run-batch CLI validation output stable
2026-06-22 12:32:26 +03:00
Yiorgis Gozadinos
b0706204bd
Allow resuming run-batch manifest replay
2026-06-22 12:22:40 +03:00
Yiorgis Gozadinos
68bbf94577
tighten run-batch manifest replay validation
2026-06-22 11:58:32 +03:00
Yiorgis Gozadinos
7251d104c4
Add run-batch manifest replay
2026-06-22 11:39:12 +03:00
Yiorgis Gozadinos
a3b542dba8
Add run-batch dry-run manifest output
2026-06-22 11:33:26 +03:00
Yiorgis Gozadinos
4b573bfebd
Add side-effect-free batch dry-run discovery
2026-06-22 11:33:26 +03:00
Yiorgis Gozadinos
23f5b3c47a
Merge pull request #449 from bd-mkt/bd_concurrency2
...
perf: move CPU-bound ingest work off the event loop
2026-06-22 11:05:14 +03:00
Yiorgis Gozadinos
d3a1011baf
write fetched bodies off the event loop
2026-06-22 10:37:48 +03:00
Yiorgis Gozadinos
183d595494
prepare stored Docling blobs off the event loop
2026-06-22 10:30:02 +03:00
Yiorgis Gozadinos
5c4b37df89
vb
2026-06-19 11:12:45 +03:00
Yiorgis Gozadinos
b67f330031
Move textual-image to base dependencies
2026-06-19 10:37:10 +03:00
bryan davis
fe954a090e
move cpu bound actions off of main loop
2026-06-18 16:35:48 -05:00
