Yiorgis Gozadinos
9deb1f2bd4
replace haiku.skills with native Pydantic AI capabilities
2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
543aba7547
Multimodal reranking: send picture chunks to vllm rerankers as images
...
reranking.multimodal (vllm provider only) attaches picture bytes to
synthetic picture chunks before rerank; VLLMReranker sends them as
content-parts documents (base64 data URI + description text) in the
same /v1/rerank request as plain text documents.
2026-07-24 12:29:20 +03:00
Yiorgis Gozadinos
1ed424b50d
Serialize metadata-only document updates with the write lock
2026-07-16 13:11:35 +03:00
Yiorgis Gozadinos
0e271eaf4b
Remove --before/--at time travel
2026-07-16 13:11:33 +03:00
Yiorgis Gozadinos
0813a1c980
Add tag CLI commands and history tag annotations
2026-07-16 13:11:27 +03:00
Yiorgis Gozadinos
0cbde6b7a2
Serialize tag operations, vacuum, and metadata refresh with writes
2026-07-16 12:32:57 +03:00
Yiorgis Gozadinos
144900d385
Remove the mxbai reranking provider
2026-07-14 11:09:55 +03:00
Yiorgis Gozadinos
e7d5a0440d
Tune two-tone visualization colors
2026-07-09 15:37:25 +03:00
Yiorgis Gozadinos
18c0f6c5e8
Add --no-expand to the visualize command for chunk-only grounding
2026-07-09 12:49:42 +03:00
Yiorgis Gozadinos
995d081aa1
Visualize the exact context the model saw via Citation.doc_item_refs
...
visualize_chunk re-expanded chunks from scratch to recover their refs,
which could not faithfully reproduce the original merge, scores, and
clip — so a visualization could highlight different pages than the
citation covered. Carry the cited items on Citation.doc_item_refs and
resolve bounding boxes from them directly; re-expansion remains only as
the fallback for callers with no stored context (CLI, inspector). Chat,
inspector, the app endpoint, and the frontend pass the refs through.
2026-07-09 11:51:37 +03:00
Yiorgis Gozadinos
6d86237dd6
Draw matched content stronger than expanded context in visualizations
2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
494f774473
Visualize all constituent chunks of a merged citation
2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
c02bcd5dc9
Pool HTTP clients for vLLM embedding and vLLM/Jina reranking
2026-07-09 10:32:44 +03:00
Yiorgis Gozadinos
43a580afb3
Dedupe picture chunks and skip small pictures at chunking
2026-07-09 10:24:32 +03:00
Yiorgis Gozadinos
a1cf405cae
Rename document_meta identity column document_id to id
2026-07-08 16:32:55 +03:00
Yiorgis Gozadinos
23bbca5998
Re-check URI under the write lock to prevent duplicate ingestion
2026-07-08 11:26:45 +03:00
Yiorgis Gozadinos
c2c8c24951
Allow updating a document's uri via update_document
2026-07-08 10:32:14 +03:00
Yiorgis Gozadinos
d2593304c5
Attach figure bytes when a result matches its caption
2026-07-03 13:22:02 +03:00
Yiorgis Gozadinos
2b2b475279
Collapse docling compression to a single function
2026-06-24 10:43:27 +03:00
Yiorgis Gozadinos
a1ec310bf4
Add VoyageAI multimodal embedder
2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
8cb52ca1ed
Merge pull request #459 from mcdonc/thread-rebuild-compress
...
Thread compress_docling_split in rebuild.py off the asyncio event loop
2026-06-23 09:23:53 +03:00
Yiorgis Gozadinos
775dc58b53
Drop redundant _apply_descriptions_sync unit test, annotate the helper
2026-06-23 09:14:36 +03:00
Yiorgis Gozadinos
0ee36a269d
Merge pull request #460 from mcdonc/thread-processing-iterate
...
Thread picture chunk merging off the asyncio event loop
2026-06-23 09:05:45 +03:00
Yiorgis Gozadinos
141e288b83
Tighten _merge_picture_chunks type annotations
...
Restore the typing the merge logic carried inline before extraction:
text_chunks: list[Chunk], existing_picture_data: dict[str, bytes] | None,
-> list[Chunk], and first_pos(c: Chunk) -> int.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 08:56:51 +03:00
Yiorgis Gozadinos
4436673f10
Tighten extract_items hoist annotation in _update_document_with_chunks
2026-06-23 08:47:55 +03:00
Chris McDonough
ccab012747
Thread picture chunk merging off the asyncio event loop
...
Extracts build_picture_chunks + iterate_items position mapping into a
sync helper and wraps it with asyncio.to_thread so image-heavy
documents don't block the event loop during chunking.
Fixes #456 .
2026-06-22 14:01:35 -04:00
Chris McDonough
cee9484f35
Fix import sort order
2026-06-22 13:49:59 -04:00
Chris McDonough
d160080ff8
Move lazy imports to module scope
2026-06-22 13:48:09 -04:00
Chris McDonough
97748c2fee
Fix import sorting lint error
2026-06-22 13:46:21 -04:00
Chris McDonough
22b4397148
Thread compress_docling_split in rebuild.py off the asyncio event loop
...
Extracts pydantic serialization and zstd compression into a sync
helper and wraps it with asyncio.to_thread so picture description
rebuilds don't block the event loop.
Fixes #457 .
2026-06-22 13:42:40 -04:00
Chris McDonough
09e78ef46a
Thread extract_items off the asyncio event loop
...
Moves CPU-bound docling document item extraction into worker threads
via asyncio.to_thread so the event loop stays responsive during
ingestion. The extract_items calls are hoisted out of the write lock
(they are pure computation with no DB I/O) and run with placeholder
document IDs that are patched after the DB create returns.
Fixes #453 .
2026-06-22 13:19:38 -04:00
Yiorgis Gozadinos
d3a1011baf
write fetched bodies off the event loop
2026-06-22 10:37:48 +03:00
Yiorgis Gozadinos
183d595494
prepare stored Docling blobs off the event loop
2026-06-22 10:30:02 +03:00
bryan davis
faa97f8bc6
improve concurrency management for pdf attachments
2026-06-18 14:30:02 -05:00
Yiorgis Gozadinos
1e65d36766
Pass fetched FetchResult to ingester metadata providers
2026-06-16 16:37:26 +03:00
Yiorgis Gozadinos
70c9c2778f
Serialize all in-process pdfium access under a shared lock
2026-06-14 10:50:42 +03:00
Yiorgis Gozadinos
2b8b9477b7
Serialize and roll back the multi-table document delete
2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
f2a5ac4246
Throttle background auto-vacuum to at most once per 5 minutes
2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
5d5d87d44c
Throttle background auto-vacuum to at most once per 5 minutes
2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
3366a6d383
Split meta document attributes into a document_meta table
2026-06-12 10:17:17 +03:00
Yiorgis Gozadinos
4bc52f0710
Serialize multi-table document writes and bound update version churn
2026-06-12 10:16:39 +03:00
Yiorgis Gozadinos
8ac670714d
Tidy PDF attachment extension fix
2026-06-12 09:44:11 +03:00
bryan davis
bce24e84a0
fix for issue with parsing pdf attachment such as *.joboptions files
2026-06-11 16:52:07 -05:00
Yiorgis Gozadinos
c62166b26e
Add /database and /config endpoints to the ingester API
2026-06-09 12:11:33 +03:00
Yiorgis Gozadinos
a7ed405d75
Add HaikuRAG.import_documents for batch document import
2026-06-09 10:23:37 +03:00
Yiorgis Gozadinos
0e3d53791f
Add rebuild --set-embedder to reconcile embedder identity
2026-06-05 10:42:15 +03:00
Yiorgis Gozadinos
651b22ddcf
Fix rebuild --embed-only corrupting picture embeddings
2026-06-03 11:46:43 +03:00
Yiorgis Gozadinos
e7c7df2915
Always use the Store-owned embedder
2026-05-29 11:36:53 +03:00
Yiorgis Gozadinos
37a78a4e9b
Cache reranker on the client instead of rebuilding per search
2026-05-29 10:33:50 +03:00
Yiorgis Gozadinos
402957d3e5
Fix CI flakes for PDF attachment extraction
2026-05-28 17:51:02 +03:00
