Yiorgis Gozadinos
2b2b475279
Collapse docling compression to a single function
2026-06-24 10:43:27 +03:00
Yiorgis Gozadinos
a1ec310bf4
Add VoyageAI multimodal embedder
2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
8cb52ca1ed
Merge pull request #459 from mcdonc/thread-rebuild-compress
...
Thread compress_docling_split in rebuild.py off the asyncio event loop
2026-06-23 09:23:53 +03:00
Yiorgis Gozadinos
775dc58b53
Drop redundant _apply_descriptions_sync unit test, annotate the helper
2026-06-23 09:14:36 +03:00
Yiorgis Gozadinos
0ee36a269d
Merge pull request #460 from mcdonc/thread-processing-iterate
...
Thread picture chunk merging off the asyncio event loop
2026-06-23 09:05:45 +03:00
Yiorgis Gozadinos
141e288b83
Tighten _merge_picture_chunks type annotations
...
Restore the typing the merge logic carried inline before extraction:
text_chunks: list[Chunk], existing_picture_data: dict[str, bytes] | None,
-> list[Chunk], and first_pos(c: Chunk) -> int.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 08:56:51 +03:00
Yiorgis Gozadinos
4436673f10
Tighten extract_items hoist annotation in _update_document_with_chunks
2026-06-23 08:47:55 +03:00
Chris McDonough
ccab012747
Thread picture chunk merging off the asyncio event loop
...
Extracts build_picture_chunks + iterate_items position mapping into a
sync helper and wraps it with asyncio.to_thread so image-heavy
documents don't block the event loop during chunking.
Fixes #456 .
2026-06-22 14:01:35 -04:00
Chris McDonough
cee9484f35
Fix import sort order
2026-06-22 13:49:59 -04:00
Chris McDonough
d160080ff8
Move lazy imports to module scope
2026-06-22 13:48:09 -04:00
Chris McDonough
97748c2fee
Fix import sorting lint error
2026-06-22 13:46:21 -04:00
Chris McDonough
22b4397148
Thread compress_docling_split in rebuild.py off the asyncio event loop
...
Extracts pydantic serialization and zstd compression into a sync
helper and wraps it with asyncio.to_thread so picture description
rebuilds don't block the event loop.
Fixes #457 .
2026-06-22 13:42:40 -04:00
Chris McDonough
09e78ef46a
Thread extract_items off the asyncio event loop
...
Moves CPU-bound docling document item extraction into worker threads
via asyncio.to_thread so the event loop stays responsive during
ingestion. The extract_items calls are hoisted out of the write lock
(they are pure computation with no DB I/O) and run with placeholder
document IDs that are patched after the DB create returns.
Fixes #453 .
2026-06-22 13:19:38 -04:00
Yiorgis Gozadinos
d3a1011baf
write fetched bodies off the event loop
2026-06-22 10:37:48 +03:00
Yiorgis Gozadinos
183d595494
prepare stored Docling blobs off the event loop
2026-06-22 10:30:02 +03:00
bryan davis
faa97f8bc6
improve concurrency management for pdf attachments
2026-06-18 14:30:02 -05:00
Yiorgis Gozadinos
1e65d36766
Pass fetched FetchResult to ingester metadata providers
2026-06-16 16:37:26 +03:00
Yiorgis Gozadinos
70c9c2778f
Serialize all in-process pdfium access under a shared lock
2026-06-14 10:50:42 +03:00
Yiorgis Gozadinos
2b8b9477b7
Serialize and roll back the multi-table document delete
2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
f2a5ac4246
Throttle background auto-vacuum to at most once per 5 minutes
2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
5d5d87d44c
Throttle background auto-vacuum to at most once per 5 minutes
2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
3366a6d383
Split meta document attributes into a document_meta table
2026-06-12 10:17:17 +03:00
Yiorgis Gozadinos
4bc52f0710
Serialize multi-table document writes and bound update version churn
2026-06-12 10:16:39 +03:00
Yiorgis Gozadinos
8ac670714d
Tidy PDF attachment extension fix
2026-06-12 09:44:11 +03:00
bryan davis
bce24e84a0
fix for issue with parsing pdf attachment such as *.joboptions files
2026-06-11 16:52:07 -05:00
Yiorgis Gozadinos
c62166b26e
Add /database and /config endpoints to the ingester API
2026-06-09 12:11:33 +03:00
Yiorgis Gozadinos
a7ed405d75
Add HaikuRAG.import_documents for batch document import
2026-06-09 10:23:37 +03:00
Yiorgis Gozadinos
0e3d53791f
Add rebuild --set-embedder to reconcile embedder identity
2026-06-05 10:42:15 +03:00
Yiorgis Gozadinos
651b22ddcf
Fix rebuild --embed-only corrupting picture embeddings
2026-06-03 11:46:43 +03:00
Yiorgis Gozadinos
e7c7df2915
Always use the Store-owned embedder
2026-05-29 11:36:53 +03:00
Yiorgis Gozadinos
37a78a4e9b
Cache reranker on the client instead of rebuilding per search
2026-05-29 10:33:50 +03:00
Yiorgis Gozadinos
402957d3e5
Fix CI flakes for PDF attachment extraction
2026-05-28 17:51:02 +03:00
Yiorgis Gozadinos
8e8c4433bd
Extract PDF /EmbeddedFiles attachments as child documents
2026-05-28 15:51:32 +03:00
Yiorgis Gozadinos
26bc71d6d8
Cascade delete_document to children via metadata.parent_uri
2026-05-28 15:36:00 +03:00
Yiorgis Gozadinos
b0d0ac588d
Split snapshot APIs and resolve_fetcher by intent
2026-05-27 14:39:54 +03:00
Yiorgis Gozadinos
d7fdd61fed
Resolve worker source by source_id, not just supports(uri)
2026-05-27 13:36:03 +03:00
Yiorgis Gozadinos
f89cc998eb
add sources_provider to WorkerPool. workers now resolve extra info through these sources
2026-05-26 11:45:35 +03:00
Yiorgis Gozadinos
73f8349a00
Use haiku.rag as telemtry scope
2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
ed36cc2230
typed UnsupportedSourceError replaces pipeline string-marker matching
2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
6bcc2f6357
PDF split-convert-merge for memory-bound large PDFs
2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
13cbadeb6f
canonical metadata keys: source_revision + content_type, bump to 0.50.0
2026-05-26 11:43:30 +03:00
Yiorgis Gozadinos
fa672c298a
logfire spans for the ingester, nested poller/job/document traces
2026-05-26 11:42:42 +03:00
Yiorgis Gozadinos
6d15f1f247
fix URL-decode file:// paths before stat/exists checks
2026-05-26 11:42:42 +03:00
Yiorgis Gozadinos
7ea61a7b10
drop the old monitor, rename serve→mcp, add e2e tests
2026-05-26 11:42:41 +03:00
Yiorgis Gozadinos
717fc3320b
route create_document_from_source through source adapters
2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
8c57a3ca99
Drop the multi-agent research workflow
2026-05-20 12:46:48 +03:00
Yiorgis Gozadinos
e3314e36fd
search: only attach picture bytes from pre-expansion chunks
2026-05-19 15:03:22 +03:00
Yiorgis Gozadinos
6f95e2bc27
Delete the standalone QA and analysis agents
2026-05-19 11:39:20 +03:00
Yiorgis Gozadinos
947a26b391
client.analyze routes through the rag-analysis skill; drop documents= and AnalysisResult.program. Re-record all cassettes that are relevant
2026-05-19 10:46:32 +03:00
Yiorgis Gozadinos
d96d2eeb0f
client.ask routes through the rag skill; always show citations in CLI
2026-05-19 10:19:39 +03:00
