Yiorgis Gozadinos
2b2b475279
Collapse docling compression to a single function
2026-06-24 10:43:27 +03:00
bryan davis
e95ac2e25d
remove redundant serialization
2026-06-24 10:42:58 +03:00
Yiorgis Gozadinos
e73a4272f8
Merge pull request #464 from bd-mkt/zstd_concurrency
...
adjust zstd handling to avoid possible core dumps with concurrency
2026-06-24 10:17:34 +03:00
Yiorgis Gozadinos
62d2326933
Use single-threaded zstd compressor
2026-06-24 10:08:25 +03:00
bryan davis
2ca12c0096
adjust zstd handling to avoid possible core dumps with concurrency
2026-06-23 15:26:33 -05:00
Yiorgis Gozadinos
65f1339aa0
vb
2026-06-23 16:59:00 +03:00
Yiorgis Gozadinos
0b822ee402
Merge pull request #463 from ggozad/chore/docling-update
...
Adapt docling-serve integration to docling-serve 1.25.0
2026-06-23 16:55:10 +03:00
Yiorgis Gozadinos
e1fbaf8aad
Re-record remaining docling-serve cassettes against 1.25.0
2026-06-23 16:44:58 +03:00
Yiorgis Gozadinos
3cc3c98986
Store docling-serve chunk bodies via raw_text
2026-06-23 16:35:21 +03:00
Yiorgis Gozadinos
68715c9657
Request page images explicitly from docling-serve
2026-06-23 16:15:02 +03:00
Yiorgis Gozadinos
f22341f003
Fix doctor's vector-index check to match documented guidance
2026-06-23 15:39:44 +03:00
Yiorgis Gozadinos
195dce7511
Merge pull request #462 from ggozad/fix/multi-modal-embedder-config
...
Decouple multimodal embedding from the provider name; add VoyageAI & Cohere multimodal embedders
2026-06-23 15:33:57 +03:00
Yiorgis Gozadinos
b098be790b
Document multimodal embedder flag and Voyage/Cohere providers
2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
3586ac30a7
Add Cohere multimodal embedder
2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
a1ec310bf4
Add VoyageAI multimodal embedder
2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
0ea251219c
Add explicit multimodal embedding flag, decoupled from provider name
2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
2c906b16cd
Merge pull request #461 from ggozad/feat/doctor
...
Add `haiku-rag doctor` health check
2026-06-23 14:43:22 +03:00
Yiorgis Gozadinos
4fae733a50
Fix doctor provider checks: custom endpoints and processing models
2026-06-23 11:48:21 +03:00
Yiorgis Gozadinos
eb855f827a
Refine doctor's content and coverage checks
2026-06-23 11:31:20 +03:00
Yiorgis Gozadinos
cc1d8d1e4c
Add provider connectivity probes to haiku-rag doctor
2026-06-23 10:34:27 +03:00
Yiorgis Gozadinos
c2cb3cedf3
Add haiku-rag doctor database health check
2026-06-23 10:03:27 +03:00
Yiorgis Gozadinos
8cb52ca1ed
Merge pull request #459 from mcdonc/thread-rebuild-compress
...
Thread compress_docling_split in rebuild.py off the asyncio event loop
2026-06-23 09:23:53 +03:00
Yiorgis Gozadinos
775dc58b53
Drop redundant _apply_descriptions_sync unit test, annotate the helper
2026-06-23 09:14:36 +03:00
Yiorgis Gozadinos
0ee36a269d
Merge pull request #460 from mcdonc/thread-processing-iterate
...
Thread picture chunk merging off the asyncio event loop
2026-06-23 09:05:45 +03:00
Yiorgis Gozadinos
141e288b83
Tighten _merge_picture_chunks type annotations
...
Restore the typing the merge logic carried inline before extraction:
text_chunks: list[Chunk], existing_picture_data: dict[str, bytes] | None,
-> list[Chunk], and first_pos(c: Chunk) -> int.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 08:56:51 +03:00
Yiorgis Gozadinos
fbb63f7e99
Merge pull request #458 from mcdonc/thread-chunker-chunk
...
Thread chunker.chunk() off the asyncio event loop
2026-06-23 08:50:13 +03:00
Yiorgis Gozadinos
4436673f10
Tighten extract_items hoist annotation in _update_document_with_chunks
2026-06-23 08:47:55 +03:00
Yiorgis Gozadinos
4cb85c5eaf
Merge pull request #454 from mcdonc/thread-extract-items
...
Thread extract_items off the asyncio event loop
2026-06-23 08:37:08 +03:00
Chris McDonough
633517bf04
Widen chunk() type to accept None, add test for None guard
2026-06-22 14:24:12 -04:00
Chris McDonough
c59f88bcd6
Add test for _merge_picture_chunks no-pictures branch
2026-06-22 14:13:02 -04:00
Chris McDonough
21a8f52893
Use patch.object to satisfy type checker
2026-06-22 14:09:01 -04:00
Chris McDonough
a137b91654
Add test for _apply_descriptions_sync to cover skip branch
2026-06-22 14:08:23 -04:00
Chris McDonough
99a200c9d2
Restore None guard, add thread-safety test for _chunk_sync
2026-06-22 14:05:56 -04:00
Chris McDonough
25b0a155b0
Remove None guard and test — type system guarantees document is not None
2026-06-22 14:04:58 -04:00
Chris McDonough
ccab012747
Thread picture chunk merging off the asyncio event loop
...
Extracts build_picture_chunks + iterate_items position mapping into a
sync helper and wraps it with asyncio.to_thread so image-heavy
documents don't block the event loop during chunking.
Fixes #456 .
2026-06-22 14:01:35 -04:00
Chris McDonough
256fd66b46
Add test for chunker.chunk(None) to cover guard clause
2026-06-22 13:58:37 -04:00
Chris McDonough
cee9484f35
Fix import sort order
2026-06-22 13:49:59 -04:00
Chris McDonough
d160080ff8
Move lazy imports to module scope
2026-06-22 13:48:09 -04:00
Chris McDonough
97748c2fee
Fix import sorting lint error
2026-06-22 13:46:21 -04:00
Chris McDonough
22b4397148
Thread compress_docling_split in rebuild.py off the asyncio event loop
...
Extracts pydantic serialization and zstd compression into a sync
helper and wraps it with asyncio.to_thread so picture description
rebuilds don't block the event loop.
Fixes #457 .
2026-06-22 13:42:40 -04:00
Chris McDonough
8fe8b6d291
Thread chunker.chunk() off the asyncio event loop
...
Extracts the CPU-bound HybridChunker/HierarchicalChunker work into
a sync helper and wraps it with asyncio.to_thread so large documents
don't block the event loop during chunking.
Fixes #455 .
2026-06-22 13:37:22 -04:00
Chris McDonough
09e78ef46a
Thread extract_items off the asyncio event loop
...
Moves CPU-bound docling document item extraction into worker threads
via asyncio.to_thread so the event loop stays responsive during
ingestion. The extract_items calls are hoisted out of the write lock
(they are pure computation with no DB I/O) and run with placeholder
document IDs that are patched after the DB create returns.
Fixes #453 .
2026-06-22 13:19:38 -04:00
Yiorgis Gozadinos
0fcf91fcf7
vb
2026-06-22 15:18:35 +03:00
Yiorgis Gozadinos
0be4f7e1bd
Merge pull request #452 from ggozad/feat/run-batch-dry-run
...
Add dry-run manifests for ingestor's run-batch
2026-06-22 15:07:13 +03:00
Yiorgis Gozadinos
463c55673e
improve coverage
2026-06-22 12:59:44 +03:00
Yiorgis Gozadinos
baf8decb27
Show progress for run-batch drains
2026-06-22 12:51:38 +03:00
Yiorgis Gozadinos
6e85bbbafe
Make run-batch CLI validation output stable
2026-06-22 12:32:26 +03:00
Yiorgis Gozadinos
b0706204bd
Allow resuming run-batch manifest replay
2026-06-22 12:22:40 +03:00
Yiorgis Gozadinos
68bbf94577
tighten run-batch manifest replay validation
2026-06-22 11:58:32 +03:00
Yiorgis Gozadinos
f0826abb52
Document run-batch dry-run manifests
2026-06-22 11:43:57 +03:00
