Yiorgis Gozadinos
c02bcd5dc9
Pool HTTP clients for vLLM embedding and vLLM/Jina reranking
2026-07-09 10:32:44 +03:00
Yiorgis Gozadinos
43a580afb3
Dedupe picture chunks and skip small pictures at chunking
2026-07-09 10:24:32 +03:00
Yiorgis Gozadinos
a1cf405cae
Rename document_meta identity column document_id to id
2026-07-08 16:32:55 +03:00
Yiorgis Gozadinos
23bbca5998
Re-check URI under the write lock to prevent duplicate ingestion
2026-07-08 11:26:45 +03:00
Yiorgis Gozadinos
3c7699f156
Fail docling-serve requests over to another instance with per-instance breaking
...
When a docling-serve instance crashes or returns 5xx, DoclingServeClient now
retries the request on a different instance (up to max_attempts) and trips a
per-instance circuit breaker so subsequent jobs skip a dead instance until its
cooldown elapses. Reuses the shared CircuitBreaker; adds max_attempts and a
nested circuit_breaker to providers.docling_serve.
Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 11:03:44 +03:00
Yiorgis Gozadinos
d781335868
Move CircuitBreaker to a shared module
...
Relocate CircuitBreaker from ingester/pollers to haiku/rag/circuit_breaker
so non-ingester callers (docling-serve provider) can reuse it without
depending on the ingester package.
Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 11:03:30 +03:00
Yiorgis Gozadinos
c2c8c24951
Allow updating a document's uri via update_document
2026-07-08 10:32:14 +03:00
Yiorgis Gozadinos
d2593304c5
Attach figure bytes when a result matches its caption
2026-07-03 13:22:02 +03:00
Yiorgis Gozadinos
ac64aa9c17
Build the skills rag_db fixture once per session
2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
e4b1f16f97
Shrink the docling-serve context-expansion cassette
2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
afd6e3ae00
Share the VLM URL builder and centralize the embedder empty guard
2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
38079ff89a
Hoist reranker empty-input guard into base; stop factory tests loading models
2026-06-29 14:20:23 +03:00
Yiorgis Gozadinos
4f8cc4eb30
Show a progress spinner while doctor runs
2026-06-28 10:44:30 +03:00
Yiorgis Gozadinos
e7b4988d82
Compute document centroids during the vector scan
2026-06-28 10:23:16 +03:00
Yiorgis Gozadinos
15aae0f242
Simplify duplicate detection to whole-document centroid similarity
2026-06-28 10:23:16 +03:00
Yiorgis Gozadinos
b94e13083f
Hard-cap context expansion at max_context_chars
...
A single oversized document_items row (e.g. a spreadsheet converted to one
table) expanded far past search.max_context_chars and could overflow the
model context window. _expand_outward only used the budget as a soft
accumulation threshold and expand_with_items never capped the joined result.
Add _clip_to_budget to clip each expanded result to max_context_chars,
returning a window centered on the matched chunk (via _evidence_anchors) so
the retrieved evidence survives the cut.
2026-06-27 09:46:36 +03:00
Yiorgis Gozadinos
73f62ab290
Enforce the duplicate candidate cap during row collection
2026-06-26 16:57:51 +03:00
Yiorgis Gozadinos
f1e1a16f9b
Add doctor --duplicates-out YAML export; summarize terminal report
2026-06-26 16:52:22 +03:00
Yiorgis Gozadinos
044ac62e49
Drop boilerplate handling; make duplicate report readable
2026-06-26 16:52:22 +03:00
Yiorgis Gozadinos
961913dde4
Ignore boilerplate chunks in duplicate-document detection
2026-06-26 16:52:21 +03:00
Yiorgis Gozadinos
43f2130b66
Add near-duplicate document detection to doctor
2026-06-26 16:52:21 +03:00
Yiorgis Gozadinos
ce27a7aae5
Clean up S3 test data per-test to keep SeaweedFS writable
2026-06-26 16:41:20 +03:00
Yiorgis Gozadinos
87f0233bcd
Cover heartbeat health, lease-renewal failure, and breaker recovery
2026-06-25 16:32:56 +03:00
Yiorgis Gozadinos
4707db780f
Renew job leases from the worker pool; lease-based reaping
2026-06-25 13:26:09 +03:00
Yiorgis Gozadinos
fd73f57649
Add last_heartbeat_at lease column to the ingester queue
2026-06-25 13:25:37 +03:00
Yiorgis Gozadinos
ba6b318ece
Make ingester worker ids globally unique
2026-06-25 13:25:37 +03:00
Yiorgis Gozadinos
b61134b747
Log FTS index build failures at WARNING
2026-06-25 13:00:29 +03:00
Yiorgis Gozadinos
2fd025951c
Treat empty env vars as unset in config expansion
2026-06-25 12:41:04 +03:00
Yiorgis Gozadinos
9f4dda9254
Reuse one MarkdownDocSerializer per document in item extraction
2026-06-24 12:02:13 +03:00
Yiorgis Gozadinos
2b2b475279
Collapse docling compression to a single function
2026-06-24 10:43:27 +03:00
bryan davis
e95ac2e25d
remove redundant serialization
2026-06-24 10:42:58 +03:00
bryan davis
2ca12c0096
adjust zstd handling to avoid possible core dumps with concurrency
2026-06-23 15:26:33 -05:00
Yiorgis Gozadinos
e1fbaf8aad
Re-record remaining docling-serve cassettes against 1.25.0
2026-06-23 16:44:58 +03:00
Yiorgis Gozadinos
3cc3c98986
Store docling-serve chunk bodies via raw_text
2026-06-23 16:35:21 +03:00
Yiorgis Gozadinos
68715c9657
Request page images explicitly from docling-serve
2026-06-23 16:15:02 +03:00
Yiorgis Gozadinos
f22341f003
Fix doctor's vector-index check to match documented guidance
2026-06-23 15:39:44 +03:00
Yiorgis Gozadinos
3586ac30a7
Add Cohere multimodal embedder
2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
a1ec310bf4
Add VoyageAI multimodal embedder
2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
0ea251219c
Add explicit multimodal embedding flag, decoupled from provider name
2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
4fae733a50
Fix doctor provider checks: custom endpoints and processing models
2026-06-23 11:48:21 +03:00
Yiorgis Gozadinos
eb855f827a
Refine doctor's content and coverage checks
2026-06-23 11:31:20 +03:00
Yiorgis Gozadinos
cc1d8d1e4c
Add provider connectivity probes to haiku-rag doctor
2026-06-23 10:34:27 +03:00
Yiorgis Gozadinos
c2cb3cedf3
Add haiku-rag doctor database health check
2026-06-23 10:03:27 +03:00
Yiorgis Gozadinos
0ee36a269d
Merge pull request #460 from mcdonc/thread-processing-iterate
...
Thread picture chunk merging off the asyncio event loop
2026-06-23 09:05:45 +03:00
Chris McDonough
633517bf04
Widen chunk() type to accept None, add test for None guard
2026-06-22 14:24:12 -04:00
Chris McDonough
c59f88bcd6
Add test for _merge_picture_chunks no-pictures branch
2026-06-22 14:13:02 -04:00
Chris McDonough
21a8f52893
Use patch.object to satisfy type checker
2026-06-22 14:09:01 -04:00
Chris McDonough
99a200c9d2
Restore None guard, add thread-safety test for _chunk_sync
2026-06-22 14:05:56 -04:00
Chris McDonough
25b0a155b0
Remove None guard and test — type system guarantees document is not None
2026-06-22 14:04:58 -04:00
Chris McDonough
256fd66b46
Add test for chunker.chunk(None) to cover guard clause
2026-06-22 13:58:37 -04:00
