Chris McDonough
22b4397148
Thread compress_docling_split in rebuild.py off the asyncio event loop
...
Extracts pydantic serialization and zstd compression into a sync
helper and wraps it with asyncio.to_thread so picture description
rebuilds don't block the event loop.
Fixes #457 .
2026-06-22 13:42:40 -04:00
Yiorgis Gozadinos
0fcf91fcf7
vb
2026-06-22 15:18:35 +03:00
Yiorgis Gozadinos
0be4f7e1bd
Merge pull request #452 from ggozad/feat/run-batch-dry-run
...
Add dry-run manifests for ingestor's run-batch
2026-06-22 15:07:13 +03:00
Yiorgis Gozadinos
463c55673e
improve coverage
2026-06-22 12:59:44 +03:00
Yiorgis Gozadinos
baf8decb27
Show progress for run-batch drains
2026-06-22 12:51:38 +03:00
Yiorgis Gozadinos
6e85bbbafe
Make run-batch CLI validation output stable
2026-06-22 12:32:26 +03:00
Yiorgis Gozadinos
b0706204bd
Allow resuming run-batch manifest replay
2026-06-22 12:22:40 +03:00
Yiorgis Gozadinos
68bbf94577
tighten run-batch manifest replay validation
2026-06-22 11:58:32 +03:00
Yiorgis Gozadinos
f0826abb52
Document run-batch dry-run manifests
2026-06-22 11:43:57 +03:00
Yiorgis Gozadinos
7251d104c4
Add run-batch manifest replay
2026-06-22 11:39:12 +03:00
Yiorgis Gozadinos
a3b542dba8
Add run-batch dry-run manifest output
2026-06-22 11:33:26 +03:00
Yiorgis Gozadinos
4b573bfebd
Add side-effect-free batch dry-run discovery
2026-06-22 11:33:26 +03:00
Yiorgis Gozadinos
6f2ab0963c
cl for #449
2026-06-22 11:33:11 +03:00
Yiorgis Gozadinos
23f5b3c47a
Merge pull request #449 from bd-mkt/bd_concurrency2
...
perf: move CPU-bound ingest work off the event loop
2026-06-22 11:05:14 +03:00
Yiorgis Gozadinos
ad3b111cf1
test: clarify off-loop thread assertions
2026-06-22 10:51:35 +03:00
Yiorgis Gozadinos
d3a1011baf
write fetched bodies off the event loop
2026-06-22 10:37:48 +03:00
Yiorgis Gozadinos
183d595494
prepare stored Docling blobs off the event loop
2026-06-22 10:30:02 +03:00
Yiorgis Gozadinos
c6514c9df4
compare off-loop work against actual event-loop thread, fix ty
2026-06-22 10:17:02 +03:00
Yiorgis Gozadinos
5c4b37df89
vb
2026-06-19 11:12:45 +03:00
Yiorgis Gozadinos
9e08ec40d5
Merge pull request #450 from ggozad/fix/textual-image-base-deps
...
Move textual-image to base dependencies
2026-06-19 10:47:38 +03:00
Yiorgis Gozadinos
b67f330031
Move textual-image to base dependencies
2026-06-19 10:37:10 +03:00
Yiorgis Gozadinos
2c55bca8bc
cl for #448
2026-06-19 10:17:18 +03:00
Yiorgis Gozadinos
d0e40d9025
Merge pull request #448 from bd-mkt/pdf_att_lock
...
improve concurrency management for pdf attachments
2026-06-19 10:15:58 +03:00
bryan davis
fe954a090e
move cpu bound actions off of main loop
2026-06-18 16:35:48 -05:00
bryan davis
faa97f8bc6
improve concurrency management for pdf attachments
2026-06-18 14:30:02 -05:00
Yiorgis Gozadinos
0c33a00b4e
claude code personal notes
2026-06-16 17:06:32 +03:00
Yiorgis Gozadinos
99011755fb
vb
2026-06-16 16:48:36 +03:00
Yiorgis Gozadinos
78f86c81a7
Merge pull request #444 from ggozad/feat/ingester-metadata-fetchresult
...
Pass fetched FetchResult to ingester metadata providers
2026-06-16 16:46:49 +03:00
Yiorgis Gozadinos
1e65d36766
Pass fetched FetchResult to ingester metadata providers
2026-06-16 16:37:26 +03:00
Yiorgis Gozadinos
3f5141abd0
Merge pull request #446 from ggozad/chore/docling-update
...
Bump docling deps and relax opencv floor
2026-06-16 16:32:31 +03:00
Yiorgis Gozadinos
89151db52a
Merge pull request #445 from ggozad/fix/queue-retry-idempotent
...
Make ingester job retry idempotent against a live sibling
2026-06-16 16:23:42 +03:00
Yiorgis Gozadinos
d38f4f9702
Merge pull request #443 from ggozad/fix/queue-reenqueue
...
Stop re-enqueuing permanently-failed ingester documents
2026-06-16 16:23:21 +03:00
Yiorgis Gozadinos
06a3f365c4
Bump docling deps and relax opencv floor
2026-06-16 16:16:32 +03:00
Yiorgis Gozadinos
6add2780e8
Make ingester job retry idempotent against a live sibling
2026-06-16 16:05:05 +03:00
Yiorgis Gozadinos
79a4f49387
Stop re-enqueuing permanently-failed ingester documents
2026-06-16 16:03:51 +03:00
Yiorgis Gozadinos
276f277232
Merge pull request #442 from ggozad/feat/custom-source
...
Custom ingester sources via entry points
2026-06-16 12:11:10 +03:00
Yiorgis Gozadinos
515f42dd5a
Add custom ingester sources via entry points
2026-06-16 10:54:24 +03:00
Yiorgis Gozadinos
2ce7d10c51
Merge pull request #441 from ggozad/fix/sqlite-pool
...
Widen SQLite ingester queue pool to serve concurrent connections
2026-06-16 09:41:11 +03:00
Yiorgis Gozadinos
7e20b47e98
Widen SQLite ingester queue pool to serve concurrent connections
2026-06-16 09:25:01 +03:00
Yiorgis Gozadinos
f94c419c31
cl
2026-06-15 10:21:09 +03:00
Yiorgis Gozadinos
6440698b80
Merge pull request #440 from ggozad/fix/webdav-follow-redirects
...
Follow HTTP redirects in the WebDAV source
2026-06-15 10:10:44 +03:00
Yiorgis Gozadinos
fac62cb347
Follow HTTP redirects in the WebDAV source
2026-06-15 09:59:13 +03:00
Yiorgis Gozadinos
84b778b4ea
Merge pull request #439 from ggozad/feat/ingestor-meta
...
Ingester metadata providers
2026-06-15 08:51:52 +03:00
Yiorgis Gozadinos
cc73a8629a
Wire metadata providers into ingester ingestion
2026-06-15 08:41:26 +03:00
Yiorgis Gozadinos
5722260857
Add metadata-provider discovery for the ingester
2026-06-15 08:05:45 +03:00
Yiorgis Gozadinos
61f27085f9
Merge pull request #438 from ggozad/fix/pdfium-concurrency
...
Serialize all in-process pdfium access under a shared lock
2026-06-14 11:08:26 +03:00
Yiorgis Gozadinos
70c9c2778f
Serialize all in-process pdfium access under a shared lock
2026-06-14 10:50:42 +03:00
Yiorgis Gozadinos
8d9dbd0abc
Merge pull request #435 from ggozad/feat/ingestor-vacuuming
...
Fix ingestion disk bloat: split mutable document attributes into a document_meta table
2026-06-12 10:29:38 +03:00
Yiorgis Gozadinos
885e7b7ce7
coverage
2026-06-12 10:17:19 +03:00
Yiorgis Gozadinos
6c04dd5881
docs
2026-06-12 10:17:18 +03:00
