danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
2242 commits 9 branches 185 tags 79 MiB
Commit graph

1055 commits

Author SHA1 Message Date
Yiorgis Gozadinos
494f774473
Visualize all constituent chunks of a merged citation 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
0bcf34363a
Carry merged chunk ids on SearchResult and Citation 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
b14152a45b
Keep picture and table hits section-bounded during context expansion 2026-07-09 11:05:40 +03:00
Yiorgis Gozadinos
c02bcd5dc9
Pool HTTP clients for vLLM embedding and vLLM/Jina reranking 2026-07-09 10:32:44 +03:00
Yiorgis Gozadinos
43a580afb3
Dedupe picture chunks and skip small pictures at chunking 2026-07-09 10:24:32 +03:00
Yiorgis Gozadinos
a1cf405cae
Rename document_meta identity column document_id to id 2026-07-08 16:32:55 +03:00
Yiorgis Gozadinos
23bbca5998
Re-check URI under the write lock to prevent duplicate ingestion 2026-07-08 11:26:45 +03:00
Yiorgis Gozadinos
3c7699f156
Fail docling-serve requests over to another instance with per-instance breaking 
When a docling-serve instance crashes or returns 5xx, DoclingServeClient now
retries the request on a different instance (up to max_attempts) and trips a
per-instance circuit breaker so subsequent jobs skip a dead instance until its
cooldown elapses. Reuses the shared CircuitBreaker; adds max_attempts and a
nested circuit_breaker to providers.docling_serve.

Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-08 11:03:44 +03:00
Yiorgis Gozadinos
d781335868
Move CircuitBreaker to a shared module 
Relocate CircuitBreaker from ingester/pollers to haiku/rag/circuit_breaker
so non-ingester callers (docling-serve provider) can reuse it without
depending on the ingester package.

Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-08 11:03:30 +03:00
Yiorgis Gozadinos
c2c8c24951
Allow updating a document's uri via update_document 2026-07-08 10:32:14 +03:00
Yiorgis Gozadinos
9a9bea86d3
vb 2026-07-03 13:35:10 +03:00
Yiorgis Gozadinos
d2593304c5
Attach figure bytes when a result matches its caption 2026-07-03 13:22:02 +03:00
Yiorgis Gozadinos
7f50d3698f
Validate reranker config before importing the provider package 2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
afd6e3ae00
Share the VLM URL builder and centralize the embedder empty guard 2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
654cb2b94c
Collapse get_reranker into a single guard and import-guarded dispatch 
Replace the seven repeated `config.reranking.model and ... == provider`
checks and six per-branch ImportError handlers with one None guard and one
try/except around the provider dispatch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
38079ff89a
Hoist reranker empty-input guard into base; stop factory tests loading models 2026-06-29 14:20:23 +03:00
Yiorgis Gozadinos
9a85632cf3
vb 2026-06-29 12:36:36 +03:00
Yiorgis Gozadinos
5fc4056dcd
Require haiku.skills>=0.18.0 2026-06-29 11:44:51 +03:00
Yiorgis Gozadinos
b2abdfaffb
vb 2026-06-28 10:57:26 +03:00
Yiorgis Gozadinos
4f8cc4eb30
Show a progress spinner while doctor runs 2026-06-28 10:44:30 +03:00
Yiorgis Gozadinos
e7b4988d82
Compute document centroids during the vector scan 2026-06-28 10:23:16 +03:00
Yiorgis Gozadinos
15aae0f242
Simplify duplicate detection to whole-document centroid similarity 2026-06-28 10:23:16 +03:00
Yiorgis Gozadinos
c7c3f75508
vb 2026-06-27 09:59:21 +03:00
Yiorgis Gozadinos
b94e13083f
Hard-cap context expansion at max_context_chars 
A single oversized document_items row (e.g. a spreadsheet converted to one
table) expanded far past search.max_context_chars and could overflow the
model context window. _expand_outward only used the budget as a soft
accumulation threshold and expand_with_items never capped the joined result.

Add _clip_to_budget to clip each expanded result to max_context_chars,
returning a window centered on the matched chunk (via _evidence_anchors) so
the retrieved evidence survives the cut.
 2026-06-27 09:46:36 +03:00
Yiorgis Gozadinos
cd1a73b389
vb 2026-06-26 17:13:03 +03:00
Yiorgis Gozadinos
73f62ab290
Enforce the duplicate candidate cap during row collection 2026-06-26 16:57:51 +03:00
Yiorgis Gozadinos
f1e1a16f9b
Add doctor --duplicates-out YAML export; summarize terminal report 2026-06-26 16:52:22 +03:00
Yiorgis Gozadinos
044ac62e49
Drop boilerplate handling; make duplicate report readable 2026-06-26 16:52:22 +03:00
Yiorgis Gozadinos
961913dde4
Ignore boilerplate chunks in duplicate-document detection 2026-06-26 16:52:21 +03:00
Yiorgis Gozadinos
43f2130b66
Add near-duplicate document detection to doctor 2026-06-26 16:52:21 +03:00
Yiorgis Gozadinos
4707db780f
Renew job leases from the worker pool; lease-based reaping 2026-06-25 13:26:09 +03:00
Yiorgis Gozadinos
fd73f57649
Add last_heartbeat_at lease column to the ingester queue 2026-06-25 13:25:37 +03:00
Yiorgis Gozadinos
ba6b318ece
Make ingester worker ids globally unique 2026-06-25 13:25:37 +03:00
Yiorgis Gozadinos
b61134b747
Log FTS index build failures at WARNING 2026-06-25 13:00:29 +03:00
Yiorgis Gozadinos
2fd025951c
Treat empty env vars as unset in config expansion 2026-06-25 12:41:04 +03:00
Yiorgis Gozadinos
197d58a602
vb 2026-06-24 18:41:05 +03:00
Yiorgis Gozadinos
b343f3687d
Depend on fastmcp directly for the MCP server 2026-06-24 18:35:00 +03:00
Yiorgis Gozadinos
ab1775dbc6
vb 2026-06-24 16:45:19 +03:00
Yiorgis Gozadinos
9f4dda9254
Reuse one MarkdownDocSerializer per document in item extraction 2026-06-24 12:02:13 +03:00
Yiorgis Gozadinos
2b2b475279
Collapse docling compression to a single function 2026-06-24 10:43:27 +03:00
bryan davis
e95ac2e25d
remove redundant serialization 2026-06-24 10:42:58 +03:00
Yiorgis Gozadinos
62d2326933
Use single-threaded zstd compressor 2026-06-24 10:08:25 +03:00
bryan davis
2ca12c0096
adjust zstd handling to avoid possible core dumps with concurrency 2026-06-23 15:26:33 -05:00
Yiorgis Gozadinos
65f1339aa0
vb 2026-06-23 16:59:00 +03:00
Yiorgis Gozadinos
3cc3c98986
Store docling-serve chunk bodies via raw_text 2026-06-23 16:35:21 +03:00
Yiorgis Gozadinos
68715c9657
Request page images explicitly from docling-serve 2026-06-23 16:15:02 +03:00
Yiorgis Gozadinos
f22341f003
Fix doctor's vector-index check to match documented guidance 2026-06-23 15:39:44 +03:00
Yiorgis Gozadinos
3586ac30a7
Add Cohere multimodal embedder 2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
a1ec310bf4
Add VoyageAI multimodal embedder 2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
0ea251219c
Add explicit multimodal embedding flag, decoupled from provider name 2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
2c906b16cd
Merge pull request #461 from ggozad/feat/doctor 
Add `haiku-rag doctor` health check
 2026-06-23 14:43:22 +03:00
Yiorgis Gozadinos
4fae733a50
Fix doctor provider checks: custom endpoints and processing models 2026-06-23 11:48:21 +03:00
Yiorgis Gozadinos
eb855f827a
Refine doctor's content and coverage checks 2026-06-23 11:31:20 +03:00
Yiorgis Gozadinos
cc1d8d1e4c
Add provider connectivity probes to haiku-rag doctor 2026-06-23 10:34:27 +03:00
Yiorgis Gozadinos
c2cb3cedf3
Add haiku-rag doctor database health check 2026-06-23 10:03:27 +03:00
Yiorgis Gozadinos
8cb52ca1ed
Merge pull request #459 from mcdonc/thread-rebuild-compress 
Thread compress_docling_split in rebuild.py off the asyncio event loop
 2026-06-23 09:23:53 +03:00
Yiorgis Gozadinos
775dc58b53
Drop redundant _apply_descriptions_sync unit test, annotate the helper 2026-06-23 09:14:36 +03:00
Yiorgis Gozadinos
0ee36a269d
Merge pull request #460 from mcdonc/thread-processing-iterate 
Thread picture chunk merging off the asyncio event loop
 2026-06-23 09:05:45 +03:00
Yiorgis Gozadinos
141e288b83
Tighten _merge_picture_chunks type annotations 
Restore the typing the merge logic carried inline before extraction:
text_chunks: list[Chunk], existing_picture_data: dict[str, bytes] | None,
-> list[Chunk], and first_pos(c: Chunk) -> int.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-23 08:56:51 +03:00
Yiorgis Gozadinos
fbb63f7e99
Merge pull request #458 from mcdonc/thread-chunker-chunk 
Thread chunker.chunk() off the asyncio event loop
 2026-06-23 08:50:13 +03:00
Yiorgis Gozadinos
4436673f10
Tighten extract_items hoist annotation in _update_document_with_chunks 2026-06-23 08:47:55 +03:00
Chris McDonough
633517bf04 Widen chunk() type to accept None, add test for None guard 2026-06-22 14:24:12 -04:00
Chris McDonough
99a200c9d2 Restore None guard, add thread-safety test for _chunk_sync 2026-06-22 14:05:56 -04:00
Chris McDonough
25b0a155b0 Remove None guard and test — type system guarantees document is not None 2026-06-22 14:04:58 -04:00
Chris McDonough
ccab012747 Thread picture chunk merging off the asyncio event loop 
Extracts build_picture_chunks + iterate_items position mapping into a
sync helper and wraps it with asyncio.to_thread so image-heavy
documents don't block the event loop during chunking.

Fixes #456.
 2026-06-22 14:01:35 -04:00
Chris McDonough
cee9484f35 Fix import sort order 2026-06-22 13:49:59 -04:00
Chris McDonough
d160080ff8 Move lazy imports to module scope 2026-06-22 13:48:09 -04:00
Chris McDonough
97748c2fee Fix import sorting lint error 2026-06-22 13:46:21 -04:00
Chris McDonough
22b4397148 Thread compress_docling_split in rebuild.py off the asyncio event loop 
Extracts pydantic serialization and zstd compression into a sync
helper and wraps it with asyncio.to_thread so picture description
rebuilds don't block the event loop.

Fixes #457.
 2026-06-22 13:42:40 -04:00
Chris McDonough
8fe8b6d291 Thread chunker.chunk() off the asyncio event loop 
Extracts the CPU-bound HybridChunker/HierarchicalChunker work into
a sync helper and wraps it with asyncio.to_thread so large documents
don't block the event loop during chunking.

Fixes #455.
 2026-06-22 13:37:22 -04:00
Chris McDonough
09e78ef46a Thread extract_items off the asyncio event loop 
Moves CPU-bound docling document item extraction into worker threads
via asyncio.to_thread so the event loop stays responsive during
ingestion. The extract_items calls are hoisted out of the write lock
(they are pure computation with no DB I/O) and run with placeholder
document IDs that are patched after the DB create returns.

Fixes #453.
 2026-06-22 13:19:38 -04:00
Yiorgis Gozadinos
0fcf91fcf7
vb 2026-06-22 15:18:35 +03:00
Yiorgis Gozadinos
463c55673e
improve coverage 2026-06-22 12:59:44 +03:00
Yiorgis Gozadinos
baf8decb27
Show progress for run-batch drains 2026-06-22 12:51:38 +03:00
Yiorgis Gozadinos
6e85bbbafe
Make run-batch CLI validation output stable 2026-06-22 12:32:26 +03:00
Yiorgis Gozadinos
b0706204bd
Allow resuming run-batch manifest replay 2026-06-22 12:22:40 +03:00
Yiorgis Gozadinos
68bbf94577
tighten run-batch manifest replay validation 2026-06-22 11:58:32 +03:00
Yiorgis Gozadinos
7251d104c4
Add run-batch manifest replay 2026-06-22 11:39:12 +03:00
Yiorgis Gozadinos
a3b542dba8
Add run-batch dry-run manifest output 2026-06-22 11:33:26 +03:00
Yiorgis Gozadinos
4b573bfebd
Add side-effect-free batch dry-run discovery 2026-06-22 11:33:26 +03:00
Yiorgis Gozadinos
23f5b3c47a
Merge pull request #449 from bd-mkt/bd_concurrency2 
perf: move CPU-bound ingest work off the event loop
 2026-06-22 11:05:14 +03:00
Yiorgis Gozadinos
d3a1011baf
write fetched bodies off the event loop 2026-06-22 10:37:48 +03:00
Yiorgis Gozadinos
183d595494
prepare stored Docling blobs off the event loop 2026-06-22 10:30:02 +03:00
Yiorgis Gozadinos
5c4b37df89
vb 2026-06-19 11:12:45 +03:00
Yiorgis Gozadinos
b67f330031
Move textual-image to base dependencies 2026-06-19 10:37:10 +03:00
bryan davis
fe954a090e
move cpu bound actions off of main loop 2026-06-18 16:35:48 -05:00
bryan davis
faa97f8bc6
improve concurrency management for pdf attachments 2026-06-18 14:30:02 -05:00
Yiorgis Gozadinos
99011755fb
vb 2026-06-16 16:48:36 +03:00
Yiorgis Gozadinos
1e65d36766
Pass fetched FetchResult to ingester metadata providers 2026-06-16 16:37:26 +03:00
Yiorgis Gozadinos
3f5141abd0
Merge pull request #446 from ggozad/chore/docling-update 
Bump docling deps and relax opencv floor
 2026-06-16 16:32:31 +03:00
Yiorgis Gozadinos
06a3f365c4
Bump docling deps and relax opencv floor 2026-06-16 16:16:32 +03:00
Yiorgis Gozadinos
6add2780e8
Make ingester job retry idempotent against a live sibling 2026-06-16 16:05:05 +03:00
Yiorgis Gozadinos
79a4f49387
Stop re-enqueuing permanently-failed ingester documents 2026-06-16 16:03:51 +03:00
Yiorgis Gozadinos
515f42dd5a
Add custom ingester sources via entry points 2026-06-16 10:54:24 +03:00
Yiorgis Gozadinos
7e20b47e98
Widen SQLite ingester queue pool to serve concurrent connections 2026-06-16 09:25:01 +03:00
Yiorgis Gozadinos
fac62cb347
Follow HTTP redirects in the WebDAV source 2026-06-15 09:59:13 +03:00
Yiorgis Gozadinos
cc73a8629a
Wire metadata providers into ingester ingestion 2026-06-15 08:41:26 +03:00
Yiorgis Gozadinos
5722260857
Add metadata-provider discovery for the ingester 2026-06-15 08:05:45 +03:00
Yiorgis Gozadinos
70c9c2778f
Serialize all in-process pdfium access under a shared lock 2026-06-14 10:50:42 +03:00
Yiorgis Gozadinos
885e7b7ce7
coverage 2026-06-12 10:17:19 +03:00
Yiorgis Gozadinos
bfc51b8b28
urface document_meta in info, history, and the inspector 2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
2b8b9477b7
Serialize and roll back the multi-table document delete 2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
f2a5ac4246
Throttle background auto-vacuum to at most once per 5 minutes 2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
5d5d87d44c
Throttle background auto-vacuum to at most once per 5 minutes 2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
df8af54298
rebase from main 2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
3366a6d383
Split meta document attributes into a document_meta table 2026-06-12 10:17:17 +03:00
Yiorgis Gozadinos
4bc52f0710
Serialize multi-table document writes and bound update version churn 2026-06-12 10:16:39 +03:00
Yiorgis Gozadinos
8ac670714d
Tidy PDF attachment extension fix 2026-06-12 09:44:11 +03:00
bryan davis
bce24e84a0
fix for issue with parsing pdf attachment such as *.joboptions files 2026-06-11 16:52:07 -05:00
Yiorgis Gozadinos
88696cc6f1
vb 2026-06-11 09:50:33 +03:00
Yiorgis Gozadinos
ceae56562f
Expand ${VAR} environment references in YAML config 2026-06-11 09:39:54 +03:00
Tres Seaver
b78f0ae9ed
feat(ingester): serve control plane under a configurable base path 
Add `ingester.api.root_path` so the HTTP control plane (dashboard + API)
can be reverse-proxied behind a sub-path (e.g. /ingester/) on a shared
origin, instead of needing nginx sub_filter URL-rewriting.

- APIConfig.root_path: normalized ('', or single leading slash, no trailing
  slash) via a field_validator; validate_assignment so CLI overrides
  normalize the same way as config-file values.
- Forwarded to FastAPI(root_path=) and uvicorn.Config(root_path=) so
  OpenAPI/docs links are prefix-aware.
- Dashboard route injects a <base href> matching root_path; all dashboard
  fetches are now base-relative, so they resolve under the prefix while
  staying identical at the root.
- `serve --root-path` CLI flag.
- Docs: "Behind a reverse proxy" section with an nginx example.

Closes #431

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-09 23:02:43 -04:00
Yiorgis Gozadinos
bdad908a60
vb 2026-06-09 15:39:14 +03:00
Yiorgis Gozadinos
54524a4ea3
Gate ingester control-plane access logs to DEBUG 2026-06-09 12:46:17 +03:00
Yiorgis Gozadinos
4c186b01c7
Add Database and Configuration panels to the ingester dashboard 2026-06-09 12:28:13 +03:00
Yiorgis Gozadinos
c62166b26e
Add /database and /config endpoints to the ingester API 2026-06-09 12:11:33 +03:00
Yiorgis Gozadinos
0be2e5b24f
Extract gather_database_info shared by the info command 2026-06-09 11:09:28 +03:00
Yiorgis Gozadinos
a7ed405d75
Add HaikuRAG.import_documents for batch document import 2026-06-09 10:23:37 +03:00
Yiorgis Gozadinos
b43843f862
Batch-capable document and document-item repositories 2026-06-09 10:01:20 +03:00
Yiorgis Gozadinos
bd548837e5
Make the ingester worker circuit breaker per-source 2026-06-09 09:41:54 +03:00
Yiorgis Gozadinos
46b8fa3fef
vb 2026-06-08 12:24:04 +03:00
Yiorgis Gozadinos
d386d7f900
Bump haiku.skills to 0.17.2 2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
cd77bd9889
Bound analysis execute_code calls to avoid request-limit nulls 2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
52738cbc0b
Always create the shared-connection lock so serialization can't be skipped 2026-06-06 10:51:07 +03:00
Yiorgis Gozadinos
4a65a69a27
Make the shared-connection lock optional for direct tool calls 2026-06-05 18:04:56 +03:00
Yiorgis Gozadinos
e59ee56002
Serialize shared-connection access across skill tools and the sandbox 2026-06-05 17:22:11 +03:00
Yiorgis Gozadinos
b47583258e
Run analysis sandbox VFS reads on the calling loop via the skill connection 2026-06-05 16:20:11 +03:00
Yiorgis Gozadinos
916b84bff4
vb 2026-06-05 12:09:57 +03:00
Yiorgis Gozadinos
7906688bf1
Fix analysis sandbox "Already borrowed" crash on VFS reads 2026-06-05 11:50:19 +03:00
Yiorgis Gozadinos
414be551fb
Open read CLI verbs read-only 2026-06-05 10:48:49 +03:00
Yiorgis Gozadinos
0e3d53791f
Add rebuild --set-embedder to reconcile embedder identity 2026-06-05 10:42:15 +03:00
Yiorgis Gozadinos
213569601b
Stop rewriting stored embedding settings on database open 2026-06-05 10:35:50 +03:00
Yiorgis Gozadinos
b6bbea3d64
Stop writing the schema version on database open 2026-06-05 10:32:30 +03:00
Yiorgis Gozadinos
e70f271ac7
vb 2026-06-04 14:13:55 +03:00
Yiorgis Gozadinos
73e8ac2dd6
Build the SQLite queue URL without reparsing the path 2026-06-04 12:14:27 +03:00
Yiorgis Gozadinos
a04dc9445b
Pre-ping the Postgres queue engine 2026-06-04 11:29:31 +03:00
Yiorgis Gozadinos
1717bd4996
Make the SQLite job claim atomic across processes 2026-06-03 16:57:57 +03:00
Yiorgis Gozadinos
5cc32f111e
Mask the dburi password in queue init/migrate output 2026-06-03 14:42:32 +03:00
Yiorgis Gozadinos
44089e5b1f
Support a dburi for the ingester queue (SQLite + Postgres) 
Migrate the ingester queue storage from raw aiosqlite to SQLAlchemy Core
async. The backend is chosen by ingester.queue.dburi: a SQLAlchemy async
URL points the queue at a database server, and SQLite remains the default
when unset. The Postgres path claims jobs with FOR UPDATE SKIP LOCKED so
multiple ingester processes can share one queue; SQLite caps the pool to a
single connection to keep the select-then-update claim atomic.
 2026-06-03 14:34:09 +03:00
Yiorgis Gozadinos
4eae86225e
vb 2026-06-03 14:26:58 +03:00
Yiorgis Gozadinos
8afdd46176
skip spurious ingester DELETE when the resource is back on its source 2026-06-03 14:05:53 +03:00
Yiorgis Gozadinos
651b22ddcf
Fix rebuild --embed-only corrupting picture embeddings 2026-06-03 11:46:43 +03:00
Yiorgis Gozadinos
46747d369a
Add retention window to ingester queue, prune terminal job rows past retention window 2026-06-03 10:38:58 +03:00
Yiorgis Gozadinos
eb74525084
Lift analysis-skill citation rate via SKILL.md tightening 2026-06-01 18:58:52 +03:00
Yiorgis Gozadinos
8dbcd4bd7a
vb, changelog 2026-06-01 18:54:22 +03:00
Yiorgis Gozadinos
0550e363d4
Merge pull request #394 from mcdonc/perf/max-file-size-validation 
perf: add configurable max_file_size to prevent OOM on large files
 2026-06-01 18:49:18 +03:00
Chris McDonough
7e4aa8c71f
Add configurable max_file_size to reject oversized files before ingestion 
Large files buffered entirely in RAM can OOM workers. Add
max_file_size to source config (default None = no limit).

FS checks stat().st_size before read_bytes(). HTTP and WebDAV issue
a HEAD request before GET when a limit is configured. S3 checks the
size from the existing head_async() call before get_async().

FileTooLargeError is classified as PermanentError so oversized files
go straight to the DLQ instead of retrying.
 2026-06-01 18:40:46 +03:00
Yiorgis Gozadinos
774ac7c350
Merge pull request #393 from mcdonc/perf/batch-sync-state-writes 
perf: batch sync_state writes during poller sweeps
 2026-06-01 18:33:40 +03:00
Yiorgis Gozadinos
2cd97880fd
Replace sync_state batch 5-tuple with a SyncRow NamedTuple 2026-06-01 18:25:15 +03:00
Chris McDonough
a0a247d18a
Batch sync_state writes during poller sweeps 
Each discovered file previously triggered a separate sync.upsert()
call with its own lock acquire + SQLite commit (fsync). On a sweep
finding 1,000 files this meant 1,000 individual commits.

Collect sync_state rows into a list during the sweep and flush them
in a single SyncStateRepo.batch_upsert() call at the end — one lock
acquisition, one commit, one fsync.
 2026-06-01 18:21:09 +03:00