Commit graph

693 commits

Author SHA1 Message Date
Yiorgis Gozadinos
e3ea207358
Merge pull request #396 from mcdonc/perf/stagger-periodic-polls
perf: stagger periodic poll sweeps to avoid thundering herd
2026-06-01 17:23:22 +03:00
Yiorgis Gozadinos
b79bbcaed4
Merge pull request #399 from mcdonc/fix/discover-stat-race
fix: handle file deleted during discover() stat() call
2026-06-01 17:18:47 +03:00
Yiorgis Gozadinos
fda798a746
Merge pull request #391 from mcdonc/perf/event-driven-worker-wakeup
perf: event-driven worker wakeup via asyncio.Condition
2026-06-01 17:06:40 +03:00
Chris McDonough
004d59563c Extract _stagger_start helper into BasePoller, add tests
The jitter-before-first-sleep block was duplicated verbatim in
PeriodicPoller.run() and FSPoller._sweep_loop(). Move it to
BasePoller._stagger_start() with a named _STAGGER_FRACTION constant.

This also gives a testable seam outside the pragma-no-cover
event-loop glue methods.
2026-06-01 09:32:18 -04:00
Chris McDonough
2a06421e7a Improve discover stat race test to actually exercise the try/except
The previous test deleted a file mid-iteration, but is_file()
caught it before stat() ran — so the new try/except never executed.
Monkeypatch Path.stat to raise FileNotFoundError on the third call
for the victim path (after is_symlink and is_file pass), simulating
the exact TOCTOU window between is_file() and stat().
2026-06-01 09:28:26 -04:00
Chris McDonough
720ff23357 Fix shutdown regression: notify idle workers on stop()
Workers parked on job_available.wait() were not woken by stop(),
causing them to sleep out the full poll_idle_interval_s before
noticing _stop. With the default 1.0s interval, stop() took ~0.8s
instead of ~0.007s.

Notify all waiters on the condition in stop() so idle workers exit
immediately. Add tests for fast job pickup via notification and
fast shutdown with idle workers.
2026-06-01 09:21:24 -04:00
Yiorgis Gozadinos
8bf1b6be1a
Merge pull request #403 from mcdonc/fix/watch-change-stat-race
fix: watch loop crash when file deleted before stat() in _handle_watch_change
2026-06-01 15:59:50 +03:00
Chris McDonough
3c7dbf0966
Classify PermissionError as PermanentError instead of TransientError
PermissionError is a subclass of OSError, so it was caught by the
broad timeout/io handler and classified as transient. An unreadable
file would retry 5 times then DLQ — permissions don't fix themselves
without operator intervention.

Add an explicit PermissionError check before the OSError catch so
unreadable files go straight to the DLQ.
2026-06-01 15:36:37 +03:00
Chris McDonough
42922cd2bd Fix watch loop crash when file is deleted before stat() in _handle_watch_change
The expression `str(path.stat().st_mtime_ns) if path.exists() else None`
has a TOCTOU race: the file can be deleted between exists() and stat().
The resulting FileNotFoundError propagates up to _watch_loop's except
handler, which records a breaker failure and terminates the loop — no
more push events are processed until restart.

Replace with a try/except around stat() and return early on
FileNotFoundError. The deletion event from watchfiles will handle
cleanup.
2026-06-01 07:46:35 -04:00
Chris McDonough
c9c48bc814 Classify FileNotFoundError as PermanentError instead of TransientError
FileNotFoundError is a subclass of OSError, so it was caught by the
broad timeout/io handler and classified as transient. A file deleted
between discovery and fetch would retry 5 times on a file that's
permanently gone, then DLQ with a confusing error message.

Add an explicit FileNotFoundError check before the OSError catch so
deleted files go straight to the DLQ.
2026-06-01 07:32:09 -04:00
Chris McDonough
7d80eb4f83 Fix discover() crash when file is deleted during stat()
A file deleted between os.walk() and path.stat() raises
FileNotFoundError, which propagated uncaught and failed the entire
discover() sweep. With enough failures this trips the circuit
breaker, silencing the poller.

Catch FileNotFoundError around the stat() call and skip the file.
The next sweep (or watchfiles) will emit the DELETE event.
2026-06-01 07:29:51 -04:00
Yiorgis Gozadinos
339d231a4e
Remove repliqa from test setup 2026-06-01 10:40:52 +03:00
Yiorgis Gozadinos
5a23e4eda6
Surface failed discovery sweeps in run-batch 2026-06-01 10:27:31 +03:00
Yiorgis Gozadinos
6f2a40c676
cover run-batch, serve, and _stop_pool with tests 2026-05-29 17:43:58 +03:00
Yiorgis Gozadinos
5ba7838b71
Add haiku-ingester run-batch, remove run-once 2026-05-29 17:00:59 +03:00
Yiorgis Gozadinos
e7c7df2915
Always use the Store-owned embedder 2026-05-29 11:36:53 +03:00
Yiorgis Gozadinos
37a78a4e9b
Cache reranker on the client instead of rebuilding per search 2026-05-29 10:33:50 +03:00
Yiorgis Gozadinos
d8d6727227
improve coverage 2026-05-28 18:06:47 +03:00
Yiorgis Gozadinos
402957d3e5
Fix CI flakes for PDF attachment extraction 2026-05-28 17:51:02 +03:00
Yiorgis Gozadinos
6eee09743b
Cover PDF attachment ingest through the full create_document_from_source path 2026-05-28 16:07:12 +03:00
Yiorgis Gozadinos
8e8c4433bd
Extract PDF /EmbeddedFiles attachments as child documents 2026-05-28 15:51:32 +03:00
Yiorgis Gozadinos
26bc71d6d8
Cascade delete_document to children via metadata.parent_uri 2026-05-28 15:36:00 +03:00
Yiorgis Gozadinos
5400085147
Auto-prune dead jobs when a sibling DELETE succeeds 2026-05-27 17:17:01 +03:00
Yiorgis Gozadinos
1e6dc34871
Skip FS DELETE enqueue when the file is already back 2026-05-27 15:18:46 +03:00
Yiorgis Gozadinos
15d13cab04
Probe docling-serve only when converter or chunker uses it 2026-05-27 15:18:25 +03:00
Yiorgis Gozadinos
b0d0ac588d
Split snapshot APIs and resolve_fetcher by intent 2026-05-27 14:39:54 +03:00
Yiorgis Gozadinos
2c63ceda0c
Backfill ingester test gaps and drop unneeded retry clamp 2026-05-27 14:24:22 +03:00
Yiorgis Gozadinos
cfbd0b09d6
Tighten HTTP config-removal handling 2026-05-27 14:07:45 +03:00
Yiorgis Gozadinos
7466539b4f
Emit DELETE for HTTP URLs removed from config 2026-05-27 13:39:18 +03:00
Yiorgis Gozadinos
d7fdd61fed
Resolve worker source by source_id, not just supports(uri) 2026-05-27 13:36:03 +03:00
Yiorgis Gozadinos
922d1d567d
Prevent DELETE/UPSERT race for the same URI 2026-05-27 13:30:46 +03:00
Yiorgis Gozadinos
1c710433c7
Collapse worker_count and max_concurrent into worker_count 2026-05-27 12:48:14 +03:00
Yiorgis Gozadinos
87af5e5139
Boot-reap stale claims at WorkerPool start 2026-05-27 11:46:37 +03:00
Yiorgis Gozadinos
1b36452629
Surface provider reachability and per-job failure context on dashboard 2026-05-27 11:38:48 +03:00
Yiorgis Gozadinos
439307d5af
Add pool-wide circuit breaker to the worker pool 2026-05-27 10:52:48 +03:00
Yiorgis Gozadinos
b491f767f3
Let httpx errors flow through DoclingServeClient 2026-05-26 17:38:49 +03:00
Yiorgis Gozadinos
fdb73fc41f
Drop defensive branches 2026-05-26 16:56:32 +03:00
Yiorgis Gozadinos
29b3ccdbda
Drain orphan cancel-cleanup releases before closing the queue 2026-05-26 16:14:17 +03:00
Yiorgis Gozadinos
3c608046e2
Bound ?limit and ?offset on /jobs and /dlq 2026-05-26 16:05:47 +03:00
Yiorgis Gozadinos
761956eb70
Extend reaper-resurrection guard to reschedule and release_if_claimed 2026-05-26 16:04:24 +03:00
Yiorgis Gozadinos
a822b754b7
Guard mark_succeeded/mark_dead against reaper resurrection 2026-05-26 14:07:44 +03:00
Yiorgis Gozadinos
9cb3ce40ad
Shield cancel-cleanup release in worker pool 2026-05-26 13:57:35 +03:00
Yiorgis Gozadinos
78fc5d0e05
Discover within-root symlinks in FSSource 2026-05-26 13:54:33 +03:00
Yiorgis Gozadinos
e5e0df6ade
Unlink PDF slice tempfiles when slice write fails 2026-05-26 13:46:24 +03:00
Yiorgis Gozadinos
98efd73177
Serialize JobRepo and SyncStateRepo on one shared lock 2026-05-26 13:11:01 +03:00
Yiorgis Gozadinos
5f31a6b12f
Fix round-robin docling-serve 2026-05-26 13:07:21 +03:00
Yiorgis Gozadinos
cf6caf14fe
Tighten retry/reap_stale guards; fix SourceSummary.type leak 2026-05-26 12:38:55 +03:00
Yiorgis Gozadinos
f89cc998eb
add sources_provider to WorkerPool. workers now resolve extra info through these sources 2026-05-26 11:45:35 +03:00
Yiorgis Gozadinos
410279cd6c
Preserve sync_state revision when poller bumps last_seen_at 2026-05-26 11:45:35 +03:00
Yiorgis Gozadinos
ef9cacf981
Harden FS source against symlink escape; surface pool/poller liveness in /health, additional auth tests 2026-05-26 11:44:47 +03:00