Commit graph

27 commits

Author SHA1 Message Date
Yiorgis Gozadinos
5ba7838b71
Add haiku-ingester run-batch, remove run-once 2026-05-29 17:00:59 +03:00
Yiorgis Gozadinos
5400085147
Auto-prune dead jobs when a sibling DELETE succeeds 2026-05-27 17:17:01 +03:00
Yiorgis Gozadinos
b0d0ac588d
Split snapshot APIs and resolve_fetcher by intent 2026-05-27 14:39:54 +03:00
Yiorgis Gozadinos
2c63ceda0c
Backfill ingester test gaps and drop unneeded retry clamp 2026-05-27 14:24:22 +03:00
Yiorgis Gozadinos
d7fdd61fed
Resolve worker source by source_id, not just supports(uri) 2026-05-27 13:36:03 +03:00
Yiorgis Gozadinos
1c710433c7
Collapse worker_count and max_concurrent into worker_count 2026-05-27 12:48:14 +03:00
Yiorgis Gozadinos
6f74fe67b3
Cap claimed-row count at max_concurren 2026-05-27 12:10:42 +03:00
Yiorgis Gozadinos
87af5e5139
Boot-reap stale claims at WorkerPool start 2026-05-27 11:46:37 +03:00
Yiorgis Gozadinos
439307d5af
Add pool-wide circuit breaker to the worker pool 2026-05-27 10:52:48 +03:00
Yiorgis Gozadinos
b491f767f3
Let httpx errors flow through DoclingServeClient 2026-05-26 17:38:49 +03:00
Yiorgis Gozadinos
fdb73fc41f
Drop defensive branches 2026-05-26 16:56:32 +03:00
Yiorgis Gozadinos
29b3ccdbda
Drain orphan cancel-cleanup releases before closing the queue 2026-05-26 16:14:17 +03:00
Yiorgis Gozadinos
761956eb70
Extend reaper-resurrection guard to reschedule and release_if_claimed 2026-05-26 16:04:24 +03:00
Yiorgis Gozadinos
a822b754b7
Guard mark_succeeded/mark_dead against reaper resurrection 2026-05-26 14:07:44 +03:00
Yiorgis Gozadinos
9cb3ce40ad
Shield cancel-cleanup release in worker pool 2026-05-26 13:57:35 +03:00
Yiorgis Gozadinos
f89cc998eb
add sources_provider to WorkerPool. workers now resolve extra info through these sources 2026-05-26 11:45:35 +03:00
Yiorgis Gozadinos
73f8349a00
Use haiku.rag as telemtry scope 2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
ef9cacf981
Harden FS source against symlink escape; surface pool/poller liveness in /health, additional auth tests 2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
da3cfe1a58
Release claim on jobs when cancelled 2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
ba623b0862
mark event-loop glue + defensive guards as no-cover 2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
ed36cc2230
typed UnsupportedSourceError replaces pipeline string-marker matching 2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
5ccbadde0a
Handle shutdown more gracefully, by stopping pollers and cancelling jobs after timeout. Skip periodic poll if a source has pending jobs 2026-05-26 11:44:03 +03:00
Yiorgis Gozadinos
13cbadeb6f
canonical metadata keys: source_revision + content_type, bump to 0.50.0 2026-05-26 11:43:30 +03:00
Yiorgis Gozadinos
fa672c298a
logfire spans for the ingester, nested poller/job/document traces 2026-05-26 11:42:42 +03:00
Yiorgis Gozadinos
1ca3c25a83
logging, sqlite cursors 2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
de3b3fa1c9
HTTP control plane 2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
bdcc1caa49
worker pool, rety policy, pipeline 2026-05-26 11:41:54 +03:00