Commit graph

7 commits

Author SHA1 Message Date
Yiorgis Gozadinos
64f2b7b7d2
Merge pull request #405 from mcdonc/fix/run-batch-hang-on-dead-workers
fix: run_batch hangs forever when all workers die
2026-06-01 17:51:49 +03:00
Yiorgis Gozadinos
c5814b31bf
Test source clients close after the worker pool stops
Workers share the pollers' Source instances for fetch(), so the httpx
clients must be closed only after the pool has stopped. Guards both
run_batch() and serve() against reintroducing the shutdown-order bug.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-01 17:29:53 +03:00
Chris McDonough
b144e620de Fix dead-worker condition and test for run_batch abort
The condition only checked claimed jobs, but queued jobs with no
live workers also hang forever. Check live_workers == 0 regardless
of whether outstanding work is queued or claimed.

Rewrite the test to actually crash workers: patch _process to raise
a bare Exception (which _worker_loop doesn't catch), use
worker_count=1 so the single crash leaves live_workers == 0, and
assert the abort log message fires.
2026-06-01 09:47:10 -04:00
Chris McDonough
da8e7dc568 Fix run_batch hanging forever when all workers die
The drain loop in run_batch() polls counts_by_status() waiting for
queued and claimed counts to reach zero. If all worker tasks crash
(unhandled exception, OOM), claimed jobs stay claimed forever and
the loop never exits — the CLI command hangs.

Check live_workers during the drain loop. If claimed jobs exist but
no workers are alive to process them, log an error and break out.
The stranded jobs will be reaped on the next start.
2026-06-01 07:52:17 -04:00
Yiorgis Gozadinos
5a23e4eda6
Surface failed discovery sweeps in run-batch 2026-06-01 10:27:31 +03:00
Yiorgis Gozadinos
6f2a40c676
cover run-batch, serve, and _stop_pool with tests 2026-05-29 17:43:58 +03:00
Yiorgis Gozadinos
5ba7838b71
Add haiku-ingester run-batch, remove run-once 2026-05-29 17:00:59 +03:00