danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
haiku.rag/tests/ingester
History
Chris McDonough da8e7dc568 Fix run_batch hanging forever when all workers die 
The drain loop in run_batch() polls counts_by_status() waiting for
queued and claimed counts to reach zero. If all worker tasks crash
(unhandled exception, OOM), claimed jobs stay claimed forever and
the loop never exits — the CLI command hangs.

Check live_workers during the drain loop. If claimed jobs exist but
no workers are alive to process them, log an error and break out.
The stranded jobs will be reaped on the next start.
2026-06-01 07:52:17 -04:00
..
__init__.py ingester: add source-adapter scaffolding + FS source 2026-05-26 11:41:53 +03:00
test_api.py Probe docling-serve only when converter or chunker uses it 2026-05-27 15:18:25 +03:00
test_circuit_breaker.py additional config, pollers, serve 2026-05-26 11:41:54 +03:00
test_cli.py Surface failed discovery sweeps in run-batch 2026-06-01 10:27:31 +03:00
test_config.py Collapse worker_count and max_concurrent into worker_count 2026-05-27 12:48:14 +03:00
test_fs_source.py Split snapshot APIs and resolve_fetcher by intent 2026-05-27 14:39:54 +03:00
test_http_source.py Split snapshot APIs and resolve_fetcher by intent 2026-05-27 14:39:54 +03:00
test_pipeline.py Backfill ingester test gaps and drop unneeded retry clamp 2026-05-27 14:24:22 +03:00
test_pollers.py Skip FS DELETE enqueue when the file is already back 2026-05-27 15:18:46 +03:00
test_queue.py Auto-prune dead jobs when a sibling DELETE succeeds 2026-05-27 17:17:01 +03:00
test_resolve_fetcher.py Split snapshot APIs and resolve_fetcher by intent 2026-05-27 14:39:54 +03:00
test_revision_round_trip.py record cassettes against new fixtures, fix stale assertions 2026-05-26 11:44:46 +03:00
test_run_batch.py Fix run_batch hanging forever when all workers die 2026-06-01 07:52:17 -04:00
test_s3_source.py Split snapshot APIs and resolve_fetcher by intent 2026-05-27 14:39:54 +03:00
test_serve_integration.py Collapse worker_count and max_concurrent into worker_count 2026-05-27 12:48:14 +03:00
test_sources_base.py route create_document_from_source through source adapters 2026-05-26 11:41:54 +03:00
test_webdav_source.py Split snapshot APIs and resolve_fetcher by intent 2026-05-27 14:39:54 +03:00
test_workers.py Auto-prune dead jobs when a sibling DELETE succeeds 2026-05-27 17:17:01 +03:00