haiku.rag/tests/ingester
Chris McDonough a0a247d18a
Batch sync_state writes during poller sweeps
Each discovered file previously triggered a separate sync.upsert()
call with its own lock acquire + SQLite commit (fsync). On a sweep
finding 1,000 files this meant 1,000 individual commits.

Collect sync_state rows into a list during the sweep and flush them
in a single SyncStateRepo.batch_upsert() call at the end — one lock
acquisition, one commit, one fsync.
2026-06-01 18:21:09 +03:00
..
__init__.py ingester: add source-adapter scaffolding + FS source 2026-05-26 11:41:53 +03:00
test_api.py Probe docling-serve only when converter or chunker uses it 2026-05-27 15:18:25 +03:00
test_circuit_breaker.py additional config, pollers, serve 2026-05-26 11:41:54 +03:00
test_cli.py Strengthen config-load assertions 2026-06-01 18:03:09 +03:00
test_config.py Collapse worker_count and max_concurrent into worker_count 2026-05-27 12:48:14 +03:00
test_filter.py Improve test coverage for cli, filter, registry, and migrations 2026-06-01 18:00:47 +03:00
test_fs_source.py Improve discover stat race test to actually exercise the try/except 2026-06-01 09:28:26 -04:00
test_http_source.py Narrow HTTP discover() exception catch to TransportError only 2026-06-01 17:57:51 +03:00
test_pipeline.py Classify IsADirectoryError and NotADirectoryError as PermanentError 2026-06-01 17:59:23 +03:00
test_pollers.py Merge pull request #396 from mcdonc/perf/stagger-periodic-polls 2026-06-01 17:23:22 +03:00
test_queue.py Batch sync_state writes during poller sweeps 2026-06-01 18:21:09 +03:00
test_resolve_fetcher.py Improve test coverage for cli, filter, registry, and migrations 2026-06-01 18:00:47 +03:00
test_revision_round_trip.py record cassettes against new fixtures, fix stale assertions 2026-05-26 11:44:46 +03:00
test_run_batch.py Merge pull request #405 from mcdonc/fix/run-batch-hang-on-dead-workers 2026-06-01 17:51:49 +03:00
test_s3_source.py Split snapshot APIs and resolve_fetcher by intent 2026-05-27 14:39:54 +03:00
test_serve_integration.py Collapse worker_count and max_concurrent into worker_count 2026-05-27 12:48:14 +03:00
test_sources_base.py Fix shutdown-order bug: close source clients after workers stop 2026-06-01 09:55:39 -04:00
test_webdav_source.py Split snapshot APIs and resolve_fetcher by intent 2026-05-27 14:39:54 +03:00
test_workers.py Merge pull request #409 from mcdonc/fix/sync-state-write-crash 2026-06-01 17:46:46 +03:00