danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
haiku.rag/tests/ingester
History
Chris McDonough 7e4aa8c71f
Add configurable max_file_size to reject oversized files before ingestion 
Large files buffered entirely in RAM can OOM workers. Add
max_file_size to source config (default None = no limit).

FS checks stat().st_size before read_bytes(). HTTP and WebDAV issue
a HEAD request before GET when a limit is configured. S3 checks the
size from the existing head_async() call before get_async().

FileTooLargeError is classified as PermanentError so oversized files
go straight to the DLQ instead of retrying.
2026-06-01 18:40:46 +03:00
..
__init__.py ingester: add source-adapter scaffolding + FS source 2026-05-26 11:41:53 +03:00
test_api.py Probe docling-serve only when converter or chunker uses it 2026-05-27 15:18:25 +03:00
test_circuit_breaker.py additional config, pollers, serve 2026-05-26 11:41:54 +03:00
test_cli.py Strengthen config-load assertions 2026-06-01 18:03:09 +03:00
test_config.py Collapse worker_count and max_concurrent into worker_count 2026-05-27 12:48:14 +03:00
test_filter.py Improve test coverage for cli, filter, registry, and migrations 2026-06-01 18:00:47 +03:00
test_fs_source.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00
test_http_source.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00
test_pipeline.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00
test_pollers.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00
test_queue.py Auto-prune dead jobs when a sibling DELETE succeeds 2026-05-27 17:17:01 +03:00
test_resolve_fetcher.py Improve test coverage for cli, filter, registry, and migrations 2026-06-01 18:00:47 +03:00
test_revision_round_trip.py record cassettes against new fixtures, fix stale assertions 2026-05-26 11:44:46 +03:00
test_run_batch.py Merge pull request #405 from mcdonc/fix/run-batch-hang-on-dead-workers 2026-06-01 17:51:49 +03:00
test_s3_source.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00
test_serve_integration.py Collapse worker_count and max_concurrent into worker_count 2026-05-27 12:48:14 +03:00
test_sources_base.py Fix shutdown-order bug: close source clients after workers stop 2026-06-01 09:55:39 -04:00
test_webdav_source.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00
test_workers.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00