danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
haiku.rag/haiku_rag_slim/haiku/rag/ingester/sources
History
Chris McDonough 7e4aa8c71f
Add configurable max_file_size to reject oversized files before ingestion 
Large files buffered entirely in RAM can OOM workers. Add
max_file_size to source config (default None = no limit).

FS checks stat().st_size before read_bytes(). HTTP and WebDAV issue
a HEAD request before GET when a limit is configured. S3 checks the
size from the existing head_async() call before get_async().

FileTooLargeError is classified as PermanentError so oversized files
go straight to the DLQ instead of retrying.
2026-06-01 18:40:46 +03:00
..
__init__.py Split snapshot APIs and resolve_fetcher by intent 2026-05-27 14:39:54 +03:00
base.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00
filter.py ingester: add source-adapter scaffolding + FS source 2026-05-26 11:41:53 +03:00
fs.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00
http.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00
registry.py Split snapshot APIs and resolve_fetcher by intent 2026-05-27 14:39:54 +03:00
s3.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00
webdav.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00