Commit graph

8 commits

Author SHA1 Message Date
Yiorgis Gozadinos
ab19f78507
Move source adapters out of the ingester package
haiku.rag.ingester.sources was never ingester-only: one-shot client
ingestion resolves adapters through it (create_document_from_source), and
convert() now fetches through HTTPSource, so the core client imported into
the ingester package to reach them.

Move the package to haiku.rag.sources and update every import. No shims:
haiku.rag.ingester.sources is gone.

The haiku.rag.sources plugin entry-point group is unchanged, so third-party
source packages need no edit — the group name now matches the module path it
always implied.

Source unit tests move to tests/sources/. test_source_plugins.py stays in
tests/ingester/: it drives a PeriodicPoller against the job repo, so it is
plugin wiring through ingester machinery rather than a source test.
2026-08-20 11:46:55 +03:00
Yiorgis Gozadinos
515f42dd5a
Add custom ingester sources via entry points 2026-06-16 10:54:24 +03:00
Chris McDonough
7e4aa8c71f
Add configurable max_file_size to reject oversized files before ingestion
Large files buffered entirely in RAM can OOM workers. Add
max_file_size to source config (default None = no limit).

FS checks stat().st_size before read_bytes(). HTTP and WebDAV issue
a HEAD request before GET when a limit is configured. S3 checks the
size from the existing head_async() call before get_async().

FileTooLargeError is classified as PermanentError so oversized files
go straight to the DLQ instead of retrying.
2026-06-01 18:40:46 +03:00
Yiorgis Gozadinos
fdb73fc41f
Drop defensive branches 2026-05-26 16:56:32 +03:00
Yiorgis Gozadinos
ba623b0862
mark event-loop glue + defensive guards as no-cover 2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
65c309e12b
Handle WebDAV as source 2026-05-26 11:43:30 +03:00
Yiorgis Gozadinos
de3b3fa1c9
HTTP control plane 2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
75c3896588
additional config, pollers, serve 2026-05-26 11:41:54 +03:00