Commit graph

13 commits

Author SHA1 Message Date
Yiorgis Gozadinos
1e8e5e9f6f
Cover MCP, ingester and converter error paths
Add tests for the MCP tools' degradation contracts, malformed WebDAV
multistatus bodies, dry-run poller sweeps including the circuit-open and
discover-failure paths, FS source scheme and symlink handling, docling-serve
zip parsing, and the remaining embedding and reranker helpers. Parametrize
_strip_etag.

Drop the misplaced pragma on the analyze handler, which sat on the return and
left the except uncovered. Add one on the FS symlink OSError guard, which
resolve(strict=False) absorbs for every real link.
2026-07-26 19:14:26 +03:00
Yiorgis Gozadinos
c6514c9df4
compare off-loop work against actual event-loop thread, fix ty 2026-06-22 10:17:02 +03:00
bryan davis
fe954a090e
move cpu bound actions off of main loop 2026-06-18 16:35:48 -05:00
Chris McDonough
7e4aa8c71f
Add configurable max_file_size to reject oversized files before ingestion
Large files buffered entirely in RAM can OOM workers. Add
max_file_size to source config (default None = no limit).

FS checks stat().st_size before read_bytes(). HTTP and WebDAV issue
a HEAD request before GET when a limit is configured. S3 checks the
size from the existing head_async() call before get_async().

FileTooLargeError is classified as PermanentError so oversized files
go straight to the DLQ instead of retrying.
2026-06-01 18:40:46 +03:00
Chris McDonough
2a06421e7a Improve discover stat race test to actually exercise the try/except
The previous test deleted a file mid-iteration, but is_file()
caught it before stat() ran — so the new try/except never executed.
Monkeypatch Path.stat to raise FileNotFoundError on the third call
for the victim path (after is_symlink and is_file pass), simulating
the exact TOCTOU window between is_file() and stat().
2026-06-01 09:28:26 -04:00
Chris McDonough
7d80eb4f83 Fix discover() crash when file is deleted during stat()
A file deleted between os.walk() and path.stat() raises
FileNotFoundError, which propagated uncaught and failed the entire
discover() sweep. With enough failures this trips the circuit
breaker, silencing the poller.

Catch FileNotFoundError around the stat() call and skip the file.
The next sweep (or watchfiles) will emit the DELETE event.
2026-06-01 07:29:51 -04:00
Yiorgis Gozadinos
b0d0ac588d
Split snapshot APIs and resolve_fetcher by intent 2026-05-27 14:39:54 +03:00
Yiorgis Gozadinos
fdb73fc41f
Drop defensive branches 2026-05-26 16:56:32 +03:00
Yiorgis Gozadinos
78fc5d0e05
Discover within-root symlinks in FSSource 2026-05-26 13:54:33 +03:00
Yiorgis Gozadinos
ef9cacf981
Harden FS source against symlink escape; surface pool/poller liveness in /health, additional auth tests 2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
7ea61a7b10
drop the old monitor, rename serve→mcp, add e2e tests 2026-05-26 11:42:41 +03:00
Yiorgis Gozadinos
717fc3320b
route create_document_from_source through source adapters 2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
d0a730ef60
ingester: add source-adapter scaffolding + FS source 2026-05-26 11:41:53 +03:00