Document s3 storage & monitoring

CHANGELOG.md

@@ -1,6 +1,18 @@
 # Changelog
 ## [Unreleased]
 
+### Added
+
+- **`s3://` is a first-class document source.** `create_document_from_source`, the CLI `haiku-rag add-src`, and the MCP `add_document_from_url` tool all dispatch on the `s3` URL scheme. Two-stage change detection keeps `metadata["md5"]` semantically uniform across all sources: HEAD ETag matching the stored `metadata["etag"]` short-circuits without GET; if ETag differs but bytes hash to the same MD5 (multipart re-upload, server-side `CopyObject`, SSE mode change), only the etag refreshes — no re-chunk or re-embed. Closes #357.
+- **S3 / object-storage monitoring.** `monitor.s3: list[S3MonitorEntry]` adds a polling watcher per bucket prefix alongside the existing local-directory watcher. Each entry has its own `poll_interval`, `include_patterns`, `ignore_patterns`, `delete_orphans`, and `storage_options`. The same `serve --monitor` flag enables both. Orphan deletion is per-entry (scoped via `uri LIKE 's3://bucket/prefix/%'`); other buckets and prefixes are never touched.
+- **`[s3]` optional extra** (`obstore>=0.9`). Required for `s3://` sources and the S3 watcher. Uses obstore — the Python binding to the same Rust `object_store` crate that LanceDB uses internally — so `monitor.s3[*].storage_options` accepts the same dict shape as `lancedb.storage_options`. Empty/missing options fall back to the AWS default credential chain.
+- **`scripts/run-integration-tests.sh`** — wraps `docker compose up --wait`, `pytest -m integration`, and tear-down so the SeaweedFS-backed integration suite is a one-liner.
+
+### Documentation
+
+- New "S3 / Object Storage Monitoring" section in `docs/server.md` and `docs/configuration/processing.md` covering the `[s3]` extra, polling cadence, ETag semantics, credentials, and CLI usage.
+- New "Deployment Pattern: One Writer, Many Readers" subsection in `docs/configuration/storage.md` documenting the recommended IAM split (one ingestion process + N read-only consumers).
+
 ## [0.45.0] - 2026-05-08
 
 ### Added

docs/cli.md

@@ -75,6 +75,17 @@ From directory (recursively adds all supported files):
 haiku-rag add-src /path/to/documents/
 ```
 
+From an S3 bucket (requires the `[s3]` extra — see [Server Mode → S3 / Object Storage Monitoring](server.md#s3-object-storage-monitoring)):
+```bash
+# AWS S3 with credentials in the default chain (env vars, IAM role, AWS profile)
+haiku-rag add-src s3://my-bucket/path/to/document.pdf
+
+# S3-compatible endpoint (SeaweedFS, MinIO, Cloudflare R2, etc.)
+AWS_ACCESS_KEY_ID=key AWS_SECRET_ACCESS_KEY=secret AWS_REGION=us-east-1 \
+  AWS_ENDPOINT_URL=http://localhost:8333 \
+  haiku-rag add-src s3://my-bucket/path/to/document.pdf
+```
+
 !!! note
     When adding a directory, the same content filters configured for [file monitoring](configuration/processing.md#filtering-monitored-files) are applied. This means `ignore_patterns` and `include_patterns` from your configuration will be used to filter which files are added.
 

docs/configuration/processing.md

@@ -322,3 +322,29 @@ Patterns follow [gitignore syntax](https://git-scm.com/docs/gitignore#_pattern_f
 - `**` matches zero or more directories
 - `?` matches any single character
 - `[abc]` matches any character in the set
+
+### S3 / Object Storage Sources
+
+In addition to local directories, the watcher can poll S3-compatible buckets (AWS S3, SeaweedFS, MinIO, Cloudflare R2, etc.). Install the `[s3]` extra and configure one or more entries under `monitor.s3`:
+
+```yaml
+monitor:
+  s3:
+    - uri: s3://my-bucket/incoming/
+      poll_interval: 300        # seconds between sweeps; default 300
+      include_patterns: ["*.pdf", "*.md"]
+      ignore_patterns: ["draft*"]
+      delete_orphans: true
+      storage_options:
+        endpoint: http://seaweed:8333
+        aws_access_key_id: ${AWS_KEY}
+        aws_secret_access_key: ${AWS_SECRET}
+        region: us-east-1
+        allow_http: "true"
+```
+
+Each entry is independent — own poll interval, own include/ignore patterns, own `delete_orphans` setting, own credentials. Omit `storage_options` to fall back to the AWS default credential chain (env vars, IAM role, AWS profile).
+
+The dict shape matches `lancedb.storage_options` — the same Rust `object_store` library is used by both, so credentials configured for the LanceDB backend can be copy-pasted here.
+
+See [Server Mode → S3 / Object Storage Monitoring](../server.md#s3-object-storage-monitoring) for behaviour details (ETag-based change detection, orphan-deletion scope, CLI `add-src s3://…`).

docs/configuration/storage.md

@@ -74,6 +74,17 @@ The `storage_options` keys are case-insensitive and passed directly to the under
 
 **Note:** Table optimization is automatically handled by LanceDB Cloud (`db://` URIs) and is disabled for better performance. For object storage backends (S3, Azure, GCS), optimization and vector indexing are still performed normally.
 
+### Deployment Pattern: One Writer, Many Readers
+
+LanceDB on S3 supports **exactly one writer + N readers** per database URI. Multiple writers against the same URI can race on the manifest commit and corrupt state — this is a LanceDB property, not something `haiku.rag` enforces.
+
+The recommended layout for production is "different buckets, same account, separate IAM roles per process":
+
+- **Ingestion process** — IAM role with `s3:Get/List` on the documents bucket and `s3:Get/Put/Delete` on the LanceDB bucket. Runs `haiku-rag serve --monitor` (with `monitor.s3` entries pointing at the documents bucket). Exactly one such process per LanceDB URI.
+- **Consumer processes** (1..N) — IAM role with `s3:Get/List` on the LanceDB bucket only. Run `haiku-rag serve --read-only --mcp`, the chat TUI, etc. They never see the documents bucket.
+
+Each process picks up its own credentials from the AWS default chain (env vars, IAM instance role, AWS profile), so no credentials are hard-coded in the configuration files.
+
 ## Database Creation
 
 Databases must be explicitly created before use:

docs/server.md

@@ -120,3 +120,53 @@ The file monitor processes documents using [Docling](https://github.com/DS4SD/do
 - RST (`.rst`)
 
 URLs are also supported - the content is fetched and converted to markdown.
+
+## S3 / Object Storage Monitoring
+
+The server can also poll S3-compatible object storage (AWS S3, SeaweedFS, MinIO, Cloudflare R2, etc.) for new, modified, and deleted objects, treating each one as a document source.
+
+Install the optional `[s3]` extra:
+
+```bash
+pip install haiku.rag-slim[s3]
+# or, for the full package:
+pip install haiku.rag[s3]
+```
+
+Configure one or more S3 sources under `monitor.s3` in `haiku.rag.yaml`:
+
+```yaml
+monitor:
+  s3:
+    - uri: s3://my-bucket/incoming/
+      poll_interval: 300        # seconds between sweeps; default 300
+      include_patterns: ["*.pdf", "*.md"]
+      ignore_patterns: ["draft*"]
+      delete_orphans: true
+      storage_options:
+        endpoint: http://seaweed:8333
+        aws_access_key_id: ${AWS_KEY}
+        aws_secret_access_key: ${AWS_SECRET}
+        region: us-east-1
+        allow_http: "true"
+```
+
+Then start the server with `--monitor` — the same flag enables both local-directory and S3 watchers:
+
+```bash
+haiku-rag serve --monitor
+```
+
+Each entry in `monitor.s3` runs as its own polling task. On every sweep the watcher lists all objects under the configured prefix, compares each object's S3 ETag against the document's stored `metadata["etag"]`, and only re-fetches keys whose ETag has changed. When the bytes turn out to match the stored MD5 (e.g. the same file was re-uploaded with a different multipart chunk size), the watcher refreshes the etag and skips re-chunking. Otherwise the document is downloaded, chunked, and re-embedded.
+
+### Credentials
+
+`storage_options` follows the same convention as `lancedb.storage_options` — the dict is passed straight to obstore (the same Rust `object_store` library LanceDB uses internally), so any keys you've configured there work here too. When `storage_options` is omitted, the watcher falls back to the AWS default credential chain (environment variables, IAM instance role, AWS profile).
+
+### Orphan deletion scope
+
+`delete_orphans: true` is per-entry: a watcher only removes documents whose URI starts with that entry's `s3://bucket/prefix/`. Documents from other buckets, prefixes, or local-file sources are never touched.
+
+### One-off ingestion
+
+`s3://` URIs are also a first-class source for `haiku-rag add-src` and the MCP `add_document_from_url` tool — see [CLI → Add Documents](cli.md#add-documents).