diff --git a/CHANGELOG.md b/CHANGELOG.md index 140f8a5a..dbb1a98d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,18 @@ # Changelog ## [Unreleased] +### Added + +- **`s3://` is a first-class document source.** `create_document_from_source`, the CLI `haiku-rag add-src`, and the MCP `add_document_from_url` tool all dispatch on the `s3` URL scheme. Two-stage change detection keeps `metadata["md5"]` semantically uniform across all sources: HEAD ETag matching the stored `metadata["etag"]` short-circuits without GET; if ETag differs but bytes hash to the same MD5 (multipart re-upload, server-side `CopyObject`, SSE mode change), only the etag refreshes — no re-chunk or re-embed. Closes #357. +- **S3 / object-storage monitoring.** `monitor.s3: list[S3MonitorEntry]` adds a polling watcher per bucket prefix alongside the existing local-directory watcher. Each entry has its own `poll_interval`, `include_patterns`, `ignore_patterns`, `delete_orphans`, and `storage_options`. The same `serve --monitor` flag enables both. Orphan deletion is per-entry (scoped via `uri LIKE 's3://bucket/prefix/%'`); other buckets and prefixes are never touched. +- **`[s3]` optional extra** (`obstore>=0.9`). Required for `s3://` sources and the S3 watcher. Uses obstore — the Python binding to the same Rust `object_store` crate that LanceDB uses internally — so `monitor.s3[*].storage_options` accepts the same dict shape as `lancedb.storage_options`. Empty/missing options fall back to the AWS default credential chain. +- **`scripts/run-integration-tests.sh`** — wraps `docker compose up --wait`, `pytest -m integration`, and tear-down so the SeaweedFS-backed integration suite is a one-liner. + +### Documentation + +- New "S3 / Object Storage Monitoring" section in `docs/server.md` and `docs/configuration/processing.md` covering the `[s3]` extra, polling cadence, ETag semantics, credentials, and CLI usage. +- New "Deployment Pattern: One Writer, Many Readers" subsection in `docs/configuration/storage.md` documenting the recommended IAM split (one ingestion process + N read-only consumers). + ## [0.45.0] - 2026-05-08 ### Added diff --git a/docs/cli.md b/docs/cli.md index 668dd451..b4c9848f 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -75,6 +75,17 @@ From directory (recursively adds all supported files): haiku-rag add-src /path/to/documents/ ``` +From an S3 bucket (requires the `[s3]` extra — see [Server Mode → S3 / Object Storage Monitoring](server.md#s3-object-storage-monitoring)): +```bash +# AWS S3 with credentials in the default chain (env vars, IAM role, AWS profile) +haiku-rag add-src s3://my-bucket/path/to/document.pdf + +# S3-compatible endpoint (SeaweedFS, MinIO, Cloudflare R2, etc.) +AWS_ACCESS_KEY_ID=key AWS_SECRET_ACCESS_KEY=secret AWS_REGION=us-east-1 \ + AWS_ENDPOINT_URL=http://localhost:8333 \ + haiku-rag add-src s3://my-bucket/path/to/document.pdf +``` + !!! note When adding a directory, the same content filters configured for [file monitoring](configuration/processing.md#filtering-monitored-files) are applied. This means `ignore_patterns` and `include_patterns` from your configuration will be used to filter which files are added. diff --git a/docs/configuration/processing.md b/docs/configuration/processing.md index c96cf48a..e48279c5 100644 --- a/docs/configuration/processing.md +++ b/docs/configuration/processing.md @@ -322,3 +322,29 @@ Patterns follow [gitignore syntax](https://git-scm.com/docs/gitignore#_pattern_f - `**` matches zero or more directories - `?` matches any single character - `[abc]` matches any character in the set + +### S3 / Object Storage Sources + +In addition to local directories, the watcher can poll S3-compatible buckets (AWS S3, SeaweedFS, MinIO, Cloudflare R2, etc.). Install the `[s3]` extra and configure one or more entries under `monitor.s3`: + +```yaml +monitor: + s3: + - uri: s3://my-bucket/incoming/ + poll_interval: 300 # seconds between sweeps; default 300 + include_patterns: ["*.pdf", "*.md"] + ignore_patterns: ["draft*"] + delete_orphans: true + storage_options: + endpoint: http://seaweed:8333 + aws_access_key_id: ${AWS_KEY} + aws_secret_access_key: ${AWS_SECRET} + region: us-east-1 + allow_http: "true" +``` + +Each entry is independent — own poll interval, own include/ignore patterns, own `delete_orphans` setting, own credentials. Omit `storage_options` to fall back to the AWS default credential chain (env vars, IAM role, AWS profile). + +The dict shape matches `lancedb.storage_options` — the same Rust `object_store` library is used by both, so credentials configured for the LanceDB backend can be copy-pasted here. + +See [Server Mode → S3 / Object Storage Monitoring](../server.md#s3-object-storage-monitoring) for behaviour details (ETag-based change detection, orphan-deletion scope, CLI `add-src s3://…`). diff --git a/docs/configuration/storage.md b/docs/configuration/storage.md index ebb233bb..2a576417 100644 --- a/docs/configuration/storage.md +++ b/docs/configuration/storage.md @@ -74,6 +74,17 @@ The `storage_options` keys are case-insensitive and passed directly to the under **Note:** Table optimization is automatically handled by LanceDB Cloud (`db://` URIs) and is disabled for better performance. For object storage backends (S3, Azure, GCS), optimization and vector indexing are still performed normally. +### Deployment Pattern: One Writer, Many Readers + +LanceDB on S3 supports **exactly one writer + N readers** per database URI. Multiple writers against the same URI can race on the manifest commit and corrupt state — this is a LanceDB property, not something `haiku.rag` enforces. + +The recommended layout for production is "different buckets, same account, separate IAM roles per process": + +- **Ingestion process** — IAM role with `s3:Get/List` on the documents bucket and `s3:Get/Put/Delete` on the LanceDB bucket. Runs `haiku-rag serve --monitor` (with `monitor.s3` entries pointing at the documents bucket). Exactly one such process per LanceDB URI. +- **Consumer processes** (1..N) — IAM role with `s3:Get/List` on the LanceDB bucket only. Run `haiku-rag serve --read-only --mcp`, the chat TUI, etc. They never see the documents bucket. + +Each process picks up its own credentials from the AWS default chain (env vars, IAM instance role, AWS profile), so no credentials are hard-coded in the configuration files. + ## Database Creation Databases must be explicitly created before use: diff --git a/docs/server.md b/docs/server.md index 8917f4e6..bda27147 100644 --- a/docs/server.md +++ b/docs/server.md @@ -120,3 +120,53 @@ The file monitor processes documents using [Docling](https://github.com/DS4SD/do - RST (`.rst`) URLs are also supported - the content is fetched and converted to markdown. + +## S3 / Object Storage Monitoring + +The server can also poll S3-compatible object storage (AWS S3, SeaweedFS, MinIO, Cloudflare R2, etc.) for new, modified, and deleted objects, treating each one as a document source. + +Install the optional `[s3]` extra: + +```bash +pip install haiku.rag-slim[s3] +# or, for the full package: +pip install haiku.rag[s3] +``` + +Configure one or more S3 sources under `monitor.s3` in `haiku.rag.yaml`: + +```yaml +monitor: + s3: + - uri: s3://my-bucket/incoming/ + poll_interval: 300 # seconds between sweeps; default 300 + include_patterns: ["*.pdf", "*.md"] + ignore_patterns: ["draft*"] + delete_orphans: true + storage_options: + endpoint: http://seaweed:8333 + aws_access_key_id: ${AWS_KEY} + aws_secret_access_key: ${AWS_SECRET} + region: us-east-1 + allow_http: "true" +``` + +Then start the server with `--monitor` — the same flag enables both local-directory and S3 watchers: + +```bash +haiku-rag serve --monitor +``` + +Each entry in `monitor.s3` runs as its own polling task. On every sweep the watcher lists all objects under the configured prefix, compares each object's S3 ETag against the document's stored `metadata["etag"]`, and only re-fetches keys whose ETag has changed. When the bytes turn out to match the stored MD5 (e.g. the same file was re-uploaded with a different multipart chunk size), the watcher refreshes the etag and skips re-chunking. Otherwise the document is downloaded, chunked, and re-embedded. + +### Credentials + +`storage_options` follows the same convention as `lancedb.storage_options` — the dict is passed straight to obstore (the same Rust `object_store` library LanceDB uses internally), so any keys you've configured there work here too. When `storage_options` is omitted, the watcher falls back to the AWS default credential chain (environment variables, IAM instance role, AWS profile). + +### Orphan deletion scope + +`delete_orphans: true` is per-entry: a watcher only removes documents whose URI starts with that entry's `s3://bucket/prefix/`. Documents from other buckets, prefixes, or local-file sources are never touched. + +### One-off ingestion + +`s3://` URIs are also a first-class source for `haiku-rag add-src` and the MCP `add_document_from_url` tool — see [CLI → Add Documents](cli.md#add-documents).