172 lines
5 KiB
Markdown
172 lines
5 KiB
Markdown
# Server Mode
|
|
|
|
The server provides automatic file monitoring and MCP functionality.
|
|
|
|
## Starting the Server
|
|
|
|
The `serve` command requires at least one service flag. You can enable file monitoring, MCP server, or both:
|
|
|
|
### MCP Server Only
|
|
|
|
```bash
|
|
haiku-rag serve --mcp
|
|
```
|
|
|
|
Transport options:
|
|
- Default - Streamable HTTP transport on port 8001
|
|
- `--stdio` - Standard input/output transport
|
|
- `--mcp-port` - Custom port (default: 8001)
|
|
|
|
### File Monitoring Only
|
|
|
|
```bash
|
|
haiku-rag serve --monitor
|
|
```
|
|
|
|
### Both Services
|
|
|
|
```bash
|
|
haiku-rag serve --monitor --mcp
|
|
```
|
|
|
|
This will start file monitoring and MCP server on port 8001.
|
|
|
|
## File Monitoring
|
|
|
|
Configure directories to monitor in your `haiku.rag.yaml` (see [Document Processing](configuration/processing.md#file-monitoring) for all options):
|
|
|
|
```yaml
|
|
monitor:
|
|
directories:
|
|
- /path/to/documents
|
|
- /another/path
|
|
```
|
|
|
|
Then start the server:
|
|
|
|
```bash
|
|
haiku-rag serve --monitor
|
|
```
|
|
|
|
### Monitoring Features
|
|
|
|
- **Startup**: Scans all monitored directories and adds new files
|
|
- **File Added/Modified**: Automatically parses and updates documents
|
|
- **File Deleted**: Removes corresponding documents from database
|
|
|
|
### Filtering Files
|
|
|
|
You can filter which files to monitor using gitignore-style patterns:
|
|
|
|
```yaml
|
|
monitor:
|
|
directories:
|
|
- /path/to/documents
|
|
|
|
# Ignore patterns (exclude files)
|
|
ignore_patterns:
|
|
- "*draft*" # Ignore draft files
|
|
- "temp/" # Ignore temp directory
|
|
- "**/archive/**" # Ignore archive directories
|
|
|
|
# Include patterns (whitelist files)
|
|
include_patterns:
|
|
- "*.md" # Only markdown files
|
|
- "**/docs/**" # Files in docs directories
|
|
```
|
|
|
|
**Pattern behavior:**
|
|
- Extension filtering is applied first (only supported file types)
|
|
- Include patterns create a whitelist (if specified)
|
|
- Ignore patterns exclude files
|
|
- Both can be combined for fine-grained control
|
|
|
|
### Supported Formats
|
|
|
|
The file monitor processes documents using [Docling](https://github.com/DS4SD/docling), which supports:
|
|
|
|
**Documents:**
|
|
- PDF (`.pdf`) - with OCR support for scanned documents
|
|
- Microsoft Word (`.docx`)
|
|
- Microsoft Excel (`.xlsx`)
|
|
- Microsoft PowerPoint (`.pptx`)
|
|
- HTML (`.html`, `.htm`)
|
|
- Markdown (`.md`)
|
|
- Quarto Markdown (`.qmd`)
|
|
- R Markdown (`.rmd`)
|
|
- LaTeX (`.tex`, `.latex`)
|
|
- AsciiDoc (`.adoc`, `.asciidoc`)
|
|
|
|
**Data formats:**
|
|
- CSV (`.csv`)
|
|
- JSON (`.json`)
|
|
- XML (`.xml`)
|
|
|
|
**Images (via OCR):**
|
|
- PNG (`.png`)
|
|
- JPEG (`.jpg`, `.jpeg`)
|
|
- TIFF (`.tiff`, `.tif`)
|
|
- BMP (`.bmp`)
|
|
|
|
**Code files:**
|
|
- Python (`.py`)
|
|
- JavaScript (`.js`)
|
|
- TypeScript (`.ts`)
|
|
- PlantUML (`.puml`, `.plantuml`, `.pu`)
|
|
- And other text-based code files
|
|
|
|
**Plain text:**
|
|
- Text files (`.txt`)
|
|
- RST (`.rst`)
|
|
|
|
URLs are also supported - the content is fetched and converted to markdown.
|
|
|
|
## S3 / Object Storage Monitoring
|
|
|
|
The server can also poll S3-compatible object storage (AWS S3, SeaweedFS, MinIO, Cloudflare R2, etc.) for new, modified, and deleted objects, treating each one as a document source.
|
|
|
|
Install the optional `[s3]` extra:
|
|
|
|
```bash
|
|
pip install haiku.rag-slim[s3]
|
|
# or, for the full package:
|
|
pip install haiku.rag[s3]
|
|
```
|
|
|
|
Configure one or more S3 sources under `monitor.s3` in `haiku.rag.yaml`:
|
|
|
|
```yaml
|
|
monitor:
|
|
s3:
|
|
- uri: s3://my-bucket/incoming/
|
|
poll_interval: 300 # seconds between sweeps; default 300
|
|
include_patterns: ["*.pdf", "*.md"]
|
|
ignore_patterns: ["draft*"]
|
|
delete_orphans: true
|
|
storage_options:
|
|
endpoint: http://seaweed:8333
|
|
aws_access_key_id: ${AWS_KEY}
|
|
aws_secret_access_key: ${AWS_SECRET}
|
|
region: us-east-1
|
|
allow_http: "true"
|
|
```
|
|
|
|
Then start the server with `--monitor` (the same flag enables both local-directory and S3 watchers):
|
|
|
|
```bash
|
|
haiku-rag serve --monitor
|
|
```
|
|
|
|
Each entry in `monitor.s3` runs as its own polling task. On every sweep the watcher lists all objects under the configured prefix, compares each object's S3 ETag against the document's stored `metadata["etag"]`, and only re-fetches keys whose ETag has changed. When the bytes turn out to match the stored MD5 (e.g. the same file was re-uploaded with a different multipart chunk size), the watcher refreshes the etag and skips re-chunking. Otherwise the document is downloaded, chunked, and re-embedded.
|
|
|
|
### Credentials
|
|
|
|
`storage_options` follows the same convention as `lancedb.storage_options`. The dict is passed straight to obstore (the same Rust `object_store` library LanceDB uses internally), so any keys you've configured there work here too. When `storage_options` is omitted, the watcher falls back to the AWS default credential chain (environment variables, IAM instance role, AWS profile).
|
|
|
|
### Orphan deletion scope
|
|
|
|
`delete_orphans: true` is per-entry: a watcher only removes documents whose URI starts with that entry's `s3://bucket/prefix/`. Documents from other buckets, prefixes, or local-file sources are never touched.
|
|
|
|
## One-off ingestion
|
|
|
|
`s3://` URIs are also a first-class source for `haiku-rag add-src` and the MCP `add_document_from_url` tool. See [CLI → Add Documents](cli.md#add-documents).
|