diff --git a/CHANGELOG.md b/CHANGELOG.md index f0b89008..4d97f041 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,9 @@ ### Added -- New `haiku-ingester` service for continuous document ingestion: persistent SQLite job queue, async worker pool with retries and a dead-letter queue, FS/HTTP/S3/WebDAV source adapters with per-source circuit breakers, and a FastAPI control plane (`/health`, `/jobs`, `/sources`, `/dlq`). Configured under `ingester:` in `haiku.rag.yaml`. Shipped behind the `[ingester]` extra, with Logfire spans (`ingester.poller.sweep` → `ingester.job` → `document.{fetch,convert,chunk,embed,store}`, plus `document.convert_slice` per slice when splitting) for traceable ingestion. See [docs/ingester.md](docs/ingester.md). +- New `haiku-ingester` service for continuous document ingestion: persistent SQLite job queue, async worker pool with retries and a dead-letter queue, FS/HTTP/S3/WebDAV source adapters with per-source circuit breakers, and a FastAPI control plane on `127.0.0.1:8765` exposing `/health`, `/sources`, `/jobs`, `/dlq`, `/stats`, and a browser dashboard at `/`. Configured under `ingester:` in `haiku.rag.yaml`. Shipped behind the `[ingester]` extra, with Logfire spans (`ingester.poller.sweep` and `ingester.poller.watch_event` → `ingester.job` → `document.{fetch,convert,chunk,embed,store}`, plus `document.convert_slice` per slice when splitting) for traceable ingestion. See [docs/ingester.md](docs/ingester.md). +- Browser dashboard at `GET /` on the ingester's control plane. Single self-contained HTML page (no CDN, no external assets — works offline). Polls every 3s and renders: queue chips (queued / claimed / succeeded / dead), per-source state (last polled, circuit-breaker, "queue busy" badge when sweeps are skipped), rolling throughput (5m / 30m / 1h succeeded), worker occupancy, oldest queued age, per-source backlog and DLQ counts, currently-claimed jobs with cancel, recent failures with retry, and the last-completed feed. Dashboard route is unauthenticated; the JS attaches the bearer token to its own JSON fetches on a `401` (token stored in `localStorage`). +- `GET /stats` endpoint on the ingester control plane: rolling counts of succeeded jobs (last 5m / 30m / 1h), worker occupancy (`busy` / `total`), oldest queued job age in seconds, and per-source breakdowns of DLQ size and queue depth. One SQL aggregation per field; cheap to poll. - `processing.split_pages` (default `0`): split large PDFs into N-page slices, convert each independently through docling-local or docling-serve, and merge with `DoclingDocument.concatenate()`. Bounds peak working set on memory-hungry docs and lets multiple docling-serve replicas parallelize per-document. `0` disables (single-pass conversion). ### Removed @@ -13,10 +15,13 @@ ### Changed - `haiku-rag serve` renamed to `haiku-rag mcp` (only MCP is left). `--mcp-port` renamed to `--port`. Update any `claude_desktop_config.json` from `["serve", "--mcp", "--stdio"]` to `["mcp", "--stdio"]`. -- `document.metadata` now uses source-agnostic keys: `source_revision` (was `etag` — S3-only and never populated for FS, so periodic sweeps re-ingested every file) and `content_type` (was `contentType`, snake_case for consistency). The v0.50.0 startup migration rewrites existing documents. All four source adapters (FS, HTTP, S3, WebDAV) now write their native revision (mtime_ns, ETag, etc.) under the same key, fixing the regression where FS sources never short-circuited on unchanged files. +- `document.metadata` now uses source-agnostic keys: `source_revision` (was `etag` — S3-only and never populated for FS, so periodic sweeps re-ingested every file) and `content_type` (was `contentType`, snake_case for consistency). The v0.50.0 startup migration rewrites existing documents and compacts the `documents` table at the end so per-row UPDATE tombstones don't keep the table at roughly doubled size for the default 24-hour vacuum retention window (measured `12 GB → 25 GB` mid-migration on a 1000-doc PDF corpus, reclaimed back to `12 GB` by the in-migration compaction). All four source adapters (FS, HTTP, S3, WebDAV) now write their native revision (mtime_ns, ETag, etc.) under the same key, fixing the regression where FS sources never short-circuited on unchanged files. - Ingester pollers skip their periodic sweep when the source already has queued or claimed jobs in the queue — saves the listing round-trip (`PROPFIND` / `S3 LIST` / FS walk) when work is backed up. FS push events from `watchfiles` keep flowing during skipped sweeps. Visible in Logfire as `ingester.poller.sweep` spans with `skipped=true reason=pending_work`. - Ingester now drains in-flight jobs on `SIGINT` / `SIGTERM` up to `workers.shutdown_grace_s` (default 60s) before cancelling. Cancelled jobs release their claim back to `queued` (and decrement `attempts` since a cancel isn't a failure) so the next process picks them up immediately instead of waiting on the reaper's `claim_timeout_s`. Bonus: the pipeline no longer wraps `KeyboardInterrupt` / `SystemExit` / `CancelledError` as `TransientError` — those now propagate as intended. - `providers.docling_serve.base_url` now accepts a list. Jobs round-robin across the entries with each job's submit/poll/result pinned to one instance (task IDs are instance-local). The counter is per-process; for cross-process load balancing or failover, put an LB in front and pass a single URL here. +- `GET /health` now returns `workers_alive` and `pollers_alive` (counts of tasks still running) alongside the configured `worker_count` and `poller_count`. `status` is `"ok"` when both shortfalls are zero and `"degraded"` when a worker or poller has died — a DB-only liveness probe couldn't distinguish "process alive but doing nothing" from "process alive and healthy", and uptime monitors can now alert on the degraded state directly. +- FS source rejects any URI whose resolved path escapes the configured `root` directory. `supports()` returns `False`, `head()` returns `None`, and `fetch()` raises `UnsupportedSourceError`. `discover()` walks with `os.walk(followlinks=False)` and skips file-level symlinks, so a symlink under the watched root pointing at `/etc/passwd` (or any path outside the root) can't be ingested. Defense-in-depth for deployments where the watched directory is multi-writer. +- Logfire spans emitted by haiku.rag now report `instrumentation_scope.name = "haiku.rag"` (was the SDK default `logfire`). Cross-library instrumentations (pydantic-ai, docling-serve, etc.) keep their own scopes. Downstream consumers filtering OTel traces by source library can now match on `scope.name = haiku.rag` instead of catching everything the SDK exports. - Drop `list_documents` and `get_document` from the default RAG skill's tool set; the skill now exposes only `search` and `cite`. Both tools dumped unbounded content into the agent's context (full document lists, full document bodies) and `get_document` returned no chunk_ids so its output was structurally uncitable. The analysis skill already covers these uses programmatically — `await list_documents()` and `Path('/documents/{id}/content.txt').read_text()` inside `execute_code`. The tool branches remain in `create_skill_tools` and the `skill_generator` `AVAILABLE_TOOLS` set so users can still opt in when building custom skills. ## [0.48.1] - 2026-05-21 diff --git a/docs/configuration/processing.md b/docs/configuration/processing.md index 871a8323..f496059f 100644 --- a/docs/configuration/processing.md +++ b/docs/configuration/processing.md @@ -1,6 +1,8 @@ -# Document Processing & Monitoring +# Document Processing -This guide covers how haiku.rag converts, chunks, and monitors documents. +This guide covers how haiku.rag converts and chunks documents. Continuous +ingestion (watching directories, polling HTTP / S3 / WebDAV sources) lives +in the [ingester](../ingester.md) service. ## Document Processing diff --git a/docs/ingester.md b/docs/ingester.md index ca557412..68793b9e 100644 --- a/docs/ingester.md +++ b/docs/ingester.md @@ -242,8 +242,9 @@ token; without one the API stays open and the service logs a warning. | Method | Path | Purpose | |---|---|---| -| `GET` | `/health` | liveness + queue counts | -| `GET` | `/sources` | configured pollers + last-poll time + breaker state | +| `GET` | `/` | browser dashboard (HTML; unauthenticated, the JS attaches the bearer on its own JSON fetches) | +| `GET` | `/health` | liveness + queue counts + live worker/poller counts; `status` is `"ok"` or `"degraded"` | +| `GET` | `/sources` | configured pollers + last-poll time + breaker state + last skip reason | | `POST` | `/sources/{id}/refresh` | force an out-of-band sweep | | `GET` | `/jobs` | filtered list (`status`, `source_id`, `uri`, `limit`, `offset`) | | `GET` | `/jobs/{id}` | one job | @@ -251,8 +252,14 @@ token; without one the API stays open and the service logs a warning. | `DELETE` | `/jobs/{id}` | cancel a queued/claimed job | | `GET` | `/dlq` | dead jobs | | `POST` | `/dlq/{id}/retry` | resurrect from DLQ | +| `GET` | `/stats` | rolling throughput (5m / 30m / 1h succeeded), worker occupancy, oldest queued age, per-source DLQ + backlog | -OpenAPI docs at `http://localhost:8765/docs`. +OpenAPI docs at `http://localhost:8765/docs`. The dashboard at `/` polls +the JSON endpoints above every few seconds and surfaces the same data +visually — queue depth chips, per-source health with a `queue busy` badge +when sweeps are skipped, throughput counters, active jobs with a Cancel +button, recent failures with a Retry button, and the last-completed +feed. ```yaml ingester: