Add `ingester.api.root_path` so the HTTP control plane (dashboard + API)
can be reverse-proxied behind a sub-path (e.g. /ingester/) on a shared
origin, instead of needing nginx sub_filter URL-rewriting.
- APIConfig.root_path: normalized ('', or single leading slash, no trailing
slash) via a field_validator; validate_assignment so CLI overrides
normalize the same way as config-file values.
- Forwarded to FastAPI(root_path=) and uvicorn.Config(root_path=) so
OpenAPI/docs links are prefix-aware.
- Dashboard route injects a <base href> matching root_path; all dashboard
fetches are now base-relative, so they resolve under the prefix while
staying identical at the root.
- `serve --root-path` CLI flag.
- Docs: "Behind a reverse proxy" section with an nginx example.
Closes#431
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate the ingester queue storage from raw aiosqlite to SQLAlchemy Core
async. The backend is chosen by ingester.queue.dburi: a SQLAlchemy async
URL points the queue at a database server, and SQLite remains the default
when unset. The Postgres path claims jobs with FOR UPDATE SKIP LOCKED so
multiple ingester processes can share one queue; SQLite caps the pool to a
single connection to keep the select-then-update claim atomic.
Large files buffered entirely in RAM can OOM workers. Add
max_file_size to source config (default None = no limit).
FS checks stat().st_size before read_bytes(). HTTP and WebDAV issue
a HEAD request before GET when a limit is configured. S3 checks the
size from the existing head_async() call before get_async().
FileTooLargeError is classified as PermanentError so oversized files
go straight to the DLQ instead of retrying.
Each discovered file previously triggered a separate sync.upsert()
call with its own lock acquire + SQLite commit (fsync). On a sweep
finding 1,000 files this meant 1,000 individual commits.
Collect sync_state rows into a list during the sweep and flush them
in a single SyncStateRepo.batch_upsert() call at the end — one lock
acquisition, one commit, one fsync.
HTTP, S3, and WebDAV sources all check `revision is not None and
snapshot.get(uri) == revision` to decide UPSERT vs UNCHANGED. When
a server returns no ETag or Last-Modified, revision is None and the
condition always fails — every sweep emits UPSERT even though the
content hasn't changed.
Now emit UNCHANGED when revision is None and the URI is already
known (has been ingested before). A first-time discovery with no
revision still correctly emits UPSERT.
