danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Document run-batch dry-run manifests

Browse source
This commit is contained in:
Yiorgis Gozadinos 2026-06-22 11:43:57 +03:00
parent 7251d104c4
commit f0826abb52
No known key found for this signature in database
2 changed files with 29 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
CHANGELOG.md
View file

@ -1,6 +1,10 @@
# Changelog # Changelog
## [Unreleased] ## [Unreleased]
### Added
- `haiku-ingester run-batch --dry-run` writes a YAML manifest of planned upserts/deletes without mutating queue jobs or `sync_state`; `run-batch --manifest <path>` replays that frozen changeset without another discovery sweep.
### Fixed ### Fixed
- CPU-bound ingest steps now run off the event loop: Docling document serialization, docling-serve zip parsing, split-PDF concatenation, fetched-body temp writes, and filesystem read/hash work no longer stall concurrent ingester workers on large image-bearing documents. - CPU-bound ingest steps now run off the event loop: Docling document serialization, docling-serve zip parsing, split-PDF concatenation, fetched-body temp writes, and filesystem read/hash work no longer stall concurrent ingester workers on large image-bearing documents.

25
docs/ingester.md
View file

@ -524,6 +524,31 @@ haiku-ingester run-batch
haiku-ingester run-batch --db rag.lancedb haiku-ingester run-batch --db rag.lancedb
``` ```
To review a batch before it mutates the document store, use `--dry-run`.
Dry-run performs the same discovery checks but writes no queue jobs and does
not update `sync_state`. It writes a YAML manifest named
`manifest-<datestamp>.yaml` by default:
```bash
haiku-ingester run-batch --dry-run
haiku-ingester run-batch --dry-run --output manifest-20260622.yaml
```
The manifest records the `upsert` and `delete` changes discovered for each
source. Replay it later to ingest exactly that changeset, without another
discovery sweep:
```bash
haiku-ingester run-batch --manifest manifest-20260622.yaml
```
Manifest replay rejects sources with queued or claimed work, preserving the
one-active-changeset-per-source pattern. Revisioned upserts are checked
against the current upstream revision before fetch; if the resource changed
after dry-run, that job dead-letters and the newer version waits for the next
dry-run. Sources that provide no revision can freeze URI discovery but cannot
prove byte identity at replay time.
Orphan deletion compares each source against `sync_state` in the queue DB, Orphan deletion compares each source against `sync_state` in the queue DB,
so persist `ingester.db` between runs for deletions to be detected. It exits so persist `ingester.db` between runs for deletions to be detected. It exits
non-zero if any job dead-letters or a source's discovery sweep does not non-zero if any job dead-letters or a source's discovery sweep does not