Document run-batch dry-run manifests
This commit is contained in:
parent
7251d104c4
commit
f0826abb52
2 changed files with 29 additions and 0 deletions
|
|
@ -1,6 +1,10 @@
|
|||
# Changelog
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- `haiku-ingester run-batch --dry-run` writes a YAML manifest of planned upserts/deletes without mutating queue jobs or `sync_state`; `run-batch --manifest <path>` replays that frozen changeset without another discovery sweep.
|
||||
|
||||
### Fixed
|
||||
|
||||
- CPU-bound ingest steps now run off the event loop: Docling document serialization, docling-serve zip parsing, split-PDF concatenation, fetched-body temp writes, and filesystem read/hash work no longer stall concurrent ingester workers on large image-bearing documents.
|
||||
|
|
|
|||
|
|
@ -524,6 +524,31 @@ haiku-ingester run-batch
|
|||
haiku-ingester run-batch --db rag.lancedb
|
||||
```
|
||||
|
||||
To review a batch before it mutates the document store, use `--dry-run`.
|
||||
Dry-run performs the same discovery checks but writes no queue jobs and does
|
||||
not update `sync_state`. It writes a YAML manifest named
|
||||
`manifest-<datestamp>.yaml` by default:
|
||||
|
||||
```bash
|
||||
haiku-ingester run-batch --dry-run
|
||||
haiku-ingester run-batch --dry-run --output manifest-20260622.yaml
|
||||
```
|
||||
|
||||
The manifest records the `upsert` and `delete` changes discovered for each
|
||||
source. Replay it later to ingest exactly that changeset, without another
|
||||
discovery sweep:
|
||||
|
||||
```bash
|
||||
haiku-ingester run-batch --manifest manifest-20260622.yaml
|
||||
```
|
||||
|
||||
Manifest replay rejects sources with queued or claimed work, preserving the
|
||||
one-active-changeset-per-source pattern. Revisioned upserts are checked
|
||||
against the current upstream revision before fetch; if the resource changed
|
||||
after dry-run, that job dead-letters and the newer version waits for the next
|
||||
dry-run. Sources that provide no revision can freeze URI discovery but cannot
|
||||
prove byte identity at replay time.
|
||||
|
||||
Orphan deletion compares each source against `sync_state` in the queue DB,
|
||||
so persist `ingester.db` between runs for deletions to be detected. It exits
|
||||
non-zero if any job dead-letters or a source's discovery sweep does not
|
||||
|
|
|
|||
Loading…
Reference in a new issue