# Design: Audit trail for chunk edits - **Issue:** #205 - **Title on issue:** [FEATURE] Audit trail for chunk edits (who, when, before/after) - **Author:** Pier-Jean Malandrino - **Date:** 2026-04-29 - **Status:** Accepted - **Target milestone:** 0.6.0 — Doc-centric ingest - **Impacted layers:** backend: domain · persistence · services · api · frontend: features/chunking · shared - **Audit dimensions likely touched:** Hexagonal Architecture · DDD · Tests · Security · Documentation - **ADR spawned?:** ADR-NNN — "Chunks become a first-class persisted entity" --- ## 1. Problem Once chunks become editable in 0.6.0 (E5), production teams will need to answer "who changed what, when, and why is the answer suddenly different?". Without an audit trail, a regression caused by a chunk edit is impossible to investigate. Today the architecture makes this hard because **chunks are not persisted as first-class records**. They live as a JSON blob inside `analysis_jobs.chunks_json` and as derived rows in OpenSearch. There is no chunk identity that survives a re-parse, no row to attach an audit record to, and no way to retrieve "the version of these chunks at the last push". This issue elevates chunks to a first-class persisted entity (with stable IDs and content versioning), introduces an immutable `chunk_edits` table, and exposes a snapshot API consumed by the visual diff (#221). ## 2. Goals - [ ] Promote chunks to a first-class persisted entity with stable IDs across edits. - [ ] Every chunk operation (create / update / delete / merge / split) writes an immutable `chunk_edits` record with: actor, timestamp, action, before-state, after-state, optional reason. - [ ] API: `GET /api/documents/{id}/chunks/history` returns the edit timeline. - [ ] API: `GET /api/documents/{id}/chunks?at=` returns the chunkset snapshot at a given push. - [ ] Backwards compatible: existing reads of `analysis_jobs.chunks_json` keep working until #206 finishes the migration. ## 3. Non-goals - Undo / redo UI — out of scope (the audit trail is the substrate; UI comes later). - Rollback to an arbitrary past state — not in 0.6.0; use the snapshot API for read-only inspection. - A diff *view* — that is **#221**. - Per-chunk content hash — that is part of **#223**, not this issue. (#204's `chunkset_hash` is at the link granularity.) - Authentication / authorization model rework — `actor` defaults to a hard-coded `"system"` until the auth layer is ready. ## 4. Context & constraints ### Existing code surface - `document-parser/domain/value_objects.py` — `ChunkResult` (transient). - `document-parser/domain/models.py` — `AnalysisJob.chunks_json` (current chunkset storage). - `document-parser/services/analysis_service.py` — chunking pipeline. - `document-parser/persistence/database.py` — schema + migrations. - `document-parser/api/` — no chunks endpoint today; needs creation. - `frontend/src/features/chunking/` — Pinia store + API client. ### Hexagonal Architecture constraints - New domain entities `Chunk` and `ChunkEdit` live in `domain/models.py`. - `ChunkRepository` and `ChunkEditRepository` are ports (`domain/ports.py`). - aiosqlite adapters in `persistence/chunk_repo.py` and `persistence/chunk_edit_repo.py`. - A new `ChunkEditingService` in `services/` orchestrates the operations and writes audit records atomically with the chunk write. - The existing `analysis_jobs.chunks_json` becomes a **legacy fallback** — read by `ChunkRepository.list_for_doc()` if no rows exist in the new `chunks` table. #206 backfills. ### Hard constraints - Stable `chunk.id` across edits. A chunk that was split into two creates two new ids; merging two chunks produces a third new id. The "lineage" is recorded in `chunk_edits`. (No "ship the same id" hack.) - Immutable audit table. Once written, never updated. - No PII leak in audit records — `actor` is whatever the auth layer hands us; `before`/`after` payloads are the chunk text and metadata only. - Atomicity: an edit + its audit row are written in the same SQL transaction. ## 5. Proposed design ### 5.1 Domain `document-parser/domain/models.py`: ```python @dataclass class Chunk: id: str # uuid4 hex document_id: str sequence: int # ordering within the doc; gaps allowed text: str headings: list[str] source_page: int | None bboxes: list[Bbox] # carried for rendering; not part of identity doc_items: list[str] token_count: int | None created_at: datetime updated_at: datetime deleted_at: datetime | None # soft delete class ChunkEditAction(StrEnum): INSERT = "insert" UPDATE = "update" DELETE = "delete" MERGE = "merge" SPLIT = "split" @dataclass(frozen=True) class ChunkEdit: id: str document_id: str chunk_id: str | None # null on MERGE result row (uses children_ids) action: ChunkEditAction actor: str # "system" until auth lands at: datetime before: dict | None # JSON snapshot — None for INSERT after: dict | None # JSON snapshot — None for DELETE parents: list[str] # for SPLIT result rows: source chunk id; for MERGE: source ids children: list[str] # inverse links reason: str | None ``` Domain operations on a chunkset (`domain/chunk_editing.py`): ```python def insert(chunks: list[Chunk], at_position: int, new_chunk: Chunk) -> list[Chunk]: ... def update(chunks: list[Chunk], chunk_id: str, *, text: str, headings: list[str]) -> list[Chunk]: ... def delete(chunks: list[Chunk], chunk_id: str) -> list[Chunk]: ... def merge(chunks: list[Chunk], chunk_ids: list[str]) -> tuple[list[Chunk], Chunk]: """Returns the updated list and the new merged chunk.""" def split(chunks: list[Chunk], chunk_id: str, at_offset: int) -> tuple[list[Chunk], Chunk, Chunk]: """Returns the updated list and the two new chunks.""" ``` These are pure. The service wraps each call with audit-record generation. ### 5.2 Persistence ```sql CREATE TABLE IF NOT EXISTS chunks ( id TEXT PRIMARY KEY, document_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE, sequence INTEGER NOT NULL, text TEXT NOT NULL, headings TEXT NOT NULL DEFAULT '[]', -- JSON source_page INTEGER, bboxes TEXT NOT NULL DEFAULT '[]', -- JSON doc_items TEXT NOT NULL DEFAULT '[]', -- JSON token_count INTEGER, created_at TEXT NOT NULL, updated_at TEXT NOT NULL, deleted_at TEXT ); CREATE INDEX IF NOT EXISTS idx_chunks_doc ON chunks(document_id); CREATE INDEX IF NOT EXISTS idx_chunks_doc_seq ON chunks(document_id, sequence); CREATE TABLE IF NOT EXISTS chunk_edits ( id TEXT PRIMARY KEY, document_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE, chunk_id TEXT, action TEXT NOT NULL, actor TEXT NOT NULL DEFAULT 'system', at TEXT NOT NULL, before_json TEXT, after_json TEXT, parents_json TEXT NOT NULL DEFAULT '[]', children_json TEXT NOT NULL DEFAULT '[]', reason TEXT ); CREATE INDEX IF NOT EXISTS idx_chunk_edits_doc_at ON chunk_edits(document_id, at); CREATE INDEX IF NOT EXISTS idx_chunk_edits_chunk ON chunk_edits(chunk_id); -- Snapshot table — captures the chunkset hash at every successful push. CREATE TABLE IF NOT EXISTS chunk_pushes ( id TEXT PRIMARY KEY, document_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE, store_id TEXT NOT NULL REFERENCES stores(id) ON DELETE CASCADE, chunkset_hash TEXT NOT NULL, chunk_ids TEXT NOT NULL, -- JSON array of chunk ids in order at push time pushed_at TEXT NOT NULL ); CREATE INDEX IF NOT EXISTS idx_chunk_pushes_doc_store ON chunk_pushes(document_id, store_id); ``` Note: `chunk_pushes.chunk_ids` materialises the chunkset at push time. Combined with the immutable `chunk_edits` history, we can reconstruct any past chunkset by replay (slower) — but `chunk_ids` is the cheap path. ### 5.3 Infra adapters None. ### 5.4 Services `document-parser/services/chunk_editing_service.py`: ```python class ChunkEditingService: def __init__(self, chunks: ChunkRepository, edits: ChunkEditRepository, ...): ... async def insert(self, doc_id, at_position, payload, *, actor, reason=None) -> Chunk: ... async def update(self, doc_id, chunk_id, payload, *, actor, reason=None) -> Chunk: ... async def delete(self, doc_id, chunk_id, *, actor, reason=None) -> None: ... async def merge(self, doc_id, chunk_ids, *, actor, reason=None) -> Chunk: ... async def split(self, doc_id, chunk_id, at_offset, *, actor, reason=None) -> tuple[Chunk, Chunk]: ... async def history(self, doc_id) -> list[ChunkEdit]: ... async def chunkset_at(self, doc_id, push_id) -> list[Chunk]: ... ``` Every mutating method runs in a single SQL transaction: 1. Apply the domain operation. 2. Persist chunk rows (insert / update / soft-delete). 3. Insert `chunk_edits` row with full before/after snapshot. 4. Trigger #204's stale detection on the doc. `history()` — paginated (default 50, max 500), ordered `at DESC`. `chunkset_at(push_id)` — read `chunk_pushes.chunk_ids`, return the matching `chunks` rows in that order. `IngestionService.ingest()` (touched in #203) writes a row to `chunk_pushes` on success. ### 5.5 API New router `document-parser/api/chunks.py`: ``` GET /api/documents/{id}/chunks List current chunks GET /api/documents/{id}/chunks?at= Snapshot at push GET /api/documents/{id}/chunks/history Edit timeline (paginated) POST /api/documents/{id}/chunks Insert chunk PATCH /api/documents/{id}/chunks/{chunk_id} Update chunk DELETE /api/documents/{id}/chunks/{chunk_id} Soft-delete chunk POST /api/documents/{id}/chunks/merge { chunkIds: [...], reason? } → new chunk POST /api/documents/{id}/chunks/{chunk_id}/split { atOffset, reason? } → two new chunks ``` DTOs (`schemas.py`, camelCase via alias): ```python class ChunkResponse(BaseModel): id: str sequence: int text: str headings: list[str] sourcePage: int | None tokenCount: int | None bboxes: list[BboxDto] docItems: list[str] updatedAt: datetime class ChunkEditResponse(BaseModel): id: str chunkId: str | None action: str actor: str at: datetime before: dict | None after: dict | None parents: list[str] children: list[str] reason: str | None ``` ### 5.6 Frontend — feature module Touched: `frontend/src/features/chunking/`. - `api.ts` — clients for the seven endpoints above. - `store.ts` — Pinia store holds `chunks`, `history`, and a draft layer for optimistic updates with rollback on failure. - `ui/` — primitives only in this issue (`ChunkRow.vue`, `ChunkActionsMenu.vue`); the full editor is built in #219 / #220 / #221 on top. - `data-e2e` selectors named upfront: `chunk-row`, `chunk-actions-menu`, `chunk-action-merge`, etc. ### 5.7 Cross-cutting - Feature flag: existing `chunking` flag (in `/api/health`) gates the editing endpoints. When `false`, the endpoints return `403`. - i18n: `chunks.action.*`, `chunks.reason.placeholder` keys. - Shared types: `DocumentChunk`, `ChunkEdit` re-exported from `shared/types.ts`. ## 6. Alternatives considered ### Alternative A — Keep chunks in `analysis_jobs.chunks_json`, version the JSON - **Summary:** Add a `version` column on `analysis_jobs` and a `chunk_edits` table referencing JSON paths inside `chunks_json`. - **Why not:** No chunk identity → `before`/`after` lookups become regex on JSON. Splits and merges have nowhere to record lineage. Reviewers and customer support cannot navigate the audit. The architectural cost is paid sooner or later — pay it now while the data is small. ### Alternative B — Use OpenSearch as the source of truth for chunks - **Summary:** Treat the OpenSearch document as the authoritative chunk; record audit metadata there. - **Why not:** OpenSearch is a derived index. It is the *projection* of the corpus, not its source. Replacing OpenSearch (Pinecone, Qdrant) would lose history. Source of truth belongs in SQLite. ### Alternative C — Event-sourcing only (no current-state table) - **Summary:** Drop `chunks` table; replay `chunk_edits` to compute current state. - **Why not:** Replay cost on every read is unacceptable for chunks-editor latency and for #221's diff. Hybrid (current state + immutable history) is the standard pragmatic shape. ## 7. API & data contract ### Endpoints See §5.5 — eight new endpoints, all behind the existing `chunking` feature flag. ### Persistence schema See §5.2. ### Env vars / config | Name | Default | Allowed | Notes | |------|---------|---------|-------| | `CHUNK_HISTORY_PAGE_DEFAULT` | `50` | int 1–500 | default page size for history | | `CHUNK_HISTORY_PAGE_MAX` | `500` | int | hard cap | ### Breaking changes None. Existing chunks in `chunks_json` remain accessible via the legacy fallback in `ChunkRepository.list_for_doc()` until #206 finishes the migration. ## 8. Risks & mitigations | Risk | Audit dimension | Likelihood | Impact | How we notice | Mitigation / rollback | |------|-----------------|------------|--------|---------------|------------------------| | Audit table growth on heavy editing customers | Performance | Medium | Medium | Slow `chunk_edits` queries | Indexed by (doc, at). Pagination on the API. Future: archival job. | | Race between two parallel edits on the same chunk | DDD | Low | Medium | Lost update | All mutations go through `ChunkEditingService` which serialises via the SQL transaction; client sends `If-Match` with `updatedAt` for optimistic concurrency on `PATCH` / `DELETE` | | Audit before/after reveals sensitive content if shared | Security | Medium | Medium | Audit export shared accidentally | The audit endpoint is gated by the same auth as chunk edits; export does not include audit by default; admin-only flag in a future release | | Schema drift: `chunks_json` and `chunks` table diverge during migration window | DDD | Medium | High | Diff between the two on the same doc | #206 is the single source of truth; new edits write to `chunks` only; reads prefer `chunks` and fall back to `chunks_json` | ## 9. Testing strategy ### Backend — pytest - **Unit (domain):** - `test_chunk_editing_pure.py` — insert / update / delete / merge / split on in-memory lists; properties verified (lengths, sequences, identity rules). - **Persistence:** - `test_chunk_repo.py` — round-trip, soft delete, ordering by sequence. - `test_chunk_edit_repo.py` — write + read history; immutability (UPDATE on `chunk_edits` is rejected by a CHECK constraint or service-level guard). - **Services / integration:** - `test_chunk_editing_service_audit_atomicity.py` — failure mid-write rolls back both chunk and audit. - `test_chunkset_at_snapshot.py` — `chunkset_at(push_id)` returns the right snapshot after edits. - `test_legacy_fallback.py` — read chunks for a doc that has only `chunks_json` (pre-migration). ### Frontend — Vitest - `features/chunking/api.test.ts` — eight endpoints round-trip. - `features/chunking/store.test.ts` — optimistic update + rollback on API failure; history pagination. ### E2E — Karate UI Out of scope here; the editor lands in #219 / #220 with E2E tags `@critical @ui`. ### Manual QA 1. Trigger an INSERT / UPDATE / DELETE / MERGE / SPLIT. 2. `GET /api/documents/{id}/chunks/history` shows the action with `before`/`after`. 3. After a successful push, `GET /api/documents/{id}/chunks?at=` returns the chunkset captured at push. ## 10. Rollout & observability ### Release branch `release/0.6.0`. ### Feature flag The existing `chunking` flag (in `/api/health`) controls whether the editing endpoints accept writes. Reads (`GET`) are always allowed (consistent with the chunks tab being read-only when the flag is off). ### Observability - Each mutation logs: `INFO event=chunk_edit doc_id= chunk_id= action= actor=`. - Counter (Prometheus, future): `chunk_edits_total{action}`. ### Rollback plan The migration is additive. Reverting the code leaves the new tables but no writers. Reads from the chunks tab fall back to the legacy `chunks_json`. Editing endpoints disappear (404). If a clean rollback is needed, drop the three new tables. ## 11. Open questions - Should `actor` carry a structured object (id, name, role) or stay as a free-form string? **Decision for 0.6.0:** free-form string (`"system"` today; whatever auth provides later). Structured upgrade is non-breaking thanks to JSON-friendly columns. - Should we generate a *human-readable* diff in the audit row (line-level), or only store before/after JSON? **Decision:** store JSON; render the diff client-side in #221. ## 12. References - **Issue:** https://github.com/scub-france/Docling-Studio/issues/205 - **Related issues:** #202 (lifecycle), #203 (per-store), #204 (auto-stale), #206 (migration), #219 (editor view), #220 (edit actions), #221 (visual diff), #222 (push) - **ADRs:** ADR — "Chunks become a first-class persisted entity" - **Project docs:** - Architecture: `docs/architecture.md` - Coding standards: `docs/architecture/coding-standards.md` - ADR guide: `docs/architecture/adr-guide.md`