From f439d5e579330338f41fbfc9e6db6f1fde99c7ce Mon Sep 17 00:00:00 2001 From: Pier-Jean Malandrino Date: Wed, 29 Apr 2026 15:24:16 +0200 Subject: [PATCH] docs(design): E1 design docs for 0.6.0 doc-centric ingest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds technical design docs for the foundation of the doc-centric ingest pivot: - 202 — Document lifecycle state machine - 203 — Per (document, store) ingestion state - 204 — Auto-detect Stale state via chunk content hash - 205 — Audit trail for chunk edits (chunks → first-class entity) - 206 — Migration of existing documents to the new model Status: Accepted on all five. Each doc spells out the domain entities, persistence schema, services orchestration, API contract, alternatives considered, risks, and testing strategy. ADR placeholders called out where load-bearing decisions warrant a follow-up document. Refs #202 #203 #204 #205 #206 --- .../202-document-lifecycle-state-machine.md | 292 ++++++++++++++ docs/design/203-per-store-ingestion-state.md | 329 ++++++++++++++++ docs/design/204-auto-stale-chunk-hash.md | 265 +++++++++++++ docs/design/205-chunk-edits-audit-trail.md | 372 ++++++++++++++++++ docs/design/206-lifecycle-state-migration.md | 265 +++++++++++++ 5 files changed, 1523 insertions(+) create mode 100644 docs/design/202-document-lifecycle-state-machine.md create mode 100644 docs/design/203-per-store-ingestion-state.md create mode 100644 docs/design/204-auto-stale-chunk-hash.md create mode 100644 docs/design/205-chunk-edits-audit-trail.md create mode 100644 docs/design/206-lifecycle-state-migration.md diff --git a/docs/design/202-document-lifecycle-state-machine.md b/docs/design/202-document-lifecycle-state-machine.md new file mode 100644 index 0000000..db205cd --- /dev/null +++ b/docs/design/202-document-lifecycle-state-machine.md @@ -0,0 +1,292 @@ +# Design: Document lifecycle state machine + +- **Issue:** #202 +- **Title on issue:** [FEATURE] Introduce Document lifecycle state machine +- **Author:** Pier-Jean Malandrino +- **Date:** 2026-04-29 +- **Status:** Accepted +- **Target milestone:** 0.6.0 — Doc-centric ingest +- **Impacted layers:** backend: domain · persistence · services · api · frontend: features/document · shared +- **Audit dimensions likely touched:** Hexagonal Architecture · DDD · Tests · Documentation +- **ADR spawned?:** no + +--- + +## 1. Problem + +Studio today tracks ingestion as a side-effect of an `AnalysisJob`. The only "state" a document carries is implicit — derived by joining the document with its latest analysis job (`PENDING`, `RUNNING`, `COMPLETED`, `FAILED`) and, separately, by checking whether OpenSearch holds rows for `doc_id`. There is no explicit, persisted lifecycle of the document itself. + +This blocks the 0.6.0 doc-centric pivot: we cannot show a document library `/docs` with status badges, we cannot tell the user "this doc is ingested but stale on store X", and we cannot drive a chunks editor that knows whether its draft is committed. Every later feature in E3, E4, and E5 reads or transitions a document state. + +This design introduces a first-class **document lifecycle** as a domain concept: a single canonical state on each document, validated transitions, persisted, and surfaced via the API. + +## 2. Goals + +- [ ] Add a `DocumentLifecycleState` enum to the domain with six states: `Uploaded`, `Parsed`, `Chunked`, `Ingested`, `Stale`, `Failed`. +- [ ] `Document` carries `lifecycle_state` (current state) + `lifecycle_state_at` (last transition timestamp). +- [ ] State transitions are validated in the domain layer; invalid transitions raise `InvalidLifecycleTransition`. +- [ ] State changes emit a domain event `DocumentLifecycleChanged(previous, current, at)`. +- [ ] Persisted via aiosqlite migration; existing rows default to `Uploaded` (refined by #206). +- [ ] Surfaced on the document API DTO (camelCase: `lifecycleState`, `lifecycleStateAt`). + +## 3. Non-goals + +- Per-`(doc, store)` state — that is **#203** and uses a different table. +- Auto-stale detection via hash — that is **#204**; this issue only declares the `Stale` state value. +- UI rendering of the badge — that is **#215**. +- Migrating production data — that is **#206**; this issue ships the schema + default value only. +- Refactoring `AnalysisJob.status`. The two concepts coexist: `AnalysisJob.status` describes **a single conversion attempt**; `Document.lifecycle_state` describes **the doc as a whole**. + +## 4. Context & constraints + +### Existing code surface + +- `document-parser/domain/models.py` — `Document` dataclass (line 27), `AnalysisJob` (line 38), `AnalysisStatus` enum. +- `document-parser/domain/value_objects.py` — value objects. +- `document-parser/persistence/database.py` — `_MIGRATIONS` list and `_run_migrations()` checking PRAGMA `table_info()`. +- `document-parser/persistence/document_repo.py` — `DocumentRepository` (aiosqlite). +- `document-parser/api/documents.py` — REST endpoints. +- `document-parser/api/schemas.py` — Pydantic DTOs with `alias_generator=_to_camel`. +- `frontend/src/features/document/store.ts` and `api.ts`. + +### Hexagonal Architecture constraints (backend) + +- `DocumentLifecycleState` is a **value object** — it lives in `domain/value_objects.py` with zero imports from `api`/`persistence`/`infra`. +- The transition rules are domain logic — they live as methods on `Document` (or a small `DocumentLifecycle` policy module in `domain/`). No HTTP, no DB. +- `DocumentRepository` (the port) gains a write path for the new fields. The aiosqlite adapter is the only place that knows about the column. +- API layer translates the enum to camelCase string via Pydantic. + +### Deployment modes + +Both `latest-local` and `latest-remote` images use the same SQLite schema → migration applies to both. No HF Space-specific concern. No feature flag (the lifecycle is always on; only UI surfaces are flagged). + +### Hard constraints + +- SQLite schema additive only. No column drops, no rename. New column has a `DEFAULT` so existing rows do not break on read. +- API contract additive only. New fields appear; nothing is removed. +- `pages_json` stays snake_case (existing exception). + +## 5. Proposed design + +### 5.1 Domain + +Add to `document-parser/domain/value_objects.py`: + +```python +from enum import StrEnum + +class DocumentLifecycleState(StrEnum): + UPLOADED = "Uploaded" + PARSED = "Parsed" + CHUNKED = "Chunked" + INGESTED = "Ingested" + STALE = "Stale" + FAILED = "Failed" +``` + +Allowed transitions (a directed graph; `* → Failed` always allowed): + +``` +Uploaded → Parsed | Failed +Parsed → Chunked | Failed +Chunked → Ingested | Chunked | Failed # re-chunking is allowed +Ingested → Stale | Chunked | Failed # re-ingest stays Ingested via 203 +Stale → Ingested | Chunked | Failed +Failed → Uploaded | Parsed | Chunked # explicit retry sets the new target +``` + +`Stale` is set by the auto-detect logic (#204) — never reached as a result of a manual action. + +Add to `document-parser/domain/models.py`: + +```python +@dataclass +class Document: + ... + lifecycle_state: DocumentLifecycleState = DocumentLifecycleState.UPLOADED + lifecycle_state_at: datetime | None = None + + def transition_to(self, target: DocumentLifecycleState, *, now: datetime) -> "DocumentLifecycleChanged": + if not _is_allowed(self.lifecycle_state, target): + raise InvalidLifecycleTransition(self.lifecycle_state, target) + previous = self.lifecycle_state + self.lifecycle_state = target + self.lifecycle_state_at = now + return DocumentLifecycleChanged(self.id, previous, target, now) +``` + +`InvalidLifecycleTransition` lives in `domain/exceptions.py` (create if missing). `_is_allowed` is a pure function over a static transition table. + +`DocumentLifecycleChanged` is a frozen dataclass in `domain/events.py`. No event bus is wired in 0.6.0 — the event is *returned* from the transition call so services can log / persist / publish later. This avoids introducing infra in this issue. + +### 5.2 Persistence + +Schema migration appended to `_MIGRATIONS` in `document-parser/persistence/database.py`: + +```sql +ALTER TABLE documents ADD COLUMN lifecycle_state TEXT NOT NULL DEFAULT 'Uploaded'; +ALTER TABLE documents ADD COLUMN lifecycle_state_at TEXT; +CREATE INDEX IF NOT EXISTS idx_documents_lifecycle_state ON documents(lifecycle_state); +``` + +The `_run_migrations()` PRAGMA check ensures idempotency — re-running on an already-migrated DB is a no-op. The index supports the `/docs` library filter by status (#212). + +`DocumentRepository.save(doc)` and `update_lifecycle(doc_id, state, at)` are the two write paths. Read returns the columns directly populated into the dataclass. + +### 5.3 Infra adapters + +None. The lifecycle is database-only. + +### 5.4 Services + +`DocumentService` and `AnalysisService` are the two callers that drive transitions. + +| Trigger | From | To | Caller | +|---|---|---|---| +| Upload completes | — | `Uploaded` | `DocumentService.upload()` | +| Parse succeeds | `Uploaded` (or `Failed` retry) | `Parsed` | `AnalysisService` | +| Chunking succeeds | `Parsed` (or `Chunked` re-chunk) | `Chunked` | `AnalysisService` | +| Ingestion succeeds | `Chunked` (or `Stale`) | `Ingested` | `IngestionService` (touched in #203) | +| Failure on any pipeline step | any | `Failed` | the failing service | + +Transition calls are atomic with the underlying write (e.g., chunking writes `chunks_json` *and* transitions to `Chunked` in the same SQL transaction). + +### 5.5 API + +`DocumentResponse` (in `document-parser/api/schemas.py`) gains: + +```python +class DocumentResponse(BaseModel): + ... + lifecycle_state: str = Field(serialization_alias="lifecycleState") + lifecycle_state_at: datetime | None = Field(default=None, serialization_alias="lifecycleStateAt") +``` + +No new endpoint. The existing `GET /api/documents` and `GET /api/documents/{id}` start returning the two fields. No 4xx removal — purely additive. + +`POST /api/documents/{id}/lifecycle` is **not** added in this issue; transitions are driven by pipeline events, not by the user. + +### 5.6 Frontend — feature module + +Touched: `frontend/src/features/document/`. + +- `api.ts` — bump the `Document` type with `lifecycleState: DocumentLifecycleState` and `lifecycleStateAt: string | null`. +- `store.ts` — store retains the new fields as-is. +- No UI work — that is #211 / #215. This issue ships the typed surface only so #211 can render against it. + +`shared/types.ts` gains a re-exported `DocumentLifecycleState` union literal type matching the backend enum. + +### 5.7 Cross-cutting + +- No feature flag. The lifecycle is permanent infrastructure. +- i18n: tooltip strings for the six states are added in `shared/i18n.ts`, keyed `lifecycle.`. Used by #215. + +## 6. Alternatives considered + +### Alternative A — Reuse `AnalysisJob.status` + +- **Summary:** Promote `AnalysisJob.status` to the document's state, removing the separate `Document.lifecycle_state`. +- **Why not:** `AnalysisJob` has a 1:N relationship with `Document` (a doc can be re-analyzed). The status of the *latest* job is not the same thing as the lifecycle of the *document*. It also conflates parse/chunk/ingest into one state machine, which #202–#205 explicitly want to separate. + +### Alternative B — Computed state, no column + +- **Summary:** Derive the state on-the-fly from analysis-job rows + OpenSearch presence. +- **Why not:** Joining + remote calls on every document list query is too expensive for `/docs` with 1k+ rows. A persisted column lets us index it for the filter (#212). The cost of denormalisation is one extra write per pipeline step. + +## 7. API & data contract + +### Endpoints + +| Method | Path | Request | Response | Breaking? | +|--------|------|---------|----------|-----------| +| GET | `/api/documents` | — | `DocumentResponse[]` (now with `lifecycleState`, `lifecycleStateAt`) | No (additive) | +| GET | `/api/documents/{id}` | — | `DocumentResponse` (now with `lifecycleState`, `lifecycleStateAt`) | No (additive) | + +### Persistence schema + +```sql +ALTER TABLE documents ADD COLUMN lifecycle_state TEXT NOT NULL DEFAULT 'Uploaded'; +ALTER TABLE documents ADD COLUMN lifecycle_state_at TEXT; +CREATE INDEX IF NOT EXISTS idx_documents_lifecycle_state ON documents(lifecycle_state); +``` + +Existing rows default to `Uploaded`; #206 refines them. + +### Env vars / config + +None. + +### Breaking changes + +Additive only. + +## 8. Risks & mitigations + +| Risk | Audit dimension | Likelihood | Impact | How we notice | Mitigation / rollback | +|------|-----------------|------------|--------|---------------|------------------------| +| Drift between `lifecycle_state` and the actual pipeline state (e.g. crash mid-write leaves `Chunked` recorded but no chunks) | DDD | Medium | Medium | Failed `/docs` rendering; tests catch on integration | Wrap the data write + transition in a single SQL transaction; never write the transition before the data | +| Existing migrations break on environments mid-flight | CI/Build | Low | High | Migration test on a snapshot DB | The migration is purely additive (`ADD COLUMN ... DEFAULT`) — safe to apply on any version | +| Confusion between `AnalysisJob.status` and `Document.lifecycle_state` for newcomers | Documentation | High | Low | Reviews ask | A short paragraph in `docs/architecture.md` describing the two concepts side by side | +| `Failed` state without a known reason makes debugging harder | Tests / Documentation | Medium | Medium | Operator complaints | The transition payload carries an optional `reason: str` persisted in a small JSON column or in the analysis-job's `error_message` | + +## 9. Testing strategy + +### Backend — pytest (`document-parser/tests/`) + +- **Unit (`tests/domain/`):** + - `test_lifecycle_transitions.py` — table-driven test of every (from, to) pair: allowed → updates state, disallowed → raises `InvalidLifecycleTransition`. + - `test_lifecycle_event.py` — successful transition returns the right `DocumentLifecycleChanged` event. +- **Persistence (`tests/persistence/`):** + - `test_document_repo_lifecycle.py` — write a doc, read it back, lifecycle fields round-trip; default value is `Uploaded`; index exists. +- **Services / integration:** + - `test_analysis_service_transitions.py` — parse pipeline drives `Uploaded → Parsed`; chunking drives `Parsed → Chunked`; failure → `Failed`. + +### Frontend — Vitest + +- `features/document/api.test.ts` — DTO parses `lifecycleState` correctly into the typed enum. +- `features/document/store.test.ts` — store exposes `lifecycleState` on the cached document. + +### E2E — Karate UI + +Out of scope for this issue. E2E coverage lands with #211 (the library page that renders the badge). + +### Manual QA + +1. `docker compose -f docker-compose.dev.yml up`. +2. Upload a doc → `GET /api/documents/{id}` returns `"lifecycleState": "Uploaded"`. +3. Run analysis → state becomes `"Parsed"` then `"Chunked"`. +4. Force a failure (corrupt PDF) → state becomes `"Failed"`. + +## 10. Rollout & observability + +### Release branch + +`release/0.6.0` (this work). + +### Feature flag / staged rollout + +None. Lifecycle persistence is always on; UI surfaces are flagged separately (per #210). + +### Observability + +- Each transition is logged at `INFO` with structured keys: `event=lifecycle_changed doc_id= from= to=`. +- No new Prometheus counter in this issue (added in #211 if useful for the library page). + +### Rollback plan + +The migration is additive. Reverting the code (without dropping the column) leaves the column populated but unused — safe. If the column itself must go, write a follow-up migration; SQLite supports column drop since 3.35 and we run a recent enough version. + +## 11. Open questions + +- Should we record the `reason` for `Failed` transitions on the document directly, or rely on the linked `AnalysisJob.error_message`? **Decision for 0.6.0:** rely on `AnalysisJob.error_message`; revisit if multiple non-analysis sources of failure emerge. +- Do we want a typed `DocumentLifecycleEvent` table for audit, or are logs enough? **Decision:** logs only in 0.6.0; #205's `chunk_edits` table is the audit substrate, doc-level events can be added later if needed. + +## 12. References + +- **Issue:** https://github.com/scub-france/Docling-Studio/issues/202 +- **Related issues:** #203 (per-store state), #204 (auto-stale), #205 (audit trail), #206 (migration), #211 (library), #215 (status badges) +- **ADRs:** none planned +- **Project docs:** + - Architecture: `docs/architecture.md` + - Coding standards: `docs/architecture/coding-standards.md` + - Audit master: `docs/audit/master.md` diff --git a/docs/design/203-per-store-ingestion-state.md b/docs/design/203-per-store-ingestion-state.md new file mode 100644 index 0000000..39baa90 --- /dev/null +++ b/docs/design/203-per-store-ingestion-state.md @@ -0,0 +1,329 @@ +# Design: Per (document, store) ingestion state + +- **Issue:** #203 +- **Title on issue:** [FEATURE] Track ingestion state per (document, store) pair +- **Author:** Pier-Jean Malandrino +- **Date:** 2026-04-29 +- **Status:** Accepted +- **Target milestone:** 0.6.0 — Doc-centric ingest +- **Impacted layers:** backend: domain · persistence · services · api · infra · frontend: features/ingestion · features/document · shared +- **Audit dimensions likely touched:** Hexagonal Architecture · DDD · Decoupling · Tests · Documentation +- **ADR spawned?:** ADR-NNN — "Multiple vector stores as first-class entities" + +--- + +## 1. Problem + +In the current architecture, "the vector store" is a single OpenSearch index hard-coded to `docling-studio-chunks`. Documents either are or are not indexed in that one place. The new doc-centric model assumes documents can live in **multiple stores** simultaneously — `rh-corpus-v3`, `legal-v1`, a snapshot for a customer A/B — and each `(doc, store)` pair has its own state. + +This is also where the killer flow lives: a customer reports bad answers from a specific store; the eng opens the doc, sees that it is `Ingested` in `rh-corpus-v3` and `Stale` in `rh-corpus-v2`, fixes the chunks, and re-ingests **the right one**. Without a per-pair state, that flow is impossible. + +## 2. Goals + +- [ ] Introduce a `Store` entity in the domain (id, name, kind, config, embedder). +- [ ] Introduce a `DocumentStoreLink` entity tying a document to a store with its own state and metadata. +- [ ] States on the link: `Ingested`, `Stale`, `Failed`. (No `Uploaded`/`Parsed`/`Chunked` — those are doc-level only.) +- [ ] The `Document.lifecycle_state` (#202) aggregates the per-store states by a documented rule. +- [ ] API exposes the per-store list on the document DTO. +- [ ] One default `Store` row is seeded on first migration so existing single-index deployments keep working. + +## 3. Non-goals + +- Auto-stale detection logic — that is **#204**; this issue ships the schema slot for the content hash but not the detection. +- Multi-tenant isolation between stores — the existing tenant model (if any) is not changed; stores are global today. +- A UI to create/manage stores — that is a follow-up. For 0.6.0 we ship the data model and the seed; a single store is enough for the killer flow on a single tenant. +- Cross-doc bulk re-ingest — that is **#213**; this issue exposes the per-link state needed by it. + +## 4. Context & constraints + +### Existing code surface + +- `document-parser/domain/ports.py` — `VectorStore` protocol (line 124). +- `document-parser/infra/opensearch_store.py` — `OpenSearchStore` adapter. +- `document-parser/services/ingestion_service.py` — `IngestionService.ingest()` and `ensure_index()`. +- `document-parser/persistence/database.py` — schema + migrations. +- `document-parser/api/ingestion.py` — endpoints. +- `frontend/src/features/ingestion/` — Pinia store + API client. + +### Hexagonal Architecture constraints (backend) + +- `Store` and `DocumentStoreLink` are domain entities (`domain/models.py`). Pure data + invariants. No ORM concern. +- `StoreRepository` and `DocumentStoreLinkRepository` are ports (`domain/ports.py`). +- aiosqlite adapters live in `persistence/store_repo.py` and `persistence/document_store_link_repo.py`. +- The existing `VectorStore` port stays — it represents the **technology** (OpenSearch). The new `Store` entity represents the **logical store** (a named, configurable target). One `VectorStore` adapter can serve many `Store` entities by namespacing the index name. + +### Deployment modes + +- Single `OpenSearchStore` adapter handles all stores via per-store `index_name = f"docling-studio-{store.slug}"`. Existing default index gets migrated to `docling-studio-default` (with a redirect alias to keep backwards-compatible reads). +- HF Space deployment ships with the same one-store seed. + +### Hard constraints + +- Existing OpenSearch index must remain readable during migration; reads against the legacy name are aliased. +- API is additive: existing `/api/ingestion/{analysis_id}` keeps working with an implicit default-store target until the UI explicitly picks one (#222). +- Tests must not require a real OpenSearch instance for the link layer — repo tests stub the adapter. + +## 5. Proposed design + +### 5.1 Domain + +`document-parser/domain/value_objects.py`: + +```python +class StoreKind(StrEnum): + OPENSEARCH = "opensearch" + # future: PINECONE, QDRANT, … + +class DocumentStoreLinkState(StrEnum): + INGESTED = "Ingested" + STALE = "Stale" + FAILED = "Failed" +``` + +`document-parser/domain/models.py`: + +```python +@dataclass +class Store: + id: str + name: str # "rh-corpus-v3" + slug: str # "rh-corpus-v3" (URL-safe; usually = name) + kind: StoreKind + embedder: str # e.g. "bge-m3" — record the embedder used for this store + config: dict # adapter-specific config (index_name override, etc.) + created_at: datetime + is_default: bool + +@dataclass +class DocumentStoreLink: + id: str + document_id: str + store_id: str + state: DocumentStoreLinkState + chunkset_hash: str | None # set by #204 on push + last_push_at: datetime | None + last_run_id: str | None # FK to a future runs table; nullable in 0.6.0 + error_message: str | None # populated on FAILED state + + def mark_ingested(self, *, hash_: str, at: datetime, run_id: str | None) -> None: ... + def mark_stale(self, *, at: datetime) -> None: ... + def mark_failed(self, *, at: datetime, error: str) -> None: ... +``` + +Aggregation rule for `Document.lifecycle_state` (defined as a pure function in `domain/lifecycle_aggregation.py`): + +``` +if any link state == FAILED → Document is FAILED +elif any link state == STALE → Document is STALE +elif any link state == INGESTED → Document is INGESTED +elif chunks present (state Chunked) → Document keeps its current state +else → keep current state +``` + +The aggregation runs as a side effect of any link write. The doc's own state column (#202) is the materialized result; it is not recomputed at read time. + +### 5.2 Persistence + +```sql +CREATE TABLE IF NOT EXISTS stores ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + slug TEXT NOT NULL UNIQUE, + kind TEXT NOT NULL, + embedder TEXT NOT NULL, + config TEXT NOT NULL DEFAULT '{}', -- JSON + is_default INTEGER NOT NULL DEFAULT 0, + created_at TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS document_store_links ( + id TEXT PRIMARY KEY, + document_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE, + store_id TEXT NOT NULL REFERENCES stores(id) ON DELETE CASCADE, + state TEXT NOT NULL, + chunkset_hash TEXT, + last_push_at TEXT, + last_run_id TEXT, + error_message TEXT, + UNIQUE (document_id, store_id) +); + +CREATE INDEX IF NOT EXISTS idx_dsl_doc ON document_store_links(document_id); +CREATE INDEX IF NOT EXISTS idx_dsl_store ON document_store_links(store_id); +CREATE INDEX IF NOT EXISTS idx_dsl_state ON document_store_links(state); +``` + +Seed migration inserts one row in `stores`: + +```sql +INSERT OR IGNORE INTO stores (id, name, slug, kind, embedder, is_default, created_at) +VALUES ('default', 'default', 'default', 'opensearch', + '', + 1, datetime('now')); +``` + +The OpenSearch index name for `slug=default` is `docling-studio-default`. To keep existing data readable, an alias migration in `OpenSearchStore.ensure_index()` adds `docling-studio-chunks` as a read-only alias to `docling-studio-default` if the legacy index exists with rows. Operators with non-trivial data trigger a one-shot reindex via #206. + +### 5.3 Infra adapters + +`OpenSearchStore` is parameterised by store slug: + +```python +class OpenSearchStore(VectorStore): + def __init__(self, client, index_prefix: str = "docling-studio") -> None: ... + def _index_for(self, store_slug: str) -> str: + return f"{self.index_prefix}-{store_slug}" +``` + +Calls become `index_chunks(store_slug, doc_id, chunks, embeddings)` etc. The protocol in `domain/ports.py` is updated to take `store_slug` as an explicit argument. + +### 5.4 Services + +New: `StoreService` (`document-parser/services/store_service.py`) — list, get, create (admin only — locked behind a future flag; for 0.6.0 only seeded rows exist). + +`IngestionService.ingest(analysis_id, store_slug = "default")`: +1. Read chunks from `analysis_jobs.chunks_json`. +2. Compute embeddings. +3. Call adapter `index_chunks(store_slug, ...)`. +4. Upsert link via `DocumentStoreLinkRepository.upsert(doc_id, store_id, state=Ingested, chunkset_hash=..., at=now)`. +5. Recompute document aggregate state via the rule in §5.1. + +If any step fails: link is marked `Failed` with the error; doc state aggregates to `Failed`. + +### 5.5 API + +`schemas.py`: + +```python +class StoreLinkResponse(BaseModel): + storeId: str + storeName: str + state: str # DocumentStoreLinkState value + chunksetHash: str | None + lastPushAt: datetime | None + lastRunId: str | None + +class DocumentResponse(BaseModel): + ... # from #202 + stores: list[StoreLinkResponse] = Field(default_factory=list) +``` + +`stores` is a read-side aggregate computed by `DocumentService.find_by_id()` joining the link table. + +New endpoint `GET /api/stores` returns the list of stores (id, name, slug, embedder). Used by #222's target picker. + +### 5.6 Frontend — feature module + +- `frontend/src/features/ingestion/` — `Ingestion.run()` accepts an optional `storeSlug`; defaults to "default" if not supplied. +- New `frontend/src/features/stores/` — Pinia store, API client, types. +- `frontend/src/features/document/api.ts` — `Document` type gains `stores: StoreLink[]`. + +### 5.7 Cross-cutting + +- Feature flag: none. +- i18n: `stores.state.` keys in `shared/i18n.ts`. +- New env var documented in `.env.example`: `DEFAULT_EMBEDDER` (existing implicitly; now formalised in the seed). + +## 6. Alternatives considered + +### Alternative A — One row per (doc, store) embedded in the analysis job + +- **Summary:** Add a JSON column `stores_json` on `analysis_jobs`. +- **Why not:** Conflates an *analysis attempt* with a *long-lived ingestion link*. Re-analysis would erase per-store state. The relational shape is necessary for filters, indexes, and idempotent updates. + +### Alternative B — One adapter per store (parallel `VectorStore` instances) + +- **Summary:** Hold a registry of `VectorStore` adapters keyed by store id. +- **Why not:** Duplicates connection pools, breaks singleton pattern in FastAPI's dependency wiring, and forces N OpenSearch clients for what is logically one client serving N indexes. Parameterising the existing single adapter is cheaper. + +## 7. API & data contract + +### Endpoints + +| Method | Path | Request | Response | Breaking? | +|--------|------|---------|----------|-----------| +| GET | `/api/documents/{id}` | — | now includes `stores: StoreLinkResponse[]` | No (additive) | +| GET | `/api/stores` | — | `StoreResponse[]` | No (new) | +| POST | `/api/ingestion/{analysis_id}` | `{ "storeSlug": "default" }` (optional) | unchanged | No (additive) | + +### Persistence schema + +See §5.2. + +### Env vars / config + +| Name | Default | Allowed | Notes | +|------|---------|---------|-------| +| `DEFAULT_EMBEDDER` | `bge-m3` | any registered embedder slug | recorded on the seeded `default` store | +| `DOCLING_STUDIO_INDEX_PREFIX` | `docling-studio` | URL-safe slug | adapter-level prefix for OpenSearch indexes | + +### Breaking changes + +Additive. + +## 8. Risks & mitigations + +| Risk | Audit dimension | Likelihood | Impact | How we notice | Mitigation / rollback | +|------|-----------------|------------|--------|---------------|------------------------| +| Existing OpenSearch data invisible after rename | Decoupling | Medium | High | Smoke test on staging | Read alias `docling-studio-chunks → docling-studio-default` set in `ensure_index()`; explicit reindex documented in runbook | +| Multiple stores with conflicting embedders mixed at search time | Security/Performance | Low | Medium | Mismatched dim error from OpenSearch | Embedder is recorded per store; queries route by store; cross-store search is out of scope for 0.6.0 | +| Link table grows large over time on prod corpus | Performance | Medium | Medium | Slow `/api/documents/{id}` join | Index on `document_id` and `store_id`; pagination on the doc list (#211) | +| Aggregation rule drifts from per-store reality on partial writes | DDD | Medium | High | Tests fail | Aggregation runs in the same transaction as the link write | + +## 9. Testing strategy + +### Backend — pytest + +- **Unit (domain):** `test_store_link_transitions.py`, `test_lifecycle_aggregation.py` (table-driven over per-store state combinations). +- **Persistence:** `test_store_repo.py`, `test_document_store_link_repo.py` (round-trip; UNIQUE constraint enforced; cascade delete works). +- **Services / integration:** `test_ingestion_service_with_store.py` (default store path), `test_ingestion_service_two_stores.py` (push to two stores → two links). +- **Architecture:** import-boundary test ensures `domain/` does not import `infra/opensearch_store.py`. + +### Frontend — Vitest + +- `features/stores/api.test.ts` — list endpoint round-trip. +- `features/document/store.test.ts` — `Document.stores` populated from API. + +### E2E — Karate UI + +Out of scope here. Lands with #211 / #218. + +### Manual QA + +1. Boot. `GET /api/stores` returns the seeded `default`. +2. Upload a doc, run ingestion → `GET /api/documents/{id}` returns `stores[0].state == "Ingested"`. +3. Inspect SQLite: `document_store_links` has the row. + +## 10. Rollout & observability + +### Release branch + +`release/0.6.0`. + +### Feature flag + +None for the data; #222 will gate the multi-store UI. + +### Observability + +- Log `store_link_changed` with `doc_id`, `store_id`, `from`, `to`, `at`. +- Counter (Prometheus, future): `ingestion_links_total{state}`. Optional in 0.6.0. + +### Rollback plan + +The migration is additive. To roll back the code: revert; the unused tables stay empty for new docs but contain rows for already-pushed docs — harmless (no read path uses them after revert). For full cleanup, a follow-up migration drops the two tables. + +## 11. Open questions + +- Should `Store.config` be typed in the domain (via a discriminated union on `kind`)? **Decision for 0.6.0:** keep as opaque `dict`; introduce a typed wrapper when we add a second `kind`. +- Cross-store search (single query → many stores) — **explicitly punted** to a later release. + +## 12. References + +- **Issue:** https://github.com/scub-france/Docling-Studio/issues/203 +- **Related issues:** #202 (lifecycle), #204 (hash), #205 (audit), #206 (migration), #211 (library), #213 (bulk push), #222 (push UI), #223 (diff-aware ingest) +- **ADRs:** ADR — "Multiple vector stores as first-class entities" (to be drafted alongside the implementation PR) +- **Project docs:** + - Architecture: `docs/architecture.md` + - Coding standards: `docs/architecture/coding-standards.md` + - ADR guide: `docs/architecture/adr-guide.md` diff --git a/docs/design/204-auto-stale-chunk-hash.md b/docs/design/204-auto-stale-chunk-hash.md new file mode 100644 index 0000000..d909e97 --- /dev/null +++ b/docs/design/204-auto-stale-chunk-hash.md @@ -0,0 +1,265 @@ +# Design: Auto-detect Stale state via chunk content hash + +- **Issue:** #204 +- **Title on issue:** [FEATURE] Auto-detect Stale state via chunk content hash +- **Author:** Pier-Jean Malandrino +- **Date:** 2026-04-29 +- **Status:** Accepted +- **Target milestone:** 0.6.0 — Doc-centric ingest +- **Impacted layers:** backend: domain · persistence · services · frontend (read-only) +- **Audit dimensions likely touched:** Hexagonal Architecture · DDD · Performance · Tests · Security +- **ADR spawned?:** no + +--- + +## 1. Problem + +Without auto-detection, a user who edits a chunk has to remember which stores hold the doc and click "re-ingest" everywhere. They will forget. The next query against the stale store returns the old embedding, and the customer reports "I fixed it yesterday and it's still wrong". Manual flagging is a bug factory. + +The detection contract is: **the system, not the user, knows when a stored chunkset diverges from the current draft chunkset.** Implementation = a deterministic hash recorded at push time and compared on read or on chunk write. + +## 2. Goals + +- [ ] Define a deterministic `chunkset_hash(chunks: list[ChunkResult])` function. +- [ ] Store the hash on each `DocumentStoreLink` at push time (#203 ships the column slot). +- [ ] On any chunk modification, recompute the current chunkset hash and compare against each link; if mismatch, mark the link `Stale` and re-aggregate the document state. +- [ ] On document read (single-doc API), compare on the fly as a safety net so older drift is caught. +- [ ] Unit tests prove: edit a chunk → link state becomes `Stale`; re-push → link state becomes `Ingested` with the new hash. + +## 3. Non-goals + +- Per-chunk change tracking — that's the audit trail (#205); this issue cares about the *aggregate* hash only. +- Background sweeper job that scans the whole corpus for drift — for 0.6.0, detection is event-driven (on chunk write) + on-read; a sweeper can come later. +- Diff-aware re-ingest at the chunk granularity — that's #223; this issue tells you *whether* a re-ingest is needed, not which chunks to re-embed. +- A user-facing toggle to "force stale" — out of scope. + +## 4. Context & constraints + +### Existing code surface + +- `document-parser/domain/value_objects.py` — `ChunkResult` (line 93). +- `document-parser/services/analysis_service.py` — chunking pipeline. +- `document-parser/persistence/database.py` — `analysis_jobs.chunks_json` (the canonical chunkset today). +- `document-parser/services/ingestion_service.py` — push pipeline. +- `document-parser/api/ingestion.py` — push endpoint. +- `document-parser/persistence/document_store_link_repo.py` (created by #203). + +### Hexagonal Architecture constraints + +- `chunkset_hash` is a **pure domain function** — lives in `domain/hashing.py`. No I/O. No timestamps. Deterministic over the input. +- Detection (compare + transition) is orchestrated in services. Persistence reads the stored hash via the link repo. +- The hash is opaque to API/frontend in 0.6.0; surfaced only as a debug field on `StoreLinkResponse` (#203 added it). + +### Hard constraints + +- Hash function must be **stable across processes / machines / Python versions** — so SHA-256 (`hashlib`), not Python `hash()`. No salt. +- Must be cheap on a 500-chunk doc — a single linear pass, no JSON re-parse on hot path. +- Result is a hex string, length 64. Stored as `TEXT` in SQLite. + +## 5. Proposed design + +### 5.1 Domain + +`document-parser/domain/hashing.py`: + +```python +import hashlib +import json +from collections.abc import Iterable +from .value_objects import ChunkResult + +def chunkset_hash(chunks: Iterable[ChunkResult]) -> str: + """ + Deterministic hash over a chunkset. + + Hashed inputs (per chunk, in chunkset order): + - text (str) + - source_page (int | None) + - headings (list[str], preserved order) + + Excluded: + - bboxes / doc_items (rendering artefacts; do not affect retrieval semantics) + - token_count (derived; unstable across tokenizers) + """ + h = hashlib.sha256() + for chunk in chunks: + payload = { + "t": chunk.text, + "p": chunk.source_page, + "h": list(chunk.headings or []), + } + h.update(b"\x1f") + h.update(json.dumps(payload, ensure_ascii=False, separators=(",", ":")).encode()) + return h.hexdigest() +``` + +Notes: +- The `\x1f` (Unit Separator) byte between chunks defends against the "join attack" where chunk A's tail and chunk B's head produce the same hash as chunk A+B merged. +- `separators=(",", ":")` is the canonical compact JSON form. +- The exclusion list is intentional and documented inline — changing it bumps every doc to `Stale` once and is a deliberate one-time event (covered in §10 Rollback). + +`detect_stale_links(current_hash, links) -> list[DocumentStoreLink]` — pure helper returning the subset of links whose `chunkset_hash != current_hash`. + +### 5.2 Persistence + +No new schema. The `chunkset_hash` column was added by #203 on `document_store_links`. + +A new index supports the on-read safety net: + +```sql +CREATE INDEX IF NOT EXISTS idx_dsl_doc_state ON document_store_links(document_id, state); +``` + +### 5.3 Infra adapters + +None. Hashing is in-memory. + +### 5.4 Services + +Two call sites: + +**A. On chunk write (event-driven detection)** + +`AnalysisService.persist_chunks(doc_id, chunks)`: +1. Persist chunks (existing path: `analysis_jobs.chunks_json`). +2. Compute `current_hash = chunkset_hash(chunks)`. +3. Read all `DocumentStoreLink` rows for `doc_id`. +4. For each link with `chunkset_hash != current_hash` and `state in (Ingested,)`: + - `link.mark_stale(at=now)` and persist. +5. Re-aggregate the document state (#202). + +This runs in the same transaction as the chunk write. The full pass is O(N_links) which is small (a doc rarely lives in more than a handful of stores). + +**B. On push completion (set the new hash)** + +Already covered by #203's `IngestionService.ingest`. Add the `current_hash` argument when calling `link.mark_ingested(hash_=current_hash, at=now, run_id=...)`. + +**C. On read (safety net)** + +`DocumentService.find_by_id(id)`: +1. Fetch document + links (existing). +2. Compute `current_hash` (cheap; cached per request). +3. Mark any `Ingested` link with mismatched hash as `Stale` and persist (best-effort, swallowed if write fails — log). + +This guards against drift caused by direct DB writes / restored backups / older deploys. + +### 5.5 API + +No new endpoint. `StoreLinkResponse.chunksetHash` (already added by #203) is now actually populated. + +### 5.6 Frontend — feature module + +No changes in this issue. #224 (stale indicator) reads the existing field. + +### 5.7 Cross-cutting + +- Feature flag: none. +- Logs: `INFO event=stale_detected doc_id= store_id= previous_hash=<8> current_hash=<8>` (truncated for privacy / log volume). +- ADR: not required. The choice of SHA-256 + canonical JSON is documented inline in `hashing.py`. + +## 6. Alternatives considered + +### Alternative A — Per-chunk hashes only (no chunkset hash) + +- **Summary:** Skip the aggregate hash; track each chunk's hash and detect "any per-chunk hash drift". +- **Why not:** Per-chunk hashing is needed for #223 (diff-aware re-embed) but not for "is this link stale at all?". A single `chunkset_hash` is one-comparison cheap; per-chunk is N-comparisons. Both will exist after #223 lands; this issue ships the cheap top-level signal. + +### Alternative B — Store updated_at instead of hash + +- **Summary:** Compare `chunks.updated_at > link.last_push_at`. +- **Why not:** Brittle. Re-running a pipeline on identical input bumps `updated_at` without semantic change. Hash is content-addressed and survives idempotent rewrites. + +### Alternative C — MD5 / xxHash + +- **Summary:** Use a faster non-cryptographic hash. +- **Why not:** SHA-256 is fast enough on the volumes in scope (`<500 chunks`, microseconds in CPython hashlib via OpenSSL bindings). The cryptographic hash also gives us collision resistance for free. xxHash would require an extra dependency. + +## 7. API & data contract + +### Endpoints + +No additions. Existing `StoreLinkResponse.chunksetHash` becomes populated. + +### Persistence schema + +```sql +CREATE INDEX IF NOT EXISTS idx_dsl_doc_state ON document_store_links(document_id, state); +``` + +### Env vars / config + +None. + +### Breaking changes + +None. + +## 8. Risks & mitigations + +| Risk | Audit dimension | Likelihood | Impact | How we notice | Mitigation / rollback | +|------|-----------------|------------|--------|---------------|------------------------| +| Hash function changes (e.g. someone adds `bboxes` to the input) silently invalidates every link | DDD | Medium | High | Every doc flips to `Stale` post-deploy | Hash function is in a single file with a docstring listing the canonical inputs; CI fixture asserts a fixed hash for a known chunkset. Bumping requires updating the fixture deliberately. | +| Read-side detection updates state under a GET (write-on-read) | Performance / Decoupling | Low | Medium | Slow / unexpected SQL in read paths | Best-effort, swallowed write; only triggered when the API actually serves the doc detail page (already a write-allowed code path). Disabled if a query param `?refresh=false` is set (future). | +| Unicode normalization issues (NFC vs NFD) produce different hashes for "the same" text | Tests | Low | Medium | Drift after a copy-paste from a Mac | Document the policy: text is stored as-is, no normalization; if drift appears, normalize at write time, not at hash time. | +| JSON ordering instability across Python versions | Tests | Low | High | Hash mismatch on different prod nodes | `separators=(",", ":")` + `ensure_ascii=False` + explicit key order in the dict literal. Reviewer checklist mentions this. | + +## 9. Testing strategy + +### Backend — pytest + +- **Unit (domain):** + - `test_chunkset_hash_determinism.py` — same input → same output across multiple invocations. + - `test_chunkset_hash_sensitivity.py` — every input field in the canonical list changes the hash; excluded fields (`bboxes`, `token_count`) do not. + - `test_chunkset_hash_join_attack.py` — separating into different chunks must produce different hash from concatenation. + - **Locked fixture** `test_chunkset_hash_fixture.py` — a hand-built 3-chunk input whose hash is hard-coded; CI fails if anyone changes the function silently. +- **Services:** + - `test_stale_detection_on_edit.py` — edit a chunk → link state becomes `Stale`. + - `test_stale_clears_on_repush.py` — re-push → link state becomes `Ingested` with the new hash. + - `test_stale_safety_net_on_read.py` — direct DB tampering → next read flips state to `Stale`. + +### Frontend — Vitest + +None new in this issue. + +### E2E — Karate UI + +Out of scope here; lands with #224. + +### Manual QA + +1. Push a doc to the default store → `chunksetHash` populated, `state == "Ingested"`. +2. Edit a chunk via API → `state == "Stale"`. +3. Re-push → `state == "Ingested"`, new hash. + +## 10. Rollout & observability + +### Release branch + +`release/0.6.0`. + +### Feature flag + +None. Detection is always on; cheap; correctness improvement. + +### Observability + +- Log lines as in §5.7. +- One-time bump scenario: if we ever change the canonical input list, every link will flip to `Stale` once. That is a deliberate decision; the operator must be informed via release notes, and a one-shot reindex job is recommended (out of scope here). + +### Rollback plan + +The migration is additive (a new index). Reverting the code leaves the existing `chunkset_hash` column populated but unused — harmless. The index can be dropped in a follow-up. + +## 11. Open questions + +- Should the safety-net read-side check be opt-in via a query param? **Decision:** always-on for 0.6.0; revisit if the cost shows up in profiles. +- Should headings include the *path* (parent → leaf) or just the leaf? **Decision:** the full ordered list as it sits on `ChunkResult.headings`. If the source mutates the list semantics, that is a separate domain concern. + +## 12. References + +- **Issue:** https://github.com/scub-france/Docling-Studio/issues/204 +- **Related issues:** #202 (lifecycle), #203 (per-store state), #205 (audit), #206 (migration), #223 (diff-aware re-ingest), #224 (stale indicator) +- **ADRs:** none planned +- **Project docs:** + - Architecture: `docs/architecture.md` + - Coding standards: `docs/architecture/coding-standards.md` diff --git a/docs/design/205-chunk-edits-audit-trail.md b/docs/design/205-chunk-edits-audit-trail.md new file mode 100644 index 0000000..a2a9023 --- /dev/null +++ b/docs/design/205-chunk-edits-audit-trail.md @@ -0,0 +1,372 @@ +# Design: Audit trail for chunk edits + +- **Issue:** #205 +- **Title on issue:** [FEATURE] Audit trail for chunk edits (who, when, before/after) +- **Author:** Pier-Jean Malandrino +- **Date:** 2026-04-29 +- **Status:** Accepted +- **Target milestone:** 0.6.0 — Doc-centric ingest +- **Impacted layers:** backend: domain · persistence · services · api · frontend: features/chunking · shared +- **Audit dimensions likely touched:** Hexagonal Architecture · DDD · Tests · Security · Documentation +- **ADR spawned?:** ADR-NNN — "Chunks become a first-class persisted entity" + +--- + +## 1. Problem + +Once chunks become editable in 0.6.0 (E5), production teams will need to answer "who changed what, when, and why is the answer suddenly different?". Without an audit trail, a regression caused by a chunk edit is impossible to investigate. + +Today the architecture makes this hard because **chunks are not persisted as first-class records**. They live as a JSON blob inside `analysis_jobs.chunks_json` and as derived rows in OpenSearch. There is no chunk identity that survives a re-parse, no row to attach an audit record to, and no way to retrieve "the version of these chunks at the last push". + +This issue elevates chunks to a first-class persisted entity (with stable IDs and content versioning), introduces an immutable `chunk_edits` table, and exposes a snapshot API consumed by the visual diff (#221). + +## 2. Goals + +- [ ] Promote chunks to a first-class persisted entity with stable IDs across edits. +- [ ] Every chunk operation (create / update / delete / merge / split) writes an immutable `chunk_edits` record with: actor, timestamp, action, before-state, after-state, optional reason. +- [ ] API: `GET /api/documents/{id}/chunks/history` returns the edit timeline. +- [ ] API: `GET /api/documents/{id}/chunks?at=` returns the chunkset snapshot at a given push. +- [ ] Backwards compatible: existing reads of `analysis_jobs.chunks_json` keep working until #206 finishes the migration. + +## 3. Non-goals + +- Undo / redo UI — out of scope (the audit trail is the substrate; UI comes later). +- Rollback to an arbitrary past state — not in 0.6.0; use the snapshot API for read-only inspection. +- A diff *view* — that is **#221**. +- Per-chunk content hash — that is part of **#223**, not this issue. (#204's `chunkset_hash` is at the link granularity.) +- Authentication / authorization model rework — `actor` defaults to a hard-coded `"system"` until the auth layer is ready. + +## 4. Context & constraints + +### Existing code surface + +- `document-parser/domain/value_objects.py` — `ChunkResult` (transient). +- `document-parser/domain/models.py` — `AnalysisJob.chunks_json` (current chunkset storage). +- `document-parser/services/analysis_service.py` — chunking pipeline. +- `document-parser/persistence/database.py` — schema + migrations. +- `document-parser/api/` — no chunks endpoint today; needs creation. +- `frontend/src/features/chunking/` — Pinia store + API client. + +### Hexagonal Architecture constraints + +- New domain entities `Chunk` and `ChunkEdit` live in `domain/models.py`. +- `ChunkRepository` and `ChunkEditRepository` are ports (`domain/ports.py`). +- aiosqlite adapters in `persistence/chunk_repo.py` and `persistence/chunk_edit_repo.py`. +- A new `ChunkEditingService` in `services/` orchestrates the operations and writes audit records atomically with the chunk write. +- The existing `analysis_jobs.chunks_json` becomes a **legacy fallback** — read by `ChunkRepository.list_for_doc()` if no rows exist in the new `chunks` table. #206 backfills. + +### Hard constraints + +- Stable `chunk.id` across edits. A chunk that was split into two creates two new ids; merging two chunks produces a third new id. The "lineage" is recorded in `chunk_edits`. (No "ship the same id" hack.) +- Immutable audit table. Once written, never updated. +- No PII leak in audit records — `actor` is whatever the auth layer hands us; `before`/`after` payloads are the chunk text and metadata only. +- Atomicity: an edit + its audit row are written in the same SQL transaction. + +## 5. Proposed design + +### 5.1 Domain + +`document-parser/domain/models.py`: + +```python +@dataclass +class Chunk: + id: str # uuid4 hex + document_id: str + sequence: int # ordering within the doc; gaps allowed + text: str + headings: list[str] + source_page: int | None + bboxes: list[Bbox] # carried for rendering; not part of identity + doc_items: list[str] + token_count: int | None + created_at: datetime + updated_at: datetime + deleted_at: datetime | None # soft delete + +class ChunkEditAction(StrEnum): + INSERT = "insert" + UPDATE = "update" + DELETE = "delete" + MERGE = "merge" + SPLIT = "split" + +@dataclass(frozen=True) +class ChunkEdit: + id: str + document_id: str + chunk_id: str | None # null on MERGE result row (uses children_ids) + action: ChunkEditAction + actor: str # "system" until auth lands + at: datetime + before: dict | None # JSON snapshot — None for INSERT + after: dict | None # JSON snapshot — None for DELETE + parents: list[str] # for SPLIT result rows: source chunk id; for MERGE: source ids + children: list[str] # inverse links + reason: str | None +``` + +Domain operations on a chunkset (`domain/chunk_editing.py`): + +```python +def insert(chunks: list[Chunk], at_position: int, new_chunk: Chunk) -> list[Chunk]: ... +def update(chunks: list[Chunk], chunk_id: str, *, text: str, headings: list[str]) -> list[Chunk]: ... +def delete(chunks: list[Chunk], chunk_id: str) -> list[Chunk]: ... +def merge(chunks: list[Chunk], chunk_ids: list[str]) -> tuple[list[Chunk], Chunk]: + """Returns the updated list and the new merged chunk.""" +def split(chunks: list[Chunk], chunk_id: str, at_offset: int) -> tuple[list[Chunk], Chunk, Chunk]: + """Returns the updated list and the two new chunks.""" +``` + +These are pure. The service wraps each call with audit-record generation. + +### 5.2 Persistence + +```sql +CREATE TABLE IF NOT EXISTS chunks ( + id TEXT PRIMARY KEY, + document_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE, + sequence INTEGER NOT NULL, + text TEXT NOT NULL, + headings TEXT NOT NULL DEFAULT '[]', -- JSON + source_page INTEGER, + bboxes TEXT NOT NULL DEFAULT '[]', -- JSON + doc_items TEXT NOT NULL DEFAULT '[]', -- JSON + token_count INTEGER, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + deleted_at TEXT +); +CREATE INDEX IF NOT EXISTS idx_chunks_doc ON chunks(document_id); +CREATE INDEX IF NOT EXISTS idx_chunks_doc_seq ON chunks(document_id, sequence); + +CREATE TABLE IF NOT EXISTS chunk_edits ( + id TEXT PRIMARY KEY, + document_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE, + chunk_id TEXT, + action TEXT NOT NULL, + actor TEXT NOT NULL DEFAULT 'system', + at TEXT NOT NULL, + before_json TEXT, + after_json TEXT, + parents_json TEXT NOT NULL DEFAULT '[]', + children_json TEXT NOT NULL DEFAULT '[]', + reason TEXT +); +CREATE INDEX IF NOT EXISTS idx_chunk_edits_doc_at ON chunk_edits(document_id, at); +CREATE INDEX IF NOT EXISTS idx_chunk_edits_chunk ON chunk_edits(chunk_id); + +-- Snapshot table — captures the chunkset hash at every successful push. +CREATE TABLE IF NOT EXISTS chunk_pushes ( + id TEXT PRIMARY KEY, + document_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE, + store_id TEXT NOT NULL REFERENCES stores(id) ON DELETE CASCADE, + chunkset_hash TEXT NOT NULL, + chunk_ids TEXT NOT NULL, -- JSON array of chunk ids in order at push time + pushed_at TEXT NOT NULL +); +CREATE INDEX IF NOT EXISTS idx_chunk_pushes_doc_store ON chunk_pushes(document_id, store_id); +``` + +Note: `chunk_pushes.chunk_ids` materialises the chunkset at push time. Combined with the immutable `chunk_edits` history, we can reconstruct any past chunkset by replay (slower) — but `chunk_ids` is the cheap path. + +### 5.3 Infra adapters + +None. + +### 5.4 Services + +`document-parser/services/chunk_editing_service.py`: + +```python +class ChunkEditingService: + def __init__(self, chunks: ChunkRepository, edits: ChunkEditRepository, ...): ... + + async def insert(self, doc_id, at_position, payload, *, actor, reason=None) -> Chunk: ... + async def update(self, doc_id, chunk_id, payload, *, actor, reason=None) -> Chunk: ... + async def delete(self, doc_id, chunk_id, *, actor, reason=None) -> None: ... + async def merge(self, doc_id, chunk_ids, *, actor, reason=None) -> Chunk: ... + async def split(self, doc_id, chunk_id, at_offset, *, actor, reason=None) -> tuple[Chunk, Chunk]: ... + + async def history(self, doc_id) -> list[ChunkEdit]: ... + async def chunkset_at(self, doc_id, push_id) -> list[Chunk]: ... +``` + +Every mutating method runs in a single SQL transaction: +1. Apply the domain operation. +2. Persist chunk rows (insert / update / soft-delete). +3. Insert `chunk_edits` row with full before/after snapshot. +4. Trigger #204's stale detection on the doc. + +`history()` — paginated (default 50, max 500), ordered `at DESC`. + +`chunkset_at(push_id)` — read `chunk_pushes.chunk_ids`, return the matching `chunks` rows in that order. + +`IngestionService.ingest()` (touched in #203) writes a row to `chunk_pushes` on success. + +### 5.5 API + +New router `document-parser/api/chunks.py`: + +``` +GET /api/documents/{id}/chunks List current chunks +GET /api/documents/{id}/chunks?at= Snapshot at push +GET /api/documents/{id}/chunks/history Edit timeline (paginated) +POST /api/documents/{id}/chunks Insert chunk +PATCH /api/documents/{id}/chunks/{chunk_id} Update chunk +DELETE /api/documents/{id}/chunks/{chunk_id} Soft-delete chunk +POST /api/documents/{id}/chunks/merge { chunkIds: [...], reason? } → new chunk +POST /api/documents/{id}/chunks/{chunk_id}/split { atOffset, reason? } → two new chunks +``` + +DTOs (`schemas.py`, camelCase via alias): + +```python +class ChunkResponse(BaseModel): + id: str + sequence: int + text: str + headings: list[str] + sourcePage: int | None + tokenCount: int | None + bboxes: list[BboxDto] + docItems: list[str] + updatedAt: datetime + +class ChunkEditResponse(BaseModel): + id: str + chunkId: str | None + action: str + actor: str + at: datetime + before: dict | None + after: dict | None + parents: list[str] + children: list[str] + reason: str | None +``` + +### 5.6 Frontend — feature module + +Touched: `frontend/src/features/chunking/`. + +- `api.ts` — clients for the seven endpoints above. +- `store.ts` — Pinia store holds `chunks`, `history`, and a draft layer for optimistic updates with rollback on failure. +- `ui/` — primitives only in this issue (`ChunkRow.vue`, `ChunkActionsMenu.vue`); the full editor is built in #219 / #220 / #221 on top. +- `data-e2e` selectors named upfront: `chunk-row`, `chunk-actions-menu`, `chunk-action-merge`, etc. + +### 5.7 Cross-cutting + +- Feature flag: existing `chunking` flag (in `/api/health`) gates the editing endpoints. When `false`, the endpoints return `403`. +- i18n: `chunks.action.*`, `chunks.reason.placeholder` keys. +- Shared types: `DocumentChunk`, `ChunkEdit` re-exported from `shared/types.ts`. + +## 6. Alternatives considered + +### Alternative A — Keep chunks in `analysis_jobs.chunks_json`, version the JSON + +- **Summary:** Add a `version` column on `analysis_jobs` and a `chunk_edits` table referencing JSON paths inside `chunks_json`. +- **Why not:** No chunk identity → `before`/`after` lookups become regex on JSON. Splits and merges have nowhere to record lineage. Reviewers and customer support cannot navigate the audit. The architectural cost is paid sooner or later — pay it now while the data is small. + +### Alternative B — Use OpenSearch as the source of truth for chunks + +- **Summary:** Treat the OpenSearch document as the authoritative chunk; record audit metadata there. +- **Why not:** OpenSearch is a derived index. It is the *projection* of the corpus, not its source. Replacing OpenSearch (Pinecone, Qdrant) would lose history. Source of truth belongs in SQLite. + +### Alternative C — Event-sourcing only (no current-state table) + +- **Summary:** Drop `chunks` table; replay `chunk_edits` to compute current state. +- **Why not:** Replay cost on every read is unacceptable for chunks-editor latency and for #221's diff. Hybrid (current state + immutable history) is the standard pragmatic shape. + +## 7. API & data contract + +### Endpoints + +See §5.5 — eight new endpoints, all behind the existing `chunking` feature flag. + +### Persistence schema + +See §5.2. + +### Env vars / config + +| Name | Default | Allowed | Notes | +|------|---------|---------|-------| +| `CHUNK_HISTORY_PAGE_DEFAULT` | `50` | int 1–500 | default page size for history | +| `CHUNK_HISTORY_PAGE_MAX` | `500` | int | hard cap | + +### Breaking changes + +None. Existing chunks in `chunks_json` remain accessible via the legacy fallback in `ChunkRepository.list_for_doc()` until #206 finishes the migration. + +## 8. Risks & mitigations + +| Risk | Audit dimension | Likelihood | Impact | How we notice | Mitigation / rollback | +|------|-----------------|------------|--------|---------------|------------------------| +| Audit table growth on heavy editing customers | Performance | Medium | Medium | Slow `chunk_edits` queries | Indexed by (doc, at). Pagination on the API. Future: archival job. | +| Race between two parallel edits on the same chunk | DDD | Low | Medium | Lost update | All mutations go through `ChunkEditingService` which serialises via the SQL transaction; client sends `If-Match` with `updatedAt` for optimistic concurrency on `PATCH` / `DELETE` | +| Audit before/after reveals sensitive content if shared | Security | Medium | Medium | Audit export shared accidentally | The audit endpoint is gated by the same auth as chunk edits; export does not include audit by default; admin-only flag in a future release | +| Schema drift: `chunks_json` and `chunks` table diverge during migration window | DDD | Medium | High | Diff between the two on the same doc | #206 is the single source of truth; new edits write to `chunks` only; reads prefer `chunks` and fall back to `chunks_json` | + +## 9. Testing strategy + +### Backend — pytest + +- **Unit (domain):** + - `test_chunk_editing_pure.py` — insert / update / delete / merge / split on in-memory lists; properties verified (lengths, sequences, identity rules). +- **Persistence:** + - `test_chunk_repo.py` — round-trip, soft delete, ordering by sequence. + - `test_chunk_edit_repo.py` — write + read history; immutability (UPDATE on `chunk_edits` is rejected by a CHECK constraint or service-level guard). +- **Services / integration:** + - `test_chunk_editing_service_audit_atomicity.py` — failure mid-write rolls back both chunk and audit. + - `test_chunkset_at_snapshot.py` — `chunkset_at(push_id)` returns the right snapshot after edits. + - `test_legacy_fallback.py` — read chunks for a doc that has only `chunks_json` (pre-migration). + +### Frontend — Vitest + +- `features/chunking/api.test.ts` — eight endpoints round-trip. +- `features/chunking/store.test.ts` — optimistic update + rollback on API failure; history pagination. + +### E2E — Karate UI + +Out of scope here; the editor lands in #219 / #220 with E2E tags `@critical @ui`. + +### Manual QA + +1. Trigger an INSERT / UPDATE / DELETE / MERGE / SPLIT. +2. `GET /api/documents/{id}/chunks/history` shows the action with `before`/`after`. +3. After a successful push, `GET /api/documents/{id}/chunks?at=` returns the chunkset captured at push. + +## 10. Rollout & observability + +### Release branch + +`release/0.6.0`. + +### Feature flag + +The existing `chunking` flag (in `/api/health`) controls whether the editing endpoints accept writes. Reads (`GET`) are always allowed (consistent with the chunks tab being read-only when the flag is off). + +### Observability + +- Each mutation logs: `INFO event=chunk_edit doc_id= chunk_id= action= actor=`. +- Counter (Prometheus, future): `chunk_edits_total{action}`. + +### Rollback plan + +The migration is additive. Reverting the code leaves the new tables but no writers. Reads from the chunks tab fall back to the legacy `chunks_json`. Editing endpoints disappear (404). If a clean rollback is needed, drop the three new tables. + +## 11. Open questions + +- Should `actor` carry a structured object (id, name, role) or stay as a free-form string? **Decision for 0.6.0:** free-form string (`"system"` today; whatever auth provides later). Structured upgrade is non-breaking thanks to JSON-friendly columns. +- Should we generate a *human-readable* diff in the audit row (line-level), or only store before/after JSON? **Decision:** store JSON; render the diff client-side in #221. + +## 12. References + +- **Issue:** https://github.com/scub-france/Docling-Studio/issues/205 +- **Related issues:** #202 (lifecycle), #203 (per-store), #204 (auto-stale), #206 (migration), #219 (editor view), #220 (edit actions), #221 (visual diff), #222 (push) +- **ADRs:** ADR — "Chunks become a first-class persisted entity" +- **Project docs:** + - Architecture: `docs/architecture.md` + - Coding standards: `docs/architecture/coding-standards.md` + - ADR guide: `docs/architecture/adr-guide.md` diff --git a/docs/design/206-lifecycle-state-migration.md b/docs/design/206-lifecycle-state-migration.md new file mode 100644 index 0000000..b61ad01 --- /dev/null +++ b/docs/design/206-lifecycle-state-migration.md @@ -0,0 +1,265 @@ +# Design: Migrate existing documents to the new lifecycle state model + +- **Issue:** #206 +- **Title on issue:** [CHORE] Migrate existing documents to the new lifecycle state model +- **Author:** Pier-Jean Malandrino +- **Date:** 2026-04-29 +- **Status:** Accepted +- **Target milestone:** 0.6.0 — Doc-centric ingest +- **Impacted layers:** backend: persistence · services · infra (CLI) · frontend (none) +- **Audit dimensions likely touched:** CI/Build · Tests · Documentation · Security +- **ADR spawned?:** no + +--- + +## 1. Problem + +#202, #203, and #205 introduce four new tables / columns: `documents.lifecycle_state`, `stores`, `document_store_links`, `chunks`, `chunk_edits`, `chunk_pushes`. Existing tenants already have documents, analysis jobs, and chunks living in `analysis_jobs.chunks_json` plus rows in OpenSearch. After deploy, these documents must appear in `/docs` with sensible state — otherwise the page looks broken (every doc shows `Uploaded` or unknown). + +This issue ships an idempotent migration that backfills: +- `documents.lifecycle_state` and `lifecycle_state_at`. +- One `stores` row (the `default` seeded by #203, no-op if already present). +- One `document_store_links` row per document already represented in OpenSearch under the legacy index. +- `chunks` rows materialized from existing `analysis_jobs.chunks_json`. +- `chunk_pushes` rows reconstructed for documents currently indexed. + +It also reindexes from `docling-studio-chunks` to `docling-studio-default` if needed, so the new store-aware code can read them. + +## 2. Goals + +- [ ] Idempotent CLI command `python -m document_parser.tools.migrate_06` (re-runs safely; second run is a no-op). +- [ ] `--dry-run` flag prints what would change without writing. +- [ ] Inference rules below produce a sensible state for every existing document. +- [ ] Per-tenant counts logged at the end (parsed=N, chunked=N, ingested=N, failed=N). +- [ ] Documented in `docs/runbooks/release-0.6.0-migration.md`. +- [ ] Tested on a snapshot DB representing the three relevant pre-states (no analysis, completed analysis, indexed in OpenSearch). + +## 3. Non-goals + +- Re-running ingestion. The migration only touches metadata; it does not re-embed. +- A web UI for migration progress — operator-only, run via CLI. +- Migrating tenants on environments that have already adopted parts of the new schema (e.g. a partial pre-release deploy) — out of scope; an emergency hotfix path is documented separately. +- Moving to a new vector backend — out of scope. + +## 4. Context & constraints + +### Existing code surface + +- `document-parser/persistence/database.py` — `_run_migrations()` runs schema DDL on app start. +- `document-parser/persistence/document_repo.py`, `analysis_repo.py` — read paths. +- `document-parser/infra/opensearch_store.py` — read access to existing OpenSearch index. +- New repos from #203 / #205: `store_repo.py`, `document_store_link_repo.py`, `chunk_repo.py`, `chunk_edit_repo.py`. + +### Hexagonal Architecture constraints + +- The migration is a **CLI script** in `document-parser/tools/migrate_06.py`. It uses repositories only — never raw SQL outside of repo modules. This keeps the upgrade path testable like any service. +- The reindex helper (`docling-studio-chunks → docling-studio-default`) lives in `infra/opensearch_store.py` as a method, called by the script. + +### Hard constraints + +- Idempotent: re-running on an already-migrated DB makes zero writes. +- Resumable: a crash mid-run can be restarted; checkpoints stored in a tiny `migration_progress` table (rows: `(name, completed_at)`). +- No assumption about uptime: the script must work on a quiesced DB OR a live one (transactions short, single-row scope). +- Read-only on OpenSearch by default; write happens only when the operator explicitly passes `--reindex-default-store`. + +### Deployment modes + +Run once per environment (`local`, `staging`, `prod`). HF Space deployments — same script, run by the deploy step before the new image starts serving traffic. + +## 5. Proposed design + +### 5.1 Domain + +No new domain code. The migration is a coordinator over existing repos. + +### 5.2 Persistence + +The schema additions ship in #202 / #203 / #205 migrations (already applied at app start). This issue contributes: + +```sql +CREATE TABLE IF NOT EXISTS migration_progress ( + name TEXT PRIMARY KEY, + completed_at TEXT NOT NULL +); +``` + +This is the only new table. It records each migration step's completion to enable resumability. + +### 5.3 Infra adapters + +`OpenSearchStore.copy_legacy_to_default(*, dry_run: bool) -> int` — +1. If `docling-studio-chunks` exists and `docling-studio-default` does not, create the new index with the same mapping. +2. Reindex via `_reindex_op` (OpenSearch reindex API) from legacy → default. +3. Add a read-only alias `docling-studio-chunks → docling-studio-default`. +4. Returns the number of documents copied. + +### 5.4 Services + +`document-parser/services/migration_06_service.py` — orchestrator with one method per step. Each step is bracketed by: + +```python +if not progress.is_done(step_name): + do_step(...) + progress.mark_done(step_name) +``` + +Steps: + +1. **`seed_default_store`** — `INSERT OR IGNORE INTO stores (...) VALUES ('default', ...)`. (Already in #203's migration; this step is a guard for older deploys that skipped the seed.) +2. **`backfill_document_lifecycle_state`** — for each document, infer state: + - Has at least one `analysis_jobs` row with `status=COMPLETED` and a non-null `chunks_json` → `Chunked` (refined below). + - Has at least one row indexed in OpenSearch (legacy or default index) → `Ingested`. + - Has only `analysis_jobs` rows with `status=COMPLETED` and no `chunks_json` → `Parsed`. + - Has only `analysis_jobs` rows with `status=FAILED` → `Failed`. + - Else → `Uploaded`. +3. **`materialize_chunks_from_chunks_json`** — for each `analysis_jobs` row with non-null `chunks_json`, parse and insert rows into `chunks` table. Stable id derivation: `f"chunk-{document_id}-{sequence:05d}-{sha256(text)[:8]}"` so re-runs are deterministic. +4. **`backfill_links_from_opensearch`** — for each document, query OpenSearch (default index, then legacy as fallback) for the count of indexed chunks. If non-zero: + - Insert `document_store_links` row (state = `Ingested`). + - Compute the chunkset hash via #204's function over the materialized `chunks`. + - Set `link.chunkset_hash`. + - Insert a `chunk_pushes` row with the materialized chunk ids (best-effort: ordered by sequence). +5. **`reaggregate_document_lifecycle`** — recompute the doc state by combining the link states (#203's aggregation rule) and update `documents.lifecycle_state` accordingly. This may upgrade or downgrade the value set in step 2 (e.g. `Chunked → Ingested`). +6. **`copy_legacy_index`** *(optional, behind `--reindex-default-store`)* — call `OpenSearchStore.copy_legacy_to_default()`. + +Each step is wrapped in its own transaction; partial failure leaves earlier steps committed. + +### 5.5 API + +None. CLI only. + +### 5.6 Frontend + +None. + +### 5.7 Cross-cutting + +- Logging: structured at `INFO` per step, at `ERROR` per failed item with `doc_id`. End summary printed to stdout in a fixed table. +- Configuration via CLI flags only (no env vars introduced for this script): + +``` +python -m document_parser.tools.migrate_06 \ + [--dry-run] \ + [--reindex-default-store] \ + [--limit N] \ + [--only-step ] +``` + +- Documentation: `docs/runbooks/release-0.6.0-migration.md` describes the operator flow (backup → run dry-run → run real → validate via `/api/documents`). + +## 6. Alternatives considered + +### Alternative A — Schema migration applies everything at app boot + +- **Summary:** Embed inference logic in `_run_migrations()` so the app starts and self-heals. +- **Why not:** Migration is observable and debuggable as a CLI; baking it into boot time risks slow startup and silent failures. Operators want to run it during a maintenance window with a dry-run first. + +### Alternative B — Migrate on demand (lazy) + +- **Summary:** Add a "needs migration" check at runtime, materialize chunks for a doc only on first read. +- **Why not:** Surfaces the partial state in the API (`stores: []` for unmigrated docs even if they are indexed). The library page (#211) becomes a half-true representation of reality. Eager migration is simpler. + +### Alternative C — Run only the schema; let users re-ingest manually + +- **Summary:** Skip backfill; users re-trigger ingestion from the UI as needed. +- **Why not:** A tenant with 10k docs cannot click 10k re-ingest buttons. The killer flow promises "your existing corpus already has state". + +## 7. API & data contract + +No API changes. + +### Persistence schema + +See §5.2 (one new table: `migration_progress`). + +### CLI + +``` +python -m document_parser.tools.migrate_06 [flags] +``` + +All flags are documented in `--help`. + +### Breaking changes + +None. + +## 8. Risks & mitigations + +| Risk | Audit dimension | Likelihood | Impact | How we notice | Mitigation / rollback | +|------|-----------------|------------|--------|---------------|------------------------| +| Migration crashes mid-run | CI/Build | Medium | Medium | Error logs | Resumable via `migration_progress`; re-run picks up where it left off | +| Wrong inference for a doc edge case (e.g. multiple stores already) | DDD | Low | Medium | Operator validation (sample 10 docs by hand) | Dry-run lists changes; operator can `--only-step` to redo a single step | +| OpenSearch reindex copies bad data | Decoupling | Low | High | Diff in document counts | Reindex is opt-in (`--reindex-default-store`); operator runs it deliberately; alias keeps legacy reads working | +| Hash mismatch after migration (chunks materialized differ from what was indexed) | DDD | Medium | Medium | Newly-migrated link shows `Stale` immediately | Acceptable behaviour: it correctly tells the user "your indexed chunks may not match the current source"; operator can choose to re-ingest or leave as-is | +| Missing analysis-job rows for a doc | Tests | Low | Low | Doc shows `Uploaded` despite being indexed | Inference falls back to OpenSearch presence; if both empty, `Uploaded` is correct | + +## 9. Testing strategy + +### Backend — pytest + +- **Unit (services):** `test_migration_inference.py` — table-driven: every `(analysis_state, has_chunks_json, indexed)` tuple → expected `lifecycle_state`. +- **Integration:** `test_migration_idempotency.py` — run twice → second run zero writes; mid-run abort + restart → final state matches one-shot run. +- **Persistence:** `test_chunks_materialization.py` — `chunks_json` → rows; ids deterministic (re-running produces same ids). + +### Snapshot fixture + +`document-parser/tests/fixtures/db_pre_06.sqlite` — handcrafted SQLite DB with three documents (uploaded only, completed analysis, indexed in OpenSearch via fake adapter). Migration runs against it and the resulting state is asserted. + +### Manual QA + +1. Snapshot prod DB. +2. Run `--dry-run` on the snapshot → review the printed plan. +3. Run for real on the snapshot → validate via `/api/documents` that every doc has a sensible `lifecycleState` and (where applicable) `stores`. +4. Run the same against prod during the maintenance window. + +### Performance + +The script is O(N_docs + N_chunks). For 10k docs / 500k chunks: target < 5 minutes on a single-node SQLite. No parallelism needed in 0.6.0. + +## 10. Rollout & observability + +### Release branch + +`release/0.6.0`. The migration ships in the same release as #202 / #203 / #204 / #205 — operators run it after deploying the new code and before sending traffic. + +### Feature flag + +None. + +### Observability + +- Stdout summary table at the end: + + ``` + step wrote skipped + seed_default_store 0 1 + backfill_document_lifecycle_state 87 0 + materialize_chunks_from_chunks_json 14502 0 + backfill_links_from_opensearch 73 14 + reaggregate_document_lifecycle 73 0 + copy_legacy_index — — + total 14735 15 + ``` + +- Logs: per-step start / finish; per-error row. + +### Rollback plan + +- `migration_progress` rows can be deleted to force re-run a specific step. +- The new tables can be truncated or dropped (data is recoverable from `analysis_jobs.chunks_json` and OpenSearch). +- Reverting application code: the new tables stay populated but unused; safe. + +## 11. Open questions + +- Should the script open a *long* SQLite transaction or many small ones? **Decision:** many small (per-doc), to keep the DB writeable for the live app if the operator chooses to run the script while traffic is on. +- Hash mismatch on freshly-migrated docs (chunks materialized may differ from what was indexed) — should we *force-mark* them `Stale`, or trust the hash compare to do it implicitly? **Decision:** trust the compare. The result is identical and the path is uniform. + +## 12. References + +- **Issue:** https://github.com/scub-france/Docling-Studio/issues/206 +- **Related issues:** #202 (lifecycle), #203 (per-store), #204 (hash), #205 (audit + chunks table) +- **ADRs:** none planned +- **Project docs:** + - Architecture: `docs/architecture.md` + - Coding standards: `docs/architecture/coding-standards.md` + - Operations playbooks: `docs/operations/`