docling-studio/docs/design/206-lifecycle-state-migration.md

# Design: Migrate existing documents to the new lifecycle state model

- **Issue:** #206
- **Title on issue:** [CHORE] Migrate existing documents to the new lifecycle state model
- **Author:** Pier-Jean Malandrino
- **Date:** 2026-04-29
- **Status:** Accepted
- **Target milestone:** 0.6.0 — Doc-centric ingest
- **Impacted layers:** backend: persistence · services · infra (CLI) · frontend (none)
- **Audit dimensions likely touched:** CI/Build · Tests · Documentation · Security
- **ADR spawned?:** no

---

## 1. Problem

#202, #203, and #205 introduce four new tables / columns: `documents.lifecycle_state`, `stores`, `document_store_links`, `chunks`, `chunk_edits`, `chunk_pushes`. Existing tenants already have documents, analysis jobs, and chunks living in `analysis_jobs.chunks_json` plus rows in OpenSearch. After deploy, these documents must appear in `/docs` with sensible state — otherwise the page looks broken (every doc shows `Uploaded` or unknown).

This issue ships an idempotent migration that backfills:
- `documents.lifecycle_state` and `lifecycle_state_at`.
- One `stores` row (the `default` seeded by #203, no-op if already present).
- One `document_store_links` row per document already represented in OpenSearch under the legacy index.
- `chunks` rows materialized from existing `analysis_jobs.chunks_json`.
- `chunk_pushes` rows reconstructed for documents currently indexed.

It also reindexes from `docling-studio-chunks` to `docling-studio-default` if needed, so the new store-aware code can read them.

## 2. Goals

- [ ] Idempotent CLI command `python -m document_parser.tools.migrate_06` (re-runs safely; second run is a no-op).
- [ ] `--dry-run` flag prints what would change without writing.
- [ ] Inference rules below produce a sensible state for every existing document.
- [ ] Per-tenant counts logged at the end (parsed=N, chunked=N, ingested=N, failed=N).
- [ ] Documented in `docs/runbooks/release-0.6.0-migration.md`.
- [ ] Tested on a snapshot DB representing the three relevant pre-states (no analysis, completed analysis, indexed in OpenSearch).

## 3. Non-goals

- Re-running ingestion. The migration only touches metadata; it does not re-embed.
- A web UI for migration progress — operator-only, run via CLI.
- Migrating tenants on environments that have already adopted parts of the new schema (e.g. a partial pre-release deploy) — out of scope; an emergency hotfix path is documented separately.
- Moving to a new vector backend — out of scope.

## 4. Context & constraints

### Existing code surface

- `document-parser/persistence/database.py` — `_run_migrations()` runs schema DDL on app start.
- `document-parser/persistence/document_repo.py`, `analysis_repo.py` — read paths.
- `document-parser/infra/opensearch_store.py` — read access to existing OpenSearch index.
- New repos from #203 / #205: `store_repo.py`, `document_store_link_repo.py`, `chunk_repo.py`, `chunk_edit_repo.py`.

### Hexagonal Architecture constraints

- The migration is a **CLI script** in `document-parser/tools/migrate_06.py`. It uses repositories only — never raw SQL outside of repo modules. This keeps the upgrade path testable like any service.
- The reindex helper (`docling-studio-chunks → docling-studio-default`) lives in `infra/opensearch_store.py` as a method, called by the script.

### Hard constraints

- Idempotent: re-running on an already-migrated DB makes zero writes.
- Resumable: a crash mid-run can be restarted; checkpoints stored in a tiny `migration_progress` table (rows: `(name, completed_at)`).
- No assumption about uptime: the script must work on a quiesced DB OR a live one (transactions short, single-row scope).
- Read-only on OpenSearch by default; write happens only when the operator explicitly passes `--reindex-default-store`.

### Deployment modes

Run once per environment (`local`, `staging`, `prod`). HF Space deployments — same script, run by the deploy step before the new image starts serving traffic.

## 5. Proposed design

### 5.1 Domain

No new domain code. The migration is a coordinator over existing repos.

### 5.2 Persistence

The schema additions ship in #202 / #203 / #205 migrations (already applied at app start). This issue contributes:

```sql
CREATE TABLE IF NOT EXISTS migration_progress (
    name        TEXT PRIMARY KEY,
    completed_at TEXT NOT NULL
);
```

This is the only new table. It records each migration step's completion to enable resumability.

### 5.3 Infra adapters

`OpenSearchStore.copy_legacy_to_default(*, dry_run: bool) -> int` —
1. If `docling-studio-chunks` exists and `docling-studio-default` does not, create the new index with the same mapping.
2. Reindex via `_reindex_op` (OpenSearch reindex API) from legacy → default.
3. Add a read-only alias `docling-studio-chunks → docling-studio-default`.
4. Returns the number of documents copied.

### 5.4 Services

`document-parser/services/migration_06_service.py` — orchestrator with one method per step. Each step is bracketed by:

```python
if not progress.is_done(step_name):
    do_step(...)
    progress.mark_done(step_name)
```

Steps:

1. **`seed_default_store`** — `INSERT OR IGNORE INTO stores (...) VALUES ('default', ...)`. (Already in #203's migration; this step is a guard for older deploys that skipped the seed.)
2. **`backfill_document_lifecycle_state`** — for each document, infer state:
   - Has at least one `analysis_jobs` row with `status=COMPLETED` and a non-null `chunks_json` → `Chunked` (refined below).
   - Has at least one row indexed in OpenSearch (legacy or default index) → `Ingested`.
   - Has only `analysis_jobs` rows with `status=COMPLETED` and no `chunks_json` → `Parsed`.
   - Has only `analysis_jobs` rows with `status=FAILED` → `Failed`.
   - Else → `Uploaded`.
3. **`materialize_chunks_from_chunks_json`** — for each `analysis_jobs` row with non-null `chunks_json`, parse and insert rows into `chunks` table. Stable id derivation: `f"chunk-{document_id}-{sequence:05d}-{sha256(text)[:8]}"` so re-runs are deterministic.
4. **`backfill_links_from_opensearch`** — for each document, query OpenSearch (default index, then legacy as fallback) for the count of indexed chunks. If non-zero:
   - Insert `document_store_links` row (state = `Ingested`).
   - Compute the chunkset hash via #204's function over the materialized `chunks`.
   - Set `link.chunkset_hash`.
   - Insert a `chunk_pushes` row with the materialized chunk ids (best-effort: ordered by sequence).
5. **`reaggregate_document_lifecycle`** — recompute the doc state by combining the link states (#203's aggregation rule) and update `documents.lifecycle_state` accordingly. This may upgrade or downgrade the value set in step 2 (e.g. `Chunked → Ingested`).
6. **`copy_legacy_index`** *(optional, behind `--reindex-default-store`)* — call `OpenSearchStore.copy_legacy_to_default()`.

Each step is wrapped in its own transaction; partial failure leaves earlier steps committed.

### 5.5 API

None. CLI only.

### 5.6 Frontend

None.

### 5.7 Cross-cutting

- Logging: structured at `INFO` per step, at `ERROR` per failed item with `doc_id`. End summary printed to stdout in a fixed table.
- Configuration via CLI flags only (no env vars introduced for this script):

```
python -m document_parser.tools.migrate_06 \
   [--dry-run] \
   [--reindex-default-store] \
   [--limit N] \
   [--only-step <step_name>]
```

- Documentation: `docs/runbooks/release-0.6.0-migration.md` describes the operator flow (backup → run dry-run → run real → validate via `/api/documents`).

## 6. Alternatives considered

### Alternative A — Schema migration applies everything at app boot

- **Summary:** Embed inference logic in `_run_migrations()` so the app starts and self-heals.
- **Why not:** Migration is observable and debuggable as a CLI; baking it into boot time risks slow startup and silent failures. Operators want to run it during a maintenance window with a dry-run first.

### Alternative B — Migrate on demand (lazy)

- **Summary:** Add a "needs migration" check at runtime, materialize chunks for a doc only on first read.
- **Why not:** Surfaces the partial state in the API (`stores: []` for unmigrated docs even if they are indexed). The library page (#211) becomes a half-true representation of reality. Eager migration is simpler.

### Alternative C — Run only the schema; let users re-ingest manually

- **Summary:** Skip backfill; users re-trigger ingestion from the UI as needed.
- **Why not:** A tenant with 10k docs cannot click 10k re-ingest buttons. The killer flow promises "your existing corpus already has state".

## 7. API & data contract

No API changes.

### Persistence schema

See §5.2 (one new table: `migration_progress`).

### CLI

```
python -m document_parser.tools.migrate_06 [flags]
```

All flags are documented in `--help`.

### Breaking changes

None.

## 8. Risks & mitigations

| Risk | Audit dimension | Likelihood | Impact | How we notice | Mitigation / rollback |
|------|-----------------|------------|--------|---------------|------------------------|
| Migration crashes mid-run | CI/Build | Medium | Medium | Error logs | Resumable via `migration_progress`; re-run picks up where it left off |
| Wrong inference for a doc edge case (e.g. multiple stores already) | DDD | Low | Medium | Operator validation (sample 10 docs by hand) | Dry-run lists changes; operator can `--only-step` to redo a single step |
| OpenSearch reindex copies bad data | Decoupling | Low | High | Diff in document counts | Reindex is opt-in (`--reindex-default-store`); operator runs it deliberately; alias keeps legacy reads working |
| Hash mismatch after migration (chunks materialized differ from what was indexed) | DDD | Medium | Medium | Newly-migrated link shows `Stale` immediately | Acceptable behaviour: it correctly tells the user "your indexed chunks may not match the current source"; operator can choose to re-ingest or leave as-is |
| Missing analysis-job rows for a doc | Tests | Low | Low | Doc shows `Uploaded` despite being indexed | Inference falls back to OpenSearch presence; if both empty, `Uploaded` is correct |

## 9. Testing strategy

### Backend — pytest

- **Unit (services):** `test_migration_inference.py` — table-driven: every `(analysis_state, has_chunks_json, indexed)` tuple → expected `lifecycle_state`.
- **Integration:** `test_migration_idempotency.py` — run twice → second run zero writes; mid-run abort + restart → final state matches one-shot run.
- **Persistence:** `test_chunks_materialization.py` — `chunks_json` → rows; ids deterministic (re-running produces same ids).

### Snapshot fixture

`document-parser/tests/fixtures/db_pre_06.sqlite` — handcrafted SQLite DB with three documents (uploaded only, completed analysis, indexed in OpenSearch via fake adapter). Migration runs against it and the resulting state is asserted.

### Manual QA

1. Snapshot prod DB.
2. Run `--dry-run` on the snapshot → review the printed plan.
3. Run for real on the snapshot → validate via `/api/documents` that every doc has a sensible `lifecycleState` and (where applicable) `stores`.
4. Run the same against prod during the maintenance window.

### Performance

The script is O(N_docs + N_chunks). For 10k docs / 500k chunks: target < 5 minutes on a single-node SQLite. No parallelism needed in 0.6.0.

## 10. Rollout & observability

### Release branch

`release/0.6.0`. The migration ships in the same release as #202 / #203 / #204 / #205 — operators run it after deploying the new code and before sending traffic.

### Feature flag

None.

### Observability

- Stdout summary table at the end:

  ```
  step                                   wrote   skipped
  seed_default_store                         0         1
  backfill_document_lifecycle_state         87         0
  materialize_chunks_from_chunks_json    14502         0
  backfill_links_from_opensearch            73        14
  reaggregate_document_lifecycle            73         0
  copy_legacy_index                          —         —
  total                                  14735        15
  ```

- Logs: per-step start / finish; per-error row.

### Rollback plan

- `migration_progress` rows can be deleted to force re-run a specific step.
- The new tables can be truncated or dropped (data is recoverable from `analysis_jobs.chunks_json` and OpenSearch).
- Reverting application code: the new tables stay populated but unused; safe.

## 11. Open questions

- Should the script open a *long* SQLite transaction or many small ones? **Decision:** many small (per-doc), to keep the DB writeable for the live app if the operator chooses to run the script while traffic is on.
- Hash mismatch on freshly-migrated docs (chunks materialized may differ from what was indexed) — should we *force-mark* them `Stale`, or trust the hash compare to do it implicitly? **Decision:** trust the compare. The result is identical and the path is uniform.

## 12. References

- **Issue:** https://github.com/scub-france/Docling-Studio/issues/206
- **Related issues:** #202 (lifecycle), #203 (per-store), #204 (hash), #205 (audit + chunks table)
- **ADRs:** none planned
- **Project docs:**
  - Architecture: `docs/architecture.md`
  - Coding standards: `docs/architecture/coding-standards.md`
  - Operations playbooks: `docs/operations/`