Étend le repo avec les méthodes manquantes pour le CRUD :
- find_by_name : valider unicité name au create/update
- update : remplace les champs mutables (id et created_at intacts)
- clear_default_except : promotion mono-default lors d'un changement de default
- delete : retourne True/False selon suppression effective
Tests dédiés sur chacune des nouvelles méthodes.
CLI script to migrate existing tenants to the data model introduced
by #202-205. Idempotent and resumable — re-running is a no-op.
tools/migrate_06.py
- Step 1: backfill_lifecycle — infers documents.lifecycle_state from
analysis_jobs.status + chunks_json presence
Failed analyses, no completed -> Failed
Completed + chunks_json -> Chunked
Completed, no chunks_json -> Parsed
Otherwise -> Uploaded
- Step 2: materialize_chunks — promotes the latest chunks_json blob
for each doc into rows in the new chunks table. Stable id derivation
via uuid5(NAMESPACE_OID, '<doc>|<seq>|<text>') so re-running lands
on the same ids
- Step 3: backfill_links — for any doc that has chunks materialized,
creates a (doc, default-store) link in Ingested state with a freshly
computed chunkset_hash. Treats 'chunks exist' as proxy for 'doc was
ingested into the legacy single index' — sufficient for the typical
pre-0.6.0 deployment, with OpenSearch reindex documented separately
- Step 4: reaggregate — applies #203's aggregate_lifecycle rule so
doc-chunked transitions to Ingested
Persistence
- migration_progress table for resumability + per-step idempotency
CLI
python -m tools.migrate_06 # full migration
python -m tools.migrate_06 --dry-run # plan only, no writes
python -m tools.migrate_06 --only-step <name> # rerun a phase
Tests
- 9 tests: inference rule per (completed, failed, chunks_json) tuple,
end-to-end migration on a hand-built three-doc snapshot, idempotency
(second run = no writes), --dry-run writes nothing, deterministic
chunk ids across reruns
Refs #206
The data and domain layers for the chunks editor (#219-224 in 0.6.0).
Chunks were previously stored as a JSON blob in analysis_jobs.chunks_json;
this commit makes them first-class persisted entities with stable IDs,
soft-delete, and an immutable audit log.
Domain
- Chunk: persistent entity with id, document_id, sequence, text,
headings, source_page, bboxes, doc_items, token_count, timestamps,
deleted_at (soft delete)
- ChunkEdit: immutable audit row (action, actor, at, before, after,
parents, children, reason)
- ChunkPush: snapshot of which chunk_ids landed in which store at push
- ChunkEditAction enum: insert/update/delete/merge/split
- domain/chunk_editing.py: pure operations on a chunkset (insert,
update, delete, merge, split). Each returns a new chunkset and the
affected chunk(s); errors raise ChunkEditingError.
Persistence
- Three new tables: chunks, chunk_edits, chunk_pushes (FK + indexes)
- SqliteChunkRepository (insert, insert_many, update, soft_delete,
find_for_document, find_by_id; respects deleted_at)
- SqliteChunkEditRepository (append-only audit log; paginated reads
ordered newest-first; per-chunk history)
- SqliteChunkPushRepository (per-(doc, store) latest snapshot)
Ports
- ChunkRepository, ChunkEditRepository, ChunkPushRepository protocols
added to domain/ports.py
Tests
- 17 tests for the pure chunk-editing operations covering insert /
update / delete / merge / split, sequence shifts, lineage, error
paths (out-of-range, missing id, deleted target, cross-document)
- 11 tests for the three repositories: round-trips, soft-delete
filtering, history ordering, lineage round-trip, cascade-delete with
document, find_latest semantics
Service orchestration (ChunkEditingService — atomic chunk + audit
write) and the API endpoints land in a follow-up commit on the same
feature branch / next release. The data + domain foundation here is
what unblocks #219-224.
Refs #205
Introduces the data layer for multi-store ingestion. Documents can now
live in multiple stores, each with its own Ingested/Stale/Failed state.
The doc-level lifecycle (#202) becomes the aggregate over all per-store
links, computed by a pure domain function.
Domain
- Store entity (name, slug, kind, embedder, config, is_default)
- DocumentStoreLink entity with mark_ingested / mark_stale / mark_failed
helpers
- StoreKind and DocumentStoreLinkState enums
- aggregate_lifecycle(): pure function — Failed > Stale > Ingested
> fallback (the doc's pre-link Uploaded/Parsed/Chunked state)
Persistence
- New tables 'stores' and 'document_store_links' with the right indexes
(doc_id, store_id, state) and a UNIQUE (doc, store) on the link
- Default 'opensearch' store seeded idempotently in init_db, embedder
pulled from DEFAULT_EMBEDDER (fallback bge-m3)
- SqliteStoreRepository (find_by_slug, find_by_id, get_default, …)
- SqliteDocumentStoreLinkRepository with ON CONFLICT … DO UPDATE upsert
Ports
- StoreRepository and DocumentStoreLinkRepository protocols added
Tests
- 14 new tests: seed idempotency, insert/find round-trips, UNIQUE
constraint, cascade delete with the document, every link state
round-trips, aggregation rule with all branches
Refs #203
Adds a first-class lifecycle state to every document, distinct from
AnalysisJob.status. The lifecycle describes the document as a whole and
is the foundation for the doc-centric pivot in 0.6.0.
Domain
- DocumentLifecycleState enum (Uploaded/Parsed/Chunked/Ingested/Stale/Failed)
- Document.lifecycle_state and lifecycle_state_at fields
- Document.transition_to() validates against a transition table
(domain/lifecycle.py) and returns a DocumentLifecycleChanged event
- InvalidLifecycleTransitionError on disallowed transitions
Persistence
- ALTER TABLE documents to add the two columns (default 'Uploaded')
- New index idx_documents_lifecycle_state for filter perf
- _COLUMN_MIGRATIONS refactored to support multiple tables
- _POST_MIGRATION_DDL list for indexes on freshly-added columns
- SqliteDocumentRepository.update_lifecycle()
Services
- AnalysisService drives transitions on parse / chunk / re-chunk / fail
via _transition_document(); idempotent and resilient (logs WARN and
continues if a stale state is somehow encountered)
API
- DocumentResponse exposes lifecycleState + lifecycleStateAt
(additive — existing 'status' field kept for backwards compat)
Frontend
- Document type extended with lifecycleState and lifecycleStateAt
- DocumentLifecycleState union literal mirroring the backend enum
Tests
- 24 new tests in test_lifecycle.py covering transitions, idempotency,
invariant preservation, and event emission
- test_repos.py: round-trip + every-enum-value check + update_lifecycle
- test_chunking.py: rechunk path now mocks document_repo correctly
Refs #202
Adds the `docling-agent` reasoning-trace viewer as a Studio tunnel, per
`docs/design/reasoning-trace.md`. Users pick an analyzed document, import
a RAGResult JSON, and the iterations are overlaid on the document graph.
Graph source is decoupled from Neo4j: a new pure builder
(`infra/docling_graph.build_graph_payload`) reads `document_json` from
SQLite and emits the same Cytoscape-shaped payload that `fetch_graph`
returns from Neo4j. Neo4j stays exclusive to the Maintain ingestion
pipeline. Shared DoclingDocument helpers live in `infra/docling_tree.py`
so TreeWriter and the builder can't drift on label taxonomy or tree walks.
Also removes the Cytoscape minimap (cytoscape-navigator) from GraphView:
second render instance hurt perf on large documents for no UX win.
Backend
- new `GET /api/documents/:id/reasoning-graph` (SQLite-only)
- new `infra/docling_tree.py`, `infra/docling_graph.py`
- `analysis_repo.find_latest_completed_by_document`
- tests: `test_docling_graph.py` (builder), `test_graph_api.py` (endpoint)
Frontend
- `features/reasoning/` — store, overlay, types, panel, import dialog,
workspace, doc picker
- new `ReasoningPage` + `/reasoning` and `/reasoning/:docId` routes
- `GraphView` gains a `fetcher` prop so reasoning can inject the
SQLite-backed fetcher while Maintain keeps using the Neo4j one
- drops minimap (nav container, dep, CSS)
- legend filters + section parenting extracted for reuse
- i18n base strings (FR + EN)
- Add DocumentRepository and AnalysisRepository protocols in domain/ports.py (#128)
- Refactor persistence repos from module functions to SqliteDocumentRepository
and SqliteAnalysisRepository classes
- Inject repos into AnalysisService and new DocumentService class via
constructor, removing direct imports of persistence and infra.settings (#129)
- Move _merge_results, _classify_error, _extract_html_body to domain/services.py (#130)
- Update main.py composition root to build and wire all dependencies
- Switch api/documents.py to Depends pattern matching api/analyses.py
- Update all tests to use injected mocks instead of module-level patches
Closes#128, closes#129, closes#130
Use Docling's native page_range parameter to split large PDFs into
sequential batches, preventing memory exhaustion and timeouts.
Progress is reported via existing polling mechanism.
Closes#56
Domain model state transitions, repository CRUD operations, and service
methods now have descriptive docstrings to lower the contribution
barrier for an open-source project.
All endpoint functions, lifespan context manager, and get_connection now
have explicit return types. Upload endpoint returns 201 Created instead
of 200 to follow REST conventions.
UPLOAD_DIR and DB_PATH were read directly from os.environ, bypassing
the Settings dataclass. This caused an inconsistency where overriding
Settings had no effect on these values. Now all modules import from
infra.settings.settings.
- Remove dead get_analysis_service() function in main.py
- Scope file deletion to UPLOAD_DIR to prevent path traversal
- Parse datetime strings back to datetime objects in repos
- Add 10-minute polling timeout in frontend analysis store
- Accept .pdf extension (not just MIME type) on drag-and-drop
- Guard localStorage access for private browsing compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
