Adds a first-class lifecycle state to every document, distinct from
AnalysisJob.status. The lifecycle describes the document as a whole and
is the foundation for the doc-centric pivot in 0.6.0.
Domain
- DocumentLifecycleState enum (Uploaded/Parsed/Chunked/Ingested/Stale/Failed)
- Document.lifecycle_state and lifecycle_state_at fields
- Document.transition_to() validates against a transition table
(domain/lifecycle.py) and returns a DocumentLifecycleChanged event
- InvalidLifecycleTransitionError on disallowed transitions
Persistence
- ALTER TABLE documents to add the two columns (default 'Uploaded')
- New index idx_documents_lifecycle_state for filter perf
- _COLUMN_MIGRATIONS refactored to support multiple tables
- _POST_MIGRATION_DDL list for indexes on freshly-added columns
- SqliteDocumentRepository.update_lifecycle()
Services
- AnalysisService drives transitions on parse / chunk / re-chunk / fail
via _transition_document(); idempotent and resilient (logs WARN and
continues if a stale state is somehow encountered)
API
- DocumentResponse exposes lifecycleState + lifecycleStateAt
(additive — existing 'status' field kept for backwards compat)
Frontend
- Document type extended with lifecycleState and lifecycleStateAt
- DocumentLifecycleState union literal mirroring the backend enum
Tests
- 24 new tests in test_lifecycle.py covering transitions, idempotency,
invariant preservation, and event emission
- test_repos.py: round-trip + every-enum-value check + update_lifecycle
- test_chunking.py: rechunk path now mocks document_repo correctly
Refs #202
ChunkWriter mirrors chunks into Neo4j after OpenSearch indexing, creating
HAS_CHUNK edges and DERIVED_FROM back-references to the source Elements
(via doc_items propagated from the local chunker).
Graph API: GET /api/documents/{id}/graph returns a cytoscape-shaped
payload with nodes + edges for Document / Element / Page / Chunk.
Hard cap at 200 pages returns HTTP 413 per design §8.4.
Frontend: new Graph tab in Studio results, rendered with Cytoscape.js +
dagre layout (lazy-loaded, ~175 KB gz). Legend, node styling per element
label, directional edges styled per edge type.
README gains a Neo4j section with the schema, three demo Cypher
queries, and env vars. Backend tests skip cleanly when the neo4j python
package is not installed locally.
Refs #186
Serialize a DoclingDocument to a Neo4j graph: Document + Page + Element
nodes with dynamic specific labels (SectionHeader, Paragraph, Table,
Figure, …), plus HAS_ROOT / PARENT_OF / NEXT / ON_PAGE edges. Replace-on-
write for idempotent re-ingestion.
The reader returns the verbatim document_json stored on the Document
node — reconstruction from graph nodes is deferred to v0.6.
Wired into AnalysisService._finalize_analysis: runs after conversion,
degrades gracefully by default, fails fast when neo4j_required is set.
Refs #186
Hybrid approach: reuse LocalChunker to chunk the DoclingDocument JSON
returned by Serve, so chunking works identically in both local and
remote modes without calling Serve's chunk endpoint.
Backend:
- _build_chunker() always returns LocalChunker (remove engine guard)
- Use docling-core[chunking] extra for required dependencies
- Skip client-side batching in remote mode (Serve manages its own
resources, and batching discards document_json needed for chunking)
- Fix Serve form fields: remove generate_page_images (not a Serve
field), use repeated form keys for to_formats and page_range
- Log Serve error response body on 4xx/5xx for diagnosis
- Fix FastAPI 204 DELETE routes missing response_model=None
Frontend:
- Update chunking feature flag to enable Prepare UI in remote mode
Closes#51
Visual stepper in Studio topbar: Embedding → Indexing → Done.
Each step shows pending/active/done with animated dot. Store tracks
currentStep through the pipeline. Auto-resets after 2s.
- Replace French mode strings (configurer/verifier/preparer) with English
equivalents (configure/verify/prepare) in StudioPage.vue and tests
- Extract _build_conversion_options, _run_conversion, _finalize_analysis
from _run_analysis_inner to respect Single Responsibility Principle
- Rename _get_default_converter to _ensure_default_converter to reflect
its lazy-init side effect
Closes#136, closes#137, closes#138
- Add DocumentRepository and AnalysisRepository protocols in domain/ports.py (#128)
- Refactor persistence repos from module functions to SqliteDocumentRepository
and SqliteAnalysisRepository classes
- Inject repos into AnalysisService and new DocumentService class via
constructor, removing direct imports of persistence and infra.settings (#129)
- Move _merge_results, _classify_error, _extract_html_body to domain/services.py (#130)
- Update main.py composition root to build and wire all dependencies
- Switch api/documents.py to Depends pattern matching api/analyses.py
- Update all tests to use injected mocks instead of module-level patches
Closes#128, closes#129, closes#130
Re-read the job from DB before mark_completed so that
progress_current/progress_total written during batched conversion
are not overwritten by the stale in-memory object.
Add regression unit test and e2e assertion on final progress values.
Replace hardcoded 5 MB upload limit with a configurable setting.
Backend exposes the value via /api/health, frontend reads it
dynamically for validation and UI messages.
Closes#48
Use Docling's native page_range parameter to split large PDFs into
sequential batches, preventing memory exhaustion and timeouts.
Progress is reported via existing polling mechanism.
Closes#56
Raw PyTorch/Docling stack traces are no longer shown to the user.
Common failures (missing compiler, OOM, lock contention, corrupted
document) are mapped to actionable messages. Unknown errors are
truncated to 200 chars.
Ref #57 (M1)
TableFormerMode.ACCURATE is very expensive on CPU (~3-5x slower).
The default can now be set to "fast" on resource-constrained
environments (HF Spaces) via the DEFAULT_TABLE_MODE env var.
User-specified table_mode in the request still takes precedence.
Ref #57 (H1)
Prevents PyTorch/Docling pipeline crashes on HF Spaces CPU by:
- Reducing max file size from 50 MB to 5 MB
- Adding configurable MAX_PAGE_COUNT setting (env var, default unlimited)
- Increasing conversion timeout from 600s to 900s
- Adding frontend upload validation with explicit error messages
- Exposing maxPageCount via /api/health for dynamic UI hints
Domain model state transitions, repository CRUD operations, and service
methods now have descriptive docstrings to lower the contribution
barrier for an open-source project.
Read uploaded files in fixed-size chunks instead of a single file.read()
to reduce peak memory pressure. Also enforces the size limit during
streaming so oversized payloads are rejected before fully buffered.
Unbounded asyncio.create_task calls could exhaust CPU and memory on
modest hardware. Add a configurable semaphore (MAX_CONCURRENT_ANALYSES,
default 3) so excess jobs queue instead of running all at once.
Replace broad `except Exception` with specific types (FileNotFoundError,
PermissionError, OSError) so errors are properly categorized in logs.
Users reporting bugs will get actionable messages instead of generic ones.
UPLOAD_DIR and DB_PATH were read directly from os.environ, bypassing
the Settings dataclass. This caused an inconsistency where overriding
Settings had no effect on these values. Now all modules import from
infra.settings.settings.
Extract bounding boxes from chunk doc_items provenance in the chunker,
propagate through domain/service/API layers, and render highlighted
bboxes on canvas when hovering a chunk card. Reset highlights on
mode and page changes to prevent stale visual state.
AnalysisService gains rechunk() and inline chunking during conversion.
ChunkingOptionsRequest/ChunkResponse schemas, POST rechunk endpoint,
and conditional chunker injection in main.py (local engine only).
- Remove dead get_analysis_service() function in main.py
- Scope file deletion to UPLOAD_DIR to prevent path traversal
- Parse datetime strings back to datetime objects in repos
- Add 10-minute polling timeout in frontend analysis store
- Accept .pdf extension (not just MIME type) on drag-and-drop
- Guard localStorage access for private browsing compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bug #1: _on_task_done now receives job_id via functools.partial and
calls _mark_failed when the background task raises or is cancelled,
preventing jobs from being stuck in RUNNING state forever.
Bug #5: _parse_response wraps json.loads in try/except JSONDecodeError
so malformed json_content strings fall back gracefully instead of crashing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract domain value objects and ports from parsing.py, move Docling-specific
code to infra/local_converter.py, and convert analysis_service to a class
with injected DocumentConverter. This prepares the codebase for plugging in
alternative conversion backends (e.g. Docling Serve) via the Protocol pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
