Commit graph

31 commits

Author SHA1 Message Date
Pier-Jean Malandrino
995e891d9b feat: orchestrated ingestion pipeline Docling → embedding → OpenSearch
Closes #72
2026-04-10 21:12:11 +02:00
Pier-Jean Malandrino
80b0c44e8c feat(chunking): soft-delete chunk with confirmation dialog
Closes #90
2026-04-10 20:28:24 +02:00
Pier-Jean Malandrino
c74a3277b8 feat(chunking): inline chunk text editing
Closes #89
2026-04-10 19:37:29 +02:00
Pier-Jean Malandrino
3d199cb783 fix(clean-code): English mode strings, SRP extraction, rename getter
- Replace French mode strings (configurer/verifier/preparer) with English
  equivalents (configure/verify/prepare) in StudioPage.vue and tests
- Extract _build_conversion_options, _run_conversion, _finalize_analysis
  from _run_analysis_inner to respect Single Responsibility Principle
- Rename _get_default_converter to _ensure_default_converter to reflect
  its lazy-init side effect

Closes #136, closes #137, closes #138
2026-04-10 13:45:13 +02:00
Pier-Jean Malandrino
9907b7f040 fix(arch): inject repositories and extract domain logic
- Add DocumentRepository and AnalysisRepository protocols in domain/ports.py (#128)
- Refactor persistence repos from module functions to SqliteDocumentRepository
  and SqliteAnalysisRepository classes
- Inject repos into AnalysisService and new DocumentService class via
  constructor, removing direct imports of persistence and infra.settings (#129)
- Move _merge_results, _classify_error, _extract_html_body to domain/services.py (#130)
- Update main.py composition root to build and wire all dependencies
- Switch api/documents.py to Depends pattern matching api/analyses.py
- Update all tests to use injected mocks instead of module-level patches

Closes #128, closes #129, closes #130
2026-04-10 11:44:33 +02:00
Pier-Jean Malandrino
d810fda333 fix: preserve batch progress on analysis completion
Re-read the job from DB before mark_completed so that
progress_current/progress_total written during batched conversion
are not overwritten by the stale in-memory object.

Add regression unit test and e2e assertion on final progress values.
2026-04-09 15:51:36 +02:00
Pier-Jean Malandrino
704ba550d4 feat: make document max size configurable via MAX_FILE_SIZE_MB env var
Replace hardcoded 5 MB upload limit with a configurable setting.
Backend exposes the value via /api/health, frontend reads it
dynamically for validation and UI messages.

Closes #48
2026-04-08 10:27:38 +02:00
Pier-Jean Malandrino
fc866ce229 feat: batch large documents with page_range and progress reporting
Use Docling's native page_range parameter to split large PDFs into
sequential batches, preventing memory exhaustion and timeouts.
Progress is reported via existing polling mechanism.

Closes #56
2026-04-07 17:54:40 +02:00
Pier-Jean Malandrino
af10c200f0 fix: cancel running conversion when analysis job is deleted (#64)
Track background tasks per job ID. On delete, cancel the task to release
the converter lock instead of letting phantom jobs run to completion.
2026-04-07 17:07:55 +02:00
Pier-Jean Malandrino
302b24cc4c fix: route _on_task_done errors through _classify_error (#63)
Unhandled task exceptions now produce user-friendly messages instead of
raw Python tracebacks in the error_message field.
2026-04-07 17:06:16 +02:00
Pier-Jean Malandrino
c281b6c551 fix: classify pipeline errors into user-friendly messages (M1)
Raw PyTorch/Docling stack traces are no longer shown to the user.
Common failures (missing compiler, OOM, lock contention, corrupted
document) are mapped to actionable messages. Unknown errors are
truncated to 200 chars.

Ref #57 (M1)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
f58b563a13 fix: make default table_mode configurable via DEFAULT_TABLE_MODE (H1)
TableFormerMode.ACCURATE is very expensive on CPU (~3-5x slower).
The default can now be set to "fast" on resource-constrained
environments (HF Spaces) via the DEFAULT_TABLE_MODE env var.

User-specified table_mode in the request still takes precedence.

Ref #57 (H1)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
09145206d2 fix: limit upload to 5 MB / 20 pages max and increase conversion timeout
Prevents PyTorch/Docling pipeline crashes on HF Spaces CPU by:
- Reducing max file size from 50 MB to 5 MB
- Adding configurable MAX_PAGE_COUNT setting (env var, default unlimited)
- Increasing conversion timeout from 600s to 900s
- Adding frontend upload validation with explicit error messages
- Exposing maxPageCount via /api/health for dynamic UI hints
2026-04-07 10:56:16 +02:00
Pier-Jean Malandrino
c82fd66ffc Add docstrings to all public methods
Domain model state transitions, repository CRUD operations, and service
methods now have descriptive docstrings to lower the contribution
barrier for an open-source project.
2026-04-03 13:54:52 +02:00
Pier-Jean Malandrino
360ea7ea8f Stream upload reads in 64 KB chunks
Read uploaded files in fixed-size chunks instead of a single file.read()
to reduce peak memory pressure. Also enforces the size limit during
streaming so oversized payloads are rejected before fully buffered.
2026-04-03 13:52:32 +02:00
Pier-Jean Malandrino
c2e91262c6 Limit concurrent analyses with asyncio.Semaphore
Unbounded asyncio.create_task calls could exhaust CPU and memory on
modest hardware. Add a configurable semaphore (MAX_CONCURRENT_ANALYSES,
default 3) so excess jobs queue instead of running all at once.
2026-04-03 13:51:53 +02:00
Pier-Jean Malandrino
69ce1df5a1 Narrow exception handlers for better diagnostics
Replace broad `except Exception` with specific types (FileNotFoundError,
PermissionError, OSError) so errors are properly categorized in logs.
Users reporting bugs will get actionable messages instead of generic ones.
2026-04-03 13:48:33 +02:00
Pier-Jean Malandrino
c50106c185 Centralize config through Settings singleton
UPLOAD_DIR and DB_PATH were read directly from os.environ, bypassing
the Settings dataclass. This caused an inconsistency where overriding
Settings had no effect on these values. Now all modules import from
infra.settings.settings.
2026-04-03 13:47:53 +02:00
Pier-Jean Malandrino
8856daf3c9 Force .pdf extension on uploaded files regardless of user input
The file content is already validated as PDF, so the user-supplied
extension should not be trusted or written to disk.
2026-04-03 13:03:21 +02:00
Pier-Jean Malandrino
4af6b5b231 Add chunk-to-bbox hover highlighting in Prepare mode
Extract bounding boxes from chunk doc_items provenance in the chunker,
propagate through domain/service/API layers, and render highlighted
bboxes on canvas when hovering a chunk card. Reset highlights on
mode and page changes to prevent stale visual state.
2026-04-02 14:47:31 +02:00
Pier-Jean Malandrino
a9517d38eb Add chunking service orchestration, API endpoints, and wiring
AnalysisService gains rechunk() and inline chunking during conversion.
ChunkingOptionsRequest/ChunkResponse schemas, POST rechunk endpoint,
and conditional chunker injection in main.py (local engine only).
2026-04-02 12:33:07 +02:00
Pier-Jean Malandrino
2b991108f7 Rebase sync 2026-03-31 16:55:17 +02:00
Pier-Jean Malandrino
001cf6807c Fix audit findings: security, robustness, and dead code
- Remove dead get_analysis_service() function in main.py
- Scope file deletion to UPLOAD_DIR to prevent path traversal
- Parse datetime strings back to datetime objects in repos
- Add 10-minute polling timeout in frontend analysis store
- Accept .pdf extension (not just MIME type) on drag-and-drop
- Guard localStorage access for private browsing compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-31 16:52:36 +02:00
Pier-Jean Malandrino
0af3b81b8f Fix zombie jobs and unprotected JSON parse
Bug #1: _on_task_done now receives job_id via functools.partial and
calls _mark_failed when the background task raises or is cancelled,
preventing jobs from being stuck in RUNNING state forever.

Bug #5: _parse_response wraps json.loads in try/except JSONDecodeError
so malformed json_content strings fall back gracefully instead of crashing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-31 16:52:36 +02:00
Pier-Jean Malandrino
a63af61b73 Fix lint ans test errors 2026-03-31 16:35:48 +02:00
Pier-Jean Malandrino
3743ed4ca8 Refactor backend to hexagonal architecture for converter extensibility
Extract domain value objects and ports from parsing.py, move Docling-specific
code to infra/local_converter.py, and convert analysis_service to a class
with injected DocumentConverter. This prepares the codebase for plugging in
alternative conversion backends (e.g. Docling Serve) via the Protocol pattern.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-31 10:34:07 +02:00
pjmalandrino
577225d10e Add qualityt check and contributing doc 2026-03-21 15:34:54 +01:00
pjmalandrino
2db251182f Refacto and add cp paste btn 2026-03-21 15:03:10 +01:00
pjmalandrino
ad1f1a81d3 Mke some fixes and refacto 2026-03-20 12:36:26 +01:00
pjmalandrino
e69ff90650 Connect options with model configuration 2026-03-20 08:59:20 +01:00
pjmalandrino
5fff141045 Radical architecture change, migration to a more lightweight 2026-03-17 16:06:27 +01:00