Commit graph

251 commits

Author SHA1 Message Date
Pier-Jean Malandrino
93f37282a0
Merge pull request #126 from scub-france/fix/batch-progress-and-version
fix: batch progress, segmented UI, version source, about section
2026-04-09 16:37:37 +02:00
Pier-Jean Malandrino
86fb98a7c7 chore: bump version to 0.3.1 and update CHANGELOG 2026-04-09 15:52:09 +02:00
Pier-Jean Malandrino
163a6f289a feat: add i18n keys for batch notice and about section (FR + EN) 2026-04-09 15:52:03 +02:00
Pier-Jean Malandrino
abf3325923 feat: single source of truth for app version from health check
Read version from the backend /api/health endpoint instead of
hardcoding from package.json at build time. Add About section
in Settings with link to the DZone design article.
2026-04-09 15:51:59 +02:00
Pier-Jean Malandrino
f487b752dc feat: informational notice when chunking is unavailable in batch mode
Show an info banner in the Prepare panel explaining that batched
analyses do not generate the internal structure required for chunking.
2026-04-09 15:51:52 +02:00
Pier-Jean Malandrino
f6992d91bc feat: segmented batch progress bar with ring indicator
Replace the basic progress bar with a ring percentage indicator,
segmented batch bar with pulse animation, and an inline mini
progress bar in the top banner visible from any tab.
2026-04-09 15:51:46 +02:00
Pier-Jean Malandrino
26fd46967c fix: forward BATCH_PAGE_SIZE env var in docker-compose
The variable was defined in settings but never passed to the container,
making batch mode silently disabled in Docker deployments.
2026-04-09 15:51:41 +02:00
Pier-Jean Malandrino
d810fda333 fix: preserve batch progress on analysis completion
Re-read the job from DB before mark_completed so that
progress_current/progress_total written during batched conversion
are not overwritten by the stale in-memory object.

Add regression unit test and e2e assertion on final progress values.
2026-04-09 15:51:36 +02:00
Pier-Jean Malandrino
0e5e088438 fix(ci): fix Trivy action version (v0.35.0) 2026-04-09 13:36:49 +02:00
Pier-Jean Malandrino
2aa56d8ca2 fix(ci): fix audit-checks crash (set -e + grep exit code + ANSI in output) 2026-04-09 13:25:15 +02:00
Pier-Jean Malandrino
3c9d09c4e2 ci: remove duplicate push trigger from release-gate 2026-04-09 13:22:25 +02:00
Pier-Jean Malandrino
941028e705 ci: add release-gate pipeline, fix CI duplication, add OSS playbook docs
- Add release-gate.yml: 11-job pipeline (lint, tests, Docker build/smoke,
  Trivy scan, dep audit, audit checks, e2e API+UI, PR summary comment)
- Fix CI double-trigger on release branches (push + PR)
- Add CODE_OF_CONDUCT.md, SECURITY.md, PR template
- Add docs: git-workflow, architecture (ADR), release, operations, community
- Add profiles/fastapi-vue for automated audit checks
- Add docs/PROCESSES.md: index of 19 available processes
2026-04-09 13:20:43 +02:00
Pier-Jean Malandrino
6d3453dc7f
Merge pull request #125 from scub-france/feature/karate-ui-e2e-tests
feat: add Karate UI e2e tests with data-e2e selectors
2026-04-09 08:34:28 +02:00
Pier-Jean Malandrino
fdbab2f49e feat: add Karate UI e2e tests with data-e2e selectors (#124)
- Add e2e/ui/ as peer project to e2e/api/ (own pom.xml, runner, config)
- 5 critical UI journeys: upload, delete, analysis, batch-progress, rechunk
- 4 local-only tests: sidebar, i18n, error-states, pipeline-options
- 1 full happy path workflow covering all modes
- Add data-e2e attributes on all tested Vue components (decoupled from CSS)
- Add CONVENTIONS.md with 7 golden rules for writing Karate UI tests
- Update CI with dedicated e2e-ui job (Chrome headless, --no-sandbox)
- Update docs/contributing.md with UI test instructions

Closes #124
2026-04-08 17:53:52 +02:00
Pier-Jean Malandrino
9696526961
Merge pull request #123 from scub-france/fix/ci-warnings
fix: resolve CI warnings (Node.js 20 deprecation + ESLint formatting)
2026-04-08 15:27:46 +02:00
Pier-Jean Malandrino
babd710d90 fix: resolve CI warnings (Node.js 20 deprecation + ESLint formatting)
Force Node.js 24 in all GitHub Actions workflows to suppress the
upcoming Node.js 20 deprecation warnings. Disable vue/html-indent and
vue/html-closing-bracket-newline ESLint rules that conflict with
Prettier formatting.

Closes #122
2026-04-08 14:54:56 +02:00
Pier-Jean Malandrino
145d4bc180 fix: forward RATE_LIMIT_RPM and MAX_FILE_SIZE_MB to backend container
These env vars were set in the CI workflow step but never passed through
docker-compose to the container, so the backend always used defaults
(100 RPM), causing 429 errors in E2E tests.
2026-04-08 14:18:22 +02:00
Pier-Jean Malandrino
06bfe9b2ba fix: use nginx port 3000 for CI health check and e2e tests
The backend container only exposes port 8000 internally (expose, not
ports). The health check and Karate tests must go through the nginx
proxy on port 3000, the only port published to the host.
2026-04-08 14:11:19 +02:00
Pier-Jean Malandrino
80fb16c85e
Merge pull request #121 from scub-france/fix/ci-e2e-fpdf2
fix: add missing fpdf2 dependency in CI e2e job
2026-04-08 14:00:11 +02:00
Pier-Jean Malandrino
db21d73202 fix: add missing fpdf2 dependency in CI e2e job
The generate-test-data.py script imports fpdf (from fpdf2) but only
pypdfium2 was installed in the CI workflow.
2026-04-08 13:58:11 +02:00
Pier-Jean Malandrino
9fadbf62af
Merge pull request #120 from scub-france/feature/e2e-karate
feat: E2E API tests with Karate V2
2026-04-08 13:50:15 +02:00
Pier-Jean Malandrino
cfc5bb5c35 feat: add E2E API tests with Karate V2
Set up a full E2E test suite (39 scenarios) using Karate against
the real API stack. Hybrid architecture: domain-based features +
cross-domain workflows, with data-driven testing and callable helpers.

Structure:
- e2e/pom.xml: Maven + karate-core 1.5
- 3 helpers (upload, analyze+poll, cleanup)
- 3 JSON schemas (health, document, analysis)
- 12 feature files across health, documents, analyses, workflows
- Tags: @smoke (2), @regression (35), @e2e (2)
- generate-test-data.py: fpdf2-based PDF generation (no binaries)

Also adds:
- RATE_LIMIT_RPM env var to make rate limiter configurable (0=disabled)
- CI job e2e with needs: [backend, frontend]
- e2e/ in .dockerignore

Closes #119
2026-04-08 13:47:03 +02:00
Pier-Jean Malandrino
ce639d934b
Merge pull request #118 from scub-france/feature/configurable-max-file-size
feat: make document max size configurable via MAX_FILE_SIZE_MB
2026-04-08 10:29:58 +02:00
Pier-Jean Malandrino
704ba550d4 feat: make document max size configurable via MAX_FILE_SIZE_MB env var
Replace hardcoded 5 MB upload limit with a configurable setting.
Backend exposes the value via /api/health, frontend reads it
dynamically for validation and UI messages.

Closes #48
2026-04-08 10:27:38 +02:00
Pier-Jean Malandrino
d62af058ed
Merge pull request #116 from scub-france/ci/auto-close-issues
ci: auto-close issues on merge into release/* branches
2026-04-08 10:21:57 +02:00
Pier-Jean Malandrino
1f85d5b9cb ci: auto-close issues on merge into release/* branches
GitHub only auto-closes issues when PRs target the default branch.
This workflow scans commit messages for Closes/Fixes patterns on
push to release/** and closes the referenced issues via gh CLI.

Closes #115
2026-04-08 10:18:49 +02:00
Pier-Jean Malandrino
ea78632070
Merge pull request #117 from scub-france/fix/ci-missing-pypdfium2
fix: add pypdfium2 to requirements.txt for CI
2026-04-08 10:17:46 +02:00
Pier-Jean Malandrino
459a720e77 fix: add pypdfium2 to requirements.txt for CI
pypdfium2 is imported in analysis_service for PDF page counting
(batch feature #56) but was missing from requirements.txt, causing
CI collection errors on all 4 test modules.
2026-04-08 10:14:39 +02:00
Pier-Jean Malandrino
a1ed48fb6f
Merge pull request #67 from scub-france/feature/batch-page-range
feat: batch large documents with page_range and progress reporting
2026-04-08 10:11:31 +02:00
Pier-Jean Malandrino
fc866ce229 feat: batch large documents with page_range and progress reporting
Use Docling's native page_range parameter to split large PDFs into
sequential batches, preventing memory exhaustion and timeouts.
Progress is reported via existing polling mechanism.

Closes #56
2026-04-07 17:54:40 +02:00
Pier-Jean Malandrino
18f3b91af4
Merge pull request #66 from scub-france/fix/audit-robustness
[FIX] Robustness audit — settings validation, timeout cascade, cancellation
2026-04-07 17:10:03 +02:00
Pier-Jean Malandrino
af10c200f0 fix: cancel running conversion when analysis job is deleted (#64)
Track background tasks per job ID. On delete, cancel the task to release
the converter lock instead of letting phantom jobs run to completion.
2026-04-07 17:07:55 +02:00
Pier-Jean Malandrino
302b24cc4c fix: route _on_task_done errors through _classify_error (#63)
Unhandled task exceptions now produce user-friendly messages instead of
raw Python tracebacks in the error_message field.
2026-04-07 17:06:16 +02:00
Pier-Jean Malandrino
8ea0cebc16 fix: validate timeout cascade ordering in Settings (#62)
Enforce document_timeout < lock_timeout < conversion_timeout at startup.
Prevents inverted timeout configurations that cause unpredictable behavior.
2026-04-07 17:04:09 +02:00
Pier-Jean Malandrino
1657bce1f0 fix: make converter lock timeout configurable via LOCK_TIMEOUT env var (#61)
Replace hardcoded _LOCK_TIMEOUT=300 with settings.lock_timeout,
readable from LOCK_TIMEOUT environment variable.
2026-04-07 17:01:10 +02:00
Pier-Jean Malandrino
2c254382c8 fix: add __post_init__ validation to Settings dataclass (#65)
Reject invalid configuration at startup: negative timeouts, zero
concurrency, bad table mode. Reports all errors at once.
2026-04-07 16:59:37 +02:00
Pier-Jean Malandrino
24cfd567f2
Merge pull request #58 from scub-france/fix/pipeline-robustness
Fix/pipeline robustness
2026-04-07 15:41:00 +02:00
Pier-Jean Malandrino
f04e5369ef style: use single import style for infra.local_converter in tests
Use `import infra.local_converter as lc_mod` consistently instead of
mixing `import` and `from ... import` for the same module.

Addresses CodeQL review comment on PR #58.
2026-04-07 15:39:02 +02:00
Pier-Jean Malandrino
6177452de1 test: add robustness tests for pipeline failure scenarios
25 tests covering all backend robustness fixes:
- TestClassifyError (9): user-friendly error message mapping
- TestDefaultTableMode (4): settings.default_table_mode injection
- TestDocumentTimeout (2): document_timeout wired into pipeline
- TestConverterLockTimeout (3): lock timeout + release guarantees
- TestConvertSyncLimits (4): max_num_pages/max_file_size forwarding
- TestGetDefaultConverterReset (3): converter singleton reset on failure

Ref #57
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
c281b6c551 fix: classify pipeline errors into user-friendly messages (M1)
Raw PyTorch/Docling stack traces are no longer shown to the user.
Common failures (missing compiler, OOM, lock contention, corrupted
document) are mapped to actionable messages. Unknown errors are
truncated to 200 chars.

Ref #57 (M1)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
f89dc51661 fix: reset _default_converter on init failure (H5)
If the lazy-init of the default converter fails (e.g. model download
error), the singleton was left as None but subsequent calls would not
retry. Now the failed state is cleared so the next request retries.

Ref #57 (H5)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
4b1a15f49a fix: add retry tolerance to frontend polling (H3)
A single transient network error (502, timeout) no longer kills the
polling loop. The frontend now tolerates up to 3 consecutive errors
before abandoning. Successful fetches reset the counter.

Also aligns frontend polling timeout (15 min) with backend timeout.

Ref #57 (H3, M5)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
f58b563a13 fix: make default table_mode configurable via DEFAULT_TABLE_MODE (H1)
TableFormerMode.ACCURATE is very expensive on CPU (~3-5x slower).
The default can now be set to "fast" on resource-constrained
environments (HF Spaces) via the DEFAULT_TABLE_MODE env var.

User-specified table_mode in the request still takes precedence.

Ref #57 (H1)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
6327b13614 fix: pass max_num_pages and max_file_size to Docling convert (C3)
Defense-in-depth: even if upload validation passes, Docling itself
now enforces page count and file size limits. Configurable via
MAX_PAGE_COUNT and MAX_FILE_SIZE env vars (0 = unlimited).

Ref #57 (C3)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
c0fb128718 fix: replace _converter_lock with timeout-based acquisition (C2)
A frozen conversion holding the lock indefinitely blocks all subsequent
jobs. Using lock.acquire(timeout=300) fails fast with a clear error
instead of waiting forever.

Ref #57 (C2)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
4c76101142 fix: add document_timeout to PdfPipelineOptions (C1)
Docling's native document_timeout is the only mechanism that can
interrupt processing inside a blocked thread (OCR, table extraction).
Without it, asyncio.wait_for cannot stop a frozen conversion.

Configurable via DOCUMENT_TIMEOUT env var (default: 120s).

Closes #57 (C1)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
5b6f138164
Update README.md 2026-04-07 12:37:17 +02:00
Pier-Jean Malandrino
05ed49b893 Add issue templates, GitHub stars badge, fix health endpoint type hint 2026-04-07 11:57:29 +02:00
Pier-Jean Malandrino
09145206d2 fix: limit upload to 5 MB / 20 pages max and increase conversion timeout
Prevents PyTorch/Docling pipeline crashes on HF Spaces CPU by:
- Reducing max file size from 50 MB to 5 MB
- Adding configurable MAX_PAGE_COUNT setting (env var, default unlimited)
- Increasing conversion timeout from 600s to 900s
- Adding frontend upload validation with explicit error messages
- Exposing maxPageCount via /api/health for dynamic UI hints
2026-04-07 10:56:16 +02:00
Pier-Jean Malandrino
11eaeb4cd6
Merge pull request #29 from scub-france/release/0.3.0
Some checks failed
Release Docker Images / Build & push — local (push) Has been cancelled
Release Docker Images / Build & push — remote (push) Has been cancelled
Release/0.3.0
2026-04-07 09:00:02 +02:00