Commit graph

239 commits

Author SHA1 Message Date
Pier-Jean Malandrino
6d3453dc7f
Merge pull request #125 from scub-france/feature/karate-ui-e2e-tests
feat: add Karate UI e2e tests with data-e2e selectors
2026-04-09 08:34:28 +02:00
Pier-Jean Malandrino
fdbab2f49e feat: add Karate UI e2e tests with data-e2e selectors (#124)
- Add e2e/ui/ as peer project to e2e/api/ (own pom.xml, runner, config)
- 5 critical UI journeys: upload, delete, analysis, batch-progress, rechunk
- 4 local-only tests: sidebar, i18n, error-states, pipeline-options
- 1 full happy path workflow covering all modes
- Add data-e2e attributes on all tested Vue components (decoupled from CSS)
- Add CONVENTIONS.md with 7 golden rules for writing Karate UI tests
- Update CI with dedicated e2e-ui job (Chrome headless, --no-sandbox)
- Update docs/contributing.md with UI test instructions

Closes #124
2026-04-08 17:53:52 +02:00
Pier-Jean Malandrino
9696526961
Merge pull request #123 from scub-france/fix/ci-warnings
fix: resolve CI warnings (Node.js 20 deprecation + ESLint formatting)
2026-04-08 15:27:46 +02:00
Pier-Jean Malandrino
babd710d90 fix: resolve CI warnings (Node.js 20 deprecation + ESLint formatting)
Force Node.js 24 in all GitHub Actions workflows to suppress the
upcoming Node.js 20 deprecation warnings. Disable vue/html-indent and
vue/html-closing-bracket-newline ESLint rules that conflict with
Prettier formatting.

Closes #122
2026-04-08 14:54:56 +02:00
Pier-Jean Malandrino
145d4bc180 fix: forward RATE_LIMIT_RPM and MAX_FILE_SIZE_MB to backend container
These env vars were set in the CI workflow step but never passed through
docker-compose to the container, so the backend always used defaults
(100 RPM), causing 429 errors in E2E tests.
2026-04-08 14:18:22 +02:00
Pier-Jean Malandrino
06bfe9b2ba fix: use nginx port 3000 for CI health check and e2e tests
The backend container only exposes port 8000 internally (expose, not
ports). The health check and Karate tests must go through the nginx
proxy on port 3000, the only port published to the host.
2026-04-08 14:11:19 +02:00
Pier-Jean Malandrino
80fb16c85e
Merge pull request #121 from scub-france/fix/ci-e2e-fpdf2
fix: add missing fpdf2 dependency in CI e2e job
2026-04-08 14:00:11 +02:00
Pier-Jean Malandrino
db21d73202 fix: add missing fpdf2 dependency in CI e2e job
The generate-test-data.py script imports fpdf (from fpdf2) but only
pypdfium2 was installed in the CI workflow.
2026-04-08 13:58:11 +02:00
Pier-Jean Malandrino
9fadbf62af
Merge pull request #120 from scub-france/feature/e2e-karate
feat: E2E API tests with Karate V2
2026-04-08 13:50:15 +02:00
Pier-Jean Malandrino
cfc5bb5c35 feat: add E2E API tests with Karate V2
Set up a full E2E test suite (39 scenarios) using Karate against
the real API stack. Hybrid architecture: domain-based features +
cross-domain workflows, with data-driven testing and callable helpers.

Structure:
- e2e/pom.xml: Maven + karate-core 1.5
- 3 helpers (upload, analyze+poll, cleanup)
- 3 JSON schemas (health, document, analysis)
- 12 feature files across health, documents, analyses, workflows
- Tags: @smoke (2), @regression (35), @e2e (2)
- generate-test-data.py: fpdf2-based PDF generation (no binaries)

Also adds:
- RATE_LIMIT_RPM env var to make rate limiter configurable (0=disabled)
- CI job e2e with needs: [backend, frontend]
- e2e/ in .dockerignore

Closes #119
2026-04-08 13:47:03 +02:00
Pier-Jean Malandrino
ce639d934b
Merge pull request #118 from scub-france/feature/configurable-max-file-size
feat: make document max size configurable via MAX_FILE_SIZE_MB
2026-04-08 10:29:58 +02:00
Pier-Jean Malandrino
704ba550d4 feat: make document max size configurable via MAX_FILE_SIZE_MB env var
Replace hardcoded 5 MB upload limit with a configurable setting.
Backend exposes the value via /api/health, frontend reads it
dynamically for validation and UI messages.

Closes #48
2026-04-08 10:27:38 +02:00
Pier-Jean Malandrino
d62af058ed
Merge pull request #116 from scub-france/ci/auto-close-issues
ci: auto-close issues on merge into release/* branches
2026-04-08 10:21:57 +02:00
Pier-Jean Malandrino
1f85d5b9cb ci: auto-close issues on merge into release/* branches
GitHub only auto-closes issues when PRs target the default branch.
This workflow scans commit messages for Closes/Fixes patterns on
push to release/** and closes the referenced issues via gh CLI.

Closes #115
2026-04-08 10:18:49 +02:00
Pier-Jean Malandrino
ea78632070
Merge pull request #117 from scub-france/fix/ci-missing-pypdfium2
fix: add pypdfium2 to requirements.txt for CI
2026-04-08 10:17:46 +02:00
Pier-Jean Malandrino
459a720e77 fix: add pypdfium2 to requirements.txt for CI
pypdfium2 is imported in analysis_service for PDF page counting
(batch feature #56) but was missing from requirements.txt, causing
CI collection errors on all 4 test modules.
2026-04-08 10:14:39 +02:00
Pier-Jean Malandrino
a1ed48fb6f
Merge pull request #67 from scub-france/feature/batch-page-range
feat: batch large documents with page_range and progress reporting
2026-04-08 10:11:31 +02:00
Pier-Jean Malandrino
fc866ce229 feat: batch large documents with page_range and progress reporting
Use Docling's native page_range parameter to split large PDFs into
sequential batches, preventing memory exhaustion and timeouts.
Progress is reported via existing polling mechanism.

Closes #56
2026-04-07 17:54:40 +02:00
Pier-Jean Malandrino
18f3b91af4
Merge pull request #66 from scub-france/fix/audit-robustness
[FIX] Robustness audit — settings validation, timeout cascade, cancellation
2026-04-07 17:10:03 +02:00
Pier-Jean Malandrino
af10c200f0 fix: cancel running conversion when analysis job is deleted (#64)
Track background tasks per job ID. On delete, cancel the task to release
the converter lock instead of letting phantom jobs run to completion.
2026-04-07 17:07:55 +02:00
Pier-Jean Malandrino
302b24cc4c fix: route _on_task_done errors through _classify_error (#63)
Unhandled task exceptions now produce user-friendly messages instead of
raw Python tracebacks in the error_message field.
2026-04-07 17:06:16 +02:00
Pier-Jean Malandrino
8ea0cebc16 fix: validate timeout cascade ordering in Settings (#62)
Enforce document_timeout < lock_timeout < conversion_timeout at startup.
Prevents inverted timeout configurations that cause unpredictable behavior.
2026-04-07 17:04:09 +02:00
Pier-Jean Malandrino
1657bce1f0 fix: make converter lock timeout configurable via LOCK_TIMEOUT env var (#61)
Replace hardcoded _LOCK_TIMEOUT=300 with settings.lock_timeout,
readable from LOCK_TIMEOUT environment variable.
2026-04-07 17:01:10 +02:00
Pier-Jean Malandrino
2c254382c8 fix: add __post_init__ validation to Settings dataclass (#65)
Reject invalid configuration at startup: negative timeouts, zero
concurrency, bad table mode. Reports all errors at once.
2026-04-07 16:59:37 +02:00
Pier-Jean Malandrino
24cfd567f2
Merge pull request #58 from scub-france/fix/pipeline-robustness
Fix/pipeline robustness
2026-04-07 15:41:00 +02:00
Pier-Jean Malandrino
f04e5369ef style: use single import style for infra.local_converter in tests
Use `import infra.local_converter as lc_mod` consistently instead of
mixing `import` and `from ... import` for the same module.

Addresses CodeQL review comment on PR #58.
2026-04-07 15:39:02 +02:00
Pier-Jean Malandrino
6177452de1 test: add robustness tests for pipeline failure scenarios
25 tests covering all backend robustness fixes:
- TestClassifyError (9): user-friendly error message mapping
- TestDefaultTableMode (4): settings.default_table_mode injection
- TestDocumentTimeout (2): document_timeout wired into pipeline
- TestConverterLockTimeout (3): lock timeout + release guarantees
- TestConvertSyncLimits (4): max_num_pages/max_file_size forwarding
- TestGetDefaultConverterReset (3): converter singleton reset on failure

Ref #57
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
c281b6c551 fix: classify pipeline errors into user-friendly messages (M1)
Raw PyTorch/Docling stack traces are no longer shown to the user.
Common failures (missing compiler, OOM, lock contention, corrupted
document) are mapped to actionable messages. Unknown errors are
truncated to 200 chars.

Ref #57 (M1)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
f89dc51661 fix: reset _default_converter on init failure (H5)
If the lazy-init of the default converter fails (e.g. model download
error), the singleton was left as None but subsequent calls would not
retry. Now the failed state is cleared so the next request retries.

Ref #57 (H5)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
4b1a15f49a fix: add retry tolerance to frontend polling (H3)
A single transient network error (502, timeout) no longer kills the
polling loop. The frontend now tolerates up to 3 consecutive errors
before abandoning. Successful fetches reset the counter.

Also aligns frontend polling timeout (15 min) with backend timeout.

Ref #57 (H3, M5)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
f58b563a13 fix: make default table_mode configurable via DEFAULT_TABLE_MODE (H1)
TableFormerMode.ACCURATE is very expensive on CPU (~3-5x slower).
The default can now be set to "fast" on resource-constrained
environments (HF Spaces) via the DEFAULT_TABLE_MODE env var.

User-specified table_mode in the request still takes precedence.

Ref #57 (H1)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
6327b13614 fix: pass max_num_pages and max_file_size to Docling convert (C3)
Defense-in-depth: even if upload validation passes, Docling itself
now enforces page count and file size limits. Configurable via
MAX_PAGE_COUNT and MAX_FILE_SIZE env vars (0 = unlimited).

Ref #57 (C3)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
c0fb128718 fix: replace _converter_lock with timeout-based acquisition (C2)
A frozen conversion holding the lock indefinitely blocks all subsequent
jobs. Using lock.acquire(timeout=300) fails fast with a clear error
instead of waiting forever.

Ref #57 (C2)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
4c76101142 fix: add document_timeout to PdfPipelineOptions (C1)
Docling's native document_timeout is the only mechanism that can
interrupt processing inside a blocked thread (OCR, table extraction).
Without it, asyncio.wait_for cannot stop a frozen conversion.

Configurable via DOCUMENT_TIMEOUT env var (default: 120s).

Closes #57 (C1)
2026-04-07 15:32:38 +02:00
Pier-Jean Malandrino
5b6f138164
Update README.md 2026-04-07 12:37:17 +02:00
Pier-Jean Malandrino
05ed49b893 Add issue templates, GitHub stars badge, fix health endpoint type hint 2026-04-07 11:57:29 +02:00
Pier-Jean Malandrino
09145206d2 fix: limit upload to 5 MB / 20 pages max and increase conversion timeout
Prevents PyTorch/Docling pipeline crashes on HF Spaces CPU by:
- Reducing max file size from 50 MB to 5 MB
- Adding configurable MAX_PAGE_COUNT setting (env var, default unlimited)
- Increasing conversion timeout from 600s to 900s
- Adding frontend upload validation with explicit error messages
- Exposing maxPageCount via /api/health for dynamic UI hints
2026-04-07 10:56:16 +02:00
Pier-Jean Malandrino
11eaeb4cd6
Merge pull request #29 from scub-france/release/0.3.0
Some checks failed
Release Docker Images / Build & push — local (push) Has been cancelled
Release Docker Images / Build & push — remote (push) Has been cancelled
Release/0.3.0
2026-04-07 09:00:02 +02:00
Pier-Jean Malandrino
2919687202
Merge pull request #45 from scub-france/feature/github-tag
Feature/GitHub tag
2026-04-07 08:58:28 +02:00
Pier-Jean Malandrino
af4400b1f7 Add Github badge 2026-04-07 08:57:39 +02:00
Pier-Jean Malandrino
098ad30596 Freeze CHANGELOG for 0.3.0 release 2026-04-07 08:53:04 +02:00
Pier-Jean Malandrino
e1ccc7a6e0 Add document size limit (50 MB) to disclaimer banner 2026-04-07 08:35:39 +02:00
Pier-Jean Malandrino
4d84c4615b
Merge pull request #43 from scub-france/feature/update-doc
Feature/update doc
2026-04-05 10:53:58 +02:00
Pier-Jean Malandrino
09c5e5e89f Replace architecture diagram with new global/docker schemas 2026-04-05 10:53:21 +02:00
Pier-Jean Malandrino
36be9027c1 Document infra layer, feature flags, rate limiting, analysis lifecycle, local vs remote modes 2026-04-05 10:45:25 +02:00
Pier-Jean Malandrino
3bbff10d1a Add chunking options and missing env vars to getting-started docs 2026-04-05 10:45:21 +02:00
Pier-Jean Malandrino
ff7010bd6b Add missing features to docs index (chunking, feature flags, rate limiting, deployment modes) 2026-04-05 10:45:17 +02:00
Pier-Jean Malandrino
2fdb41e79e Revert "Update frontend test count in CLAUDE.md (129 tests)"
This reverts commit aa73c209bc.
2026-04-05 10:34:14 +02:00
Pier-Jean Malandrino
1b359b8faf Fix remaining stale test count in README backend structure (99 → 199) 2026-04-05 10:27:58 +02:00
Pier-Jean Malandrino
18316f513e Fix Docker image tag in docs quick start (latest → latest-local) 2026-04-05 10:25:57 +02:00