diff --git a/.env.example b/.env.example index 1e0a95d..dbcdc9a 100644 --- a/.env.example +++ b/.env.example @@ -12,6 +12,16 @@ # Max seconds per conversion (default: 600) # CONVERSION_TIMEOUT=600 +# Max parallel analysis jobs (default: 3) +# MAX_CONCURRENT_ANALYSES=3 + +# Deployment mode: "self-hosted" (default) or "huggingface" +# Shows disclaimer banner when set to "huggingface" +# DEPLOYMENT_MODE=self-hosted + +# Application version (set automatically by CI/Docker build) +# APP_VERSION=dev + # CORS (comma-separated origins, only needed for custom deployments) # CORS_ORIGINS=http://localhost:3000,https://your-domain.com diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 50b4605..baef553 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -67,11 +67,11 @@ npx prettier --write src/ # auto-format ## Running Tests ```bash -# Backend (99 tests) +# Backend (199 tests) cd document-parser pytest tests/ -v -# Frontend (81 tests) +# Frontend (129 tests) cd frontend npm run test:run ``` diff --git a/README.md b/README.md index cee5b1f..548e3fa 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,7 @@ document-parser/ ├── services/ # Use case orchestration │ ├── document_service.py # Upload, delete, preview │ └── analysis_service.py # Async Docling processing -└── tests/ # 99 tests (pytest) +└── tests/ # 199 tests (pytest) ``` ### Frontend structure (feature-based) @@ -149,12 +149,12 @@ npm run dev ### Running Tests ```bash -# Backend (99 tests) +# Backend (199 tests) cd document-parser pip install pytest pytest-asyncio httpx pytest tests/ -v -# Frontend (81 tests) +# Frontend (129 tests) cd frontend npm run test:run ``` diff --git a/docs/architecture.md b/docs/architecture.md index f51795f..764795a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -2,7 +2,7 @@ ## Overview -![Docling Studio architecture](images/archi.png){ width="700" } +![Docling Studio architecture](images/global.png){ width="700" } Two services communicating via REST. The frontend is a Vue 3 SPA served by Nginx in production. The backend is a FastAPI app that wraps Docling's document conversion engine. @@ -40,37 +40,47 @@ The backend follows a strict layered architecture. Dependencies flow inward: API ``` document-parser/ -├── main.py # FastAPI app, CORS, lifespan +├── main.py # FastAPI app, CORS, lifespan, health endpoint │ ├── domain/ # Pure domain — no HTTP, no DB │ ├── models.py # Document, AnalysisJob dataclasses -│ ├── parsing.py # Docling conversion & page extraction +│ ├── ports.py # Abstract protocols (DocumentConverter, DocumentChunker) +│ ├── value_objects.py # ConversionResult, ChunkingOptions, ChunkResult │ └── bbox.py # Bounding box coordinate normalization │ ├── api/ # HTTP layer (FastAPI routers) │ ├── schemas.py # Pydantic DTOs (camelCase serialization) │ ├── documents.py # /api/documents endpoints -│ └── analyses.py # /api/analyses endpoints +│ └── analyses.py # /api/analyses endpoints (create, rechunk, delete) │ ├── persistence/ # Data layer (SQLite via aiosqlite) │ ├── database.py # Connection management, schema init │ ├── document_repo.py # Document CRUD │ └── analysis_repo.py # AnalysisJob CRUD │ +├── infra/ # Infrastructure adapters +│ ├── settings.py # Environment-based configuration +│ ├── local_converter.py # In-process Docling converter (local mode) +│ ├── serve_converter.py # HTTP client for Docling Serve (remote mode) +│ ├── local_chunker.py # In-process chunking (HierarchicalChunker, HybridChunker) +│ ├── rate_limiter.py # Sliding-window rate limiting middleware +│ └── bbox.py # Bbox coordinate normalization helpers +│ ├── services/ # Use case orchestration │ ├── document_service.py # Upload, delete, preview -│ └── analysis_service.py # Async Docling processing +│ └── analysis_service.py # Async Docling processing + chunking │ -└── tests/ # pytest +└── tests/ # pytest (199 tests) ``` ### Layer responsibilities | Layer | Role | Depends on | |-------|------|------------| -| **domain** | Dataclasses, value objects, ports | Nothing (pure Python) | +| **domain** | Dataclasses, value objects, abstract ports | Nothing (pure Python) | | **persistence** | SQLite CRUD, aiosqlite | domain (models) | -| **services** | Orchestrate use cases, call Docling | domain + persistence | +| **infra** | Adapters: converters, chunker, rate limiter, settings | domain (ports, value objects) | +| **services** | Orchestrate use cases, call converters/chunkers | domain + persistence + infra | | **api** | HTTP endpoints, Pydantic DTOs, error handling | services | ### API contract @@ -101,7 +111,9 @@ frontend/src/ │ │ ├── AnalysisPanel.vue │ │ ├── StructureViewer.vue │ │ └── ... +│ ├── chunking/ # Chunk panel UI + rechunk action │ ├── document/ # Document store, API, upload +│ ├── feature-flags/ # Feature flag store (reads /api/health) │ ├── history/ # History store, navigation │ └── settings/ # Theme, locale, API URL │ @@ -126,3 +138,73 @@ Backend response → Pinia store state → Vue reactivity → UI update - **TypeScript strict mode** with shared interfaces in `shared/types.ts`. - **No component library** — custom CSS with CSS variables for theming. - **vue-tsc** in CI to catch type errors before merge. + +## Feature Flags + +The frontend adapts its UI based on the backend's capabilities. On startup, the feature flag store fetches `/api/health` and reads the `engine` and `deploymentMode` fields. + +| Flag | Condition | Effect | +|------|-----------|--------| +| `chunking` | `engine === 'local'` | Shows chunking options in the analysis panel | +| `disclaimer` | `deploymentMode === 'huggingface'` | Shows a disclaimer banner at the top of the app | + +This allows the same frontend build to work with both local and remote backends without conditional compilation. + +## Rate Limiting + +The backend applies a sliding-window rate limiter as middleware: + +- **60 requests** per **60 seconds** per client IP +- The `/api/health` endpoint is excluded +- When the limit is exceeded, the API returns `429 Too Many Requests` with a `Retry-After` header + +## Analysis Lifecycle + +An analysis job follows this state machine: + +``` +PENDING → RUNNING → COMPLETED + → FAILED +``` + +| Status | Description | +|--------|-------------| +| `PENDING` | Job created, waiting for a processing slot | +| `RUNNING` | Docling conversion in progress | +| `COMPLETED` | Conversion finished — results available (markdown, HTML, pages, chunks) | +| `FAILED` | Conversion error — `error_message` contains details | + +The backend limits parallel jobs via `MAX_CONCURRENT_ANALYSES` (default: 3) to avoid overloading the CPU during Docling processing. + +## Local vs Remote Mode + +The backend supports two conversion engines, selected via the `CONVERSION_ENGINE` environment variable: + +| | Local | Remote | +|---|---|---| +| **Engine** | In-process Docling (PyTorch) | HTTP client to [Docling Serve](https://github.com/DS4SD/docling-serve) | +| **Chunking** | Available (in-process) | Not available | +| **Docker image** | `latest-local` (~1.9 GB) | `latest-remote` (~270 MB) | +| **ML models** | Downloaded on first run (~400 MB) | Managed by Docling Serve | +| **CPU/RAM** | 4+ CPUs, 6+ GB RAM | 2 CPUs, 2 GB RAM | + +The converter is selected at startup in `main.py` via `_build_converter()`. The chunker (`_build_chunker()`) is only instantiated in local mode — in remote mode, the chunking feature flag is disabled and the UI hides the chunking panel. + +## Health Endpoint + +`GET /api/health` returns the backend status: + +```json +{ + "status": "ok", + "engine": "local", + "version": "0.3.0", + "deploymentMode": "self-hosted" +} +``` + +The frontend uses this response to: + +1. Verify the backend is reachable +2. Evaluate feature flags (chunking, disclaimer) +3. Display the app version diff --git a/docs/getting-started.md b/docs/getting-started.md index 539dd4f..b2ee2b8 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -7,6 +7,8 @@ Docling Studio ships two Docker image variants: | **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance | | **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only (downloads ML models on first run) | +![Docker architecture](images/docker.png){ width="600" } + ## Docker — remote mode (fastest) ```bash @@ -99,6 +101,28 @@ These options map directly to Docling's [`PdfPipelineOptions`](https://docling-p | `generate_page_images` | `false` | Rasterize each page as an image | | `images_scale` | `1.0` | Scale factor for generated images (0.1–10) | +## Chunking Options + +!!! note + Chunking is only available in **local** mode. The chunking UI is hidden when using remote mode (Docling Serve). + +After a document is analyzed, you can split the extracted content into semantic chunks. Chunking can be configured at analysis time or re-run later with different options via the **rechunk** action. + +| Option | Default | Description | +|--------|---------|-------------| +| `chunker_type` | `hybrid` | `hybrid` (semantic + structural), `hierarchical` (heading-based), or `page` (one chunk per page) | +| `max_tokens` | `512` | Maximum tokens per chunk | +| `merge_peers` | `true` | Merge sibling elements under the same heading | +| `repeat_table_header` | `true` | Repeat table headers when a table is split across chunks | + +Each chunk includes: + +- **text** — the chunk content +- **headings** — heading hierarchy leading to the chunk +- **source_page** — the page number the chunk originates from +- **token_count** — number of tokens in the chunk +- **bboxes** — bounding boxes of the chunk's source elements (page + coordinates) + ## Configuration All configuration is done via environment variables: @@ -112,6 +136,9 @@ All configuration is done via environment variables: | `UPLOAD_DIR` | `./uploads` | File storage directory | | `DB_PATH` | `./data/docling_studio.db` | SQLite database path | | `CONVERSION_TIMEOUT` | `600` | Max seconds per Docling conversion | +| `MAX_CONCURRENT_ANALYSES` | `3` | Maximum parallel analysis jobs | +| `DEPLOYMENT_MODE` | `self-hosted` | `self-hosted` or `huggingface` (shows disclaimer banner) | +| `APP_VERSION` | `dev` | Application version (set automatically by CI/Docker) | ## System Requirements diff --git a/docs/images/archi.png b/docs/images/archi.png deleted file mode 100644 index f81569d..0000000 Binary files a/docs/images/archi.png and /dev/null differ diff --git a/docs/images/docker.png b/docs/images/docker.png new file mode 100644 index 0000000..67ede30 Binary files /dev/null and b/docs/images/docker.png differ diff --git a/docs/images/global.png b/docs/images/global.png new file mode 100644 index 0000000..8fa4299 Binary files /dev/null and b/docs/images/global.png differ diff --git a/docs/index.md b/docs/index.md index 1da8358..b476021 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,18 +4,23 @@ A visual document analysis studio powered by [Docling](https://github.com/DS4SD/ Upload a PDF, configure the extraction pipeline, and visualize the results — text, tables, images, formulas, bounding boxes — all from your browser. -![Docling Studio architecture](images/archi.png){ width="600" } +![Docling Studio architecture](images/global.png){ width="600" } ![Docling Studio — Execution Result](screenshots/DS-execution-result.png) ## Features - **PDF viewer** with page navigation, bounding box overlay, and resizable results panel -- **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description +- **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description, image generation - **Bounding box visualization** — color-coded element overlay directly on the PDF +- **Chunking** — split extracted content into semantic chunks (hierarchical, hybrid, or page-based) with configurable token limits - **Markdown & HTML export** of extracted content - **Document management** — upload, list, delete - **Analysis history** — re-visit and open past analyses +- **Feature flags** — capabilities adapt to the conversion engine (local vs remote) +- **Rate limiting** — 60 requests per minute per IP to protect the backend +- **Deployment modes** — self-hosted (default) or HuggingFace Spaces (with disclaimer banner) +- **Health endpoint** — `/api/health` reports engine type, deployment mode, and database status - **Dark / Light theme** and **FR / EN** localization ## Tech Stack @@ -31,7 +36,7 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t ```bash # Docker (fastest) -docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest +docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local ``` Open [http://localhost:3000](http://localhost:3000) and upload a PDF.