Merge pull request #43 from scub-france/feature/update-doc

Feature/update doc
2026-04-05 10:53:58 +02:00 · 2026-04-05 10:53:58 +02:00 · 4d84c4615b
commit 4d84c4615b
parent ee57b04f84 09c5e5e89f
9 changed files with 140 additions and 16 deletions
--- a/.env.example
+++ b/.env.example
@ -12,6 +12,16 @@
 # Max seconds per conversion (default: 600)
 # CONVERSION_TIMEOUT=600

+# Max parallel analysis jobs (default: 3)
+# MAX_CONCURRENT_ANALYSES=3
+
+# Deployment mode: "self-hosted" (default) or "huggingface"
+# Shows disclaimer banner when set to "huggingface"
+# DEPLOYMENT_MODE=self-hosted
+
+# Application version (set automatically by CI/Docker build)
+# APP_VERSION=dev
+
 # CORS (comma-separated origins, only needed for custom deployments)
 # CORS_ORIGINS=http://localhost:3000,https://your-domain.com

--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@ -67,11 +67,11 @@ npx prettier --write src/   # auto-format
 ## Running Tests

 ```bash
-# Backend (99 tests)
+# Backend (199 tests)
 cd document-parser
 pytest tests/ -v

-# Frontend (81 tests)
+# Frontend (129 tests)
 cd frontend
 npm run test:run
 ```
--- a/README.md
+++ b/README.md
@ -61,7 +61,7 @@ document-parser/
 ├── services/                 # Use case orchestration
 │   ├── document_service.py   # Upload, delete, preview
 │   └── analysis_service.py   # Async Docling processing
-└── tests/                    # 99 tests (pytest)
+└── tests/                    # 199 tests (pytest)
 ```

 ### Frontend structure (feature-based)
@ -149,12 +149,12 @@ npm run dev
 ### Running Tests

 ```bash
-# Backend (99 tests)
+# Backend (199 tests)
 cd document-parser
 pip install pytest pytest-asyncio httpx
 pytest tests/ -v

-# Frontend (81 tests)
+# Frontend (129 tests)
 cd frontend
 npm run test:run
 ```
--- a/docs/architecture.md
+++ b/docs/architecture.md
@ -2,7 +2,7 @@

 ## Overview

-![Docling Studio architecture](images/archi.png){ width="700" }
+![Docling Studio architecture](images/global.png){ width="700" }

 Two services communicating via REST. The frontend is a Vue 3 SPA served by Nginx in production. The backend is a FastAPI app that wraps Docling's document conversion engine.

@ -40,37 +40,47 @@ The backend follows a strict layered architecture. Dependencies flow inward: API

 ```
 document-parser/
-├── main.py                   # FastAPI app, CORS, lifespan
+├── main.py                   # FastAPI app, CORS, lifespan, health endpoint
 │
 ├── domain/                   # Pure domain — no HTTP, no DB
 │   ├── models.py             # Document, AnalysisJob dataclasses
-│   ├── parsing.py            # Docling conversion & page extraction
+│   ├── ports.py              # Abstract protocols (DocumentConverter, DocumentChunker)
+│   ├── value_objects.py      # ConversionResult, ChunkingOptions, ChunkResult
 │   └── bbox.py               # Bounding box coordinate normalization
 │
 ├── api/                      # HTTP layer (FastAPI routers)
 │   ├── schemas.py            # Pydantic DTOs (camelCase serialization)
 │   ├── documents.py          # /api/documents endpoints
-│   └── analyses.py           # /api/analyses endpoints
+│   └── analyses.py           # /api/analyses endpoints (create, rechunk, delete)
 │
 ├── persistence/              # Data layer (SQLite via aiosqlite)
 │   ├── database.py           # Connection management, schema init
 │   ├── document_repo.py      # Document CRUD
 │   └── analysis_repo.py      # AnalysisJob CRUD
 │
+├── infra/                    # Infrastructure adapters
+│   ├── settings.py           # Environment-based configuration
+│   ├── local_converter.py    # In-process Docling converter (local mode)
+│   ├── serve_converter.py    # HTTP client for Docling Serve (remote mode)
+│   ├── local_chunker.py      # In-process chunking (HierarchicalChunker, HybridChunker)
+│   ├── rate_limiter.py       # Sliding-window rate limiting middleware
+│   └── bbox.py               # Bbox coordinate normalization helpers
+│
 ├── services/                 # Use case orchestration
 │   ├── document_service.py   # Upload, delete, preview
-│   └── analysis_service.py   # Async Docling processing
+│   └── analysis_service.py   # Async Docling processing + chunking
 │
-└── tests/                    # pytest
+└── tests/                    # pytest (199 tests)
 ```

 ### Layer responsibilities

 | Layer | Role | Depends on |
 |-------|------|------------|
-| **domain** | Dataclasses, value objects, ports | Nothing (pure Python) |
+| **domain** | Dataclasses, value objects, abstract ports | Nothing (pure Python) |
 | **persistence** | SQLite CRUD, aiosqlite | domain (models) |
-| **services** | Orchestrate use cases, call Docling | domain + persistence |
+| **infra** | Adapters: converters, chunker, rate limiter, settings | domain (ports, value objects) |
+| **services** | Orchestrate use cases, call converters/chunkers | domain + persistence + infra |
 | **api** | HTTP endpoints, Pydantic DTOs, error handling | services |

 ### API contract
@ -101,7 +111,9 @@ frontend/src/
 │   │       ├── AnalysisPanel.vue
 │   │       ├── StructureViewer.vue
 │   │       └── ...
+│   ├── chunking/             # Chunk panel UI + rechunk action
 │   ├── document/             # Document store, API, upload
+│   ├── feature-flags/        # Feature flag store (reads /api/health)
 │   ├── history/              # History store, navigation
 │   └── settings/             # Theme, locale, API URL
 │
@ -126,3 +138,73 @@ Backend response → Pinia store state → Vue reactivity → UI update
 - **TypeScript strict mode** with shared interfaces in `shared/types.ts`.
 - **No component library** — custom CSS with CSS variables for theming.
 - **vue-tsc** in CI to catch type errors before merge.
+
+## Feature Flags
+
+The frontend adapts its UI based on the backend's capabilities. On startup, the feature flag store fetches `/api/health` and reads the `engine` and `deploymentMode` fields.
+
+| Flag | Condition | Effect |
+|------|-----------|--------|
+| `chunking` | `engine === 'local'` | Shows chunking options in the analysis panel |
+| `disclaimer` | `deploymentMode === 'huggingface'` | Shows a disclaimer banner at the top of the app |
+
+This allows the same frontend build to work with both local and remote backends without conditional compilation.
+
+## Rate Limiting
+
+The backend applies a sliding-window rate limiter as middleware:
+
+- **60 requests** per **60 seconds** per client IP
+- The `/api/health` endpoint is excluded
+- When the limit is exceeded, the API returns `429 Too Many Requests` with a `Retry-After` header
+
+## Analysis Lifecycle
+
+An analysis job follows this state machine:
+
+```
+PENDING → RUNNING → COMPLETED
+                  → FAILED
+```
+
+| Status | Description |
+|--------|-------------|
+| `PENDING` | Job created, waiting for a processing slot |
+| `RUNNING` | Docling conversion in progress |
+| `COMPLETED` | Conversion finished — results available (markdown, HTML, pages, chunks) |
+| `FAILED` | Conversion error — `error_message` contains details |
+
+The backend limits parallel jobs via `MAX_CONCURRENT_ANALYSES` (default: 3) to avoid overloading the CPU during Docling processing.
+
+## Local vs Remote Mode
+
+The backend supports two conversion engines, selected via the `CONVERSION_ENGINE` environment variable:
+
+| | Local | Remote |
+|---|---|---|
+| **Engine** | In-process Docling (PyTorch) | HTTP client to [Docling Serve](https://github.com/DS4SD/docling-serve) |
+| **Chunking** | Available (in-process) | Not available |
+| **Docker image** | `latest-local` (~1.9 GB) | `latest-remote` (~270 MB) |
+| **ML models** | Downloaded on first run (~400 MB) | Managed by Docling Serve |
+| **CPU/RAM** | 4+ CPUs, 6+ GB RAM | 2 CPUs, 2 GB RAM |
+
+The converter is selected at startup in `main.py` via `_build_converter()`. The chunker (`_build_chunker()`) is only instantiated in local mode — in remote mode, the chunking feature flag is disabled and the UI hides the chunking panel.
+
+## Health Endpoint
+
+`GET /api/health` returns the backend status:
+
+```json
+{
+  "status": "ok",
+  "engine": "local",
+  "version": "0.3.0",
+  "deploymentMode": "self-hosted"
+}
+```
+
+The frontend uses this response to:
+
+1. Verify the backend is reachable
+2. Evaluate feature flags (chunking, disclaimer)
+3. Display the app version
--- a/docs/getting-started.md
+++ b/docs/getting-started.md
@ -7,6 +7,8 @@ Docling Studio ships two Docker image variants:
 | **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
 | **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only (downloads ML models on first run) |

+![Docker architecture](images/docker.png){ width="600" }
+
 ## Docker — remote mode (fastest)

 ```bash
@ -99,6 +101,28 @@ These options map directly to Docling's [`PdfPipelineOptions`](https://docling-p
 | `generate_page_images` | `false` | Rasterize each page as an image |
 | `images_scale` | `1.0` | Scale factor for generated images (0.1–10) |

+## Chunking Options
+
+!!! note
+    Chunking is only available in **local** mode. The chunking UI is hidden when using remote mode (Docling Serve).
+
+After a document is analyzed, you can split the extracted content into semantic chunks. Chunking can be configured at analysis time or re-run later with different options via the **rechunk** action.
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `chunker_type` | `hybrid` | `hybrid` (semantic + structural), `hierarchical` (heading-based), or `page` (one chunk per page) |
+| `max_tokens` | `512` | Maximum tokens per chunk |
+| `merge_peers` | `true` | Merge sibling elements under the same heading |
+| `repeat_table_header` | `true` | Repeat table headers when a table is split across chunks |
+
+Each chunk includes:
+
+- **text** — the chunk content
+- **headings** — heading hierarchy leading to the chunk
+- **source_page** — the page number the chunk originates from
+- **token_count** — number of tokens in the chunk
+- **bboxes** — bounding boxes of the chunk's source elements (page + coordinates)
+
 ## Configuration

 All configuration is done via environment variables:
@ -112,6 +136,9 @@ All configuration is done via environment variables:
 | `UPLOAD_DIR` | `./uploads` | File storage directory |
 | `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
 | `CONVERSION_TIMEOUT` | `600` | Max seconds per Docling conversion |
+| `MAX_CONCURRENT_ANALYSES` | `3` | Maximum parallel analysis jobs |
+| `DEPLOYMENT_MODE` | `self-hosted` | `self-hosted` or `huggingface` (shows disclaimer banner) |
+| `APP_VERSION` | `dev` | Application version (set automatically by CI/Docker) |

 ## System Requirements

--- a/docs/images/archi.png
+++ b/docs/images/archi.png
--- a/docs/images/docker.png
+++ b/docs/images/docker.png
--- a/docs/images/global.png
+++ b/docs/images/global.png
--- a/docs/index.md
+++ b/docs/index.md
@ -4,18 +4,23 @@ A visual document analysis studio powered by [Docling](https://github.com/DS4SD/

 Upload a PDF, configure the extraction pipeline, and visualize the results — text, tables, images, formulas, bounding boxes — all from your browser.

-![Docling Studio architecture](images/archi.png){ width="600" }
+![Docling Studio architecture](images/global.png){ width="600" }

 ![Docling Studio — Execution Result](screenshots/DS-execution-result.png)

 ## Features

 - **PDF viewer** with page navigation, bounding box overlay, and resizable results panel
- **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description
+- **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description, image generation
 - **Bounding box visualization** — color-coded element overlay directly on the PDF
+- **Chunking** — split extracted content into semantic chunks (hierarchical, hybrid, or page-based) with configurable token limits
 - **Markdown & HTML export** of extracted content
 - **Document management** — upload, list, delete
 - **Analysis history** — re-visit and open past analyses
+- **Feature flags** — capabilities adapt to the conversion engine (local vs remote)
+- **Rate limiting** — 60 requests per minute per IP to protect the backend
+- **Deployment modes** — self-hosted (default) or HuggingFace Spaces (with disclaimer banner)
+- **Health endpoint** — `/api/health` reports engine type, deployment mode, and database status
 - **Dark / Light theme** and **FR / EN** localization

 ## Tech Stack
@ -31,7 +36,7 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t

 ```bash
 # Docker (fastest)
-docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest
+docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
 ```

 Open [http://localhost:3000](http://localhost:3000) and upload a PDF.