diff --git a/README.md b/README.md index dd3c57e..39ea588 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,9 @@ ![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg) ![Python](https://img.shields.io/badge/python-3.12+-blue) +![Node](https://img.shields.io/badge/node-20+-green) ![Docling](https://img.shields.io/badge/powered%20by-Docling-orange) - +![CI](https://github.com/scub-france/Docling-Studio/actions/workflows/ci.yml/badge.svg) A visual document analysis studio powered by [Docling](https://github.com/DS4SD/docling). Upload a PDF, configure the extraction pipeline, and visualize the results — text, tables, images, formulas, bounding boxes — all from your browser. @@ -12,13 +13,15 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t ## Features -- **PDF viewer** with page navigation and visual overlay toggle -- **Configurable Docling pipeline** — OCR on/off, table extraction mode (fast/accurate) -- **Bounding box visualization** — overlay extracted elements directly on the PDF with color-coded types +- **Home page** with quick upload and recent documents +- **PDF viewer** with page navigation, bounding box overlay, and resizable results panel +- **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description, image generation +- **Bounding box visualization** — color-coded element overlay directly on the PDF - **Per-page results** — right panel syncs with the current PDF page -- **Document hierarchy** — heading levels and structure preserved from Docling's `iterate_items()` API - **Markdown & HTML export** of extracted content -- **Analysis history** — re-visit past analyses +- **Document management** — upload, list, delete +- **Analysis history** — re-visit and open past analyses +- **Dark / Light theme** and **FR / EN** localization
More screenshots @@ -43,27 +46,48 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t | Service | Stack | Role | |---------|-------|------| | **frontend** | Vue 3, Vite, Pinia | UI, PDF viewer, results display | -| **document-parser** | FastAPI, Docling, SQLite, pdf2image | REST API, document parsing, storage, persistence | +| **document-parser** | FastAPI, Docling, SQLite, pdf2image | REST API, document parsing, storage | -### Python project structure (clean architecture) +### Backend structure (clean architecture) ``` document-parser/ ├── main.py # FastAPI app, CORS, lifespan -├── domain/ # Pure domain models & Docling logic +├── domain/ # Pure domain — no HTTP, no DB │ ├── models.py # Document, AnalysisJob dataclasses -│ └── parsing.py # Docling conversion & page extraction +│ ├── parsing.py # Docling conversion & page extraction +│ └── bbox.py # Bounding box coordinate normalization ├── api/ # HTTP layer (FastAPI routers) │ ├── schemas.py # Pydantic DTOs (camelCase serialization) │ ├── documents.py # /api/documents endpoints │ └── analyses.py # /api/analyses endpoints -├── persistence/ # Data layer (SQLite) +├── persistence/ # Data layer (SQLite via aiosqlite) │ ├── database.py # Connection management, schema init │ ├── document_repo.py # Document CRUD │ └── analysis_repo.py # AnalysisJob CRUD -└── services/ # Use case orchestration - ├── document_service.py # Upload, delete, preview - └── analysis_service.py # Async Docling processing +├── services/ # Use case orchestration +│ ├── document_service.py # Upload, delete, preview +│ └── analysis_service.py # Async Docling processing +└── tests/ # 99 tests (pytest) +``` + +### Frontend structure (feature-based) + +``` +frontend/src/ +├── app/ # App shell, router, global styles +├── pages/ # Route-level pages +│ ├── HomePage.vue # Landing page with upload & stats +│ ├── StudioPage.vue # PDF viewer + config + results +│ ├── DocumentsPage.vue # Document management +│ ├── HistoryPage.vue # Past analyses +│ └── SettingsPage.vue # Theme, language, API URL +├── features/ # Feature modules +│ ├── analysis/ # Analysis store, API, bbox, UI components +│ ├── document/ # Document store, API, upload, list +│ ├── history/ # History store, API, navigation +│ └── settings/ # Settings store +└── shared/ # Shared utilities (i18n, http, format) ``` ## Quick Start @@ -71,24 +95,18 @@ document-parser/ ### Docker Compose (recommended) ```bash -# Clone the repo -git clone https://github.com/scub-france/docling-studio.git -cd docling-studio - -# (Optional) customize settings -cp .env.example .env - -# Start all services +git clone https://github.com/scub-france/Docling-Studio.git +cd Docling-Studio docker compose up --build ``` Open [http://localhost:3000](http://localhost:3000) -> **Note:** The first analysis takes a bit longer as Docling downloads and caches its ML models (~400 MB). Subsequent runs are fast. +> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast. ### Local Development -**Document Parser** (Python 3.12+): +**Backend** (Python 3.12+): ```bash cd document-parser python -m venv .venv && source .venv/bin/activate @@ -103,66 +121,82 @@ npm install npm run dev ``` -## Docling Integration +### Running Tests -The document parser wraps [Docling](https://github.com/DS4SD/docling) with configurable pipeline options exposed as query parameters on the `/parse` endpoint: +```bash +# Backend (99 tests) +cd document-parser +pip install pytest pytest-asyncio httpx +pytest tests/ -v -| Parameter | Default | Description | -|-----------|---------|-------------| -| `do_ocr` | `true` | Enable OCR for scanned documents | -| `do_table_structure` | `true` | Enable table structure extraction | -| `table_mode` | `accurate` | Table extraction mode: `accurate` or `fast` | +# Frontend (87 tests) +cd frontend +npm run test:run +``` -Element types are detected using `isinstance()` checks against Docling's type hierarchy (`TextItem`, `TableItem`, `PictureItem`, `SectionHeaderItem`, etc.) and the document tree depth from `iterate_items()` is preserved for heading-level reconstruction. +## Pipeline Options + +These options map directly to Docling's [`PdfPipelineOptions`](https://docling-project.github.io/docling/usage/). See the [Docling documentation](https://docling-project.github.io/docling/) for details on each feature. + +| Option | Default | Description | +|--------|---------|-------------| +| `do_ocr` | `true` | OCR for scanned pages and embedded images | +| `do_table_structure` | `true` | Table detection and row/column reconstruction | +| `table_mode` | `accurate` | `accurate` (TableFormer) or `fast` | +| `do_code_enrichment` | `false` | Specialized OCR for code blocks | +| `do_formula_enrichment` | `false` | Math formula recognition (LaTeX output) | +| `do_picture_classification` | `false` | Classify images by type (chart, photo, diagram…) | +| `do_picture_description` | `false` | Generate image descriptions via VLM | +| `generate_picture_images` | `false` | Extract detected images as separate files | +| `generate_page_images` | `false` | Rasterize each page as an image | +| `images_scale` | `1.0` | Scale factor for generated images (0.1–10) | ## Configuration -All configuration is done via environment variables. See [`.env.example`](.env.example) for available options. +All configuration is done via environment variables. See [`.env.example`](.env.example). | Variable | Default | Description | |----------|---------|-------------| | `CORS_ORIGINS` | `http://localhost:3000,...` | CORS allowed origins (comma-separated) | | `UPLOAD_DIR` | `./uploads` | File storage directory | | `DB_PATH` | `./data/docling_studio.db` | SQLite database path | +| `CONVERSION_TIMEOUT` | `600` | Max seconds for a single Docling conversion | + +## CI + +GitHub Actions runs on every push to `main` and on pull requests: + +- **Backend job** — Python 3.12, pytest (99 tests) +- **Frontend job** — Node 20, vitest (87 tests) + vite build + +Both jobs run in parallel. See [`.github/workflows/ci.yml`](.github/workflows/ci.yml). ## Performance & System Requirements -Docling leverages optimized ML models (layout analysis, OCR, table structure) that run efficiently on CPU. The first analysis takes slightly longer as models are downloaded and cached (~400 MB). Subsequent runs are fast, even on large documents. - | Document type | Pages | Approx. time (CPU) | |---------------|-------|---------------------| -| Simple report | 5-10 | ~30s-1 min | -| Research paper | 10-30 | ~1-2 min | -| Large document | 100+ | ~2-5 min | +| Simple report | 5–10 | ~30s–1 min | +| Research paper | 10–30 | ~1–2 min | +| Large document | 100+ | ~2–5 min | ### Docker Desktop settings -The document parser needs **at least 4 GB of RAM**. Recommended Docker Desktop allocation: +The document parser needs **at least 4 GB of RAM**: | Resource | Minimum | Recommended | |----------|---------|-------------| | Memory | 6 GB | 8 GB+ | | CPUs | 4 | 8+ | -> On **macOS**: Docker Desktop > Settings > Resources -> On **Windows**: Docker Desktop > Settings > Resources > WSL 2 - ### Platform support -All Docker images are **multi-arch** (linux/amd64 + linux/arm64). All processing runs on **CPU** — no GPU required. - -| Platform | Architecture | -|----------|-------------| -| **macOS Apple Silicon** (M1/M2/M3/M4) | arm64 | -| **macOS Intel** | amd64 | -| **Linux x86_64** | amd64 | -| **Linux ARM** (Raspberry Pi 5, Ampere) | arm64 | -| **Windows + WSL2** | amd64 | +All Docker images are multi-arch (linux/amd64 + linux/arm64). No GPU required. ## Tech Stack -- **Frontend**: Vue 3 + Vite + Pinia -- **Backend**: FastAPI + Docling 2.x + SQLite + pdf2image +- **Frontend**: Vue 3, Vite, Pinia, DOMPurify +- **Backend**: FastAPI, Docling 2.x, SQLite (aiosqlite), pdf2image +- **CI**: GitHub Actions - **Infra**: Docker Compose + Nginx ## Contributing diff --git a/docs/screenshots/DS-execution-result.png b/docs/screenshots/DS-execution-result.png index 8f3f241..4b0876d 100644 Binary files a/docs/screenshots/DS-execution-result.png and b/docs/screenshots/DS-execution-result.png differ