danvics/docling-studio
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
docling-studio/README.md
Pier-Jean Malandrino ba54427445 feat(#180): feature-flag ingestion pipeline and add brainless one-liner Quick Start 
- Conditionally mount ingestion router only when OpenSearch + embedding are configured
- Add `ingestionAvailable` field to /api/health response
- Add `ingestion` feature flag to frontend (hides Search nav, Ingest button,
  OpenSearch badge, indexed badges/filters when disabled)
- Skip ingestion polling when flag is off
- Make OpenSearch + embedding optional in docker-compose via profiles
- Add docker-compose.ingestion.yml override for full-stack ingestion
- Set BATCH_PAGE_SIZE=5 default in Docker local image
- Lead Quick Start with one-liner docker run command
- Document ingestion as opt-in with dedicated section
- Add BATCH_PAGE_SIZE, MAX_FILE_SIZE_MB, MAX_PAGE_COUNT, RATE_LIMIT_RPM to config tables
- Update test counts (380 backend, 159 frontend)
- Date CHANGELOG 0.4.0, bump frontend version to 0.4.0
- Sync CONTRIBUTING.md with E2E Karate test sections

Closes #180
2026-04-13 11:18:56 +02:00

292 lines
12 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Docling Studio
![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)
![Python](https://img.shields.io/badge/python-3.12+-blue)
![Node](https://img.shields.io/badge/node-20+-green)
![Docling](https://img.shields.io/badge/powered%20by-Docling-orange)
![CI](https://github.com/scub-france/Docling-Studio/actions/workflows/ci.yml/badge.svg)
[![GitHub Stars](https://img.shields.io/github/stars/scub-france/Docling-Studio?style=flat-square&logo=github&label=Stars)](https://github.com/scub-france/Docling-Studio)
A visual document analysis studio powered by [Docling](https://github.com/DS4SD/docling).
Upload a PDF, configure the extraction pipeline, and visualize the results — text, tables, images, formulas, bounding boxes — all from your browser.
![Docling Studio — Presentation](docs/screenshots/presentation.gif)
## Star History
<a href="https://www.star-history.com/?repos=scub-france%2FDocling-Studio&type=timeline&legend=top-left">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=scub-france/Docling-Studio&type=timeline&theme=dark&legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=scub-france/Docling-Studio&type=timeline&legend=top-left" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=scub-france/Docling-Studio&type=timeline&legend=top-left" />
</picture>
</a>
## Features
- **Home page** with quick upload and recent documents
- **PDF viewer** with page navigation, bounding box overlay, and resizable results panel
- **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description, image generation
- **Bounding box visualization** — color-coded element overlay directly on the PDF
- **Per-page results** — right panel syncs with the current PDF page
- **Chunking** — split extracted content into semantic chunks (hierarchical, hybrid, or page-based) with configurable token limits and inline editing
- **Ingestion pipeline** — Docling → chunking → embedding → OpenSearch vector indexing (one-click from Studio)
- **Markdown & HTML export** of extracted content
- **Document management** — upload, list, delete, search, filter by indexing status
- **Analysis history** — re-visit and open past analyses
- **Upload limits** — configurable max file size and max page count per document
- **Rate limiting** — configurable requests per minute per IP
- **Dark / Light theme** and **FR / EN** localization
## Architecture
```
┌────────────┐ ┌──────────────────────┐
│ Frontend │────────▶│ Document Parser │
│ Vue 3 │ /api/* │ FastAPI + Docling │
│ port 3000 │ │ SQLite + file storage│
└────────────┘ │ port 8000 │
└──────────────────────┘
```
| Service | Stack | Role |
|---------|-------|------|
| **frontend** | Vue 3, TypeScript, Vite, Pinia | UI, PDF viewer, results display |
| **document-parser** | FastAPI, Docling, SQLite, pdf2image | REST API, document parsing, storage |
### Backend structure (clean architecture)
```
document-parser/
├── main.py # FastAPI app, CORS, lifespan
├── domain/ # Pure domain — no HTTP, no DB
│ ├── models.py # Document, AnalysisJob dataclasses
│ ├── ports.py # Abstract protocols (converter, chunker)
│ └── value_objects.py # ConversionResult, PageDetail, ChunkResult
├── api/ # HTTP layer (FastAPI routers)
│ ├── schemas.py # Pydantic DTOs (camelCase serialization)
│ ├── documents.py # /api/documents endpoints
│ └── analyses.py # /api/analyses endpoints
├── persistence/ # Data layer (SQLite via aiosqlite)
│ ├── database.py # Connection management, schema init
│ ├── document_repo.py # Document CRUD
│ └── analysis_repo.py # AnalysisJob CRUD
├── services/ # Use case orchestration
│ ├── document_service.py # Upload, delete, preview
│ └── analysis_service.py # Async Docling processing
└── tests/ # 377 tests (pytest)
```
### Frontend structure (feature-based)
```
frontend/src/
├── app/ # App shell, router, global styles
├── pages/ # Route-level pages
│ ├── HomePage.vue # Landing page with upload & stats
│ ├── StudioPage.vue # PDF viewer + config + results
│ ├── DocumentsPage.vue # Document management
│ ├── HistoryPage.vue # Past analyses
│ └── SettingsPage.vue # Theme, language, API URL
├── features/ # Feature modules
│ ├── analysis/ # Analysis store, API, bbox, UI components
│ ├── document/ # Document store, API, upload, list
│ ├── history/ # History store, API, navigation
│ └── settings/ # Settings store
└── shared/ # Shared utilities (types, i18n, http, format)
```
## Quick Start
One command, nothing else to install:
```bash
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
```
Open [http://localhost:3000](http://localhost:3000), upload a PDF, and get results. That's it.
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
### Image variants
| Variant | Image tag | Size | Description |
|---------|-----------|------|-------------|
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only |
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
For remote mode:
```bash
docker run -p 3000:3000 \
-e DOCLING_SERVE_URL=http://your-docling-serve:5001 \
ghcr.io/scub-france/docling-studio:latest-remote
```
### Docker Compose
```bash
git clone https://github.com/scub-france/Docling-Studio.git
cd Docling-Studio
# Simple mode (backend + frontend only)
docker compose up --build
# With ingestion pipeline (OpenSearch + embeddings)
docker compose --profile ingestion -f docker-compose.yml -f docker-compose.ingestion.yml up --build
```
### Local Development
**Backend** (Python 3.12+):
```bash
cd document-parser
python -m venv .venv && source .venv/bin/activate
# Remote mode (lightweight)
pip install -r requirements.txt
# Local mode (with Docling)
pip install -r requirements-local.txt
uvicorn main:app --reload --port 8000
```
**Frontend** (Node 20+):
```bash
cd frontend
npm install
npm run dev
```
### Running Tests
```bash
# Backend (377 tests)
cd document-parser
pip install pytest pytest-asyncio httpx
pytest tests/ -v
# Frontend (156 tests)
cd frontend
npm run test:run
```
## Pipeline Options
These options map directly to Docling's [`PdfPipelineOptions`](https://docling-project.github.io/docling/usage/). See the [Docling documentation](https://docling-project.github.io/docling/) for details on each feature.
| Option | Default | Description |
|--------|---------|-------------|
| `do_ocr` | `true` | OCR for scanned pages and embedded images |
| `do_table_structure` | `true` | Table detection and row/column reconstruction |
| `table_mode` | `accurate` | `accurate` (TableFormer) or `fast` |
| `do_code_enrichment` | `false` | Specialized OCR for code blocks |
| `do_formula_enrichment` | `false` | Math formula recognition (LaTeX output) |
| `do_picture_classification` | `false` | Classify images by type (chart, photo, diagram…) |
| `do_picture_description` | `false` | Generate image descriptions via VLM |
| `generate_picture_images` | `false` | Extract detected images as separate files |
| `generate_page_images` | `false` | Rasterize each page as an image |
| `images_scale` | `1.0` | Scale factor for generated images (0.1–10) |
## Configuration
All configuration is done via environment variables. See [`.env.example`](.env.example).
| Variable | Default | Description |
|----------|---------|-------------|
| `CONVERSION_ENGINE` | `local` | `local` (in-process Docling) or `remote` (Docling Serve) |
| `DOCLING_SERVE_URL` | `http://localhost:5001` | Docling Serve endpoint (remote mode only) |
| `DOCLING_SERVE_API_KEY` | — | API key for Docling Serve (optional) |
| `CORS_ORIGINS` | `http://localhost:3000,...` | CORS allowed origins (comma-separated) |
| `UPLOAD_DIR` | `./uploads` | File storage directory |
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
| `CONVERSION_TIMEOUT` | `600` | Max seconds for a single Docling conversion |
| `BATCH_PAGE_SIZE` | `5` (Docker) / `0` | Pages per batch (`0` = process all at once) |
| `MAX_FILE_SIZE_MB` | `50` | Maximum upload file size in MB (`0` = unlimited) |
| `MAX_PAGE_COUNT` | `0` | Maximum number of pages per document (`0` = unlimited) |
| `RATE_LIMIT_RPM` | `100` | Max requests per minute per IP (`0` = disabled) |
## Upload Limits
Docling Studio enforces configurable limits on uploaded documents to protect the server against oversized files and long-running analyses:
- **`MAX_FILE_SIZE_MB`** (default `50`) — rejects uploads exceeding this size. Validated at two levels: early `Content-Length` check and streaming byte count.
- **`MAX_PAGE_COUNT`** (default `0` = unlimited) — rejects documents with more pages than allowed. Useful on shared instances or Hugging Face Spaces to cap processing time.
Both limits are exposed in the `/api/health` endpoint so the frontend can display them to the user before upload. Set either to `0` to disable the corresponding check.
## Ingestion Pipeline (opt-in)
Docling Studio can optionally index extracted chunks into [OpenSearch](https://opensearch.org/) for vector and full-text search. This requires two additional services (OpenSearch + embedding) and is **disabled by default**.
To enable ingestion with Docker Compose:
```bash
docker compose --profile ingestion \
-f docker-compose.yml -f docker-compose.ingestion.yml \
up --build
```
When ingestion is enabled, the UI shows:
- An **Ingest** button in Studio to push chunks to OpenSearch
- An **OpenSearch** connection status badge in the sidebar
- **Indexed / Not indexed** filters on the Documents page
- A **Search** page for full-text and vector search across indexed documents
| Variable | Default | Description |
|----------|---------|-------------|
| `OPENSEARCH_URL` | — | OpenSearch endpoint (empty = ingestion disabled) |
| `EMBEDDING_URL` | — | Embedding service endpoint (empty = ingestion disabled) |
| `EMBEDDING_DIMENSION` | `384` | Vector dimension (must match embedding model) |
## CI / Release
GitHub Actions pipelines (see [`.github/workflows/`](.github/workflows/)):
| Workflow | Trigger | What it does |
|----------|---------|--------------|
| **CI** | push to `main`, pull requests | Lint + type check + Backend tests + Frontend tests + build |
| **Release** | push tag `v*` | Build & push **two** multi-arch Docker images (`remote` + `local`) to `ghcr.io` |
| **Docs** | push to `main` (docs changes) | Build & deploy MkDocs to GitHub Pages |
We follow [Semantic Versioning](https://semver.org/) with a simplified Git Flow. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full release process.
## Performance & System Requirements
| Document type | Pages | Approx. time (CPU) |
|---------------|-------|---------------------|
| Simple report | 5–10 | ~30s–1 min |
| Research paper | 10–30 | ~1–2 min |
| Large document | 100+ | ~2–5 min |
### Docker Desktop settings
| | Remote image | Local image |
|---|---|---|
| **Image size** | ~270 MB | ~1.9 GB |
| **Memory** | 2 GB | 6 GB (recommended 8 GB+) |
| **CPUs** | 2 | 4 (recommended 8+) |
### Platform support
All Docker images are multi-arch (linux/amd64 + linux/arm64). No GPU required.
## Tech Stack
- **Frontend**: Vue 3, TypeScript, Vite, Pinia, DOMPurify
- **Backend**: FastAPI, Docling 2.x, SQLite (aiosqlite), pdf2image
- **CI**: GitHub Actions
- **Infra**: Docker Compose + Nginx
## Contributing
Contributions are welcome! Please open an issue first to discuss what you'd like to change.
## License
[MIT](LICENSE) — Pier-Jean Malandrino
Reference in a new issue View git blame Copy permalink