Work on full Docker integration
This commit is contained in:
pjmalandrino
parent
e76012f341
commit
4c6aff3029
20 changed files with 653 additions and 187 deletions
17
.env.example
Normal file
17
.env.example
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
# PostgreSQL
|
||||
POSTGRES_USER=app
|
||||
POSTGRES_PASSWORD=changeme
|
||||
POSTGRES_DB=docling_studio
|
||||
|
||||
# Backend (Spring Boot)
|
||||
SPRING_DATASOURCE_USERNAME=app
|
||||
SPRING_DATASOURCE_PASSWORD=changeme
|
||||
|
||||
# Document parser URL (only change if running parser on a different host)
|
||||
# APP_DOCUMENT-PARSER_BASE-URL=http://document-parser:8000
|
||||
|
||||
# CORS (comma-separated origins, only needed for custom deployments)
|
||||
# APP_CORS_ALLOWED_ORIGINS=http://localhost:3000,https://your-domain.com
|
||||
|
||||
# File storage path (inside backend container)
|
||||
# APP_STORAGE_PATH=./uploads
|
||||
12
.gitignore
vendored
12
.gitignore
vendored
|
|
@ -18,11 +18,23 @@ Thumbs.db
|
|||
# Env
|
||||
.env
|
||||
.env.local
|
||||
.env.production
|
||||
|
||||
# Uploads
|
||||
uploads/
|
||||
backend/uploads/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.venv/
|
||||
*.egg-info/
|
||||
|
||||
# Java
|
||||
*.class
|
||||
*.jar
|
||||
*.log
|
||||
hs_err_pid*
|
||||
|
||||
# Docker
|
||||
docker-compose.override.yml
|
||||
|
|
|
|||
21
LICENSE
Normal file
21
LICENSE
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2025 Pier-Jean Malandrino
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
152
README.md
152
README.md
|
|
@ -1,51 +1,175 @@
|
|||
# Docling Studio
|
||||
|
||||
A professional document analysis studio powered by [Docling](https://github.com/DS4SD/docling), inspired by MistralAI Studio.
|
||||
A visual document analysis studio powered by [Docling](https://github.com/DS4SD/docling).
|
||||
Upload a PDF, configure the extraction pipeline, and visualize the results — text, tables, images, formulas, bounding boxes — all from your browser.
|
||||
|
||||
|
||||
|
||||
## Features
|
||||
|
||||
- **PDF viewer** with page navigation and visual overlay toggle
|
||||
- **Configurable Docling pipeline** — OCR on/off, table extraction mode (fast/accurate)
|
||||
- **Bounding box visualization** — overlay extracted elements directly on the PDF with color-coded types
|
||||
- **Per-page results** — right panel syncs with the current PDF page
|
||||
- **Document hierarchy** — heading levels and structure preserved from Docling's `iterate_items()` API
|
||||
- **Markdown & HTML export** of extracted content
|
||||
- **Analysis history** — re-visit past analyses
|
||||
|
||||
<details>
|
||||
<summary>More screenshots</summary>
|
||||
|
||||
| Import | Configure | Results |
|
||||
|--------|-----------|---------|
|
||||
|  |  |  |
|
||||
|
||||
</details>
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
frontend/ → Vue 3 + Vite + Pinia (port 3000)
|
||||
backend/ → Spring Boot 3.3.5 / Java 21 (port 8081)
|
||||
document-parser/ → FastAPI + Docling (port 8000)
|
||||
┌────────────┐ ┌──────────────┐ ┌──────────────────┐
|
||||
│ Frontend │────▶│ Backend │────▶│ Document Parser │
|
||||
│ Vue 3 │ │ Spring Boot │ │ FastAPI + Docling │
|
||||
│ port 3000 │ │ port 8081 │ │ port 8000 │
|
||||
└────────────┘ └──────┬───────┘ └──────────────────┘
|
||||
│
|
||||
┌──────▼───────┐
|
||||
│ PostgreSQL │
|
||||
│ port 5432 │
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
| Service | Stack | Role |
|
||||
|---------|-------|------|
|
||||
| **frontend** | Vue 3, Vite, Pinia | UI, PDF viewer, results display |
|
||||
| **backend** | Spring Boot 3.3, Java 21, Liquibase | REST API, storage, orchestration |
|
||||
| **document-parser** | FastAPI, Docling, pdf2image | PDF parsing with configurable pipeline |
|
||||
| **postgres** | PostgreSQL 16 | Documents & analysis persistence |
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Docker Compose (recommended)
|
||||
|
||||
```bash
|
||||
docker-compose up --build
|
||||
# Clone the repo
|
||||
git clone https://github.com/pjmalandrino/docling-studio.git
|
||||
cd docling-studio
|
||||
|
||||
# (Optional) customize credentials
|
||||
cp .env.example .env
|
||||
|
||||
# Start all services
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
Open [http://localhost:3000](http://localhost:3000)
|
||||
|
||||
> **Note:** First analysis may take a few minutes as Docling downloads its ML models (~40 MB) on first run.
|
||||
|
||||
### Local Development
|
||||
|
||||
**Document Parser:**
|
||||
Start only PostgreSQL via Docker:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
```
|
||||
|
||||
Then run each service locally:
|
||||
|
||||
**Document Parser** (Python 3.12+):
|
||||
```bash
|
||||
cd document-parser
|
||||
python -m venv .venv && source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
uvicorn main:app --reload --port 8000
|
||||
```
|
||||
|
||||
**Backend:**
|
||||
**Backend** (Java 21+):
|
||||
```bash
|
||||
cd backend
|
||||
./mvnw spring-boot:run
|
||||
```
|
||||
|
||||
**Frontend:**
|
||||
**Frontend** (Node 20+):
|
||||
```bash
|
||||
cd frontend
|
||||
npm install
|
||||
npm run dev
|
||||
```
|
||||
|
||||
## Features
|
||||
## Docling Integration
|
||||
|
||||
- PDF upload and document analysis via Docling
|
||||
- Extracted content viewing (Markdown, HTML)
|
||||
- Document structure visualization with bounding boxes
|
||||
- Image detection
|
||||
- Analysis history
|
||||
The document parser wraps [Docling](https://github.com/DS4SD/docling) with configurable pipeline options exposed as query parameters on the `/parse` endpoint:
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `do_ocr` | `true` | Enable OCR for scanned documents |
|
||||
| `do_table_structure` | `true` | Enable table structure extraction |
|
||||
| `table_mode` | `accurate` | Table extraction mode: `accurate` or `fast` |
|
||||
|
||||
Element types are detected using `isinstance()` checks against Docling's type hierarchy (`TextItem`, `TableItem`, `PictureItem`, `SectionHeaderItem`, etc.) and the document tree depth from `iterate_items()` is preserved for heading-level reconstruction.
|
||||
|
||||
## Configuration
|
||||
|
||||
All configuration is done via environment variables. See [`.env.example`](.env.example) for available options.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `POSTGRES_USER` | `app` | Database user |
|
||||
| `POSTGRES_PASSWORD` | `app` | Database password |
|
||||
| `POSTGRES_DB` | `docling_studio` | Database name |
|
||||
| `APP_CORS_ALLOWED_ORIGINS` | `http://localhost:3000,...` | CORS allowed origins (comma-separated) |
|
||||
| `APP_DOCUMENT-PARSER_BASE-URL` | `http://localhost:8000` | Document parser URL |
|
||||
| `APP_STORAGE_PATH` | `./uploads` | File storage directory |
|
||||
|
||||
## Performance & System Requirements
|
||||
|
||||
Docling runs ML models (layout analysis, OCR, table structure) on **CPU by default**. Processing time depends on document size and complexity.
|
||||
|
||||
| Document type | Pages | Approx. time (CPU) |
|
||||
|---------------|-------|---------------------|
|
||||
| Simple report | 5-10 | 1-3 min |
|
||||
| Research paper | 15-30 | 5-10 min |
|
||||
| Dense PDF with tables | 30+ | 10-20 min |
|
||||
|
||||
### Docker Desktop settings
|
||||
|
||||
The document parser needs **at least 4 GB of RAM**. Recommended Docker Desktop allocation:
|
||||
|
||||
| Resource | Minimum | Recommended |
|
||||
|----------|---------|-------------|
|
||||
| Memory | 6 GB | 8 GB+ |
|
||||
| CPUs | 4 | 8+ |
|
||||
|
||||
> On **macOS**: Docker Desktop > Settings > Resources
|
||||
> On **Windows**: Docker Desktop > Settings > Resources > WSL 2
|
||||
|
||||
### Platform support
|
||||
|
||||
All Docker images are **multi-arch** (linux/amd64 + linux/arm64). Works natively on:
|
||||
|
||||
| Platform | Architecture | GPU acceleration |
|
||||
|----------|-------------|-----------------|
|
||||
| **macOS Apple Silicon** (M1/M2/M3) | arm64 | Not in Docker (MPS unavailable). Run parser locally for GPU. |
|
||||
| **macOS Intel** | amd64 | N/A |
|
||||
| **Linux x86_64** | amd64 | NVIDIA GPU via `docker compose --profile gpu` (coming soon) |
|
||||
| **Linux ARM** (Raspberry Pi 5, Ampere) | arm64 | CPU only |
|
||||
| **Windows + WSL2** | amd64 | NVIDIA GPU passthrough supported |
|
||||
|
||||
> **Tip for Mac users:** For faster processing, run the document parser **locally** (outside Docker) to leverage Apple Silicon's MPS acceleration when supported by PyTorch/Docling.
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Frontend**: Vue 3 + Vite + Pinia
|
||||
- **Backend**: Spring Boot 3.3 + Java 21 + Liquibase + PDFBox
|
||||
- **Parser**: FastAPI + Docling 2.x + PyTorch + pdf2image
|
||||
- **Database**: PostgreSQL 16
|
||||
- **Infra**: Docker Compose + Nginx
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please open an issue first to discuss what you'd like to change.
|
||||
|
||||
## License
|
||||
|
||||
[MIT](LICENSE) — Pier-Jean Malandrino
|
||||
|
|
|
|||
|
|
@ -47,6 +47,11 @@
|
|||
<groupId>org.liquibase</groupId>
|
||||
<artifactId>liquibase-core</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.pdfbox</groupId>
|
||||
<artifactId>pdfbox</artifactId>
|
||||
<version>3.0.3</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
|
|
|
|||
|
|
@ -20,7 +20,11 @@ public class AnalysisController {
|
|||
|
||||
@PostMapping
|
||||
public AnalysisResponse create(@RequestBody Map<String, String> body) {
|
||||
UUID documentId = UUID.fromString(body.get("documentId"));
|
||||
String raw = body.get("documentId");
|
||||
if (raw == null || raw.isBlank()) {
|
||||
throw new IllegalArgumentException("documentId is required");
|
||||
}
|
||||
UUID documentId = UUID.fromString(raw);
|
||||
AnalysisJob job = service.create(documentId);
|
||||
return AnalysisResponse.from(job);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,78 @@
|
|||
package com.docling.studio.analysis;
|
||||
|
||||
import com.docling.studio.document.Document;
|
||||
import com.docling.studio.document.DocumentParserClient;
|
||||
import com.docling.studio.document.DocumentService;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.scheduling.annotation.Async;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
|
||||
/**
|
||||
* Separated from AnalysisService so that Spring's @Async proxy works correctly.
|
||||
* (Self-invocation within the same class bypasses the AOP proxy.)
|
||||
*/
|
||||
@Component
|
||||
public class AnalysisRunner {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(AnalysisRunner.class);
|
||||
|
||||
private final AnalysisJobRepository repository;
|
||||
private final DocumentService documentService;
|
||||
private final DocumentParserClient parserClient;
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
public AnalysisRunner(
|
||||
AnalysisJobRepository repository,
|
||||
DocumentService documentService,
|
||||
DocumentParserClient parserClient,
|
||||
ObjectMapper objectMapper
|
||||
) {
|
||||
this.repository = repository;
|
||||
this.documentService = documentService;
|
||||
this.parserClient = parserClient;
|
||||
this.objectMapper = objectMapper;
|
||||
}
|
||||
|
||||
@Async("analysisExecutor")
|
||||
public void runAnalysis(UUID jobId) {
|
||||
AnalysisJob job = repository.findById(jobId).orElseThrow();
|
||||
job.markRunning();
|
||||
repository.save(job);
|
||||
|
||||
log.info("Starting analysis for document: {}", job.getDocument().getFilename());
|
||||
|
||||
try {
|
||||
Path filePath = documentService.getFilePath(job.getDocument().getId());
|
||||
Map<String, Object> result = parserClient.parse(filePath, job.getDocument().getFilename());
|
||||
|
||||
String markdown = (String) result.getOrDefault("content_markdown", "");
|
||||
String html = (String) result.getOrDefault("content_html", "");
|
||||
Object pages = result.get("pages");
|
||||
String pagesJson = pages != null ? objectMapper.writeValueAsString(pages) : "[]";
|
||||
|
||||
// Update page count on document if available
|
||||
Object pageCount = result.get("page_count");
|
||||
if (pageCount instanceof Number n && n.intValue() > 0) {
|
||||
Document doc = job.getDocument();
|
||||
doc.setPageCount(n.intValue());
|
||||
documentService.save(doc);
|
||||
}
|
||||
|
||||
job.markCompleted(markdown, html, pagesJson);
|
||||
repository.save(job);
|
||||
|
||||
log.info("Analysis completed for document: {}", job.getDocument().getFilename());
|
||||
|
||||
} catch (Exception e) {
|
||||
log.error("Analysis failed for document: {}", job.getDocument().getFilename(), e);
|
||||
job.markFailed(e.getMessage());
|
||||
repository.save(job);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,16 +1,12 @@
|
|||
package com.docling.studio.analysis;
|
||||
|
||||
import com.docling.studio.document.Document;
|
||||
import com.docling.studio.document.DocumentParserClient;
|
||||
import com.docling.studio.document.DocumentService;
|
||||
import com.docling.studio.shared.exception.ResourceNotFoundException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.springframework.scheduling.annotation.Async;
|
||||
import org.springframework.stereotype.Service;
|
||||
import org.springframework.transaction.annotation.Transactional;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
|
||||
@Service
|
||||
|
|
@ -18,70 +14,40 @@ public class AnalysisService {
|
|||
|
||||
private final AnalysisJobRepository repository;
|
||||
private final DocumentService documentService;
|
||||
private final DocumentParserClient parserClient;
|
||||
private final ObjectMapper objectMapper;
|
||||
private final AnalysisRunner runner;
|
||||
|
||||
public AnalysisService(
|
||||
AnalysisJobRepository repository,
|
||||
DocumentService documentService,
|
||||
DocumentParserClient parserClient,
|
||||
ObjectMapper objectMapper
|
||||
AnalysisRunner runner
|
||||
) {
|
||||
this.repository = repository;
|
||||
this.documentService = documentService;
|
||||
this.parserClient = parserClient;
|
||||
this.objectMapper = objectMapper;
|
||||
this.runner = runner;
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public AnalysisJob create(UUID documentId) {
|
||||
Document doc = documentService.findById(documentId);
|
||||
AnalysisJob job = new AnalysisJob(doc);
|
||||
repository.save(job);
|
||||
runAnalysis(job.getId());
|
||||
// Delegate to a separate bean so @Async proxy works (no self-invocation)
|
||||
runner.runAnalysis(job.getId());
|
||||
return job;
|
||||
}
|
||||
|
||||
@Async("analysisExecutor")
|
||||
public void runAnalysis(UUID jobId) {
|
||||
AnalysisJob job = repository.findById(jobId).orElseThrow();
|
||||
job.markRunning();
|
||||
repository.save(job);
|
||||
|
||||
try {
|
||||
Path filePath = documentService.getFilePath(job.getDocument().getId());
|
||||
Map<String, Object> result = parserClient.parse(filePath, job.getDocument().getFilename());
|
||||
|
||||
String markdown = (String) result.getOrDefault("content_markdown", "");
|
||||
String html = (String) result.getOrDefault("content_html", "");
|
||||
Object pages = result.get("pages");
|
||||
String pagesJson = pages != null ? objectMapper.writeValueAsString(pages) : "[]";
|
||||
|
||||
// Update page count on document if available
|
||||
Object pageCount = result.get("page_count");
|
||||
if (pageCount instanceof Number n && n.intValue() > 0) {
|
||||
Document doc = job.getDocument();
|
||||
doc.setPageCount(n.intValue());
|
||||
documentService.save(doc);
|
||||
}
|
||||
|
||||
job.markCompleted(markdown, html, pagesJson);
|
||||
repository.save(job);
|
||||
|
||||
} catch (Exception e) {
|
||||
job.markFailed(e.getMessage());
|
||||
repository.save(job);
|
||||
}
|
||||
}
|
||||
|
||||
@Transactional(readOnly = true)
|
||||
public AnalysisJob findById(UUID id) {
|
||||
return repository.findById(id)
|
||||
.orElseThrow(() -> new ResourceNotFoundException("Analysis not found: " + id));
|
||||
}
|
||||
|
||||
@Transactional(readOnly = true)
|
||||
public List<AnalysisJob> findAll() {
|
||||
return repository.findAllByOrderByCreatedAtDesc();
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public void delete(UUID id) {
|
||||
AnalysisJob job = findById(id);
|
||||
repository.delete(job);
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
package com.docling.studio.config;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.web.servlet.config.annotation.CorsRegistry;
|
||||
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
|
||||
|
|
@ -7,10 +8,13 @@ import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
|
|||
@Configuration
|
||||
public class WebConfig implements WebMvcConfigurer {
|
||||
|
||||
@Value("${app.cors.allowed-origins:http://localhost:3000,http://localhost:5173}")
|
||||
private String allowedOrigins;
|
||||
|
||||
@Override
|
||||
public void addCorsMappings(CorsRegistry registry) {
|
||||
registry.addMapping("/api/**")
|
||||
.allowedOrigins("http://localhost:3000", "http://localhost:5173")
|
||||
.allowedOrigins(allowedOrigins.split(","))
|
||||
.allowedMethods("GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS")
|
||||
.allowedHeaders("*")
|
||||
.allowCredentials(true);
|
||||
|
|
|
|||
|
|
@ -1,19 +1,26 @@
|
|||
package com.docling.studio.document;
|
||||
|
||||
import com.docling.studio.config.DocumentParserProperties;
|
||||
import com.docling.studio.shared.exception.ServiceException;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.client.MultipartBodyBuilder;
|
||||
import org.springframework.stereotype.Component;
|
||||
import org.springframework.web.reactive.function.BodyInserters;
|
||||
import org.springframework.web.reactive.function.client.WebClient;
|
||||
import org.springframework.web.reactive.function.client.WebClientResponseException;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
|
||||
@Component
|
||||
public class DocumentParserClient {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(DocumentParserClient.class);
|
||||
|
||||
private final WebClient webClient;
|
||||
|
||||
public DocumentParserClient(DocumentParserProperties props) {
|
||||
|
|
@ -30,12 +37,26 @@ public class DocumentParserClient {
|
|||
.filename(filename)
|
||||
.contentType(MediaType.APPLICATION_PDF);
|
||||
|
||||
try {
|
||||
return webClient.post()
|
||||
.uri("/parse")
|
||||
.body(BodyInserters.fromMultipartData(builder.build()))
|
||||
.retrieve()
|
||||
.bodyToMono(Map.class)
|
||||
.block();
|
||||
.block(Duration.ofMinutes(10));
|
||||
} catch (WebClientResponseException e) {
|
||||
log.error("Parser returned HTTP {}: {}", e.getStatusCode().value(), e.getResponseBodyAsString());
|
||||
throw new ServiceException("Document parser error: " + extractDetail(e));
|
||||
} catch (java.lang.IllegalStateException e) {
|
||||
if (e.getMessage() != null && e.getMessage().contains("Timeout")) {
|
||||
log.error("Parser timed out after 10 minutes for file: {}", filename);
|
||||
throw new ServiceException("Document parsing timed out. The document may be too large or complex.");
|
||||
}
|
||||
throw new ServiceException("Document parser error: " + e.getMessage());
|
||||
} catch (Exception e) {
|
||||
log.error("Failed to reach document parser", e);
|
||||
throw new ServiceException("Document parser unavailable: " + e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
public byte[] preview(Path filePath, String filename, int page, int dpi) {
|
||||
|
|
@ -53,9 +74,24 @@ public class DocumentParserClient {
|
|||
.body(BodyInserters.fromMultipartData(builder.build()))
|
||||
.retrieve()
|
||||
.bodyToMono(byte[].class)
|
||||
.block();
|
||||
.block(Duration.ofSeconds(30));
|
||||
} catch (Exception e) {
|
||||
log.warn("Preview generation failed for page {}: {}", page, e.getMessage());
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Extract the "detail" field from a FastAPI error response, or fall back to the raw body. */
|
||||
private String extractDetail(WebClientResponseException e) {
|
||||
String body = e.getResponseBodyAsString();
|
||||
try {
|
||||
var tree = new com.fasterxml.jackson.databind.ObjectMapper().readTree(body);
|
||||
if (tree.has("detail")) {
|
||||
return tree.get("detail").asText();
|
||||
}
|
||||
} catch (Exception ignored) {
|
||||
// Not valid JSON — fall through to raw body
|
||||
}
|
||||
return body.length() > 200 ? body.substring(0, 200) : body;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,8 +2,13 @@ package com.docling.studio.document;
|
|||
|
||||
import com.docling.studio.shared.exception.ResourceNotFoundException;
|
||||
import com.docling.studio.shared.exception.ServiceException;
|
||||
import org.apache.pdfbox.Loader;
|
||||
import org.apache.pdfbox.pdmodel.PDDocument;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.stereotype.Service;
|
||||
import org.springframework.transaction.annotation.Transactional;
|
||||
import org.springframework.web.multipart.MultipartFile;
|
||||
|
||||
import java.io.IOException;
|
||||
|
|
@ -15,6 +20,8 @@ import java.util.UUID;
|
|||
@Service
|
||||
public class DocumentService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(DocumentService.class);
|
||||
|
||||
private final DocumentRepository repository;
|
||||
private final DocumentParserClient parserClient;
|
||||
private final Path storagePath;
|
||||
|
|
@ -29,6 +36,7 @@ public class DocumentService {
|
|||
this.storagePath = Path.of(storagePath);
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public Document upload(MultipartFile file) {
|
||||
if (file.isEmpty() || file.getOriginalFilename() == null) {
|
||||
throw new IllegalArgumentException("File is empty or has no name");
|
||||
|
|
@ -46,6 +54,7 @@ public class DocumentService {
|
|||
file.getSize(),
|
||||
target.toString()
|
||||
);
|
||||
doc.setPageCount(countPages(target));
|
||||
return repository.save(doc);
|
||||
|
||||
} catch (IOException e) {
|
||||
|
|
@ -66,6 +75,7 @@ public class DocumentService {
|
|||
.orElseThrow(() -> new ResourceNotFoundException("Document not found: " + id));
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public void delete(UUID id) {
|
||||
Document doc = findById(id);
|
||||
try {
|
||||
|
|
@ -85,4 +95,13 @@ public class DocumentService {
|
|||
Document doc = findById(id);
|
||||
return Path.of(doc.getStoragePath());
|
||||
}
|
||||
|
||||
private int countPages(Path pdfPath) {
|
||||
try (PDDocument pdf = Loader.loadPDF(pdfPath.toFile())) {
|
||||
return pdf.getNumberOfPages();
|
||||
} catch (Exception e) {
|
||||
log.warn("Could not count pages for {}: {}", pdfPath.getFileName(), e.getMessage());
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -3,9 +3,9 @@ server:
|
|||
|
||||
spring:
|
||||
datasource:
|
||||
url: jdbc:postgresql://localhost:5432/docling_studio
|
||||
username: app
|
||||
password: app
|
||||
url: jdbc:postgresql://${DB_HOST:localhost}:5432/${DB_NAME:docling_studio}
|
||||
username: ${SPRING_DATASOURCE_USERNAME:app}
|
||||
password: ${SPRING_DATASOURCE_PASSWORD:app}
|
||||
jpa:
|
||||
hibernate:
|
||||
ddl-auto: validate
|
||||
|
|
@ -19,6 +19,8 @@ spring:
|
|||
|
||||
app:
|
||||
document-parser:
|
||||
base-url: http://localhost:8000
|
||||
base-url: ${APP_DOCUMENT-PARSER_BASE-URL:http://localhost:8000}
|
||||
storage:
|
||||
path: ./uploads
|
||||
path: ${APP_STORAGE_PATH:./uploads}
|
||||
cors:
|
||||
allowed-origins: ${APP_CORS_ALLOWED_ORIGINS:http://localhost:3000,http://localhost:5173}
|
||||
|
|
|
|||
|
|
@ -2,15 +2,15 @@ services:
|
|||
postgres:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_USER: app
|
||||
POSTGRES_PASSWORD: app
|
||||
POSTGRES_DB: docling_studio
|
||||
POSTGRES_USER: ${POSTGRES_USER:-app}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-app}
|
||||
POSTGRES_DB: ${POSTGRES_DB:-docling_studio}
|
||||
ports:
|
||||
- "5432:5432"
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U app -d docling_studio"]
|
||||
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-app} -d ${POSTGRES_DB:-docling_studio}"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
|
|
|
|||
|
|
@ -2,15 +2,15 @@ services:
|
|||
postgres:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_USER: app
|
||||
POSTGRES_PASSWORD: app
|
||||
POSTGRES_DB: docling_studio
|
||||
POSTGRES_USER: ${POSTGRES_USER:-app}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-app}
|
||||
POSTGRES_DB: ${POSTGRES_DB:-docling_studio}
|
||||
ports:
|
||||
- "5432:5432"
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U app -d docling_studio"]
|
||||
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-app} -d ${POSTGRES_DB:-docling_studio}"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
|
|
@ -20,6 +20,10 @@ services:
|
|||
context: ./document-parser
|
||||
ports:
|
||||
- "8000:8000"
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 4g
|
||||
|
||||
backend:
|
||||
build:
|
||||
|
|
@ -27,10 +31,12 @@ services:
|
|||
ports:
|
||||
- "8081:8081"
|
||||
environment:
|
||||
SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/docling_studio
|
||||
SPRING_DATASOURCE_USERNAME: app
|
||||
SPRING_DATASOURCE_PASSWORD: app
|
||||
SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/${POSTGRES_DB:-docling_studio}
|
||||
SPRING_DATASOURCE_USERNAME: ${POSTGRES_USER:-app}
|
||||
SPRING_DATASOURCE_PASSWORD: ${POSTGRES_PASSWORD:-app}
|
||||
APP_DOCUMENT-PARSER_BASE-URL: http://document-parser:8000
|
||||
volumes:
|
||||
- uploads_data:/app/uploads
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
|
|
@ -45,3 +51,4 @@ services:
|
|||
|
||||
volumes:
|
||||
postgres_data:
|
||||
uploads_data:
|
||||
|
|
|
|||
12
document-parser/.dockerignore
Normal file
12
document-parser/.dockerignore
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.egg-info/
|
||||
.git/
|
||||
.gitignore
|
||||
.env
|
||||
*.log
|
||||
.mypy_cache/
|
||||
.pytest_cache/
|
||||
.ruff_cache/
|
||||
|
|
@ -2,6 +2,8 @@ FROM python:3.12-slim
|
|||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
poppler-utils \
|
||||
libgl1 \
|
||||
libglib2.0-0 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
|
|
|||
|
|
@ -1,33 +1,114 @@
|
|||
"""Docling Studio — Document Parser service.
|
||||
|
||||
A FastAPI microservice wrapping the Docling library for structured document
|
||||
extraction. Provides parse (full extraction) and preview (page image) endpoints
|
||||
with configurable pipeline options.
|
||||
"""
|
||||
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import FastAPI, UploadFile, HTTPException, Query
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
from docling.document_converter import DocumentConverter
|
||||
from pdf2image import convert_from_bytes
|
||||
from PIL import Image
|
||||
|
||||
from docling.document_converter import DocumentConverter, PdfFormatOption
|
||||
from docling.datamodel.base_models import InputFormat
|
||||
from docling.datamodel.pipeline_options import (
|
||||
PdfPipelineOptions,
|
||||
TableFormerMode,
|
||||
TableStructureOptions,
|
||||
)
|
||||
from docling_core.types.doc import (
|
||||
CodeItem,
|
||||
FloatingItem,
|
||||
FormulaItem,
|
||||
GroupItem,
|
||||
ListItem,
|
||||
PictureItem,
|
||||
SectionHeaderItem,
|
||||
TableItem,
|
||||
TextItem,
|
||||
TitleItem,
|
||||
)
|
||||
|
||||
from bbox import to_topleft_list
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
app = FastAPI(title="Docling Studio - Document Parser")
|
||||
|
||||
converter = DocumentConverter()
|
||||
app = FastAPI(title="Docling Studio — Document Parser")
|
||||
|
||||
MAX_FILE_SIZE = 50 * 1024 * 1024 # 50 MB
|
||||
|
||||
# Thread lock for converter — DocumentConverter is not thread-safe
|
||||
_converter_lock = threading.Lock()
|
||||
|
||||
# --- Response models ---
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pipeline factory
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _build_converter(
|
||||
do_ocr: bool = True,
|
||||
do_table_structure: bool = True,
|
||||
table_mode: str = "accurate",
|
||||
) -> DocumentConverter:
|
||||
"""Build a DocumentConverter with the given pipeline options.
|
||||
|
||||
Only exposes options that work out of the box (no extra model downloads).
|
||||
"""
|
||||
table_options = TableStructureOptions(
|
||||
do_cell_matching=True,
|
||||
mode=TableFormerMode.ACCURATE if table_mode == "accurate" else TableFormerMode.FAST,
|
||||
)
|
||||
|
||||
pipeline_options = PdfPipelineOptions(
|
||||
do_ocr=do_ocr,
|
||||
do_table_structure=do_table_structure,
|
||||
table_structure_options=table_options,
|
||||
# These require VLM model downloads — disabled by default
|
||||
do_code_enrichment=False,
|
||||
do_formula_enrichment=False,
|
||||
do_picture_classification=False,
|
||||
do_picture_description=False,
|
||||
# Page images are handled by pdf2image in /preview, not needed here
|
||||
generate_page_images=False,
|
||||
generate_picture_images=False,
|
||||
)
|
||||
|
||||
return DocumentConverter(
|
||||
format_options={
|
||||
InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# Default converter (lazy-init on first request)
|
||||
_default_converter: Optional[DocumentConverter] = None
|
||||
|
||||
|
||||
def _get_default_converter() -> DocumentConverter:
|
||||
global _default_converter
|
||||
if _default_converter is None:
|
||||
_default_converter = _build_converter()
|
||||
return _default_converter
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Response models
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class PageElement(BaseModel):
|
||||
type: str
|
||||
bbox: list[float]
|
||||
content: str
|
||||
level: int = 0 # Hierarchy depth from iterate_items()
|
||||
|
||||
|
||||
class PageDetail(BaseModel):
|
||||
|
|
@ -43,56 +124,94 @@ class ParseResponse(BaseModel):
|
|||
content_markdown: str
|
||||
content_html: str
|
||||
pages: list[PageDetail]
|
||||
skipped_items: int = 0 # Transparency on extraction failures
|
||||
|
||||
|
||||
# --- Helpers ---
|
||||
# ---------------------------------------------------------------------------
|
||||
# Element type detection — isinstance-based (no string matching)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def extract_pages_detail(doc_result) -> list[PageDetail]:
|
||||
def _get_element_type(item) -> str:
|
||||
"""Determine the element type using isinstance checks on Docling classes.
|
||||
|
||||
Order matters: more specific types are checked before their parent classes.
|
||||
"""
|
||||
if isinstance(item, TableItem):
|
||||
return "table"
|
||||
if isinstance(item, PictureItem):
|
||||
return "picture"
|
||||
if isinstance(item, TitleItem):
|
||||
return "title"
|
||||
if isinstance(item, SectionHeaderItem):
|
||||
return "section_header"
|
||||
if isinstance(item, ListItem):
|
||||
return "list"
|
||||
if isinstance(item, FormulaItem):
|
||||
return "formula"
|
||||
if isinstance(item, CodeItem):
|
||||
return "code"
|
||||
if isinstance(item, FloatingItem):
|
||||
return "floating"
|
||||
if isinstance(item, GroupItem):
|
||||
return "group"
|
||||
if isinstance(item, TextItem):
|
||||
return "text"
|
||||
return "text"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Page extraction
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def extract_pages_detail(doc_result) -> tuple[list[PageDetail], int]:
|
||||
"""Extract per-page element details with bounding boxes from Docling result.
|
||||
|
||||
Uses Docling's iterate_items() API (preferred) or falls back to document.texts.
|
||||
Both provide a flat iteration over all content items, avoiding duplicates.
|
||||
Returns (pages, skipped_count) for transparent error reporting.
|
||||
"""
|
||||
pages: dict[int, PageDetail] = {}
|
||||
document = doc_result.document
|
||||
skipped = 0
|
||||
|
||||
# Get page dimensions from document pages
|
||||
if hasattr(document, 'pages') and document.pages:
|
||||
# Populate page dimensions from document metadata
|
||||
if hasattr(document, "pages") and document.pages:
|
||||
for page_key, page_obj in document.pages.items():
|
||||
page_no = int(page_key) if isinstance(page_key, str) else page_key
|
||||
width = page_obj.size.width if hasattr(page_obj, 'size') and page_obj.size else 612.0
|
||||
height = page_obj.size.height if hasattr(page_obj, 'size') and page_obj.size else 792.0
|
||||
width = page_obj.size.width if hasattr(page_obj, "size") and page_obj.size else 612.0
|
||||
height = page_obj.size.height if hasattr(page_obj, "size") and page_obj.size else 792.0
|
||||
pages[page_no] = PageDetail(
|
||||
page_number=page_no,
|
||||
width=width,
|
||||
height=height,
|
||||
elements=[]
|
||||
page_number=page_no, width=width, height=height, elements=[]
|
||||
)
|
||||
|
||||
# Use iterate_items() (Docling v2 API) — avoids duplicates
|
||||
if hasattr(document, 'iterate_items'):
|
||||
for item, _level in document.iterate_items():
|
||||
_process_content_item(item, pages)
|
||||
elif hasattr(document, 'texts'):
|
||||
# Use iterate_items() — the Docling v2 API that yields (item, level)
|
||||
if hasattr(document, "iterate_items"):
|
||||
for item, level in document.iterate_items():
|
||||
ok = _process_content_item(item, level, pages)
|
||||
if not ok:
|
||||
skipped += 1
|
||||
elif hasattr(document, "texts"):
|
||||
for text_item in document.texts:
|
||||
_process_content_item(text_item, pages)
|
||||
ok = _process_content_item(text_item, 0, pages)
|
||||
if not ok:
|
||||
skipped += 1
|
||||
|
||||
# Sort by page number
|
||||
return sorted(pages.values(), key=lambda p: p.page_number)
|
||||
sorted_pages = sorted(pages.values(), key=lambda p: p.page_number)
|
||||
return sorted_pages, skipped
|
||||
|
||||
|
||||
def _process_content_item(item, pages: dict[int, PageDetail]):
|
||||
def _process_content_item(item, level: int, pages: dict[int, PageDetail]) -> bool:
|
||||
"""Process a single content item and add it to the appropriate page.
|
||||
|
||||
Silently skips items that lack provenance or fail to process,
|
||||
so one bad item doesn't break the whole extraction.
|
||||
Returns True on success, False if the item was skipped.
|
||||
"""
|
||||
if not hasattr(item, 'prov') or not item.prov:
|
||||
return
|
||||
# Skip groups and items without provenance
|
||||
if isinstance(item, GroupItem):
|
||||
return True # Groups are structural, not content — not an error
|
||||
if not hasattr(item, "prov") or not item.prov:
|
||||
return False
|
||||
|
||||
for prov in item.prov:
|
||||
try:
|
||||
page_no = prov.page_no if hasattr(prov, 'page_no') else 1
|
||||
page_no = prov.page_no if hasattr(prov, "page_no") else 1
|
||||
|
||||
if page_no not in pages:
|
||||
pages[page_no] = PageDetail(
|
||||
|
|
@ -101,108 +220,129 @@ def _process_content_item(item, pages: dict[int, PageDetail]):
|
|||
|
||||
page_height = pages[page_no].height
|
||||
|
||||
bbox = [0, 0, 0, 0]
|
||||
if hasattr(prov, 'bbox') and prov.bbox:
|
||||
bbox = [0.0, 0.0, 0.0, 0.0]
|
||||
if hasattr(prov, "bbox") and prov.bbox:
|
||||
b = prov.bbox
|
||||
if hasattr(b, 'l'):
|
||||
if hasattr(b, "l"):
|
||||
bbox = to_topleft_list(b, page_height)
|
||||
elif isinstance(b, (list, tuple)) and len(b) >= 4:
|
||||
bbox = list(b[:4])
|
||||
|
||||
element_type = _get_element_type(item)
|
||||
content = ""
|
||||
if hasattr(item, 'text'):
|
||||
content = item.text or ""
|
||||
|
||||
pages[page_no].elements.append(PageElement(
|
||||
content = ""
|
||||
if hasattr(item, "text"):
|
||||
content = item.text or ""
|
||||
# For tables, try to export structured content
|
||||
if isinstance(item, TableItem) and hasattr(item, "export_to_markdown"):
|
||||
try:
|
||||
content = item.export_to_markdown()
|
||||
except Exception:
|
||||
pass # Fall back to .text
|
||||
|
||||
pages[page_no].elements.append(
|
||||
PageElement(
|
||||
type=element_type,
|
||||
bbox=bbox,
|
||||
content=content[:500]
|
||||
))
|
||||
content=content,
|
||||
level=level,
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("Skipping item %s: failed to process", type(item).__name__, exc_info=True)
|
||||
logger.warning(
|
||||
"Skipping item %s on page %s",
|
||||
type(item).__name__,
|
||||
getattr(prov, "page_no", "?"),
|
||||
exc_info=True,
|
||||
)
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
|
||||
def _get_element_type(item) -> str:
|
||||
"""Determine the element type from a Docling document item."""
|
||||
type_name = type(item).__name__.lower()
|
||||
if 'table' in type_name:
|
||||
return 'table'
|
||||
if 'picture' in type_name or 'image' in type_name or 'figure' in type_name:
|
||||
return 'picture'
|
||||
if 'section' in type_name or 'heading' in type_name:
|
||||
return 'section_header'
|
||||
if 'list' in type_name:
|
||||
return 'list'
|
||||
if 'formula' in type_name or 'equation' in type_name:
|
||||
return 'formula'
|
||||
if 'caption' in type_name:
|
||||
return 'caption'
|
||||
if hasattr(item, 'label'):
|
||||
label = str(item.label).lower()
|
||||
if 'table' in label:
|
||||
return 'table'
|
||||
if 'picture' in label or 'figure' in label:
|
||||
return 'picture'
|
||||
if 'section' in label or 'head' in label:
|
||||
return 'section_header'
|
||||
if 'list' in label:
|
||||
return 'list'
|
||||
if 'formula' in label:
|
||||
return 'formula'
|
||||
return 'text'
|
||||
|
||||
|
||||
# --- Endpoints ---
|
||||
# ---------------------------------------------------------------------------
|
||||
# Endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@app.post("/parse", response_model=ParseResponse)
|
||||
async def parse(file: UploadFile):
|
||||
async def parse(
|
||||
file: UploadFile,
|
||||
do_ocr: bool = Query(True, description="Enable OCR for scanned documents"),
|
||||
do_table_structure: bool = Query(True, description="Enable table structure extraction"),
|
||||
table_mode: str = Query("accurate", regex="^(accurate|fast)$", description="Table extraction mode"),
|
||||
):
|
||||
"""Parse a document and return structured content with per-page elements."""
|
||||
if not file.filename:
|
||||
raise HTTPException(status_code=400, detail="No filename provided")
|
||||
|
||||
content = await file.read()
|
||||
if len(content) > MAX_FILE_SIZE:
|
||||
raise HTTPException(status_code=413, detail="File too large (max 50MB)")
|
||||
file_content = await file.read()
|
||||
if len(file_content) > MAX_FILE_SIZE:
|
||||
raise HTTPException(status_code=413, detail="File too large (max 50 MB)")
|
||||
|
||||
suffix = Path(file.filename).suffix
|
||||
tmp_path = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
|
||||
tmp.write(content)
|
||||
tmp.write(file_content)
|
||||
tmp_path = tmp.name
|
||||
|
||||
result = converter.convert(tmp_path)
|
||||
# Build converter with requested options (or reuse default)
|
||||
if do_ocr and do_table_structure and table_mode == "accurate":
|
||||
conv = _get_default_converter()
|
||||
else:
|
||||
conv = _build_converter(
|
||||
do_ocr=do_ocr,
|
||||
do_table_structure=do_table_structure,
|
||||
table_mode=table_mode,
|
||||
)
|
||||
|
||||
with _converter_lock:
|
||||
result = conv.convert(tmp_path)
|
||||
|
||||
doc = result.document
|
||||
|
||||
content_markdown = doc.export_to_markdown()
|
||||
content_html = doc.export_to_html() if hasattr(doc, 'export_to_html') else ""
|
||||
content_html = doc.export_to_html() if hasattr(doc, "export_to_html") else ""
|
||||
|
||||
page_count = 0
|
||||
if hasattr(doc, 'pages') and doc.pages:
|
||||
page_count = len(doc.pages)
|
||||
page_count = len(doc.pages) if hasattr(doc, "pages") and doc.pages else 0
|
||||
|
||||
pages_detail = extract_pages_detail(result)
|
||||
pages_detail, skipped = extract_pages_detail(result)
|
||||
|
||||
# Ensure we have page entries even if no elements were extracted
|
||||
if not pages_detail and page_count > 0:
|
||||
pages_detail = [
|
||||
PageDetail(page_number=i + 1, width=612.0, height=792.0, elements=[])
|
||||
for i in range(page_count)
|
||||
]
|
||||
|
||||
if skipped > 0:
|
||||
logger.info(
|
||||
"Parsed %s: %d pages, %d items skipped",
|
||||
file.filename,
|
||||
page_count,
|
||||
skipped,
|
||||
)
|
||||
|
||||
return ParseResponse(
|
||||
filename=file.filename,
|
||||
page_count=page_count or len(pages_detail) or 1,
|
||||
content_markdown=content_markdown,
|
||||
content_html=content_html,
|
||||
pages=pages_detail,
|
||||
skipped_items=skipped,
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.exception("Failed to parse document: %s", file.filename)
|
||||
raise HTTPException(status_code=422, detail=f"Failed to parse document: {str(e)}")
|
||||
raise HTTPException(status_code=422, detail=f"Failed to parse document: {e}")
|
||||
finally:
|
||||
if tmp_path and os.path.exists(tmp_path):
|
||||
try:
|
||||
os.unlink(tmp_path)
|
||||
except OSError:
|
||||
logger.warning("Could not delete temp file: %s", tmp_path)
|
||||
|
||||
|
||||
@app.post("/preview")
|
||||
|
|
@ -211,16 +351,16 @@ async def preview(
|
|||
page: int = Query(1, ge=1),
|
||||
dpi: int = Query(150, ge=72, le=300),
|
||||
):
|
||||
"""Generate a PNG preview of a specific page."""
|
||||
"""Generate a PNG preview of a specific PDF page."""
|
||||
if not file.filename:
|
||||
raise HTTPException(status_code=400, detail="No filename provided")
|
||||
|
||||
content = await file.read()
|
||||
if len(content) > MAX_FILE_SIZE:
|
||||
raise HTTPException(status_code=413, detail="File too large (max 50MB)")
|
||||
file_content = await file.read()
|
||||
if len(file_content) > MAX_FILE_SIZE:
|
||||
raise HTTPException(status_code=413, detail="File too large (max 50 MB)")
|
||||
|
||||
try:
|
||||
images = convert_from_bytes(content, first_page=page, last_page=page, dpi=dpi)
|
||||
images = convert_from_bytes(file_content, first_page=page, last_page=page, dpi=dpi)
|
||||
if not images:
|
||||
raise HTTPException(status_code=404, detail=f"Page {page} not found")
|
||||
|
||||
|
|
@ -229,10 +369,14 @@ async def preview(
|
|||
buf.seek(0)
|
||||
return StreamingResponse(buf, media_type="image/png")
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=422, detail=f"Failed to generate preview: {str(e)}")
|
||||
logger.exception("Failed to generate preview for page %d", page)
|
||||
raise HTTPException(status_code=422, detail=f"Failed to generate preview: {e}")
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health():
|
||||
"""Health check endpoint."""
|
||||
return {"status": "ok"}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
docling
|
||||
fastapi
|
||||
uvicorn[standard]
|
||||
python-multipart
|
||||
pdf2image
|
||||
pillow
|
||||
docling>=2.80.0,<3.0.0
|
||||
docling-core>=2.0.0,<3.0.0
|
||||
fastapi>=0.115.0,<1.0.0
|
||||
uvicorn[standard]>=0.32.0,<1.0.0
|
||||
python-multipart>=0.0.12
|
||||
pdf2image>=1.17.0,<2.0.0
|
||||
pillow>=10.0.0,<11.0.0
|
||||
|
|
|
|||
|
|
@ -40,12 +40,14 @@ import { ref, computed, watch, nextTick, reactive, onMounted, onBeforeUnmount }
|
|||
import { computeScale, bboxToRect, pointInRect } from '../bboxScaling.js'
|
||||
|
||||
const ELEMENT_COLORS = {
|
||||
title: '#EF4444',
|
||||
section_header: '#F97316',
|
||||
text: '#3B82F6',
|
||||
table: '#8B5CF6',
|
||||
picture: '#22C55E',
|
||||
list: '#06B6D4',
|
||||
formula: '#EC4899',
|
||||
code: '#14B8A6',
|
||||
caption: '#EAB308'
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -83,17 +83,27 @@ const pageMarkdown = computed(() => {
|
|||
|
||||
function formatElement(el) {
|
||||
if (!el.content) return ''
|
||||
const indent = ' '.repeat(Math.max(0, (el.level || 0) - 1))
|
||||
switch (el.type) {
|
||||
case 'section_header':
|
||||
return `## ${el.content}`
|
||||
case 'title':
|
||||
return `# ${el.content}`
|
||||
case 'section_header': {
|
||||
// Use hierarchy level for heading depth (h2-h4)
|
||||
const depth = Math.min(Math.max(el.level || 2, 2), 4)
|
||||
return `${'#'.repeat(depth)} ${el.content}`
|
||||
}
|
||||
case 'caption':
|
||||
return `*${el.content}*`
|
||||
return `${indent}*${el.content}*`
|
||||
case 'table':
|
||||
return el.content
|
||||
case 'formula':
|
||||
return `$$${el.content}$$`
|
||||
return `${indent}$$${el.content}$$`
|
||||
case 'code':
|
||||
return `${indent}\`\`\`\n${el.content}\n\`\`\``
|
||||
case 'list':
|
||||
return `${indent}- ${el.content}`
|
||||
default:
|
||||
return el.content
|
||||
return `${indent}${el.content}`
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
|
|
|||
Loading…
Reference in a new issue