Merge pull request #181 from scub-france/feature/feature-flag-ingestion
feat(#180): feature-flag ingestion pipeline and brainless one-liner Quick Start
This commit is contained in:
Pier-Jean Malandrino
GitHub
commit
2477a2af4b
26 changed files with 317 additions and 425 deletions
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -44,5 +44,8 @@ hs_err_pid*
|
|||
# Docker
|
||||
docker-compose.override.yml
|
||||
|
||||
# Audit profiles (internal tooling)
|
||||
profiles/
|
||||
|
||||
# E2E tests — Maven build outputs & Chrome user data
|
||||
e2e/**/target/
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ All notable changes to Docling Studio will be documented in this file.
|
|||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/).
|
||||
|
||||
## [0.4.0] - Unreleased
|
||||
## [0.4.0] - 2026-04-13
|
||||
|
||||
### Added
|
||||
|
||||
|
|
|
|||
|
|
@ -96,15 +96,43 @@ npx prettier --write src/ # auto-format
|
|||
## Running Tests
|
||||
|
||||
```bash
|
||||
# Backend (199 tests)
|
||||
# Backend (377 tests)
|
||||
cd document-parser
|
||||
pytest tests/ -v
|
||||
|
||||
# Frontend (129 tests)
|
||||
# Frontend (156 tests)
|
||||
cd frontend
|
||||
npm run test:run
|
||||
```
|
||||
|
||||
### E2E API (Karate)
|
||||
|
||||
```bash
|
||||
# Generate test PDFs + start stack
|
||||
python e2e/generate-test-data.py
|
||||
docker compose up -d --wait
|
||||
|
||||
# Run all API tests
|
||||
mvn test -f e2e/api/pom.xml
|
||||
|
||||
# Or by tag: @smoke, @regression, @e2e
|
||||
mvn test -f e2e/api/pom.xml -Dkarate.options="--tags @smoke"
|
||||
```
|
||||
|
||||
### E2E UI (Karate UI)
|
||||
|
||||
```bash
|
||||
# Generate test PDFs + start stack (if not already running)
|
||||
python e2e/generate-test-data.py
|
||||
docker compose up -d --wait
|
||||
|
||||
# Run critical UI tests (CI scope)
|
||||
mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @critical"
|
||||
|
||||
# Run all UI tests (local scope)
|
||||
mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @ui"
|
||||
```
|
||||
|
||||
All tests must pass before submitting a PR.
|
||||
|
||||
## Submitting Changes
|
||||
|
|
|
|||
83
README.md
83
README.md
|
|
@ -31,9 +31,13 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t
|
|||
- **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description, image generation
|
||||
- **Bounding box visualization** — color-coded element overlay directly on the PDF
|
||||
- **Per-page results** — right panel syncs with the current PDF page
|
||||
- **Chunking** — split extracted content into semantic chunks (hierarchical, hybrid, or page-based) with configurable token limits and inline editing
|
||||
- **Ingestion pipeline** — Docling → chunking → embedding → OpenSearch vector indexing (one-click from Studio)
|
||||
- **Markdown & HTML export** of extracted content
|
||||
- **Document management** — upload, list, delete
|
||||
- **Document management** — upload, list, delete, search, filter by indexing status
|
||||
- **Analysis history** — re-visit and open past analyses
|
||||
- **Upload limits** — configurable max file size and max page count per document
|
||||
- **Rate limiting** — configurable requests per minute per IP
|
||||
- **Dark / Light theme** and **FR / EN** localization
|
||||
|
||||
|
||||
|
|
@ -74,7 +78,7 @@ document-parser/
|
|||
├── services/ # Use case orchestration
|
||||
│ ├── document_service.py # Upload, delete, preview
|
||||
│ └── analysis_service.py # Async Docling processing
|
||||
└── tests/ # 199 tests (pytest)
|
||||
└── tests/ # 377 tests (pytest)
|
||||
```
|
||||
|
||||
### Frontend structure (feature-based)
|
||||
|
|
@ -98,14 +102,24 @@ frontend/src/
|
|||
|
||||
## Quick Start
|
||||
|
||||
Docling Studio ships two Docker image variants:
|
||||
One command, nothing else to install:
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
|
||||
```
|
||||
|
||||
Open [http://localhost:3000](http://localhost:3000), upload a PDF, and get results. That's it.
|
||||
|
||||
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
|
||||
|
||||
### Image variants
|
||||
|
||||
| Variant | Image tag | Size | Description |
|
||||
|---------|-----------|------|-------------|
|
||||
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only |
|
||||
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
|
||||
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only (downloads ML models on first run) |
|
||||
|
||||
### Docker — remote mode (fastest)
|
||||
For remote mode:
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 \
|
||||
|
|
@ -113,27 +127,17 @@ docker run -p 3000:3000 \
|
|||
ghcr.io/scub-france/docling-studio:latest-remote
|
||||
```
|
||||
|
||||
### Docker — local mode (self-contained)
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
|
||||
```
|
||||
|
||||
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
|
||||
|
||||
Open [http://localhost:3000](http://localhost:3000)
|
||||
|
||||
### Docker Compose (for development)
|
||||
### Docker Compose
|
||||
|
||||
```bash
|
||||
git clone https://github.com/scub-france/Docling-Studio.git
|
||||
cd Docling-Studio
|
||||
|
||||
# Local mode (default)
|
||||
# Simple mode (backend + frontend only)
|
||||
docker compose up --build
|
||||
|
||||
# Remote mode
|
||||
CONVERSION_MODE=remote DOCLING_SERVE_URL=http://your-docling-serve:5001 docker compose up --build
|
||||
# With ingestion pipeline (OpenSearch + embeddings)
|
||||
docker compose --profile ingestion -f docker-compose.yml -f docker-compose.ingestion.yml up --build
|
||||
```
|
||||
|
||||
### Local Development
|
||||
|
|
@ -162,12 +166,12 @@ npm run dev
|
|||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Backend (199 tests)
|
||||
# Backend (377 tests)
|
||||
cd document-parser
|
||||
pip install pytest pytest-asyncio httpx
|
||||
pytest tests/ -v
|
||||
|
||||
# Frontend (129 tests)
|
||||
# Frontend (156 tests)
|
||||
cd frontend
|
||||
npm run test:run
|
||||
```
|
||||
|
|
@ -202,6 +206,43 @@ All configuration is done via environment variables. See [`.env.example`](.env.e
|
|||
| `UPLOAD_DIR` | `./uploads` | File storage directory |
|
||||
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
|
||||
| `CONVERSION_TIMEOUT` | `600` | Max seconds for a single Docling conversion |
|
||||
| `BATCH_PAGE_SIZE` | `10` | Pages per batch (`0` = process all at once) |
|
||||
| `MAX_FILE_SIZE_MB` | `50` | Maximum upload file size in MB (`0` = unlimited) |
|
||||
| `MAX_PAGE_COUNT` | `0` | Maximum number of pages per document (`0` = unlimited) |
|
||||
| `RATE_LIMIT_RPM` | `100` | Max requests per minute per IP (`0` = disabled) |
|
||||
|
||||
## Upload Limits
|
||||
|
||||
Docling Studio enforces configurable limits on uploaded documents to protect the server against oversized files and long-running analyses:
|
||||
|
||||
- **`MAX_FILE_SIZE_MB`** (default `50`) — rejects uploads exceeding this size. Validated at two levels: early `Content-Length` check and streaming byte count.
|
||||
- **`MAX_PAGE_COUNT`** (default `0` = unlimited) — rejects documents with more pages than allowed. Useful on shared instances or Hugging Face Spaces to cap processing time.
|
||||
|
||||
Both limits are exposed in the `/api/health` endpoint so the frontend can display them to the user before upload. Set either to `0` to disable the corresponding check.
|
||||
|
||||
## Ingestion Pipeline (opt-in)
|
||||
|
||||
Docling Studio can optionally index extracted chunks into [OpenSearch](https://opensearch.org/) for vector and full-text search. This requires two additional services (OpenSearch + embedding) and is **disabled by default**.
|
||||
|
||||
To enable ingestion with Docker Compose:
|
||||
|
||||
```bash
|
||||
docker compose --profile ingestion \
|
||||
-f docker-compose.yml -f docker-compose.ingestion.yml \
|
||||
up --build
|
||||
```
|
||||
|
||||
When ingestion is enabled, the UI shows:
|
||||
- An **Ingest** button in Studio to push chunks to OpenSearch
|
||||
- An **OpenSearch** connection status badge in the sidebar
|
||||
- **Indexed / Not indexed** filters on the Documents page
|
||||
- A **Search** page for full-text and vector search across indexed documents
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `OPENSEARCH_URL` | — | OpenSearch endpoint (empty = ingestion disabled) |
|
||||
| `EMBEDDING_URL` | — | Embedding service endpoint (empty = ingestion disabled) |
|
||||
| `EMBEDDING_DIMENSION` | `384` | Vector dimension (must match embedding model) |
|
||||
|
||||
## CI / Release
|
||||
|
||||
|
|
|
|||
|
|
@ -74,7 +74,7 @@ services:
|
|||
DOCLING_SERVE_API_KEY: ${DOCLING_SERVE_API_KEY:-}
|
||||
RATE_LIMIT_RPM: ${RATE_LIMIT_RPM:-100}
|
||||
MAX_FILE_SIZE_MB: ${MAX_FILE_SIZE_MB:-50}
|
||||
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-0}
|
||||
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-10}
|
||||
OPENSEARCH_URL: http://opensearch:9200
|
||||
EMBEDDING_URL: http://embedding:8001
|
||||
command: ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
|
||||
|
|
|
|||
19
docker-compose.ingestion.yml
Normal file
19
docker-compose.ingestion.yml
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
# Override to enable the ingestion pipeline (OpenSearch + embeddings).
|
||||
#
|
||||
# Usage:
|
||||
# docker compose --profile ingestion -f docker-compose.yml -f docker-compose.ingestion.yml up --build
|
||||
#
|
||||
# This wires the backend to the OpenSearch and embedding services started
|
||||
# by the "ingestion" profile and ensures they are healthy before the
|
||||
# backend starts.
|
||||
|
||||
services:
|
||||
document-parser:
|
||||
environment:
|
||||
OPENSEARCH_URL: http://opensearch:9200
|
||||
EMBEDDING_URL: http://embedding:8001
|
||||
depends_on:
|
||||
opensearch:
|
||||
condition: service_healthy
|
||||
embedding:
|
||||
condition: service_healthy
|
||||
|
|
@ -1,6 +1,7 @@
|
|||
services:
|
||||
# --- OpenSearch (single-node, security disabled) ---
|
||||
opensearch:
|
||||
profiles: ["ingestion"]
|
||||
image: opensearchproject/opensearch:2
|
||||
environment:
|
||||
discovery.type: single-node
|
||||
|
|
@ -16,6 +17,7 @@ services:
|
|||
|
||||
# --- Embedding service (sentence-transformers) ---
|
||||
embedding:
|
||||
profiles: ["ingestion"]
|
||||
build:
|
||||
context: ./embedding-service
|
||||
environment:
|
||||
|
|
@ -48,14 +50,9 @@ services:
|
|||
DOCLING_SERVE_API_KEY: ${DOCLING_SERVE_API_KEY:-}
|
||||
RATE_LIMIT_RPM: ${RATE_LIMIT_RPM:-100}
|
||||
MAX_FILE_SIZE_MB: ${MAX_FILE_SIZE_MB:-50}
|
||||
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-0}
|
||||
OPENSEARCH_URL: http://opensearch:9200
|
||||
EMBEDDING_URL: http://embedding:8001
|
||||
depends_on:
|
||||
opensearch:
|
||||
condition: service_healthy
|
||||
embedding:
|
||||
condition: service_healthy
|
||||
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-10}
|
||||
OPENSEARCH_URL: ${OPENSEARCH_URL:-}
|
||||
EMBEDDING_URL: ${EMBEDDING_URL:-}
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
|
|
|
|||
|
|
@ -1,15 +1,28 @@
|
|||
# Getting Started
|
||||
|
||||
Docling Studio ships two Docker image variants:
|
||||
## Quick Start
|
||||
|
||||
| Variant | Image tag | Size | Description |
|
||||
|---------|-----------|------|-------------|
|
||||
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
|
||||
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only (downloads ML models on first run) |
|
||||
One command, nothing else to install:
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
|
||||
```
|
||||
|
||||
Open [http://localhost:3000](http://localhost:3000), upload a PDF, and get results. That's it.
|
||||
|
||||
!!! note
|
||||
The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
|
||||
|
||||
{ width="600" }
|
||||
|
||||
## Docker — remote mode (fastest)
|
||||
## Image Variants
|
||||
|
||||
| Variant | Image tag | Size | Description |
|
||||
|---------|-----------|------|-------------|
|
||||
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only |
|
||||
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
|
||||
|
||||
For remote mode:
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 \
|
||||
|
|
@ -17,27 +30,19 @@ docker run -p 3000:3000 \
|
|||
ghcr.io/scub-france/docling-studio:latest-remote
|
||||
```
|
||||
|
||||
## Docker — local mode (self-contained)
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
|
||||
```
|
||||
|
||||
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
|
||||
|
||||
Open [http://localhost:3000](http://localhost:3000).
|
||||
|
||||
## Docker Compose (recommended for development)
|
||||
## Docker Compose
|
||||
|
||||
```bash
|
||||
git clone https://github.com/scub-france/Docling-Studio.git
|
||||
cd Docling-Studio
|
||||
|
||||
# Local mode (default)
|
||||
# Simple mode (backend + frontend only)
|
||||
docker compose up --build
|
||||
|
||||
# Remote mode
|
||||
CONVERSION_MODE=remote DOCLING_SERVE_URL=http://your-docling-serve:5001 docker compose up --build
|
||||
# With ingestion pipeline (OpenSearch + embeddings)
|
||||
docker compose --profile ingestion \
|
||||
-f docker-compose.yml -f docker-compose.ingestion.yml \
|
||||
up --build
|
||||
```
|
||||
|
||||
## Local Development
|
||||
|
|
@ -136,10 +141,48 @@ All configuration is done via environment variables:
|
|||
| `UPLOAD_DIR` | `./uploads` | File storage directory |
|
||||
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
|
||||
| `CONVERSION_TIMEOUT` | `600` | Max seconds per Docling conversion |
|
||||
| `BATCH_PAGE_SIZE` | `10` | Pages per batch (`0` = process all at once) |
|
||||
| `MAX_CONCURRENT_ANALYSES` | `3` | Maximum parallel analysis jobs |
|
||||
| `DEPLOYMENT_MODE` | `self-hosted` | `self-hosted` or `huggingface` (shows disclaimer banner) |
|
||||
| `MAX_FILE_SIZE_MB` | `50` | Maximum upload file size in MB (`0` = unlimited) |
|
||||
| `MAX_PAGE_COUNT` | `0` | Maximum number of pages per document (`0` = unlimited) |
|
||||
| `RATE_LIMIT_RPM` | `100` | Max requests per minute per IP (`0` = disabled) |
|
||||
| `APP_VERSION` | `dev` | Application version (set automatically by CI/Docker) |
|
||||
|
||||
## Upload Limits
|
||||
|
||||
Docling Studio enforces configurable limits on uploaded documents to protect the server against oversized files and long-running analyses:
|
||||
|
||||
- **`MAX_FILE_SIZE_MB`** (default `50`) — rejects uploads exceeding this size. Validated at two levels: early `Content-Length` check and streaming byte count.
|
||||
- **`MAX_PAGE_COUNT`** (default `0` = unlimited) — rejects documents with more pages than allowed. Useful on shared instances or Hugging Face Spaces to cap processing time.
|
||||
|
||||
Both limits are exposed in the `/api/health` endpoint so the frontend can display them to the user before upload. Set either to `0` to disable the corresponding check.
|
||||
|
||||
## Ingestion Pipeline (opt-in)
|
||||
|
||||
Docling Studio can optionally index extracted chunks into [OpenSearch](https://opensearch.org/) for vector and full-text search. This requires two additional services (OpenSearch + embedding) and is **disabled by default**.
|
||||
|
||||
To enable ingestion with Docker Compose:
|
||||
|
||||
```bash
|
||||
docker compose --profile ingestion \
|
||||
-f docker-compose.yml -f docker-compose.ingestion.yml \
|
||||
up --build
|
||||
```
|
||||
|
||||
When ingestion is enabled, the UI shows:
|
||||
|
||||
- An **Ingest** button in Studio to push chunks to OpenSearch
|
||||
- An **OpenSearch** connection status badge in the sidebar
|
||||
- **Indexed / Not indexed** filters on the Documents page
|
||||
- A **Search** page for full-text and vector search across indexed documents
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `OPENSEARCH_URL` | — | OpenSearch endpoint (empty = ingestion disabled) |
|
||||
| `EMBEDDING_URL` | — | Embedding service endpoint (empty = ingestion disabled) |
|
||||
| `EMBEDDING_DIMENSION` | `384` | Vector dimension (must match embedding model) |
|
||||
|
||||
## System Requirements
|
||||
|
||||
| | Remote image | Local image |
|
||||
|
|
|
|||
|
|
@ -18,7 +18,8 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t
|
|||
- **Document management** — upload, list, delete
|
||||
- **Analysis history** — re-visit and open past analyses
|
||||
- **Feature flags** — capabilities adapt to the conversion engine (local vs remote)
|
||||
- **Rate limiting** — 60 requests per minute per IP to protect the backend
|
||||
- **Upload limits** — configurable max file size (`MAX_FILE_SIZE_MB`) and max page count (`MAX_PAGE_COUNT`) per document
|
||||
- **Rate limiting** — configurable requests per minute per IP (`RATE_LIMIT_RPM`)
|
||||
- **Deployment modes** — self-hosted (default) or HuggingFace Spaces (with disclaimer banner)
|
||||
- **Health endpoint** — `/api/health` reports engine type, deployment mode, and database status
|
||||
- **Dark / Light theme** and **FR / EN** localization
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ class HealthResponse(_CamelModel):
|
|||
database: str
|
||||
max_page_count: int | None = None
|
||||
max_file_size_mb: int | None = None
|
||||
ingestion_available: bool = False
|
||||
|
||||
|
||||
class DocumentResponse(_CamelModel):
|
||||
|
|
|
|||
|
|
@ -94,7 +94,7 @@ class Settings:
|
|||
max_file_size=int(os.environ.get("MAX_FILE_SIZE", "0")),
|
||||
max_file_size_mb=int(os.environ.get("MAX_FILE_SIZE_MB", "50")),
|
||||
rate_limit_rpm=int(os.environ.get("RATE_LIMIT_RPM", "100")),
|
||||
batch_page_size=int(os.environ.get("BATCH_PAGE_SIZE", "0")),
|
||||
batch_page_size=int(os.environ.get("BATCH_PAGE_SIZE", "10")),
|
||||
opensearch_url=os.environ.get("OPENSEARCH_URL", ""),
|
||||
embedding_url=os.environ.get("EMBEDDING_URL", ""),
|
||||
embedding_dimension=int(os.environ.get("EMBEDDING_DIMENSION", "384")),
|
||||
|
|
|
|||
|
|
@ -138,7 +138,11 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
|||
document_repo, analysis_repo = _build_repos()
|
||||
app.state.analysis_service = _build_analysis_service(document_repo, analysis_repo)
|
||||
app.state.document_service = _build_document_service(document_repo, analysis_repo)
|
||||
app.state.ingestion_service = _build_ingestion_service()
|
||||
ingestion_service = _build_ingestion_service()
|
||||
app.state.ingestion_service = ingestion_service
|
||||
if ingestion_service is not None:
|
||||
app.include_router(ingestion_router)
|
||||
logger.info("Ingestion router mounted")
|
||||
logger.info("Docling Studio backend ready (engine=%s)", settings.conversion_engine)
|
||||
yield
|
||||
|
||||
|
|
@ -165,7 +169,6 @@ if settings.rate_limit_rpm > 0:
|
|||
|
||||
app.include_router(documents_router)
|
||||
app.include_router(analyses_router)
|
||||
app.include_router(ingestion_router)
|
||||
|
||||
|
||||
@app.get("/api/health", response_model=HealthResponse)
|
||||
|
|
@ -188,4 +191,5 @@ async def health() -> HealthResponse:
|
|||
database=db_status,
|
||||
max_page_count=settings.max_page_count if settings.max_page_count > 0 else None,
|
||||
max_file_size_mb=settings.max_file_size_mb if settings.max_file_size_mb > 0 else None,
|
||||
ingestion_available=getattr(app.state, "ingestion_service", None) is not None,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -51,6 +51,23 @@ class TestHealthEndpoint:
|
|||
assert "maxFileSizeMb" in data
|
||||
assert data["maxFileSizeMb"] == 50
|
||||
|
||||
def test_health_exposes_ingestion_available_false(self, client):
|
||||
original = getattr(app.state, "ingestion_service", None)
|
||||
app.state.ingestion_service = None
|
||||
resp = client.get("/api/health")
|
||||
app.state.ingestion_service = original
|
||||
data = resp.json()
|
||||
assert "ingestionAvailable" in data
|
||||
assert data["ingestionAvailable"] is False
|
||||
|
||||
def test_health_exposes_ingestion_available_true(self, client):
|
||||
original = getattr(app.state, "ingestion_service", None)
|
||||
app.state.ingestion_service = MagicMock()
|
||||
resp = client.get("/api/health")
|
||||
app.state.ingestion_service = original
|
||||
data = resp.json()
|
||||
assert data["ingestionAvailable"] is True
|
||||
|
||||
|
||||
class TestDocumentEndpoints:
|
||||
def test_list_documents(self, client, mock_document_service):
|
||||
|
|
|
|||
|
|
@ -104,7 +104,18 @@ class TestIngestionStatus:
|
|||
|
||||
|
||||
class TestIngestionDisabled:
|
||||
def test_returns_503_when_disabled(self) -> None:
|
||||
def test_router_not_mounted_returns_404(self) -> None:
|
||||
"""When ingestion service is None, router should not be mounted → 404."""
|
||||
app = FastAPI()
|
||||
# Do NOT include ingestion router — simulates main.py conditional mount
|
||||
app.state.ingestion_service = None
|
||||
app.state.analysis_service = AsyncMock()
|
||||
tc = TestClient(app)
|
||||
resp = tc.post("/api/ingestion/job-1")
|
||||
assert resp.status_code == 404
|
||||
|
||||
def test_status_still_returns_503_when_router_mounted_but_service_none(self) -> None:
|
||||
"""If router is mounted but service is None, endpoints return 503."""
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
app.state.ingestion_service = None
|
||||
|
|
|
|||
4
frontend/package-lock.json
generated
4
frontend/package-lock.json
generated
|
|
@ -1,12 +1,12 @@
|
|||
{
|
||||
"name": "docling-studio",
|
||||
"version": "0.3.0",
|
||||
"version": "0.3.1",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "docling-studio",
|
||||
"version": "0.3.0",
|
||||
"version": "0.3.1",
|
||||
"dependencies": {
|
||||
"dompurify": "^3.3.3",
|
||||
"marked": "^17.0.4",
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"name": "docling-studio",
|
||||
"version": "0.3.1",
|
||||
"version": "0.4.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
|
|
|
|||
|
|
@ -202,7 +202,10 @@
|
|||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chunk-empty" v-else-if="!chunkingStore.rechunking">
|
||||
<div
|
||||
class="chunk-empty"
|
||||
v-if="!pageChunks.length && !chunkingStore.rechunking && deleteConfirmIdx === -1"
|
||||
>
|
||||
<p>
|
||||
{{ chunks.length ? t('chunking.noChunksOnPage') : t('chunking.noChunks') }}
|
||||
</p>
|
||||
|
|
|
|||
|
|
@ -82,6 +82,38 @@ describe('useFeatureFlagStore', () => {
|
|||
expect(store.maxFileSizeMb).toBe(0)
|
||||
})
|
||||
|
||||
it('enables ingestion when ingestionAvailable is true', async () => {
|
||||
mockApiFetch.mockResolvedValue({
|
||||
status: 'ok',
|
||||
engine: 'local',
|
||||
ingestionAvailable: true,
|
||||
})
|
||||
const store = useFeatureFlagStore()
|
||||
await store.load()
|
||||
expect(store.ingestionAvailable).toBe(true)
|
||||
expect(store.isEnabled('ingestion')).toBe(true)
|
||||
})
|
||||
|
||||
it('disables ingestion when ingestionAvailable is false', async () => {
|
||||
mockApiFetch.mockResolvedValue({
|
||||
status: 'ok',
|
||||
engine: 'local',
|
||||
ingestionAvailable: false,
|
||||
})
|
||||
const store = useFeatureFlagStore()
|
||||
await store.load()
|
||||
expect(store.ingestionAvailable).toBe(false)
|
||||
expect(store.isEnabled('ingestion')).toBe(false)
|
||||
})
|
||||
|
||||
it('defaults ingestionAvailable to false when missing', async () => {
|
||||
mockApiFetch.mockResolvedValue({ status: 'ok', engine: 'local' })
|
||||
const store = useFeatureFlagStore()
|
||||
await store.load()
|
||||
expect(store.ingestionAvailable).toBe(false)
|
||||
expect(store.isEnabled('ingestion')).toBe(false)
|
||||
})
|
||||
|
||||
it('handles health endpoint failure gracefully', async () => {
|
||||
mockApiFetch.mockRejectedValue(new Error('Network error'))
|
||||
const store = useFeatureFlagStore()
|
||||
|
|
|
|||
|
|
@ -13,9 +13,10 @@ interface HealthResponse {
|
|||
deploymentMode?: DeploymentMode
|
||||
maxPageCount?: number
|
||||
maxFileSizeMb?: number
|
||||
ingestionAvailable?: boolean
|
||||
}
|
||||
|
||||
export type FeatureFlag = 'chunking' | 'disclaimer'
|
||||
export type FeatureFlag = 'chunking' | 'disclaimer' | 'ingestion'
|
||||
|
||||
interface FeatureFlagDef {
|
||||
description: string
|
||||
|
|
@ -25,6 +26,7 @@ interface FeatureFlagDef {
|
|||
interface FeatureFlagContext {
|
||||
engine: ConversionEngine | null
|
||||
deploymentMode: DeploymentMode | null
|
||||
ingestionAvailable: boolean
|
||||
}
|
||||
|
||||
const featureRegistry: Record<FeatureFlag, FeatureFlagDef> = {
|
||||
|
|
@ -36,6 +38,10 @@ const featureRegistry: Record<FeatureFlag, FeatureFlagDef> = {
|
|||
description: 'Show shared-instance disclaimer banner',
|
||||
isEnabled: (ctx) => ctx.deploymentMode === 'huggingface',
|
||||
},
|
||||
ingestion: {
|
||||
description: 'OpenSearch ingestion pipeline (embedding + vector indexing)',
|
||||
isEnabled: (ctx) => ctx.ingestionAvailable,
|
||||
},
|
||||
}
|
||||
|
||||
export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
||||
|
|
@ -43,6 +49,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
|||
const deploymentMode = ref<DeploymentMode | null>(null)
|
||||
const maxPageCount = ref<number>(0)
|
||||
const maxFileSizeMb = ref<number>(0)
|
||||
const ingestionAvailable = ref(false)
|
||||
const appVersion = ref<string>(__APP_VERSION__)
|
||||
const loaded = ref(false)
|
||||
const error = ref<string | null>(null)
|
||||
|
|
@ -50,6 +57,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
|||
const context = computed<FeatureFlagContext>(() => ({
|
||||
engine: engine.value,
|
||||
deploymentMode: deploymentMode.value,
|
||||
ingestionAvailable: ingestionAvailable.value,
|
||||
}))
|
||||
|
||||
function isEnabled(flag: FeatureFlag): boolean {
|
||||
|
|
@ -65,6 +73,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
|||
deploymentMode.value = data.deploymentMode ?? 'self-hosted'
|
||||
maxPageCount.value = data.maxPageCount ?? 0
|
||||
maxFileSizeMb.value = data.maxFileSizeMb ?? 0
|
||||
ingestionAvailable.value = data.ingestionAvailable ?? false
|
||||
appMaxFileSizeMb.value = maxFileSizeMb.value
|
||||
appMaxPageCount.value = maxPageCount.value
|
||||
if (data.version) appVersion.value = data.version
|
||||
|
|
@ -81,6 +90,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
|||
deploymentMode,
|
||||
maxPageCount,
|
||||
maxFileSizeMb,
|
||||
ingestionAvailable,
|
||||
appVersion,
|
||||
loaded,
|
||||
error,
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
class="search-input"
|
||||
:placeholder="t('ingestion.search')"
|
||||
/>
|
||||
<div class="filter-group">
|
||||
<div v-if="ingestionEnabled" class="filter-group">
|
||||
<button
|
||||
v-for="f in filters"
|
||||
:key="f.value"
|
||||
|
|
@ -54,17 +54,19 @@
|
|||
</div>
|
||||
</div>
|
||||
<div class="doc-row-actions">
|
||||
<span
|
||||
v-if="ingestionStore.ingestedDocs[doc.id]"
|
||||
class="status-badge indexed"
|
||||
:title="t('ingestion.chunksIndexed', { n: ingestionStore.ingestedDocs[doc.id] })"
|
||||
>
|
||||
{{ t('ingestion.indexed') }}
|
||||
<span class="badge-count">{{ ingestionStore.ingestedDocs[doc.id] }}</span>
|
||||
</span>
|
||||
<span v-else class="status-badge not-indexed">
|
||||
{{ t('ingestion.notIndexed') }}
|
||||
</span>
|
||||
<template v-if="ingestionEnabled">
|
||||
<span
|
||||
v-if="ingestionStore.ingestedDocs[doc.id]"
|
||||
class="status-badge indexed"
|
||||
:title="t('ingestion.chunksIndexed', { n: ingestionStore.ingestedDocs[doc.id] })"
|
||||
>
|
||||
{{ t('ingestion.indexed') }}
|
||||
<span class="badge-count">{{ ingestionStore.ingestedDocs[doc.id] }}</span>
|
||||
</span>
|
||||
<span v-else class="status-badge not-indexed">
|
||||
{{ t('ingestion.notIndexed') }}
|
||||
</span>
|
||||
</template>
|
||||
<button
|
||||
class="action-btn"
|
||||
:title="t('ingestion.openInStudio')"
|
||||
|
|
@ -77,7 +79,7 @@
|
|||
</svg>
|
||||
</button>
|
||||
<button
|
||||
v-if="ingestionStore.ingestedDocs[doc.id]"
|
||||
v-if="ingestionEnabled && ingestionStore.ingestedDocs[doc.id]"
|
||||
class="action-btn unindex"
|
||||
:title="t('ingestion.deleteIndex')"
|
||||
@click="confirmRemoveFromIndex(doc.id)"
|
||||
|
|
@ -110,13 +112,16 @@
|
|||
import { computed, onMounted, ref } from 'vue'
|
||||
import { useRouter } from 'vue-router'
|
||||
import { useDocumentStore } from '../features/document/store'
|
||||
import { useFeatureFlagStore } from '../features/feature-flags/store'
|
||||
import { useIngestionStore } from '../features/ingestion/store'
|
||||
import { useI18n } from '../shared/i18n'
|
||||
import { formatSize } from '../shared/format'
|
||||
import type { Document } from '../shared/types'
|
||||
|
||||
const docStore = useDocumentStore()
|
||||
const featureStore = useFeatureFlagStore()
|
||||
const ingestionStore = useIngestionStore()
|
||||
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
|
||||
const router = useRouter()
|
||||
const { t } = useI18n()
|
||||
|
||||
|
|
@ -173,7 +178,7 @@ function confirmRemoveFromIndex(docId: string) {
|
|||
}
|
||||
|
||||
async function handleDelete(docId: string) {
|
||||
if (ingestionStore.ingestedDocs[docId]) {
|
||||
if (ingestionEnabled.value && ingestionStore.ingestedDocs[docId]) {
|
||||
await ingestionStore.deleteIngested(docId)
|
||||
}
|
||||
await docStore.remove(docId)
|
||||
|
|
@ -181,7 +186,9 @@ async function handleDelete(docId: string) {
|
|||
|
||||
onMounted(() => {
|
||||
docStore.load()
|
||||
ingestionStore.checkAvailability()
|
||||
if (ingestionEnabled.value) {
|
||||
ingestionStore.checkAvailability()
|
||||
}
|
||||
})
|
||||
</script>
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
<h1 class="page-title">{{ t('nav.search') }}</h1>
|
||||
</div>
|
||||
|
||||
<div v-if="!ingestionStore.available" class="tab-empty">
|
||||
<div v-if="!ingestionEnabled || !ingestionStore.available" class="tab-empty">
|
||||
{{ t('ingestion.unavailable') }}
|
||||
</div>
|
||||
|
||||
|
|
@ -50,13 +50,16 @@
|
|||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { onMounted, ref } from 'vue'
|
||||
import { computed, onMounted, ref } from 'vue'
|
||||
import { useSearchStore } from '../features/search/store'
|
||||
import { useFeatureFlagStore } from '../features/feature-flags/store'
|
||||
import { useIngestionStore } from '../features/ingestion/store'
|
||||
import { useI18n } from '../shared/i18n'
|
||||
|
||||
const searchStore = useSearchStore()
|
||||
const featureStore = useFeatureFlagStore()
|
||||
const ingestionStore = useIngestionStore()
|
||||
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
|
||||
const { t } = useI18n()
|
||||
|
||||
const searchInput = ref('')
|
||||
|
|
@ -70,7 +73,9 @@ function runSearch() {
|
|||
}
|
||||
|
||||
onMounted(() => {
|
||||
ingestionStore.checkAvailability()
|
||||
if (ingestionEnabled.value) {
|
||||
ingestionStore.checkAvailability()
|
||||
}
|
||||
})
|
||||
</script>
|
||||
|
||||
|
|
|
|||
|
|
@ -67,7 +67,7 @@
|
|||
{{ t('studio.prepare') }}
|
||||
</button>
|
||||
<button
|
||||
v-if="chunkingEnabled && ingestionStore.available"
|
||||
v-if="chunkingEnabled && ingestionEnabled && ingestionStore.available"
|
||||
class="toggle-btn"
|
||||
data-e2e="toggle-btn"
|
||||
:class="{ active: mode === 'ingest' }"
|
||||
|
|
@ -504,6 +504,7 @@ const analysisStore = useAnalysisStore()
|
|||
const ingestionStore = useIngestionStore()
|
||||
const { t } = useI18n()
|
||||
const chunkingEnabled = useFeatureFlag('chunking')
|
||||
const ingestionEnabled = useFeatureFlag('ingestion')
|
||||
|
||||
const mode = ref('configure')
|
||||
const currentPage = ref(1)
|
||||
|
|
@ -636,7 +637,9 @@ watch(
|
|||
onMounted(async () => {
|
||||
await documentStore.load()
|
||||
analysisStore.load()
|
||||
ingestionStore.checkAvailability()
|
||||
if (ingestionEnabled.value) {
|
||||
ingestionStore.checkAvailability()
|
||||
}
|
||||
|
||||
// Restore analysis from history via query param
|
||||
const analysisId = route.query.analysisId
|
||||
|
|
|
|||
|
|
@ -130,7 +130,7 @@ const messages: Messages = {
|
|||
'chunking.deleteConfirm':
|
||||
'Supprimer ce chunk ? Il sera marqué comme supprimé jusqu\u2019à la prochaine synchronisation.',
|
||||
'chunking.batchNotice':
|
||||
'Le chunking n\u2019est pas disponible pour cette analyse. Les documents volumineux trait\u00e9s par batch ne g\u00e9n\u00e8rent pas la structure interne n\u00e9cessaire au d\u00e9coupage.',
|
||||
'Le chunking n\u2019est pas disponible pour cette analyse. Les documents volumineux trait\u00e9s par batch ne g\u00e9n\u00e8rent pas la structure interne n\u00e9cessaire au d\u00e9coupage. Coming soon !',
|
||||
|
||||
// Search
|
||||
'nav.search': 'Recherche',
|
||||
|
|
@ -299,7 +299,7 @@ const messages: Messages = {
|
|||
'chunking.deleteConfirm':
|
||||
'Delete this chunk? It will be marked as deleted until the next sync.',
|
||||
'chunking.batchNotice':
|
||||
'Chunking is not available for this analysis. Large documents processed in batch mode do not generate the internal structure required for chunking.',
|
||||
'Chunking is not available for this analysis. Large documents processed in batch mode do not generate the internal structure required for chunking. Coming soon!',
|
||||
|
||||
'nav.search': 'Search',
|
||||
'search.hint': 'Enter a term to search through indexed chunks.',
|
||||
|
|
|
|||
|
|
@ -46,6 +46,7 @@
|
|||
</RouterLink>
|
||||
|
||||
<RouterLink
|
||||
v-if="ingestionEnabled"
|
||||
to="/search"
|
||||
class="nav-item"
|
||||
data-e2e="nav-search"
|
||||
|
|
@ -96,7 +97,7 @@
|
|||
|
||||
<div class="sidebar-footer">
|
||||
<div
|
||||
v-if="ingestionStore.available"
|
||||
v-if="ingestionEnabled && ingestionStore.available"
|
||||
class="opensearch-status"
|
||||
:title="
|
||||
ingestionStore.opensearchConnected
|
||||
|
|
@ -136,6 +137,7 @@ import { useIngestionStore } from '../../features/ingestion/store'
|
|||
|
||||
const featureStore = useFeatureFlagStore()
|
||||
const ingestionStore = useIngestionStore()
|
||||
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
|
||||
const version = computed(() => featureStore.appVersion)
|
||||
const route = useRoute()
|
||||
const { t } = useI18n()
|
||||
|
|
@ -145,8 +147,10 @@ defineProps({
|
|||
})
|
||||
|
||||
onMounted(() => {
|
||||
ingestionStore.checkAvailability()
|
||||
ingestionStore.startPolling(30_000)
|
||||
if (ingestionEnabled.value) {
|
||||
ingestionStore.checkAvailability()
|
||||
ingestionStore.startPolling(30_000)
|
||||
}
|
||||
})
|
||||
|
||||
onBeforeUnmount(() => {
|
||||
|
|
|
|||
|
|
@ -1,267 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
# ============================================================================
|
||||
# Docling Studio — Automated Audit Checks (FastAPI + Vue 3 profile)
|
||||
# ============================================================================
|
||||
# Runs verification commands for each of the 12 release audits.
|
||||
# Usage: bash profiles/fastapi-vue/commands.sh
|
||||
# Run from the repository root.
|
||||
# ============================================================================
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m'
|
||||
|
||||
PASS=0
|
||||
WARN=0
|
||||
FAIL=0
|
||||
|
||||
pass() { echo -e " ${GREEN}PASS${NC} $1"; ((PASS++)); }
|
||||
warn() { echo -e " ${YELLOW}WARN${NC} $1"; ((WARN++)); }
|
||||
fail() { echo -e " ${RED}FAIL${NC} $1"; ((FAIL++)); }
|
||||
|
||||
# ── 01. Clean Architecture ─────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "== 01. Clean Architecture =="
|
||||
|
||||
# Domain must not import from api, persistence, infra
|
||||
if grep -rn "from api\|from persistence\|from infra\|import fastapi\|import aiosqlite" document-parser/domain/ 2>/dev/null; then
|
||||
fail "Domain layer imports forbidden modules"
|
||||
else
|
||||
pass "Domain layer has no forbidden imports"
|
||||
fi
|
||||
|
||||
# API must not import from persistence directly
|
||||
if grep -rn "from persistence" document-parser/api/ 2>/dev/null; then
|
||||
fail "API layer imports directly from persistence"
|
||||
else
|
||||
pass "API layer does not import from persistence"
|
||||
fi
|
||||
|
||||
# ── 02. DDD ────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "== 02. DDD =="
|
||||
|
||||
# Domain models exist
|
||||
if [ -f document-parser/domain/models.py ] && [ -f document-parser/domain/ports.py ]; then
|
||||
pass "Domain models and ports exist"
|
||||
else
|
||||
fail "Missing domain/models.py or domain/ports.py"
|
||||
fi
|
||||
|
||||
# Value objects exist
|
||||
if [ -f document-parser/domain/value_objects.py ]; then
|
||||
pass "Value objects defined"
|
||||
else
|
||||
warn "No value_objects.py in domain"
|
||||
fi
|
||||
|
||||
# ── 03. Clean Code ─────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "== 03. Clean Code =="
|
||||
|
||||
# Check for files > 300 lines (backend)
|
||||
LARGE_FILES=$(find document-parser -name "*.py" ! -path "*/.venv/*" ! -path "*/__pycache__/*" ! -path "*/tests/*" -exec awk 'END{if(NR>300) print FILENAME": "NR" lines"}' {} \;)
|
||||
if [ -n "$LARGE_FILES" ]; then
|
||||
warn "Large Python files (>300 lines):"
|
||||
echo "$LARGE_FILES"
|
||||
else
|
||||
pass "No Python files exceed 300 lines"
|
||||
fi
|
||||
|
||||
# Check for files > 300 lines (frontend)
|
||||
LARGE_VUE=$(find frontend/src -name "*.vue" -o -name "*.ts" | xargs awk 'END{if(NR>300) print FILENAME": "NR" lines"}' 2>/dev/null)
|
||||
if [ -n "$LARGE_VUE" ]; then
|
||||
warn "Large frontend files (>300 lines):"
|
||||
echo "$LARGE_VUE"
|
||||
else
|
||||
pass "No frontend files exceed 300 lines"
|
||||
fi
|
||||
|
||||
# ── 04. KISS ───────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "== 04. KISS =="
|
||||
|
||||
# Check for overly complex patterns
|
||||
if grep -rn "type:\s*ignore" document-parser/ --include="*.py" ! -path "*/.venv/*" 2>/dev/null; then
|
||||
warn "Found type: ignore comments (review if justified)"
|
||||
else
|
||||
pass "No unjustified type: ignore"
|
||||
fi
|
||||
|
||||
# ── 05. DRY ────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "== 05. DRY =="
|
||||
|
||||
# Check for magic numbers in backend
|
||||
if grep -rn "[^a-zA-Z_][0-9]\{3,\}[^0-9]" document-parser/ --include="*.py" ! -path "*/.venv/*" ! -path "*/tests/*" 2>/dev/null | grep -v "port\|version\|status\|#\|MAX_\|DEFAULT_\|LIMIT_" | head -5; then
|
||||
warn "Possible magic numbers found (review above)"
|
||||
else
|
||||
pass "No obvious magic numbers"
|
||||
fi
|
||||
|
||||
# ── 06. SOLID ──────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "== 06. SOLID =="
|
||||
|
||||
# Check that ports (interfaces) exist
|
||||
if grep -l "Protocol\|ABC\|abstractmethod" document-parser/domain/ports.py 2>/dev/null; then
|
||||
pass "Domain ports use Protocol/ABC (Dependency Inversion)"
|
||||
else
|
||||
fail "No abstract ports found in domain"
|
||||
fi
|
||||
|
||||
# ── 07. Decoupling ─────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "== 07. Decoupling =="
|
||||
|
||||
# Frontend should not hardcode backend URLs (except in config)
|
||||
if grep -rn "localhost:8000\|127.0.0.1:8000" frontend/src/ --include="*.ts" --include="*.vue" 2>/dev/null | grep -v "config\|env\|http.ts"; then
|
||||
fail "Frontend hardcodes backend URL outside config"
|
||||
else
|
||||
pass "Frontend backend URL is configurable"
|
||||
fi
|
||||
|
||||
# ── 08. Security ───────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "== 08. Security =="
|
||||
|
||||
# Check for hardcoded secrets
|
||||
if grep -rni "password\s*=\s*['\"].\+['\"\|secret\s*=\s*['\"].\+['\"\|api_key\s*=\s*['\"].\+['\"]" document-parser/ --include="*.py" ! -path "*/.venv/*" ! -path "*/tests/*" 2>/dev/null; then
|
||||
fail "Possible hardcoded secrets found"
|
||||
else
|
||||
pass "No hardcoded secrets detected"
|
||||
fi
|
||||
|
||||
# Check for eval/exec
|
||||
if grep -rn "\beval(\|exec(" document-parser/ --include="*.py" ! -path "*/.venv/*" 2>/dev/null; then
|
||||
fail "eval() or exec() found in backend"
|
||||
else
|
||||
pass "No eval/exec in backend"
|
||||
fi
|
||||
|
||||
# Check CORS configuration exists
|
||||
if grep -rn "CORSMiddleware" document-parser/ --include="*.py" ! -path "*/.venv/*" 2>/dev/null > /dev/null; then
|
||||
pass "CORS middleware is configured"
|
||||
else
|
||||
warn "No CORS middleware found"
|
||||
fi
|
||||
|
||||
# ── 09. Tests ──────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "== 09. Tests =="
|
||||
|
||||
# Backend tests exist
|
||||
BACKEND_TESTS=$(find document-parser/tests -name "test_*.py" 2>/dev/null | wc -l)
|
||||
if [ "$BACKEND_TESTS" -gt 0 ]; then
|
||||
pass "Backend: $BACKEND_TESTS test files found"
|
||||
else
|
||||
fail "No backend test files found"
|
||||
fi
|
||||
|
||||
# Frontend tests exist
|
||||
FRONTEND_TESTS=$(find frontend/src -name "*.test.*" 2>/dev/null | wc -l)
|
||||
if [ "$FRONTEND_TESTS" -gt 0 ]; then
|
||||
pass "Frontend: $FRONTEND_TESTS test files found"
|
||||
else
|
||||
fail "No frontend test files found"
|
||||
fi
|
||||
|
||||
# E2E tests exist
|
||||
E2E_TESTS=$(find e2e -name "*.feature" 2>/dev/null | wc -l)
|
||||
if [ "$E2E_TESTS" -gt 0 ]; then
|
||||
pass "E2E: $E2E_TESTS feature files found"
|
||||
else
|
||||
warn "No e2e feature files found"
|
||||
fi
|
||||
|
||||
# Check for skipped tests
|
||||
if grep -rn "@skip\|@ignore\|xit(\|xdescribe(\|pytest.mark.skip" document-parser/tests/ frontend/src/ 2>/dev/null | grep -v "helpers"; then
|
||||
warn "Skipped tests found (review if intentional)"
|
||||
else
|
||||
pass "No skipped tests"
|
||||
fi
|
||||
|
||||
# ── 10. CI / Build ────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "== 10. CI / Build =="
|
||||
|
||||
# CI workflow exists
|
||||
if [ -f .github/workflows/ci.yml ]; then
|
||||
pass "CI workflow exists"
|
||||
else
|
||||
fail "No CI workflow found"
|
||||
fi
|
||||
|
||||
# Dockerfile exists
|
||||
if [ -f Dockerfile ]; then
|
||||
pass "Dockerfile exists"
|
||||
else
|
||||
fail "No Dockerfile found"
|
||||
fi
|
||||
|
||||
# Health check in docker-compose
|
||||
if grep -q "healthcheck" docker-compose.yml 2>/dev/null; then
|
||||
pass "Docker Compose has health check"
|
||||
else
|
||||
warn "No health check in docker-compose.yml"
|
||||
fi
|
||||
|
||||
# ── 11. Documentation ─────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "== 11. Documentation =="
|
||||
|
||||
# CHANGELOG exists and has content
|
||||
if [ -f CHANGELOG.md ] && [ -s CHANGELOG.md ]; then
|
||||
pass "CHANGELOG.md exists and is not empty"
|
||||
else
|
||||
fail "CHANGELOG.md missing or empty"
|
||||
fi
|
||||
|
||||
# README exists
|
||||
if [ -f README.md ]; then
|
||||
pass "README.md exists"
|
||||
else
|
||||
fail "README.md missing"
|
||||
fi
|
||||
|
||||
# Check for TODO/FIXME without issue reference
|
||||
TODOS=$(grep -rn "TODO\|FIXME" document-parser/ frontend/src/ --include="*.py" --include="*.ts" --include="*.vue" ! -path "*/.venv/*" ! -path "*/node_modules/*" 2>/dev/null | grep -v "#[0-9]" | head -5)
|
||||
if [ -n "$TODOS" ]; then
|
||||
warn "TODO/FIXME without issue reference:"
|
||||
echo "$TODOS"
|
||||
else
|
||||
pass "No orphaned TODO/FIXME"
|
||||
fi
|
||||
|
||||
# ── 12. Performance ───────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "== 12. Performance =="
|
||||
|
||||
# Check for synchronous file I/O in async context
|
||||
if grep -rn "open(" document-parser/api/ document-parser/services/ --include="*.py" 2>/dev/null | grep -v "aiofiles\|async\|#"; then
|
||||
warn "Synchronous file I/O in async code (review above)"
|
||||
else
|
||||
pass "No synchronous file I/O in async endpoints"
|
||||
fi
|
||||
|
||||
# Check for N+1 patterns (loop with DB call)
|
||||
if grep -rn "for.*in.*:" document-parser/services/ --include="*.py" -A5 2>/dev/null | grep "await.*repo\|await.*db"; then
|
||||
warn "Possible N+1 query pattern (review above)"
|
||||
else
|
||||
pass "No obvious N+1 patterns"
|
||||
fi
|
||||
|
||||
# ── Summary ────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "============================================"
|
||||
echo -e " ${GREEN}PASS${NC}: $PASS"
|
||||
echo -e " ${YELLOW}WARN${NC}: $WARN"
|
||||
echo -e " ${RED}FAIL${NC}: $FAIL"
|
||||
echo "============================================"
|
||||
|
||||
if [ "$FAIL" -gt 0 ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
# Stack Profile — FastAPI + Vue 3
|
||||
|
||||
Profile for running the 12 release audits on Docling Studio.
|
||||
|
||||
## Layer Mapping
|
||||
|
||||
| Generic Layer | Docling Studio Path | Language |
|
||||
|---------------|---------------------|----------|
|
||||
| **Domain** | `document-parser/domain/` | Python |
|
||||
| **Services** | `document-parser/services/` | Python |
|
||||
| **API** | `document-parser/api/` | Python |
|
||||
| **Infrastructure** | `document-parser/infra/` | Python |
|
||||
| **Persistence** | `document-parser/persistence/` | Python |
|
||||
| **Frontend** | `frontend/src/` | TypeScript / Vue |
|
||||
| **Tests (backend)** | `document-parser/tests/` | Python |
|
||||
| **Tests (frontend)** | `frontend/src/**/*.test.*` | TypeScript |
|
||||
| **Tests (e2e API)** | `e2e/api/` | Karate (Gherkin) |
|
||||
| **Tests (e2e UI)** | `e2e/ui/` | Karate UI (Gherkin) |
|
||||
| **CI/CD** | `.github/workflows/` | YAML |
|
||||
| **Docker** | `Dockerfile`, `docker-compose.yml`, `nginx.conf` | Docker / Nginx |
|
||||
|
||||
## Excluded Paths
|
||||
|
||||
These paths are excluded from audits:
|
||||
|
||||
- `document-parser/.venv/`
|
||||
- `document-parser/__pycache__/`
|
||||
- `frontend/node_modules/`
|
||||
- `frontend/dist/`
|
||||
- `e2e/**/target/`
|
||||
|
||||
## Framework Detection
|
||||
|
||||
Imports that should NOT appear in the domain layer:
|
||||
|
||||
```python
|
||||
# Forbidden in document-parser/domain/
|
||||
from fastapi import ...
|
||||
from pydantic import ... # except BaseModel for value objects
|
||||
import aiosqlite
|
||||
from infra import ...
|
||||
from persistence import ...
|
||||
from api import ...
|
||||
```
|
||||
|
||||
Imports that should NOT cross feature boundaries in the frontend:
|
||||
|
||||
```typescript
|
||||
// features/analysis/ should NOT import from features/chunking/store
|
||||
// features/document/ should NOT import from features/analysis/store
|
||||
// Cross-feature communication goes through shared/ or events
|
||||
```
|
||||
|
||||
## Tools & Commands
|
||||
|
||||
| Task | Command |
|
||||
|------|---------|
|
||||
| Backend lint | `cd document-parser && ruff check .` |
|
||||
| Backend format | `cd document-parser && ruff format --check .` |
|
||||
| Backend tests | `cd document-parser && pytest tests/ -v` |
|
||||
| Frontend lint | `cd frontend && npx eslint src/` |
|
||||
| Frontend type-check | `cd frontend && npm run type-check` |
|
||||
| Frontend format | `cd frontend && npx prettier --check src/` |
|
||||
| Frontend tests | `cd frontend && npm run test:run` |
|
||||
| E2E API tests | `mvn test -f e2e/api/pom.xml -Dkarate.options="--tags @smoke"` |
|
||||
| E2E UI tests | `mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @critical"` |
|
||||
| Docker build | `docker compose build` |
|
||||
| Docker health | `curl -s http://localhost:3000/api/health` |
|
||||
| Dependency audit (Python) | `cd document-parser && pip audit` |
|
||||
| Dependency audit (Node) | `cd frontend && npm audit` |
|
||||
Loading…
Reference in a new issue