Merge pull request #181 from scub-france/feature/feature-flag-ingestion
feat(#180): feature-flag ingestion pipeline and brainless one-liner Quick Start
This commit is contained in:
commit
2477a2af4b
26 changed files with 317 additions and 425 deletions
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -44,5 +44,8 @@ hs_err_pid*
|
||||||
# Docker
|
# Docker
|
||||||
docker-compose.override.yml
|
docker-compose.override.yml
|
||||||
|
|
||||||
|
# Audit profiles (internal tooling)
|
||||||
|
profiles/
|
||||||
|
|
||||||
# E2E tests — Maven build outputs & Chrome user data
|
# E2E tests — Maven build outputs & Chrome user data
|
||||||
e2e/**/target/
|
e2e/**/target/
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ All notable changes to Docling Studio will be documented in this file.
|
||||||
|
|
||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/).
|
The format is based on [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/).
|
||||||
|
|
||||||
## [0.4.0] - Unreleased
|
## [0.4.0] - 2026-04-13
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -96,15 +96,43 @@ npx prettier --write src/ # auto-format
|
||||||
## Running Tests
|
## Running Tests
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Backend (199 tests)
|
# Backend (377 tests)
|
||||||
cd document-parser
|
cd document-parser
|
||||||
pytest tests/ -v
|
pytest tests/ -v
|
||||||
|
|
||||||
# Frontend (129 tests)
|
# Frontend (156 tests)
|
||||||
cd frontend
|
cd frontend
|
||||||
npm run test:run
|
npm run test:run
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### E2E API (Karate)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate test PDFs + start stack
|
||||||
|
python e2e/generate-test-data.py
|
||||||
|
docker compose up -d --wait
|
||||||
|
|
||||||
|
# Run all API tests
|
||||||
|
mvn test -f e2e/api/pom.xml
|
||||||
|
|
||||||
|
# Or by tag: @smoke, @regression, @e2e
|
||||||
|
mvn test -f e2e/api/pom.xml -Dkarate.options="--tags @smoke"
|
||||||
|
```
|
||||||
|
|
||||||
|
### E2E UI (Karate UI)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate test PDFs + start stack (if not already running)
|
||||||
|
python e2e/generate-test-data.py
|
||||||
|
docker compose up -d --wait
|
||||||
|
|
||||||
|
# Run critical UI tests (CI scope)
|
||||||
|
mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @critical"
|
||||||
|
|
||||||
|
# Run all UI tests (local scope)
|
||||||
|
mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @ui"
|
||||||
|
```
|
||||||
|
|
||||||
All tests must pass before submitting a PR.
|
All tests must pass before submitting a PR.
|
||||||
|
|
||||||
## Submitting Changes
|
## Submitting Changes
|
||||||
|
|
|
||||||
83
README.md
83
README.md
|
|
@ -31,9 +31,13 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t
|
||||||
- **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description, image generation
|
- **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description, image generation
|
||||||
- **Bounding box visualization** — color-coded element overlay directly on the PDF
|
- **Bounding box visualization** — color-coded element overlay directly on the PDF
|
||||||
- **Per-page results** — right panel syncs with the current PDF page
|
- **Per-page results** — right panel syncs with the current PDF page
|
||||||
|
- **Chunking** — split extracted content into semantic chunks (hierarchical, hybrid, or page-based) with configurable token limits and inline editing
|
||||||
|
- **Ingestion pipeline** — Docling → chunking → embedding → OpenSearch vector indexing (one-click from Studio)
|
||||||
- **Markdown & HTML export** of extracted content
|
- **Markdown & HTML export** of extracted content
|
||||||
- **Document management** — upload, list, delete
|
- **Document management** — upload, list, delete, search, filter by indexing status
|
||||||
- **Analysis history** — re-visit and open past analyses
|
- **Analysis history** — re-visit and open past analyses
|
||||||
|
- **Upload limits** — configurable max file size and max page count per document
|
||||||
|
- **Rate limiting** — configurable requests per minute per IP
|
||||||
- **Dark / Light theme** and **FR / EN** localization
|
- **Dark / Light theme** and **FR / EN** localization
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -74,7 +78,7 @@ document-parser/
|
||||||
├── services/ # Use case orchestration
|
├── services/ # Use case orchestration
|
||||||
│ ├── document_service.py # Upload, delete, preview
|
│ ├── document_service.py # Upload, delete, preview
|
||||||
│ └── analysis_service.py # Async Docling processing
|
│ └── analysis_service.py # Async Docling processing
|
||||||
└── tests/ # 199 tests (pytest)
|
└── tests/ # 377 tests (pytest)
|
||||||
```
|
```
|
||||||
|
|
||||||
### Frontend structure (feature-based)
|
### Frontend structure (feature-based)
|
||||||
|
|
@ -98,14 +102,24 @@ frontend/src/
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
Docling Studio ships two Docker image variants:
|
One command, nothing else to install:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
|
||||||
|
```
|
||||||
|
|
||||||
|
Open [http://localhost:3000](http://localhost:3000), upload a PDF, and get results. That's it.
|
||||||
|
|
||||||
|
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
|
||||||
|
|
||||||
|
### Image variants
|
||||||
|
|
||||||
| Variant | Image tag | Size | Description |
|
| Variant | Image tag | Size | Description |
|
||||||
|---------|-----------|------|-------------|
|
|---------|-----------|------|-------------|
|
||||||
|
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only |
|
||||||
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
|
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
|
||||||
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only (downloads ML models on first run) |
|
|
||||||
|
|
||||||
### Docker — remote mode (fastest)
|
For remote mode:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker run -p 3000:3000 \
|
docker run -p 3000:3000 \
|
||||||
|
|
@ -113,27 +127,17 @@ docker run -p 3000:3000 \
|
||||||
ghcr.io/scub-france/docling-studio:latest-remote
|
ghcr.io/scub-france/docling-studio:latest-remote
|
||||||
```
|
```
|
||||||
|
|
||||||
### Docker — local mode (self-contained)
|
### Docker Compose
|
||||||
|
|
||||||
```bash
|
|
||||||
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
|
|
||||||
|
|
||||||
Open [http://localhost:3000](http://localhost:3000)
|
|
||||||
|
|
||||||
### Docker Compose (for development)
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://github.com/scub-france/Docling-Studio.git
|
git clone https://github.com/scub-france/Docling-Studio.git
|
||||||
cd Docling-Studio
|
cd Docling-Studio
|
||||||
|
|
||||||
# Local mode (default)
|
# Simple mode (backend + frontend only)
|
||||||
docker compose up --build
|
docker compose up --build
|
||||||
|
|
||||||
# Remote mode
|
# With ingestion pipeline (OpenSearch + embeddings)
|
||||||
CONVERSION_MODE=remote DOCLING_SERVE_URL=http://your-docling-serve:5001 docker compose up --build
|
docker compose --profile ingestion -f docker-compose.yml -f docker-compose.ingestion.yml up --build
|
||||||
```
|
```
|
||||||
|
|
||||||
### Local Development
|
### Local Development
|
||||||
|
|
@ -162,12 +166,12 @@ npm run dev
|
||||||
### Running Tests
|
### Running Tests
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Backend (199 tests)
|
# Backend (377 tests)
|
||||||
cd document-parser
|
cd document-parser
|
||||||
pip install pytest pytest-asyncio httpx
|
pip install pytest pytest-asyncio httpx
|
||||||
pytest tests/ -v
|
pytest tests/ -v
|
||||||
|
|
||||||
# Frontend (129 tests)
|
# Frontend (156 tests)
|
||||||
cd frontend
|
cd frontend
|
||||||
npm run test:run
|
npm run test:run
|
||||||
```
|
```
|
||||||
|
|
@ -202,6 +206,43 @@ All configuration is done via environment variables. See [`.env.example`](.env.e
|
||||||
| `UPLOAD_DIR` | `./uploads` | File storage directory |
|
| `UPLOAD_DIR` | `./uploads` | File storage directory |
|
||||||
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
|
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
|
||||||
| `CONVERSION_TIMEOUT` | `600` | Max seconds for a single Docling conversion |
|
| `CONVERSION_TIMEOUT` | `600` | Max seconds for a single Docling conversion |
|
||||||
|
| `BATCH_PAGE_SIZE` | `10` | Pages per batch (`0` = process all at once) |
|
||||||
|
| `MAX_FILE_SIZE_MB` | `50` | Maximum upload file size in MB (`0` = unlimited) |
|
||||||
|
| `MAX_PAGE_COUNT` | `0` | Maximum number of pages per document (`0` = unlimited) |
|
||||||
|
| `RATE_LIMIT_RPM` | `100` | Max requests per minute per IP (`0` = disabled) |
|
||||||
|
|
||||||
|
## Upload Limits
|
||||||
|
|
||||||
|
Docling Studio enforces configurable limits on uploaded documents to protect the server against oversized files and long-running analyses:
|
||||||
|
|
||||||
|
- **`MAX_FILE_SIZE_MB`** (default `50`) — rejects uploads exceeding this size. Validated at two levels: early `Content-Length` check and streaming byte count.
|
||||||
|
- **`MAX_PAGE_COUNT`** (default `0` = unlimited) — rejects documents with more pages than allowed. Useful on shared instances or Hugging Face Spaces to cap processing time.
|
||||||
|
|
||||||
|
Both limits are exposed in the `/api/health` endpoint so the frontend can display them to the user before upload. Set either to `0` to disable the corresponding check.
|
||||||
|
|
||||||
|
## Ingestion Pipeline (opt-in)
|
||||||
|
|
||||||
|
Docling Studio can optionally index extracted chunks into [OpenSearch](https://opensearch.org/) for vector and full-text search. This requires two additional services (OpenSearch + embedding) and is **disabled by default**.
|
||||||
|
|
||||||
|
To enable ingestion with Docker Compose:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose --profile ingestion \
|
||||||
|
-f docker-compose.yml -f docker-compose.ingestion.yml \
|
||||||
|
up --build
|
||||||
|
```
|
||||||
|
|
||||||
|
When ingestion is enabled, the UI shows:
|
||||||
|
- An **Ingest** button in Studio to push chunks to OpenSearch
|
||||||
|
- An **OpenSearch** connection status badge in the sidebar
|
||||||
|
- **Indexed / Not indexed** filters on the Documents page
|
||||||
|
- A **Search** page for full-text and vector search across indexed documents
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
|----------|---------|-------------|
|
||||||
|
| `OPENSEARCH_URL` | — | OpenSearch endpoint (empty = ingestion disabled) |
|
||||||
|
| `EMBEDDING_URL` | — | Embedding service endpoint (empty = ingestion disabled) |
|
||||||
|
| `EMBEDDING_DIMENSION` | `384` | Vector dimension (must match embedding model) |
|
||||||
|
|
||||||
## CI / Release
|
## CI / Release
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -74,7 +74,7 @@ services:
|
||||||
DOCLING_SERVE_API_KEY: ${DOCLING_SERVE_API_KEY:-}
|
DOCLING_SERVE_API_KEY: ${DOCLING_SERVE_API_KEY:-}
|
||||||
RATE_LIMIT_RPM: ${RATE_LIMIT_RPM:-100}
|
RATE_LIMIT_RPM: ${RATE_LIMIT_RPM:-100}
|
||||||
MAX_FILE_SIZE_MB: ${MAX_FILE_SIZE_MB:-50}
|
MAX_FILE_SIZE_MB: ${MAX_FILE_SIZE_MB:-50}
|
||||||
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-0}
|
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-10}
|
||||||
OPENSEARCH_URL: http://opensearch:9200
|
OPENSEARCH_URL: http://opensearch:9200
|
||||||
EMBEDDING_URL: http://embedding:8001
|
EMBEDDING_URL: http://embedding:8001
|
||||||
command: ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
|
command: ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
|
||||||
|
|
|
||||||
19
docker-compose.ingestion.yml
Normal file
19
docker-compose.ingestion.yml
Normal file
|
|
@ -0,0 +1,19 @@
|
||||||
|
# Override to enable the ingestion pipeline (OpenSearch + embeddings).
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# docker compose --profile ingestion -f docker-compose.yml -f docker-compose.ingestion.yml up --build
|
||||||
|
#
|
||||||
|
# This wires the backend to the OpenSearch and embedding services started
|
||||||
|
# by the "ingestion" profile and ensures they are healthy before the
|
||||||
|
# backend starts.
|
||||||
|
|
||||||
|
services:
|
||||||
|
document-parser:
|
||||||
|
environment:
|
||||||
|
OPENSEARCH_URL: http://opensearch:9200
|
||||||
|
EMBEDDING_URL: http://embedding:8001
|
||||||
|
depends_on:
|
||||||
|
opensearch:
|
||||||
|
condition: service_healthy
|
||||||
|
embedding:
|
||||||
|
condition: service_healthy
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
services:
|
services:
|
||||||
# --- OpenSearch (single-node, security disabled) ---
|
# --- OpenSearch (single-node, security disabled) ---
|
||||||
opensearch:
|
opensearch:
|
||||||
|
profiles: ["ingestion"]
|
||||||
image: opensearchproject/opensearch:2
|
image: opensearchproject/opensearch:2
|
||||||
environment:
|
environment:
|
||||||
discovery.type: single-node
|
discovery.type: single-node
|
||||||
|
|
@ -16,6 +17,7 @@ services:
|
||||||
|
|
||||||
# --- Embedding service (sentence-transformers) ---
|
# --- Embedding service (sentence-transformers) ---
|
||||||
embedding:
|
embedding:
|
||||||
|
profiles: ["ingestion"]
|
||||||
build:
|
build:
|
||||||
context: ./embedding-service
|
context: ./embedding-service
|
||||||
environment:
|
environment:
|
||||||
|
|
@ -48,14 +50,9 @@ services:
|
||||||
DOCLING_SERVE_API_KEY: ${DOCLING_SERVE_API_KEY:-}
|
DOCLING_SERVE_API_KEY: ${DOCLING_SERVE_API_KEY:-}
|
||||||
RATE_LIMIT_RPM: ${RATE_LIMIT_RPM:-100}
|
RATE_LIMIT_RPM: ${RATE_LIMIT_RPM:-100}
|
||||||
MAX_FILE_SIZE_MB: ${MAX_FILE_SIZE_MB:-50}
|
MAX_FILE_SIZE_MB: ${MAX_FILE_SIZE_MB:-50}
|
||||||
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-0}
|
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-10}
|
||||||
OPENSEARCH_URL: http://opensearch:9200
|
OPENSEARCH_URL: ${OPENSEARCH_URL:-}
|
||||||
EMBEDDING_URL: http://embedding:8001
|
EMBEDDING_URL: ${EMBEDDING_URL:-}
|
||||||
depends_on:
|
|
||||||
opensearch:
|
|
||||||
condition: service_healthy
|
|
||||||
embedding:
|
|
||||||
condition: service_healthy
|
|
||||||
deploy:
|
deploy:
|
||||||
resources:
|
resources:
|
||||||
limits:
|
limits:
|
||||||
|
|
|
||||||
|
|
@ -1,15 +1,28 @@
|
||||||
# Getting Started
|
# Getting Started
|
||||||
|
|
||||||
Docling Studio ships two Docker image variants:
|
## Quick Start
|
||||||
|
|
||||||
| Variant | Image tag | Size | Description |
|
One command, nothing else to install:
|
||||||
|---------|-----------|------|-------------|
|
|
||||||
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
|
```bash
|
||||||
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only (downloads ML models on first run) |
|
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
|
||||||
|
```
|
||||||
|
|
||||||
|
Open [http://localhost:3000](http://localhost:3000), upload a PDF, and get results. That's it.
|
||||||
|
|
||||||
|
!!! note
|
||||||
|
The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
|
||||||
|
|
||||||
{ width="600" }
|
{ width="600" }
|
||||||
|
|
||||||
## Docker — remote mode (fastest)
|
## Image Variants
|
||||||
|
|
||||||
|
| Variant | Image tag | Size | Description |
|
||||||
|
|---------|-----------|------|-------------|
|
||||||
|
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only |
|
||||||
|
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
|
||||||
|
|
||||||
|
For remote mode:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker run -p 3000:3000 \
|
docker run -p 3000:3000 \
|
||||||
|
|
@ -17,27 +30,19 @@ docker run -p 3000:3000 \
|
||||||
ghcr.io/scub-france/docling-studio:latest-remote
|
ghcr.io/scub-france/docling-studio:latest-remote
|
||||||
```
|
```
|
||||||
|
|
||||||
## Docker — local mode (self-contained)
|
## Docker Compose
|
||||||
|
|
||||||
```bash
|
|
||||||
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
|
|
||||||
|
|
||||||
Open [http://localhost:3000](http://localhost:3000).
|
|
||||||
|
|
||||||
## Docker Compose (recommended for development)
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://github.com/scub-france/Docling-Studio.git
|
git clone https://github.com/scub-france/Docling-Studio.git
|
||||||
cd Docling-Studio
|
cd Docling-Studio
|
||||||
|
|
||||||
# Local mode (default)
|
# Simple mode (backend + frontend only)
|
||||||
docker compose up --build
|
docker compose up --build
|
||||||
|
|
||||||
# Remote mode
|
# With ingestion pipeline (OpenSearch + embeddings)
|
||||||
CONVERSION_MODE=remote DOCLING_SERVE_URL=http://your-docling-serve:5001 docker compose up --build
|
docker compose --profile ingestion \
|
||||||
|
-f docker-compose.yml -f docker-compose.ingestion.yml \
|
||||||
|
up --build
|
||||||
```
|
```
|
||||||
|
|
||||||
## Local Development
|
## Local Development
|
||||||
|
|
@ -136,10 +141,48 @@ All configuration is done via environment variables:
|
||||||
| `UPLOAD_DIR` | `./uploads` | File storage directory |
|
| `UPLOAD_DIR` | `./uploads` | File storage directory |
|
||||||
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
|
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
|
||||||
| `CONVERSION_TIMEOUT` | `600` | Max seconds per Docling conversion |
|
| `CONVERSION_TIMEOUT` | `600` | Max seconds per Docling conversion |
|
||||||
|
| `BATCH_PAGE_SIZE` | `10` | Pages per batch (`0` = process all at once) |
|
||||||
| `MAX_CONCURRENT_ANALYSES` | `3` | Maximum parallel analysis jobs |
|
| `MAX_CONCURRENT_ANALYSES` | `3` | Maximum parallel analysis jobs |
|
||||||
| `DEPLOYMENT_MODE` | `self-hosted` | `self-hosted` or `huggingface` (shows disclaimer banner) |
|
| `DEPLOYMENT_MODE` | `self-hosted` | `self-hosted` or `huggingface` (shows disclaimer banner) |
|
||||||
|
| `MAX_FILE_SIZE_MB` | `50` | Maximum upload file size in MB (`0` = unlimited) |
|
||||||
|
| `MAX_PAGE_COUNT` | `0` | Maximum number of pages per document (`0` = unlimited) |
|
||||||
|
| `RATE_LIMIT_RPM` | `100` | Max requests per minute per IP (`0` = disabled) |
|
||||||
| `APP_VERSION` | `dev` | Application version (set automatically by CI/Docker) |
|
| `APP_VERSION` | `dev` | Application version (set automatically by CI/Docker) |
|
||||||
|
|
||||||
|
## Upload Limits
|
||||||
|
|
||||||
|
Docling Studio enforces configurable limits on uploaded documents to protect the server against oversized files and long-running analyses:
|
||||||
|
|
||||||
|
- **`MAX_FILE_SIZE_MB`** (default `50`) — rejects uploads exceeding this size. Validated at two levels: early `Content-Length` check and streaming byte count.
|
||||||
|
- **`MAX_PAGE_COUNT`** (default `0` = unlimited) — rejects documents with more pages than allowed. Useful on shared instances or Hugging Face Spaces to cap processing time.
|
||||||
|
|
||||||
|
Both limits are exposed in the `/api/health` endpoint so the frontend can display them to the user before upload. Set either to `0` to disable the corresponding check.
|
||||||
|
|
||||||
|
## Ingestion Pipeline (opt-in)
|
||||||
|
|
||||||
|
Docling Studio can optionally index extracted chunks into [OpenSearch](https://opensearch.org/) for vector and full-text search. This requires two additional services (OpenSearch + embedding) and is **disabled by default**.
|
||||||
|
|
||||||
|
To enable ingestion with Docker Compose:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose --profile ingestion \
|
||||||
|
-f docker-compose.yml -f docker-compose.ingestion.yml \
|
||||||
|
up --build
|
||||||
|
```
|
||||||
|
|
||||||
|
When ingestion is enabled, the UI shows:
|
||||||
|
|
||||||
|
- An **Ingest** button in Studio to push chunks to OpenSearch
|
||||||
|
- An **OpenSearch** connection status badge in the sidebar
|
||||||
|
- **Indexed / Not indexed** filters on the Documents page
|
||||||
|
- A **Search** page for full-text and vector search across indexed documents
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
|----------|---------|-------------|
|
||||||
|
| `OPENSEARCH_URL` | — | OpenSearch endpoint (empty = ingestion disabled) |
|
||||||
|
| `EMBEDDING_URL` | — | Embedding service endpoint (empty = ingestion disabled) |
|
||||||
|
| `EMBEDDING_DIMENSION` | `384` | Vector dimension (must match embedding model) |
|
||||||
|
|
||||||
## System Requirements
|
## System Requirements
|
||||||
|
|
||||||
| | Remote image | Local image |
|
| | Remote image | Local image |
|
||||||
|
|
|
||||||
|
|
@ -18,7 +18,8 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t
|
||||||
- **Document management** — upload, list, delete
|
- **Document management** — upload, list, delete
|
||||||
- **Analysis history** — re-visit and open past analyses
|
- **Analysis history** — re-visit and open past analyses
|
||||||
- **Feature flags** — capabilities adapt to the conversion engine (local vs remote)
|
- **Feature flags** — capabilities adapt to the conversion engine (local vs remote)
|
||||||
- **Rate limiting** — 60 requests per minute per IP to protect the backend
|
- **Upload limits** — configurable max file size (`MAX_FILE_SIZE_MB`) and max page count (`MAX_PAGE_COUNT`) per document
|
||||||
|
- **Rate limiting** — configurable requests per minute per IP (`RATE_LIMIT_RPM`)
|
||||||
- **Deployment modes** — self-hosted (default) or HuggingFace Spaces (with disclaimer banner)
|
- **Deployment modes** — self-hosted (default) or HuggingFace Spaces (with disclaimer banner)
|
||||||
- **Health endpoint** — `/api/health` reports engine type, deployment mode, and database status
|
- **Health endpoint** — `/api/health` reports engine type, deployment mode, and database status
|
||||||
- **Dark / Light theme** and **FR / EN** localization
|
- **Dark / Light theme** and **FR / EN** localization
|
||||||
|
|
|
||||||
|
|
@ -34,6 +34,7 @@ class HealthResponse(_CamelModel):
|
||||||
database: str
|
database: str
|
||||||
max_page_count: int | None = None
|
max_page_count: int | None = None
|
||||||
max_file_size_mb: int | None = None
|
max_file_size_mb: int | None = None
|
||||||
|
ingestion_available: bool = False
|
||||||
|
|
||||||
|
|
||||||
class DocumentResponse(_CamelModel):
|
class DocumentResponse(_CamelModel):
|
||||||
|
|
|
||||||
|
|
@ -94,7 +94,7 @@ class Settings:
|
||||||
max_file_size=int(os.environ.get("MAX_FILE_SIZE", "0")),
|
max_file_size=int(os.environ.get("MAX_FILE_SIZE", "0")),
|
||||||
max_file_size_mb=int(os.environ.get("MAX_FILE_SIZE_MB", "50")),
|
max_file_size_mb=int(os.environ.get("MAX_FILE_SIZE_MB", "50")),
|
||||||
rate_limit_rpm=int(os.environ.get("RATE_LIMIT_RPM", "100")),
|
rate_limit_rpm=int(os.environ.get("RATE_LIMIT_RPM", "100")),
|
||||||
batch_page_size=int(os.environ.get("BATCH_PAGE_SIZE", "0")),
|
batch_page_size=int(os.environ.get("BATCH_PAGE_SIZE", "10")),
|
||||||
opensearch_url=os.environ.get("OPENSEARCH_URL", ""),
|
opensearch_url=os.environ.get("OPENSEARCH_URL", ""),
|
||||||
embedding_url=os.environ.get("EMBEDDING_URL", ""),
|
embedding_url=os.environ.get("EMBEDDING_URL", ""),
|
||||||
embedding_dimension=int(os.environ.get("EMBEDDING_DIMENSION", "384")),
|
embedding_dimension=int(os.environ.get("EMBEDDING_DIMENSION", "384")),
|
||||||
|
|
|
||||||
|
|
@ -138,7 +138,11 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||||
document_repo, analysis_repo = _build_repos()
|
document_repo, analysis_repo = _build_repos()
|
||||||
app.state.analysis_service = _build_analysis_service(document_repo, analysis_repo)
|
app.state.analysis_service = _build_analysis_service(document_repo, analysis_repo)
|
||||||
app.state.document_service = _build_document_service(document_repo, analysis_repo)
|
app.state.document_service = _build_document_service(document_repo, analysis_repo)
|
||||||
app.state.ingestion_service = _build_ingestion_service()
|
ingestion_service = _build_ingestion_service()
|
||||||
|
app.state.ingestion_service = ingestion_service
|
||||||
|
if ingestion_service is not None:
|
||||||
|
app.include_router(ingestion_router)
|
||||||
|
logger.info("Ingestion router mounted")
|
||||||
logger.info("Docling Studio backend ready (engine=%s)", settings.conversion_engine)
|
logger.info("Docling Studio backend ready (engine=%s)", settings.conversion_engine)
|
||||||
yield
|
yield
|
||||||
|
|
||||||
|
|
@ -165,7 +169,6 @@ if settings.rate_limit_rpm > 0:
|
||||||
|
|
||||||
app.include_router(documents_router)
|
app.include_router(documents_router)
|
||||||
app.include_router(analyses_router)
|
app.include_router(analyses_router)
|
||||||
app.include_router(ingestion_router)
|
|
||||||
|
|
||||||
|
|
||||||
@app.get("/api/health", response_model=HealthResponse)
|
@app.get("/api/health", response_model=HealthResponse)
|
||||||
|
|
@ -188,4 +191,5 @@ async def health() -> HealthResponse:
|
||||||
database=db_status,
|
database=db_status,
|
||||||
max_page_count=settings.max_page_count if settings.max_page_count > 0 else None,
|
max_page_count=settings.max_page_count if settings.max_page_count > 0 else None,
|
||||||
max_file_size_mb=settings.max_file_size_mb if settings.max_file_size_mb > 0 else None,
|
max_file_size_mb=settings.max_file_size_mb if settings.max_file_size_mb > 0 else None,
|
||||||
|
ingestion_available=getattr(app.state, "ingestion_service", None) is not None,
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -51,6 +51,23 @@ class TestHealthEndpoint:
|
||||||
assert "maxFileSizeMb" in data
|
assert "maxFileSizeMb" in data
|
||||||
assert data["maxFileSizeMb"] == 50
|
assert data["maxFileSizeMb"] == 50
|
||||||
|
|
||||||
|
def test_health_exposes_ingestion_available_false(self, client):
|
||||||
|
original = getattr(app.state, "ingestion_service", None)
|
||||||
|
app.state.ingestion_service = None
|
||||||
|
resp = client.get("/api/health")
|
||||||
|
app.state.ingestion_service = original
|
||||||
|
data = resp.json()
|
||||||
|
assert "ingestionAvailable" in data
|
||||||
|
assert data["ingestionAvailable"] is False
|
||||||
|
|
||||||
|
def test_health_exposes_ingestion_available_true(self, client):
|
||||||
|
original = getattr(app.state, "ingestion_service", None)
|
||||||
|
app.state.ingestion_service = MagicMock()
|
||||||
|
resp = client.get("/api/health")
|
||||||
|
app.state.ingestion_service = original
|
||||||
|
data = resp.json()
|
||||||
|
assert data["ingestionAvailable"] is True
|
||||||
|
|
||||||
|
|
||||||
class TestDocumentEndpoints:
|
class TestDocumentEndpoints:
|
||||||
def test_list_documents(self, client, mock_document_service):
|
def test_list_documents(self, client, mock_document_service):
|
||||||
|
|
|
||||||
|
|
@ -104,7 +104,18 @@ class TestIngestionStatus:
|
||||||
|
|
||||||
|
|
||||||
class TestIngestionDisabled:
|
class TestIngestionDisabled:
|
||||||
def test_returns_503_when_disabled(self) -> None:
|
def test_router_not_mounted_returns_404(self) -> None:
|
||||||
|
"""When ingestion service is None, router should not be mounted → 404."""
|
||||||
|
app = FastAPI()
|
||||||
|
# Do NOT include ingestion router — simulates main.py conditional mount
|
||||||
|
app.state.ingestion_service = None
|
||||||
|
app.state.analysis_service = AsyncMock()
|
||||||
|
tc = TestClient(app)
|
||||||
|
resp = tc.post("/api/ingestion/job-1")
|
||||||
|
assert resp.status_code == 404
|
||||||
|
|
||||||
|
def test_status_still_returns_503_when_router_mounted_but_service_none(self) -> None:
|
||||||
|
"""If router is mounted but service is None, endpoints return 503."""
|
||||||
app = FastAPI()
|
app = FastAPI()
|
||||||
app.include_router(router)
|
app.include_router(router)
|
||||||
app.state.ingestion_service = None
|
app.state.ingestion_service = None
|
||||||
|
|
|
||||||
4
frontend/package-lock.json
generated
4
frontend/package-lock.json
generated
|
|
@ -1,12 +1,12 @@
|
||||||
{
|
{
|
||||||
"name": "docling-studio",
|
"name": "docling-studio",
|
||||||
"version": "0.3.0",
|
"version": "0.3.1",
|
||||||
"lockfileVersion": 3,
|
"lockfileVersion": 3,
|
||||||
"requires": true,
|
"requires": true,
|
||||||
"packages": {
|
"packages": {
|
||||||
"": {
|
"": {
|
||||||
"name": "docling-studio",
|
"name": "docling-studio",
|
||||||
"version": "0.3.0",
|
"version": "0.3.1",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"dompurify": "^3.3.3",
|
"dompurify": "^3.3.3",
|
||||||
"marked": "^17.0.4",
|
"marked": "^17.0.4",
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
{
|
{
|
||||||
"name": "docling-studio",
|
"name": "docling-studio",
|
||||||
"version": "0.3.1",
|
"version": "0.4.0",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
|
|
|
||||||
|
|
@ -202,7 +202,10 @@
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="chunk-empty" v-else-if="!chunkingStore.rechunking">
|
<div
|
||||||
|
class="chunk-empty"
|
||||||
|
v-if="!pageChunks.length && !chunkingStore.rechunking && deleteConfirmIdx === -1"
|
||||||
|
>
|
||||||
<p>
|
<p>
|
||||||
{{ chunks.length ? t('chunking.noChunksOnPage') : t('chunking.noChunks') }}
|
{{ chunks.length ? t('chunking.noChunksOnPage') : t('chunking.noChunks') }}
|
||||||
</p>
|
</p>
|
||||||
|
|
|
||||||
|
|
@ -82,6 +82,38 @@ describe('useFeatureFlagStore', () => {
|
||||||
expect(store.maxFileSizeMb).toBe(0)
|
expect(store.maxFileSizeMb).toBe(0)
|
||||||
})
|
})
|
||||||
|
|
||||||
|
it('enables ingestion when ingestionAvailable is true', async () => {
|
||||||
|
mockApiFetch.mockResolvedValue({
|
||||||
|
status: 'ok',
|
||||||
|
engine: 'local',
|
||||||
|
ingestionAvailable: true,
|
||||||
|
})
|
||||||
|
const store = useFeatureFlagStore()
|
||||||
|
await store.load()
|
||||||
|
expect(store.ingestionAvailable).toBe(true)
|
||||||
|
expect(store.isEnabled('ingestion')).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('disables ingestion when ingestionAvailable is false', async () => {
|
||||||
|
mockApiFetch.mockResolvedValue({
|
||||||
|
status: 'ok',
|
||||||
|
engine: 'local',
|
||||||
|
ingestionAvailable: false,
|
||||||
|
})
|
||||||
|
const store = useFeatureFlagStore()
|
||||||
|
await store.load()
|
||||||
|
expect(store.ingestionAvailable).toBe(false)
|
||||||
|
expect(store.isEnabled('ingestion')).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('defaults ingestionAvailable to false when missing', async () => {
|
||||||
|
mockApiFetch.mockResolvedValue({ status: 'ok', engine: 'local' })
|
||||||
|
const store = useFeatureFlagStore()
|
||||||
|
await store.load()
|
||||||
|
expect(store.ingestionAvailable).toBe(false)
|
||||||
|
expect(store.isEnabled('ingestion')).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
it('handles health endpoint failure gracefully', async () => {
|
it('handles health endpoint failure gracefully', async () => {
|
||||||
mockApiFetch.mockRejectedValue(new Error('Network error'))
|
mockApiFetch.mockRejectedValue(new Error('Network error'))
|
||||||
const store = useFeatureFlagStore()
|
const store = useFeatureFlagStore()
|
||||||
|
|
|
||||||
|
|
@ -13,9 +13,10 @@ interface HealthResponse {
|
||||||
deploymentMode?: DeploymentMode
|
deploymentMode?: DeploymentMode
|
||||||
maxPageCount?: number
|
maxPageCount?: number
|
||||||
maxFileSizeMb?: number
|
maxFileSizeMb?: number
|
||||||
|
ingestionAvailable?: boolean
|
||||||
}
|
}
|
||||||
|
|
||||||
export type FeatureFlag = 'chunking' | 'disclaimer'
|
export type FeatureFlag = 'chunking' | 'disclaimer' | 'ingestion'
|
||||||
|
|
||||||
interface FeatureFlagDef {
|
interface FeatureFlagDef {
|
||||||
description: string
|
description: string
|
||||||
|
|
@ -25,6 +26,7 @@ interface FeatureFlagDef {
|
||||||
interface FeatureFlagContext {
|
interface FeatureFlagContext {
|
||||||
engine: ConversionEngine | null
|
engine: ConversionEngine | null
|
||||||
deploymentMode: DeploymentMode | null
|
deploymentMode: DeploymentMode | null
|
||||||
|
ingestionAvailable: boolean
|
||||||
}
|
}
|
||||||
|
|
||||||
const featureRegistry: Record<FeatureFlag, FeatureFlagDef> = {
|
const featureRegistry: Record<FeatureFlag, FeatureFlagDef> = {
|
||||||
|
|
@ -36,6 +38,10 @@ const featureRegistry: Record<FeatureFlag, FeatureFlagDef> = {
|
||||||
description: 'Show shared-instance disclaimer banner',
|
description: 'Show shared-instance disclaimer banner',
|
||||||
isEnabled: (ctx) => ctx.deploymentMode === 'huggingface',
|
isEnabled: (ctx) => ctx.deploymentMode === 'huggingface',
|
||||||
},
|
},
|
||||||
|
ingestion: {
|
||||||
|
description: 'OpenSearch ingestion pipeline (embedding + vector indexing)',
|
||||||
|
isEnabled: (ctx) => ctx.ingestionAvailable,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
||||||
|
|
@ -43,6 +49,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
||||||
const deploymentMode = ref<DeploymentMode | null>(null)
|
const deploymentMode = ref<DeploymentMode | null>(null)
|
||||||
const maxPageCount = ref<number>(0)
|
const maxPageCount = ref<number>(0)
|
||||||
const maxFileSizeMb = ref<number>(0)
|
const maxFileSizeMb = ref<number>(0)
|
||||||
|
const ingestionAvailable = ref(false)
|
||||||
const appVersion = ref<string>(__APP_VERSION__)
|
const appVersion = ref<string>(__APP_VERSION__)
|
||||||
const loaded = ref(false)
|
const loaded = ref(false)
|
||||||
const error = ref<string | null>(null)
|
const error = ref<string | null>(null)
|
||||||
|
|
@ -50,6 +57,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
||||||
const context = computed<FeatureFlagContext>(() => ({
|
const context = computed<FeatureFlagContext>(() => ({
|
||||||
engine: engine.value,
|
engine: engine.value,
|
||||||
deploymentMode: deploymentMode.value,
|
deploymentMode: deploymentMode.value,
|
||||||
|
ingestionAvailable: ingestionAvailable.value,
|
||||||
}))
|
}))
|
||||||
|
|
||||||
function isEnabled(flag: FeatureFlag): boolean {
|
function isEnabled(flag: FeatureFlag): boolean {
|
||||||
|
|
@ -65,6 +73,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
||||||
deploymentMode.value = data.deploymentMode ?? 'self-hosted'
|
deploymentMode.value = data.deploymentMode ?? 'self-hosted'
|
||||||
maxPageCount.value = data.maxPageCount ?? 0
|
maxPageCount.value = data.maxPageCount ?? 0
|
||||||
maxFileSizeMb.value = data.maxFileSizeMb ?? 0
|
maxFileSizeMb.value = data.maxFileSizeMb ?? 0
|
||||||
|
ingestionAvailable.value = data.ingestionAvailable ?? false
|
||||||
appMaxFileSizeMb.value = maxFileSizeMb.value
|
appMaxFileSizeMb.value = maxFileSizeMb.value
|
||||||
appMaxPageCount.value = maxPageCount.value
|
appMaxPageCount.value = maxPageCount.value
|
||||||
if (data.version) appVersion.value = data.version
|
if (data.version) appVersion.value = data.version
|
||||||
|
|
@ -81,6 +90,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
||||||
deploymentMode,
|
deploymentMode,
|
||||||
maxPageCount,
|
maxPageCount,
|
||||||
maxFileSizeMb,
|
maxFileSizeMb,
|
||||||
|
ingestionAvailable,
|
||||||
appVersion,
|
appVersion,
|
||||||
loaded,
|
loaded,
|
||||||
error,
|
error,
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,7 @@
|
||||||
class="search-input"
|
class="search-input"
|
||||||
:placeholder="t('ingestion.search')"
|
:placeholder="t('ingestion.search')"
|
||||||
/>
|
/>
|
||||||
<div class="filter-group">
|
<div v-if="ingestionEnabled" class="filter-group">
|
||||||
<button
|
<button
|
||||||
v-for="f in filters"
|
v-for="f in filters"
|
||||||
:key="f.value"
|
:key="f.value"
|
||||||
|
|
@ -54,17 +54,19 @@
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="doc-row-actions">
|
<div class="doc-row-actions">
|
||||||
<span
|
<template v-if="ingestionEnabled">
|
||||||
v-if="ingestionStore.ingestedDocs[doc.id]"
|
<span
|
||||||
class="status-badge indexed"
|
v-if="ingestionStore.ingestedDocs[doc.id]"
|
||||||
:title="t('ingestion.chunksIndexed', { n: ingestionStore.ingestedDocs[doc.id] })"
|
class="status-badge indexed"
|
||||||
>
|
:title="t('ingestion.chunksIndexed', { n: ingestionStore.ingestedDocs[doc.id] })"
|
||||||
{{ t('ingestion.indexed') }}
|
>
|
||||||
<span class="badge-count">{{ ingestionStore.ingestedDocs[doc.id] }}</span>
|
{{ t('ingestion.indexed') }}
|
||||||
</span>
|
<span class="badge-count">{{ ingestionStore.ingestedDocs[doc.id] }}</span>
|
||||||
<span v-else class="status-badge not-indexed">
|
</span>
|
||||||
{{ t('ingestion.notIndexed') }}
|
<span v-else class="status-badge not-indexed">
|
||||||
</span>
|
{{ t('ingestion.notIndexed') }}
|
||||||
|
</span>
|
||||||
|
</template>
|
||||||
<button
|
<button
|
||||||
class="action-btn"
|
class="action-btn"
|
||||||
:title="t('ingestion.openInStudio')"
|
:title="t('ingestion.openInStudio')"
|
||||||
|
|
@ -77,7 +79,7 @@
|
||||||
</svg>
|
</svg>
|
||||||
</button>
|
</button>
|
||||||
<button
|
<button
|
||||||
v-if="ingestionStore.ingestedDocs[doc.id]"
|
v-if="ingestionEnabled && ingestionStore.ingestedDocs[doc.id]"
|
||||||
class="action-btn unindex"
|
class="action-btn unindex"
|
||||||
:title="t('ingestion.deleteIndex')"
|
:title="t('ingestion.deleteIndex')"
|
||||||
@click="confirmRemoveFromIndex(doc.id)"
|
@click="confirmRemoveFromIndex(doc.id)"
|
||||||
|
|
@ -110,13 +112,16 @@
|
||||||
import { computed, onMounted, ref } from 'vue'
|
import { computed, onMounted, ref } from 'vue'
|
||||||
import { useRouter } from 'vue-router'
|
import { useRouter } from 'vue-router'
|
||||||
import { useDocumentStore } from '../features/document/store'
|
import { useDocumentStore } from '../features/document/store'
|
||||||
|
import { useFeatureFlagStore } from '../features/feature-flags/store'
|
||||||
import { useIngestionStore } from '../features/ingestion/store'
|
import { useIngestionStore } from '../features/ingestion/store'
|
||||||
import { useI18n } from '../shared/i18n'
|
import { useI18n } from '../shared/i18n'
|
||||||
import { formatSize } from '../shared/format'
|
import { formatSize } from '../shared/format'
|
||||||
import type { Document } from '../shared/types'
|
import type { Document } from '../shared/types'
|
||||||
|
|
||||||
const docStore = useDocumentStore()
|
const docStore = useDocumentStore()
|
||||||
|
const featureStore = useFeatureFlagStore()
|
||||||
const ingestionStore = useIngestionStore()
|
const ingestionStore = useIngestionStore()
|
||||||
|
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
|
||||||
const router = useRouter()
|
const router = useRouter()
|
||||||
const { t } = useI18n()
|
const { t } = useI18n()
|
||||||
|
|
||||||
|
|
@ -173,7 +178,7 @@ function confirmRemoveFromIndex(docId: string) {
|
||||||
}
|
}
|
||||||
|
|
||||||
async function handleDelete(docId: string) {
|
async function handleDelete(docId: string) {
|
||||||
if (ingestionStore.ingestedDocs[docId]) {
|
if (ingestionEnabled.value && ingestionStore.ingestedDocs[docId]) {
|
||||||
await ingestionStore.deleteIngested(docId)
|
await ingestionStore.deleteIngested(docId)
|
||||||
}
|
}
|
||||||
await docStore.remove(docId)
|
await docStore.remove(docId)
|
||||||
|
|
@ -181,7 +186,9 @@ async function handleDelete(docId: string) {
|
||||||
|
|
||||||
onMounted(() => {
|
onMounted(() => {
|
||||||
docStore.load()
|
docStore.load()
|
||||||
ingestionStore.checkAvailability()
|
if (ingestionEnabled.value) {
|
||||||
|
ingestionStore.checkAvailability()
|
||||||
|
}
|
||||||
})
|
})
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@
|
||||||
<h1 class="page-title">{{ t('nav.search') }}</h1>
|
<h1 class="page-title">{{ t('nav.search') }}</h1>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div v-if="!ingestionStore.available" class="tab-empty">
|
<div v-if="!ingestionEnabled || !ingestionStore.available" class="tab-empty">
|
||||||
{{ t('ingestion.unavailable') }}
|
{{ t('ingestion.unavailable') }}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
|
@ -50,13 +50,16 @@
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { onMounted, ref } from 'vue'
|
import { computed, onMounted, ref } from 'vue'
|
||||||
import { useSearchStore } from '../features/search/store'
|
import { useSearchStore } from '../features/search/store'
|
||||||
|
import { useFeatureFlagStore } from '../features/feature-flags/store'
|
||||||
import { useIngestionStore } from '../features/ingestion/store'
|
import { useIngestionStore } from '../features/ingestion/store'
|
||||||
import { useI18n } from '../shared/i18n'
|
import { useI18n } from '../shared/i18n'
|
||||||
|
|
||||||
const searchStore = useSearchStore()
|
const searchStore = useSearchStore()
|
||||||
|
const featureStore = useFeatureFlagStore()
|
||||||
const ingestionStore = useIngestionStore()
|
const ingestionStore = useIngestionStore()
|
||||||
|
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
|
||||||
const { t } = useI18n()
|
const { t } = useI18n()
|
||||||
|
|
||||||
const searchInput = ref('')
|
const searchInput = ref('')
|
||||||
|
|
@ -70,7 +73,9 @@ function runSearch() {
|
||||||
}
|
}
|
||||||
|
|
||||||
onMounted(() => {
|
onMounted(() => {
|
||||||
ingestionStore.checkAvailability()
|
if (ingestionEnabled.value) {
|
||||||
|
ingestionStore.checkAvailability()
|
||||||
|
}
|
||||||
})
|
})
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -67,7 +67,7 @@
|
||||||
{{ t('studio.prepare') }}
|
{{ t('studio.prepare') }}
|
||||||
</button>
|
</button>
|
||||||
<button
|
<button
|
||||||
v-if="chunkingEnabled && ingestionStore.available"
|
v-if="chunkingEnabled && ingestionEnabled && ingestionStore.available"
|
||||||
class="toggle-btn"
|
class="toggle-btn"
|
||||||
data-e2e="toggle-btn"
|
data-e2e="toggle-btn"
|
||||||
:class="{ active: mode === 'ingest' }"
|
:class="{ active: mode === 'ingest' }"
|
||||||
|
|
@ -504,6 +504,7 @@ const analysisStore = useAnalysisStore()
|
||||||
const ingestionStore = useIngestionStore()
|
const ingestionStore = useIngestionStore()
|
||||||
const { t } = useI18n()
|
const { t } = useI18n()
|
||||||
const chunkingEnabled = useFeatureFlag('chunking')
|
const chunkingEnabled = useFeatureFlag('chunking')
|
||||||
|
const ingestionEnabled = useFeatureFlag('ingestion')
|
||||||
|
|
||||||
const mode = ref('configure')
|
const mode = ref('configure')
|
||||||
const currentPage = ref(1)
|
const currentPage = ref(1)
|
||||||
|
|
@ -636,7 +637,9 @@ watch(
|
||||||
onMounted(async () => {
|
onMounted(async () => {
|
||||||
await documentStore.load()
|
await documentStore.load()
|
||||||
analysisStore.load()
|
analysisStore.load()
|
||||||
ingestionStore.checkAvailability()
|
if (ingestionEnabled.value) {
|
||||||
|
ingestionStore.checkAvailability()
|
||||||
|
}
|
||||||
|
|
||||||
// Restore analysis from history via query param
|
// Restore analysis from history via query param
|
||||||
const analysisId = route.query.analysisId
|
const analysisId = route.query.analysisId
|
||||||
|
|
|
||||||
|
|
@ -130,7 +130,7 @@ const messages: Messages = {
|
||||||
'chunking.deleteConfirm':
|
'chunking.deleteConfirm':
|
||||||
'Supprimer ce chunk ? Il sera marqué comme supprimé jusqu\u2019à la prochaine synchronisation.',
|
'Supprimer ce chunk ? Il sera marqué comme supprimé jusqu\u2019à la prochaine synchronisation.',
|
||||||
'chunking.batchNotice':
|
'chunking.batchNotice':
|
||||||
'Le chunking n\u2019est pas disponible pour cette analyse. Les documents volumineux trait\u00e9s par batch ne g\u00e9n\u00e8rent pas la structure interne n\u00e9cessaire au d\u00e9coupage.',
|
'Le chunking n\u2019est pas disponible pour cette analyse. Les documents volumineux trait\u00e9s par batch ne g\u00e9n\u00e8rent pas la structure interne n\u00e9cessaire au d\u00e9coupage. Coming soon !',
|
||||||
|
|
||||||
// Search
|
// Search
|
||||||
'nav.search': 'Recherche',
|
'nav.search': 'Recherche',
|
||||||
|
|
@ -299,7 +299,7 @@ const messages: Messages = {
|
||||||
'chunking.deleteConfirm':
|
'chunking.deleteConfirm':
|
||||||
'Delete this chunk? It will be marked as deleted until the next sync.',
|
'Delete this chunk? It will be marked as deleted until the next sync.',
|
||||||
'chunking.batchNotice':
|
'chunking.batchNotice':
|
||||||
'Chunking is not available for this analysis. Large documents processed in batch mode do not generate the internal structure required for chunking.',
|
'Chunking is not available for this analysis. Large documents processed in batch mode do not generate the internal structure required for chunking. Coming soon!',
|
||||||
|
|
||||||
'nav.search': 'Search',
|
'nav.search': 'Search',
|
||||||
'search.hint': 'Enter a term to search through indexed chunks.',
|
'search.hint': 'Enter a term to search through indexed chunks.',
|
||||||
|
|
|
||||||
|
|
@ -46,6 +46,7 @@
|
||||||
</RouterLink>
|
</RouterLink>
|
||||||
|
|
||||||
<RouterLink
|
<RouterLink
|
||||||
|
v-if="ingestionEnabled"
|
||||||
to="/search"
|
to="/search"
|
||||||
class="nav-item"
|
class="nav-item"
|
||||||
data-e2e="nav-search"
|
data-e2e="nav-search"
|
||||||
|
|
@ -96,7 +97,7 @@
|
||||||
|
|
||||||
<div class="sidebar-footer">
|
<div class="sidebar-footer">
|
||||||
<div
|
<div
|
||||||
v-if="ingestionStore.available"
|
v-if="ingestionEnabled && ingestionStore.available"
|
||||||
class="opensearch-status"
|
class="opensearch-status"
|
||||||
:title="
|
:title="
|
||||||
ingestionStore.opensearchConnected
|
ingestionStore.opensearchConnected
|
||||||
|
|
@ -136,6 +137,7 @@ import { useIngestionStore } from '../../features/ingestion/store'
|
||||||
|
|
||||||
const featureStore = useFeatureFlagStore()
|
const featureStore = useFeatureFlagStore()
|
||||||
const ingestionStore = useIngestionStore()
|
const ingestionStore = useIngestionStore()
|
||||||
|
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
|
||||||
const version = computed(() => featureStore.appVersion)
|
const version = computed(() => featureStore.appVersion)
|
||||||
const route = useRoute()
|
const route = useRoute()
|
||||||
const { t } = useI18n()
|
const { t } = useI18n()
|
||||||
|
|
@ -145,8 +147,10 @@ defineProps({
|
||||||
})
|
})
|
||||||
|
|
||||||
onMounted(() => {
|
onMounted(() => {
|
||||||
ingestionStore.checkAvailability()
|
if (ingestionEnabled.value) {
|
||||||
ingestionStore.startPolling(30_000)
|
ingestionStore.checkAvailability()
|
||||||
|
ingestionStore.startPolling(30_000)
|
||||||
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
onBeforeUnmount(() => {
|
onBeforeUnmount(() => {
|
||||||
|
|
|
||||||
|
|
@ -1,267 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
# ============================================================================
|
|
||||||
# Docling Studio — Automated Audit Checks (FastAPI + Vue 3 profile)
|
|
||||||
# ============================================================================
|
|
||||||
# Runs verification commands for each of the 12 release audits.
|
|
||||||
# Usage: bash profiles/fastapi-vue/commands.sh
|
|
||||||
# Run from the repository root.
|
|
||||||
# ============================================================================
|
|
||||||
|
|
||||||
set -uo pipefail
|
|
||||||
|
|
||||||
RED='\033[0;31m'
|
|
||||||
GREEN='\033[0;32m'
|
|
||||||
YELLOW='\033[1;33m'
|
|
||||||
NC='\033[0m'
|
|
||||||
|
|
||||||
PASS=0
|
|
||||||
WARN=0
|
|
||||||
FAIL=0
|
|
||||||
|
|
||||||
pass() { echo -e " ${GREEN}PASS${NC} $1"; ((PASS++)); }
|
|
||||||
warn() { echo -e " ${YELLOW}WARN${NC} $1"; ((WARN++)); }
|
|
||||||
fail() { echo -e " ${RED}FAIL${NC} $1"; ((FAIL++)); }
|
|
||||||
|
|
||||||
# ── 01. Clean Architecture ─────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "== 01. Clean Architecture =="
|
|
||||||
|
|
||||||
# Domain must not import from api, persistence, infra
|
|
||||||
if grep -rn "from api\|from persistence\|from infra\|import fastapi\|import aiosqlite" document-parser/domain/ 2>/dev/null; then
|
|
||||||
fail "Domain layer imports forbidden modules"
|
|
||||||
else
|
|
||||||
pass "Domain layer has no forbidden imports"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# API must not import from persistence directly
|
|
||||||
if grep -rn "from persistence" document-parser/api/ 2>/dev/null; then
|
|
||||||
fail "API layer imports directly from persistence"
|
|
||||||
else
|
|
||||||
pass "API layer does not import from persistence"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── 02. DDD ────────────────────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "== 02. DDD =="
|
|
||||||
|
|
||||||
# Domain models exist
|
|
||||||
if [ -f document-parser/domain/models.py ] && [ -f document-parser/domain/ports.py ]; then
|
|
||||||
pass "Domain models and ports exist"
|
|
||||||
else
|
|
||||||
fail "Missing domain/models.py or domain/ports.py"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Value objects exist
|
|
||||||
if [ -f document-parser/domain/value_objects.py ]; then
|
|
||||||
pass "Value objects defined"
|
|
||||||
else
|
|
||||||
warn "No value_objects.py in domain"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── 03. Clean Code ─────────────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "== 03. Clean Code =="
|
|
||||||
|
|
||||||
# Check for files > 300 lines (backend)
|
|
||||||
LARGE_FILES=$(find document-parser -name "*.py" ! -path "*/.venv/*" ! -path "*/__pycache__/*" ! -path "*/tests/*" -exec awk 'END{if(NR>300) print FILENAME": "NR" lines"}' {} \;)
|
|
||||||
if [ -n "$LARGE_FILES" ]; then
|
|
||||||
warn "Large Python files (>300 lines):"
|
|
||||||
echo "$LARGE_FILES"
|
|
||||||
else
|
|
||||||
pass "No Python files exceed 300 lines"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check for files > 300 lines (frontend)
|
|
||||||
LARGE_VUE=$(find frontend/src -name "*.vue" -o -name "*.ts" | xargs awk 'END{if(NR>300) print FILENAME": "NR" lines"}' 2>/dev/null)
|
|
||||||
if [ -n "$LARGE_VUE" ]; then
|
|
||||||
warn "Large frontend files (>300 lines):"
|
|
||||||
echo "$LARGE_VUE"
|
|
||||||
else
|
|
||||||
pass "No frontend files exceed 300 lines"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── 04. KISS ───────────────────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "== 04. KISS =="
|
|
||||||
|
|
||||||
# Check for overly complex patterns
|
|
||||||
if grep -rn "type:\s*ignore" document-parser/ --include="*.py" ! -path "*/.venv/*" 2>/dev/null; then
|
|
||||||
warn "Found type: ignore comments (review if justified)"
|
|
||||||
else
|
|
||||||
pass "No unjustified type: ignore"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── 05. DRY ────────────────────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "== 05. DRY =="
|
|
||||||
|
|
||||||
# Check for magic numbers in backend
|
|
||||||
if grep -rn "[^a-zA-Z_][0-9]\{3,\}[^0-9]" document-parser/ --include="*.py" ! -path "*/.venv/*" ! -path "*/tests/*" 2>/dev/null | grep -v "port\|version\|status\|#\|MAX_\|DEFAULT_\|LIMIT_" | head -5; then
|
|
||||||
warn "Possible magic numbers found (review above)"
|
|
||||||
else
|
|
||||||
pass "No obvious magic numbers"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── 06. SOLID ──────────────────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "== 06. SOLID =="
|
|
||||||
|
|
||||||
# Check that ports (interfaces) exist
|
|
||||||
if grep -l "Protocol\|ABC\|abstractmethod" document-parser/domain/ports.py 2>/dev/null; then
|
|
||||||
pass "Domain ports use Protocol/ABC (Dependency Inversion)"
|
|
||||||
else
|
|
||||||
fail "No abstract ports found in domain"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── 07. Decoupling ─────────────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "== 07. Decoupling =="
|
|
||||||
|
|
||||||
# Frontend should not hardcode backend URLs (except in config)
|
|
||||||
if grep -rn "localhost:8000\|127.0.0.1:8000" frontend/src/ --include="*.ts" --include="*.vue" 2>/dev/null | grep -v "config\|env\|http.ts"; then
|
|
||||||
fail "Frontend hardcodes backend URL outside config"
|
|
||||||
else
|
|
||||||
pass "Frontend backend URL is configurable"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── 08. Security ───────────────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "== 08. Security =="
|
|
||||||
|
|
||||||
# Check for hardcoded secrets
|
|
||||||
if grep -rni "password\s*=\s*['\"].\+['\"\|secret\s*=\s*['\"].\+['\"\|api_key\s*=\s*['\"].\+['\"]" document-parser/ --include="*.py" ! -path "*/.venv/*" ! -path "*/tests/*" 2>/dev/null; then
|
|
||||||
fail "Possible hardcoded secrets found"
|
|
||||||
else
|
|
||||||
pass "No hardcoded secrets detected"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check for eval/exec
|
|
||||||
if grep -rn "\beval(\|exec(" document-parser/ --include="*.py" ! -path "*/.venv/*" 2>/dev/null; then
|
|
||||||
fail "eval() or exec() found in backend"
|
|
||||||
else
|
|
||||||
pass "No eval/exec in backend"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check CORS configuration exists
|
|
||||||
if grep -rn "CORSMiddleware" document-parser/ --include="*.py" ! -path "*/.venv/*" 2>/dev/null > /dev/null; then
|
|
||||||
pass "CORS middleware is configured"
|
|
||||||
else
|
|
||||||
warn "No CORS middleware found"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── 09. Tests ──────────────────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "== 09. Tests =="
|
|
||||||
|
|
||||||
# Backend tests exist
|
|
||||||
BACKEND_TESTS=$(find document-parser/tests -name "test_*.py" 2>/dev/null | wc -l)
|
|
||||||
if [ "$BACKEND_TESTS" -gt 0 ]; then
|
|
||||||
pass "Backend: $BACKEND_TESTS test files found"
|
|
||||||
else
|
|
||||||
fail "No backend test files found"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Frontend tests exist
|
|
||||||
FRONTEND_TESTS=$(find frontend/src -name "*.test.*" 2>/dev/null | wc -l)
|
|
||||||
if [ "$FRONTEND_TESTS" -gt 0 ]; then
|
|
||||||
pass "Frontend: $FRONTEND_TESTS test files found"
|
|
||||||
else
|
|
||||||
fail "No frontend test files found"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# E2E tests exist
|
|
||||||
E2E_TESTS=$(find e2e -name "*.feature" 2>/dev/null | wc -l)
|
|
||||||
if [ "$E2E_TESTS" -gt 0 ]; then
|
|
||||||
pass "E2E: $E2E_TESTS feature files found"
|
|
||||||
else
|
|
||||||
warn "No e2e feature files found"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check for skipped tests
|
|
||||||
if grep -rn "@skip\|@ignore\|xit(\|xdescribe(\|pytest.mark.skip" document-parser/tests/ frontend/src/ 2>/dev/null | grep -v "helpers"; then
|
|
||||||
warn "Skipped tests found (review if intentional)"
|
|
||||||
else
|
|
||||||
pass "No skipped tests"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── 10. CI / Build ────────────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "== 10. CI / Build =="
|
|
||||||
|
|
||||||
# CI workflow exists
|
|
||||||
if [ -f .github/workflows/ci.yml ]; then
|
|
||||||
pass "CI workflow exists"
|
|
||||||
else
|
|
||||||
fail "No CI workflow found"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Dockerfile exists
|
|
||||||
if [ -f Dockerfile ]; then
|
|
||||||
pass "Dockerfile exists"
|
|
||||||
else
|
|
||||||
fail "No Dockerfile found"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Health check in docker-compose
|
|
||||||
if grep -q "healthcheck" docker-compose.yml 2>/dev/null; then
|
|
||||||
pass "Docker Compose has health check"
|
|
||||||
else
|
|
||||||
warn "No health check in docker-compose.yml"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── 11. Documentation ─────────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "== 11. Documentation =="
|
|
||||||
|
|
||||||
# CHANGELOG exists and has content
|
|
||||||
if [ -f CHANGELOG.md ] && [ -s CHANGELOG.md ]; then
|
|
||||||
pass "CHANGELOG.md exists and is not empty"
|
|
||||||
else
|
|
||||||
fail "CHANGELOG.md missing or empty"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# README exists
|
|
||||||
if [ -f README.md ]; then
|
|
||||||
pass "README.md exists"
|
|
||||||
else
|
|
||||||
fail "README.md missing"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check for TODO/FIXME without issue reference
|
|
||||||
TODOS=$(grep -rn "TODO\|FIXME" document-parser/ frontend/src/ --include="*.py" --include="*.ts" --include="*.vue" ! -path "*/.venv/*" ! -path "*/node_modules/*" 2>/dev/null | grep -v "#[0-9]" | head -5)
|
|
||||||
if [ -n "$TODOS" ]; then
|
|
||||||
warn "TODO/FIXME without issue reference:"
|
|
||||||
echo "$TODOS"
|
|
||||||
else
|
|
||||||
pass "No orphaned TODO/FIXME"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── 12. Performance ───────────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "== 12. Performance =="
|
|
||||||
|
|
||||||
# Check for synchronous file I/O in async context
|
|
||||||
if grep -rn "open(" document-parser/api/ document-parser/services/ --include="*.py" 2>/dev/null | grep -v "aiofiles\|async\|#"; then
|
|
||||||
warn "Synchronous file I/O in async code (review above)"
|
|
||||||
else
|
|
||||||
pass "No synchronous file I/O in async endpoints"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check for N+1 patterns (loop with DB call)
|
|
||||||
if grep -rn "for.*in.*:" document-parser/services/ --include="*.py" -A5 2>/dev/null | grep "await.*repo\|await.*db"; then
|
|
||||||
warn "Possible N+1 query pattern (review above)"
|
|
||||||
else
|
|
||||||
pass "No obvious N+1 patterns"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── Summary ────────────────────────────────────────────────────────────────
|
|
||||||
echo ""
|
|
||||||
echo "============================================"
|
|
||||||
echo -e " ${GREEN}PASS${NC}: $PASS"
|
|
||||||
echo -e " ${YELLOW}WARN${NC}: $WARN"
|
|
||||||
echo -e " ${RED}FAIL${NC}: $FAIL"
|
|
||||||
echo "============================================"
|
|
||||||
|
|
||||||
if [ "$FAIL" -gt 0 ]; then
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
@ -1,70 +0,0 @@
|
||||||
# Stack Profile — FastAPI + Vue 3
|
|
||||||
|
|
||||||
Profile for running the 12 release audits on Docling Studio.
|
|
||||||
|
|
||||||
## Layer Mapping
|
|
||||||
|
|
||||||
| Generic Layer | Docling Studio Path | Language |
|
|
||||||
|---------------|---------------------|----------|
|
|
||||||
| **Domain** | `document-parser/domain/` | Python |
|
|
||||||
| **Services** | `document-parser/services/` | Python |
|
|
||||||
| **API** | `document-parser/api/` | Python |
|
|
||||||
| **Infrastructure** | `document-parser/infra/` | Python |
|
|
||||||
| **Persistence** | `document-parser/persistence/` | Python |
|
|
||||||
| **Frontend** | `frontend/src/` | TypeScript / Vue |
|
|
||||||
| **Tests (backend)** | `document-parser/tests/` | Python |
|
|
||||||
| **Tests (frontend)** | `frontend/src/**/*.test.*` | TypeScript |
|
|
||||||
| **Tests (e2e API)** | `e2e/api/` | Karate (Gherkin) |
|
|
||||||
| **Tests (e2e UI)** | `e2e/ui/` | Karate UI (Gherkin) |
|
|
||||||
| **CI/CD** | `.github/workflows/` | YAML |
|
|
||||||
| **Docker** | `Dockerfile`, `docker-compose.yml`, `nginx.conf` | Docker / Nginx |
|
|
||||||
|
|
||||||
## Excluded Paths
|
|
||||||
|
|
||||||
These paths are excluded from audits:
|
|
||||||
|
|
||||||
- `document-parser/.venv/`
|
|
||||||
- `document-parser/__pycache__/`
|
|
||||||
- `frontend/node_modules/`
|
|
||||||
- `frontend/dist/`
|
|
||||||
- `e2e/**/target/`
|
|
||||||
|
|
||||||
## Framework Detection
|
|
||||||
|
|
||||||
Imports that should NOT appear in the domain layer:
|
|
||||||
|
|
||||||
```python
|
|
||||||
# Forbidden in document-parser/domain/
|
|
||||||
from fastapi import ...
|
|
||||||
from pydantic import ... # except BaseModel for value objects
|
|
||||||
import aiosqlite
|
|
||||||
from infra import ...
|
|
||||||
from persistence import ...
|
|
||||||
from api import ...
|
|
||||||
```
|
|
||||||
|
|
||||||
Imports that should NOT cross feature boundaries in the frontend:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// features/analysis/ should NOT import from features/chunking/store
|
|
||||||
// features/document/ should NOT import from features/analysis/store
|
|
||||||
// Cross-feature communication goes through shared/ or events
|
|
||||||
```
|
|
||||||
|
|
||||||
## Tools & Commands
|
|
||||||
|
|
||||||
| Task | Command |
|
|
||||||
|------|---------|
|
|
||||||
| Backend lint | `cd document-parser && ruff check .` |
|
|
||||||
| Backend format | `cd document-parser && ruff format --check .` |
|
|
||||||
| Backend tests | `cd document-parser && pytest tests/ -v` |
|
|
||||||
| Frontend lint | `cd frontend && npx eslint src/` |
|
|
||||||
| Frontend type-check | `cd frontend && npm run type-check` |
|
|
||||||
| Frontend format | `cd frontend && npx prettier --check src/` |
|
|
||||||
| Frontend tests | `cd frontend && npm run test:run` |
|
|
||||||
| E2E API tests | `mvn test -f e2e/api/pom.xml -Dkarate.options="--tags @smoke"` |
|
|
||||||
| E2E UI tests | `mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @critical"` |
|
|
||||||
| Docker build | `docker compose build` |
|
|
||||||
| Docker health | `curl -s http://localhost:3000/api/health` |
|
|
||||||
| Dependency audit (Python) | `cd document-parser && pip audit` |
|
|
||||||
| Dependency audit (Node) | `cd frontend && npm audit` |
|
|
||||||
Loading…
Reference in a new issue