feat(#180): feature-flag ingestion pipeline and add brainless one-liner Quick Start
- Conditionally mount ingestion router only when OpenSearch + embedding are configured - Add `ingestionAvailable` field to /api/health response - Add `ingestion` feature flag to frontend (hides Search nav, Ingest button, OpenSearch badge, indexed badges/filters when disabled) - Skip ingestion polling when flag is off - Make OpenSearch + embedding optional in docker-compose via profiles - Add docker-compose.ingestion.yml override for full-stack ingestion - Set BATCH_PAGE_SIZE=5 default in Docker local image - Lead Quick Start with one-liner docker run command - Document ingestion as opt-in with dedicated section - Add BATCH_PAGE_SIZE, MAX_FILE_SIZE_MB, MAX_PAGE_COUNT, RATE_LIMIT_RPM to config tables - Update test counts (380 backend, 159 frontend) - Date CHANGELOG 0.4.0, bump frontend version to 0.4.0 - Sync CONTRIBUTING.md with E2E Karate test sections Closes #180
This commit is contained in:
Pier-Jean Malandrino
parent
6f008ec262
commit
ba54427445
21 changed files with 310 additions and 83 deletions
|
|
@ -4,7 +4,7 @@ All notable changes to Docling Studio will be documented in this file.
|
|||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/).
|
||||
|
||||
## [0.4.0] - Unreleased
|
||||
## [0.4.0] - 2026-04-13
|
||||
|
||||
### Added
|
||||
|
||||
|
|
|
|||
|
|
@ -96,15 +96,43 @@ npx prettier --write src/ # auto-format
|
|||
## Running Tests
|
||||
|
||||
```bash
|
||||
# Backend (199 tests)
|
||||
# Backend (377 tests)
|
||||
cd document-parser
|
||||
pytest tests/ -v
|
||||
|
||||
# Frontend (129 tests)
|
||||
# Frontend (156 tests)
|
||||
cd frontend
|
||||
npm run test:run
|
||||
```
|
||||
|
||||
### E2E API (Karate)
|
||||
|
||||
```bash
|
||||
# Generate test PDFs + start stack
|
||||
python e2e/generate-test-data.py
|
||||
docker compose up -d --wait
|
||||
|
||||
# Run all API tests
|
||||
mvn test -f e2e/api/pom.xml
|
||||
|
||||
# Or by tag: @smoke, @regression, @e2e
|
||||
mvn test -f e2e/api/pom.xml -Dkarate.options="--tags @smoke"
|
||||
```
|
||||
|
||||
### E2E UI (Karate UI)
|
||||
|
||||
```bash
|
||||
# Generate test PDFs + start stack (if not already running)
|
||||
python e2e/generate-test-data.py
|
||||
docker compose up -d --wait
|
||||
|
||||
# Run critical UI tests (CI scope)
|
||||
mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @critical"
|
||||
|
||||
# Run all UI tests (local scope)
|
||||
mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @ui"
|
||||
```
|
||||
|
||||
All tests must pass before submitting a PR.
|
||||
|
||||
## Submitting Changes
|
||||
|
|
|
|||
|
|
@ -76,3 +76,4 @@ RUN pip install --no-cache-dir torch torchvision --index-url https://download.py
|
|||
RUN chown -R appuser:appuser /app \
|
||||
&& chown -R appuser:appuser /usr/local/lib/python3.12/site-packages/rapidocr/models
|
||||
ENV CONVERSION_ENGINE=local
|
||||
ENV BATCH_PAGE_SIZE=5
|
||||
|
|
|
|||
83
README.md
83
README.md
|
|
@ -31,9 +31,13 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t
|
|||
- **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description, image generation
|
||||
- **Bounding box visualization** — color-coded element overlay directly on the PDF
|
||||
- **Per-page results** — right panel syncs with the current PDF page
|
||||
- **Chunking** — split extracted content into semantic chunks (hierarchical, hybrid, or page-based) with configurable token limits and inline editing
|
||||
- **Ingestion pipeline** — Docling → chunking → embedding → OpenSearch vector indexing (one-click from Studio)
|
||||
- **Markdown & HTML export** of extracted content
|
||||
- **Document management** — upload, list, delete
|
||||
- **Document management** — upload, list, delete, search, filter by indexing status
|
||||
- **Analysis history** — re-visit and open past analyses
|
||||
- **Upload limits** — configurable max file size and max page count per document
|
||||
- **Rate limiting** — configurable requests per minute per IP
|
||||
- **Dark / Light theme** and **FR / EN** localization
|
||||
|
||||
|
||||
|
|
@ -74,7 +78,7 @@ document-parser/
|
|||
├── services/ # Use case orchestration
|
||||
│ ├── document_service.py # Upload, delete, preview
|
||||
│ └── analysis_service.py # Async Docling processing
|
||||
└── tests/ # 199 tests (pytest)
|
||||
└── tests/ # 377 tests (pytest)
|
||||
```
|
||||
|
||||
### Frontend structure (feature-based)
|
||||
|
|
@ -98,14 +102,24 @@ frontend/src/
|
|||
|
||||
## Quick Start
|
||||
|
||||
Docling Studio ships two Docker image variants:
|
||||
One command, nothing else to install:
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
|
||||
```
|
||||
|
||||
Open [http://localhost:3000](http://localhost:3000), upload a PDF, and get results. That's it.
|
||||
|
||||
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
|
||||
|
||||
### Image variants
|
||||
|
||||
| Variant | Image tag | Size | Description |
|
||||
|---------|-----------|------|-------------|
|
||||
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only |
|
||||
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
|
||||
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only (downloads ML models on first run) |
|
||||
|
||||
### Docker — remote mode (fastest)
|
||||
For remote mode:
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 \
|
||||
|
|
@ -113,27 +127,17 @@ docker run -p 3000:3000 \
|
|||
ghcr.io/scub-france/docling-studio:latest-remote
|
||||
```
|
||||
|
||||
### Docker — local mode (self-contained)
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
|
||||
```
|
||||
|
||||
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
|
||||
|
||||
Open [http://localhost:3000](http://localhost:3000)
|
||||
|
||||
### Docker Compose (for development)
|
||||
### Docker Compose
|
||||
|
||||
```bash
|
||||
git clone https://github.com/scub-france/Docling-Studio.git
|
||||
cd Docling-Studio
|
||||
|
||||
# Local mode (default)
|
||||
# Simple mode (backend + frontend only)
|
||||
docker compose up --build
|
||||
|
||||
# Remote mode
|
||||
CONVERSION_MODE=remote DOCLING_SERVE_URL=http://your-docling-serve:5001 docker compose up --build
|
||||
# With ingestion pipeline (OpenSearch + embeddings)
|
||||
docker compose --profile ingestion -f docker-compose.yml -f docker-compose.ingestion.yml up --build
|
||||
```
|
||||
|
||||
### Local Development
|
||||
|
|
@ -162,12 +166,12 @@ npm run dev
|
|||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Backend (199 tests)
|
||||
# Backend (377 tests)
|
||||
cd document-parser
|
||||
pip install pytest pytest-asyncio httpx
|
||||
pytest tests/ -v
|
||||
|
||||
# Frontend (129 tests)
|
||||
# Frontend (156 tests)
|
||||
cd frontend
|
||||
npm run test:run
|
||||
```
|
||||
|
|
@ -202,6 +206,43 @@ All configuration is done via environment variables. See [`.env.example`](.env.e
|
|||
| `UPLOAD_DIR` | `./uploads` | File storage directory |
|
||||
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
|
||||
| `CONVERSION_TIMEOUT` | `600` | Max seconds for a single Docling conversion |
|
||||
| `BATCH_PAGE_SIZE` | `5` (Docker) / `0` | Pages per batch (`0` = process all at once) |
|
||||
| `MAX_FILE_SIZE_MB` | `50` | Maximum upload file size in MB (`0` = unlimited) |
|
||||
| `MAX_PAGE_COUNT` | `0` | Maximum number of pages per document (`0` = unlimited) |
|
||||
| `RATE_LIMIT_RPM` | `100` | Max requests per minute per IP (`0` = disabled) |
|
||||
|
||||
## Upload Limits
|
||||
|
||||
Docling Studio enforces configurable limits on uploaded documents to protect the server against oversized files and long-running analyses:
|
||||
|
||||
- **`MAX_FILE_SIZE_MB`** (default `50`) — rejects uploads exceeding this size. Validated at two levels: early `Content-Length` check and streaming byte count.
|
||||
- **`MAX_PAGE_COUNT`** (default `0` = unlimited) — rejects documents with more pages than allowed. Useful on shared instances or Hugging Face Spaces to cap processing time.
|
||||
|
||||
Both limits are exposed in the `/api/health` endpoint so the frontend can display them to the user before upload. Set either to `0` to disable the corresponding check.
|
||||
|
||||
## Ingestion Pipeline (opt-in)
|
||||
|
||||
Docling Studio can optionally index extracted chunks into [OpenSearch](https://opensearch.org/) for vector and full-text search. This requires two additional services (OpenSearch + embedding) and is **disabled by default**.
|
||||
|
||||
To enable ingestion with Docker Compose:
|
||||
|
||||
```bash
|
||||
docker compose --profile ingestion \
|
||||
-f docker-compose.yml -f docker-compose.ingestion.yml \
|
||||
up --build
|
||||
```
|
||||
|
||||
When ingestion is enabled, the UI shows:
|
||||
- An **Ingest** button in Studio to push chunks to OpenSearch
|
||||
- An **OpenSearch** connection status badge in the sidebar
|
||||
- **Indexed / Not indexed** filters on the Documents page
|
||||
- A **Search** page for full-text and vector search across indexed documents
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `OPENSEARCH_URL` | — | OpenSearch endpoint (empty = ingestion disabled) |
|
||||
| `EMBEDDING_URL` | — | Embedding service endpoint (empty = ingestion disabled) |
|
||||
| `EMBEDDING_DIMENSION` | `384` | Vector dimension (must match embedding model) |
|
||||
|
||||
## CI / Release
|
||||
|
||||
|
|
|
|||
19
docker-compose.ingestion.yml
Normal file
19
docker-compose.ingestion.yml
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
# Override to enable the ingestion pipeline (OpenSearch + embeddings).
|
||||
#
|
||||
# Usage:
|
||||
# docker compose --profile ingestion -f docker-compose.yml -f docker-compose.ingestion.yml up --build
|
||||
#
|
||||
# This wires the backend to the OpenSearch and embedding services started
|
||||
# by the "ingestion" profile and ensures they are healthy before the
|
||||
# backend starts.
|
||||
|
||||
services:
|
||||
document-parser:
|
||||
environment:
|
||||
OPENSEARCH_URL: http://opensearch:9200
|
||||
EMBEDDING_URL: http://embedding:8001
|
||||
depends_on:
|
||||
opensearch:
|
||||
condition: service_healthy
|
||||
embedding:
|
||||
condition: service_healthy
|
||||
|
|
@ -1,6 +1,7 @@
|
|||
services:
|
||||
# --- OpenSearch (single-node, security disabled) ---
|
||||
opensearch:
|
||||
profiles: ["ingestion"]
|
||||
image: opensearchproject/opensearch:2
|
||||
environment:
|
||||
discovery.type: single-node
|
||||
|
|
@ -16,6 +17,7 @@ services:
|
|||
|
||||
# --- Embedding service (sentence-transformers) ---
|
||||
embedding:
|
||||
profiles: ["ingestion"]
|
||||
build:
|
||||
context: ./embedding-service
|
||||
environment:
|
||||
|
|
@ -49,13 +51,8 @@ services:
|
|||
RATE_LIMIT_RPM: ${RATE_LIMIT_RPM:-100}
|
||||
MAX_FILE_SIZE_MB: ${MAX_FILE_SIZE_MB:-50}
|
||||
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-0}
|
||||
OPENSEARCH_URL: http://opensearch:9200
|
||||
EMBEDDING_URL: http://embedding:8001
|
||||
depends_on:
|
||||
opensearch:
|
||||
condition: service_healthy
|
||||
embedding:
|
||||
condition: service_healthy
|
||||
OPENSEARCH_URL: ${OPENSEARCH_URL:-}
|
||||
EMBEDDING_URL: ${EMBEDDING_URL:-}
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
|
|
|
|||
|
|
@ -1,15 +1,28 @@
|
|||
# Getting Started
|
||||
|
||||
Docling Studio ships two Docker image variants:
|
||||
## Quick Start
|
||||
|
||||
| Variant | Image tag | Size | Description |
|
||||
|---------|-----------|------|-------------|
|
||||
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
|
||||
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only (downloads ML models on first run) |
|
||||
One command, nothing else to install:
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
|
||||
```
|
||||
|
||||
Open [http://localhost:3000](http://localhost:3000), upload a PDF, and get results. That's it.
|
||||
|
||||
!!! note
|
||||
The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
|
||||
|
||||
{ width="600" }
|
||||
|
||||
## Docker — remote mode (fastest)
|
||||
## Image Variants
|
||||
|
||||
| Variant | Image tag | Size | Description |
|
||||
|---------|-----------|------|-------------|
|
||||
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only |
|
||||
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
|
||||
|
||||
For remote mode:
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 \
|
||||
|
|
@ -17,27 +30,19 @@ docker run -p 3000:3000 \
|
|||
ghcr.io/scub-france/docling-studio:latest-remote
|
||||
```
|
||||
|
||||
## Docker — local mode (self-contained)
|
||||
|
||||
```bash
|
||||
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
|
||||
```
|
||||
|
||||
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
|
||||
|
||||
Open [http://localhost:3000](http://localhost:3000).
|
||||
|
||||
## Docker Compose (recommended for development)
|
||||
## Docker Compose
|
||||
|
||||
```bash
|
||||
git clone https://github.com/scub-france/Docling-Studio.git
|
||||
cd Docling-Studio
|
||||
|
||||
# Local mode (default)
|
||||
# Simple mode (backend + frontend only)
|
||||
docker compose up --build
|
||||
|
||||
# Remote mode
|
||||
CONVERSION_MODE=remote DOCLING_SERVE_URL=http://your-docling-serve:5001 docker compose up --build
|
||||
# With ingestion pipeline (OpenSearch + embeddings)
|
||||
docker compose --profile ingestion \
|
||||
-f docker-compose.yml -f docker-compose.ingestion.yml \
|
||||
up --build
|
||||
```
|
||||
|
||||
## Local Development
|
||||
|
|
@ -136,10 +141,48 @@ All configuration is done via environment variables:
|
|||
| `UPLOAD_DIR` | `./uploads` | File storage directory |
|
||||
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
|
||||
| `CONVERSION_TIMEOUT` | `600` | Max seconds per Docling conversion |
|
||||
| `BATCH_PAGE_SIZE` | `5` (Docker) / `0` | Pages per batch (`0` = process all at once) |
|
||||
| `MAX_CONCURRENT_ANALYSES` | `3` | Maximum parallel analysis jobs |
|
||||
| `DEPLOYMENT_MODE` | `self-hosted` | `self-hosted` or `huggingface` (shows disclaimer banner) |
|
||||
| `MAX_FILE_SIZE_MB` | `50` | Maximum upload file size in MB (`0` = unlimited) |
|
||||
| `MAX_PAGE_COUNT` | `0` | Maximum number of pages per document (`0` = unlimited) |
|
||||
| `RATE_LIMIT_RPM` | `100` | Max requests per minute per IP (`0` = disabled) |
|
||||
| `APP_VERSION` | `dev` | Application version (set automatically by CI/Docker) |
|
||||
|
||||
## Upload Limits
|
||||
|
||||
Docling Studio enforces configurable limits on uploaded documents to protect the server against oversized files and long-running analyses:
|
||||
|
||||
- **`MAX_FILE_SIZE_MB`** (default `50`) — rejects uploads exceeding this size. Validated at two levels: early `Content-Length` check and streaming byte count.
|
||||
- **`MAX_PAGE_COUNT`** (default `0` = unlimited) — rejects documents with more pages than allowed. Useful on shared instances or Hugging Face Spaces to cap processing time.
|
||||
|
||||
Both limits are exposed in the `/api/health` endpoint so the frontend can display them to the user before upload. Set either to `0` to disable the corresponding check.
|
||||
|
||||
## Ingestion Pipeline (opt-in)
|
||||
|
||||
Docling Studio can optionally index extracted chunks into [OpenSearch](https://opensearch.org/) for vector and full-text search. This requires two additional services (OpenSearch + embedding) and is **disabled by default**.
|
||||
|
||||
To enable ingestion with Docker Compose:
|
||||
|
||||
```bash
|
||||
docker compose --profile ingestion \
|
||||
-f docker-compose.yml -f docker-compose.ingestion.yml \
|
||||
up --build
|
||||
```
|
||||
|
||||
When ingestion is enabled, the UI shows:
|
||||
|
||||
- An **Ingest** button in Studio to push chunks to OpenSearch
|
||||
- An **OpenSearch** connection status badge in the sidebar
|
||||
- **Indexed / Not indexed** filters on the Documents page
|
||||
- A **Search** page for full-text and vector search across indexed documents
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `OPENSEARCH_URL` | — | OpenSearch endpoint (empty = ingestion disabled) |
|
||||
| `EMBEDDING_URL` | — | Embedding service endpoint (empty = ingestion disabled) |
|
||||
| `EMBEDDING_DIMENSION` | `384` | Vector dimension (must match embedding model) |
|
||||
|
||||
## System Requirements
|
||||
|
||||
| | Remote image | Local image |
|
||||
|
|
|
|||
|
|
@ -18,7 +18,8 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t
|
|||
- **Document management** — upload, list, delete
|
||||
- **Analysis history** — re-visit and open past analyses
|
||||
- **Feature flags** — capabilities adapt to the conversion engine (local vs remote)
|
||||
- **Rate limiting** — 60 requests per minute per IP to protect the backend
|
||||
- **Upload limits** — configurable max file size (`MAX_FILE_SIZE_MB`) and max page count (`MAX_PAGE_COUNT`) per document
|
||||
- **Rate limiting** — configurable requests per minute per IP (`RATE_LIMIT_RPM`)
|
||||
- **Deployment modes** — self-hosted (default) or HuggingFace Spaces (with disclaimer banner)
|
||||
- **Health endpoint** — `/api/health` reports engine type, deployment mode, and database status
|
||||
- **Dark / Light theme** and **FR / EN** localization
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ class HealthResponse(_CamelModel):
|
|||
database: str
|
||||
max_page_count: int | None = None
|
||||
max_file_size_mb: int | None = None
|
||||
ingestion_available: bool = False
|
||||
|
||||
|
||||
class DocumentResponse(_CamelModel):
|
||||
|
|
|
|||
|
|
@ -138,7 +138,11 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
|||
document_repo, analysis_repo = _build_repos()
|
||||
app.state.analysis_service = _build_analysis_service(document_repo, analysis_repo)
|
||||
app.state.document_service = _build_document_service(document_repo, analysis_repo)
|
||||
app.state.ingestion_service = _build_ingestion_service()
|
||||
ingestion_service = _build_ingestion_service()
|
||||
app.state.ingestion_service = ingestion_service
|
||||
if ingestion_service is not None:
|
||||
app.include_router(ingestion_router)
|
||||
logger.info("Ingestion router mounted")
|
||||
logger.info("Docling Studio backend ready (engine=%s)", settings.conversion_engine)
|
||||
yield
|
||||
|
||||
|
|
@ -165,7 +169,6 @@ if settings.rate_limit_rpm > 0:
|
|||
|
||||
app.include_router(documents_router)
|
||||
app.include_router(analyses_router)
|
||||
app.include_router(ingestion_router)
|
||||
|
||||
|
||||
@app.get("/api/health", response_model=HealthResponse)
|
||||
|
|
@ -188,4 +191,5 @@ async def health() -> HealthResponse:
|
|||
database=db_status,
|
||||
max_page_count=settings.max_page_count if settings.max_page_count > 0 else None,
|
||||
max_file_size_mb=settings.max_file_size_mb if settings.max_file_size_mb > 0 else None,
|
||||
ingestion_available=getattr(app.state, "ingestion_service", None) is not None,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -51,6 +51,23 @@ class TestHealthEndpoint:
|
|||
assert "maxFileSizeMb" in data
|
||||
assert data["maxFileSizeMb"] == 50
|
||||
|
||||
def test_health_exposes_ingestion_available_false(self, client):
|
||||
original = getattr(app.state, "ingestion_service", None)
|
||||
app.state.ingestion_service = None
|
||||
resp = client.get("/api/health")
|
||||
app.state.ingestion_service = original
|
||||
data = resp.json()
|
||||
assert "ingestionAvailable" in data
|
||||
assert data["ingestionAvailable"] is False
|
||||
|
||||
def test_health_exposes_ingestion_available_true(self, client):
|
||||
original = getattr(app.state, "ingestion_service", None)
|
||||
app.state.ingestion_service = MagicMock()
|
||||
resp = client.get("/api/health")
|
||||
app.state.ingestion_service = original
|
||||
data = resp.json()
|
||||
assert data["ingestionAvailable"] is True
|
||||
|
||||
|
||||
class TestDocumentEndpoints:
|
||||
def test_list_documents(self, client, mock_document_service):
|
||||
|
|
|
|||
|
|
@ -104,7 +104,18 @@ class TestIngestionStatus:
|
|||
|
||||
|
||||
class TestIngestionDisabled:
|
||||
def test_returns_503_when_disabled(self) -> None:
|
||||
def test_router_not_mounted_returns_404(self) -> None:
|
||||
"""When ingestion service is None, router should not be mounted → 404."""
|
||||
app = FastAPI()
|
||||
# Do NOT include ingestion router — simulates main.py conditional mount
|
||||
app.state.ingestion_service = None
|
||||
app.state.analysis_service = AsyncMock()
|
||||
tc = TestClient(app)
|
||||
resp = tc.post("/api/ingestion/job-1")
|
||||
assert resp.status_code == 404
|
||||
|
||||
def test_status_still_returns_503_when_router_mounted_but_service_none(self) -> None:
|
||||
"""If router is mounted but service is None, endpoints return 503."""
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
app.state.ingestion_service = None
|
||||
|
|
|
|||
4
frontend/package-lock.json
generated
4
frontend/package-lock.json
generated
|
|
@ -1,12 +1,12 @@
|
|||
{
|
||||
"name": "docling-studio",
|
||||
"version": "0.3.0",
|
||||
"version": "0.3.1",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "docling-studio",
|
||||
"version": "0.3.0",
|
||||
"version": "0.3.1",
|
||||
"dependencies": {
|
||||
"dompurify": "^3.3.3",
|
||||
"marked": "^17.0.4",
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"name": "docling-studio",
|
||||
"version": "0.3.1",
|
||||
"version": "0.4.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
|
|
|
|||
|
|
@ -202,7 +202,10 @@
|
|||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chunk-empty" v-else-if="!chunkingStore.rechunking">
|
||||
<div
|
||||
class="chunk-empty"
|
||||
v-if="!pageChunks.length && !chunkingStore.rechunking && deleteConfirmIdx === -1"
|
||||
>
|
||||
<p>
|
||||
{{ chunks.length ? t('chunking.noChunksOnPage') : t('chunking.noChunks') }}
|
||||
</p>
|
||||
|
|
|
|||
|
|
@ -82,6 +82,38 @@ describe('useFeatureFlagStore', () => {
|
|||
expect(store.maxFileSizeMb).toBe(0)
|
||||
})
|
||||
|
||||
it('enables ingestion when ingestionAvailable is true', async () => {
|
||||
mockApiFetch.mockResolvedValue({
|
||||
status: 'ok',
|
||||
engine: 'local',
|
||||
ingestionAvailable: true,
|
||||
})
|
||||
const store = useFeatureFlagStore()
|
||||
await store.load()
|
||||
expect(store.ingestionAvailable).toBe(true)
|
||||
expect(store.isEnabled('ingestion')).toBe(true)
|
||||
})
|
||||
|
||||
it('disables ingestion when ingestionAvailable is false', async () => {
|
||||
mockApiFetch.mockResolvedValue({
|
||||
status: 'ok',
|
||||
engine: 'local',
|
||||
ingestionAvailable: false,
|
||||
})
|
||||
const store = useFeatureFlagStore()
|
||||
await store.load()
|
||||
expect(store.ingestionAvailable).toBe(false)
|
||||
expect(store.isEnabled('ingestion')).toBe(false)
|
||||
})
|
||||
|
||||
it('defaults ingestionAvailable to false when missing', async () => {
|
||||
mockApiFetch.mockResolvedValue({ status: 'ok', engine: 'local' })
|
||||
const store = useFeatureFlagStore()
|
||||
await store.load()
|
||||
expect(store.ingestionAvailable).toBe(false)
|
||||
expect(store.isEnabled('ingestion')).toBe(false)
|
||||
})
|
||||
|
||||
it('handles health endpoint failure gracefully', async () => {
|
||||
mockApiFetch.mockRejectedValue(new Error('Network error'))
|
||||
const store = useFeatureFlagStore()
|
||||
|
|
|
|||
|
|
@ -13,9 +13,10 @@ interface HealthResponse {
|
|||
deploymentMode?: DeploymentMode
|
||||
maxPageCount?: number
|
||||
maxFileSizeMb?: number
|
||||
ingestionAvailable?: boolean
|
||||
}
|
||||
|
||||
export type FeatureFlag = 'chunking' | 'disclaimer'
|
||||
export type FeatureFlag = 'chunking' | 'disclaimer' | 'ingestion'
|
||||
|
||||
interface FeatureFlagDef {
|
||||
description: string
|
||||
|
|
@ -25,6 +26,7 @@ interface FeatureFlagDef {
|
|||
interface FeatureFlagContext {
|
||||
engine: ConversionEngine | null
|
||||
deploymentMode: DeploymentMode | null
|
||||
ingestionAvailable: boolean
|
||||
}
|
||||
|
||||
const featureRegistry: Record<FeatureFlag, FeatureFlagDef> = {
|
||||
|
|
@ -36,6 +38,10 @@ const featureRegistry: Record<FeatureFlag, FeatureFlagDef> = {
|
|||
description: 'Show shared-instance disclaimer banner',
|
||||
isEnabled: (ctx) => ctx.deploymentMode === 'huggingface',
|
||||
},
|
||||
ingestion: {
|
||||
description: 'OpenSearch ingestion pipeline (embedding + vector indexing)',
|
||||
isEnabled: (ctx) => ctx.ingestionAvailable,
|
||||
},
|
||||
}
|
||||
|
||||
export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
||||
|
|
@ -43,6 +49,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
|||
const deploymentMode = ref<DeploymentMode | null>(null)
|
||||
const maxPageCount = ref<number>(0)
|
||||
const maxFileSizeMb = ref<number>(0)
|
||||
const ingestionAvailable = ref(false)
|
||||
const appVersion = ref<string>(__APP_VERSION__)
|
||||
const loaded = ref(false)
|
||||
const error = ref<string | null>(null)
|
||||
|
|
@ -50,6 +57,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
|||
const context = computed<FeatureFlagContext>(() => ({
|
||||
engine: engine.value,
|
||||
deploymentMode: deploymentMode.value,
|
||||
ingestionAvailable: ingestionAvailable.value,
|
||||
}))
|
||||
|
||||
function isEnabled(flag: FeatureFlag): boolean {
|
||||
|
|
@ -65,6 +73,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
|||
deploymentMode.value = data.deploymentMode ?? 'self-hosted'
|
||||
maxPageCount.value = data.maxPageCount ?? 0
|
||||
maxFileSizeMb.value = data.maxFileSizeMb ?? 0
|
||||
ingestionAvailable.value = data.ingestionAvailable ?? false
|
||||
appMaxFileSizeMb.value = maxFileSizeMb.value
|
||||
appMaxPageCount.value = maxPageCount.value
|
||||
if (data.version) appVersion.value = data.version
|
||||
|
|
@ -81,6 +90,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
|
|||
deploymentMode,
|
||||
maxPageCount,
|
||||
maxFileSizeMb,
|
||||
ingestionAvailable,
|
||||
appVersion,
|
||||
loaded,
|
||||
error,
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
class="search-input"
|
||||
:placeholder="t('ingestion.search')"
|
||||
/>
|
||||
<div class="filter-group">
|
||||
<div v-if="ingestionEnabled" class="filter-group">
|
||||
<button
|
||||
v-for="f in filters"
|
||||
:key="f.value"
|
||||
|
|
@ -54,6 +54,7 @@
|
|||
</div>
|
||||
</div>
|
||||
<div class="doc-row-actions">
|
||||
<template v-if="ingestionEnabled">
|
||||
<span
|
||||
v-if="ingestionStore.ingestedDocs[doc.id]"
|
||||
class="status-badge indexed"
|
||||
|
|
@ -65,6 +66,7 @@
|
|||
<span v-else class="status-badge not-indexed">
|
||||
{{ t('ingestion.notIndexed') }}
|
||||
</span>
|
||||
</template>
|
||||
<button
|
||||
class="action-btn"
|
||||
:title="t('ingestion.openInStudio')"
|
||||
|
|
@ -77,7 +79,7 @@
|
|||
</svg>
|
||||
</button>
|
||||
<button
|
||||
v-if="ingestionStore.ingestedDocs[doc.id]"
|
||||
v-if="ingestionEnabled && ingestionStore.ingestedDocs[doc.id]"
|
||||
class="action-btn unindex"
|
||||
:title="t('ingestion.deleteIndex')"
|
||||
@click="confirmRemoveFromIndex(doc.id)"
|
||||
|
|
@ -110,13 +112,16 @@
|
|||
import { computed, onMounted, ref } from 'vue'
|
||||
import { useRouter } from 'vue-router'
|
||||
import { useDocumentStore } from '../features/document/store'
|
||||
import { useFeatureFlagStore } from '../features/feature-flags/store'
|
||||
import { useIngestionStore } from '../features/ingestion/store'
|
||||
import { useI18n } from '../shared/i18n'
|
||||
import { formatSize } from '../shared/format'
|
||||
import type { Document } from '../shared/types'
|
||||
|
||||
const docStore = useDocumentStore()
|
||||
const featureStore = useFeatureFlagStore()
|
||||
const ingestionStore = useIngestionStore()
|
||||
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
|
||||
const router = useRouter()
|
||||
const { t } = useI18n()
|
||||
|
||||
|
|
@ -173,7 +178,7 @@ function confirmRemoveFromIndex(docId: string) {
|
|||
}
|
||||
|
||||
async function handleDelete(docId: string) {
|
||||
if (ingestionStore.ingestedDocs[docId]) {
|
||||
if (ingestionEnabled.value && ingestionStore.ingestedDocs[docId]) {
|
||||
await ingestionStore.deleteIngested(docId)
|
||||
}
|
||||
await docStore.remove(docId)
|
||||
|
|
@ -181,7 +186,9 @@ async function handleDelete(docId: string) {
|
|||
|
||||
onMounted(() => {
|
||||
docStore.load()
|
||||
if (ingestionEnabled.value) {
|
||||
ingestionStore.checkAvailability()
|
||||
}
|
||||
})
|
||||
</script>
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
<h1 class="page-title">{{ t('nav.search') }}</h1>
|
||||
</div>
|
||||
|
||||
<div v-if="!ingestionStore.available" class="tab-empty">
|
||||
<div v-if="!ingestionEnabled || !ingestionStore.available" class="tab-empty">
|
||||
{{ t('ingestion.unavailable') }}
|
||||
</div>
|
||||
|
||||
|
|
@ -50,13 +50,16 @@
|
|||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { onMounted, ref } from 'vue'
|
||||
import { computed, onMounted, ref } from 'vue'
|
||||
import { useSearchStore } from '../features/search/store'
|
||||
import { useFeatureFlagStore } from '../features/feature-flags/store'
|
||||
import { useIngestionStore } from '../features/ingestion/store'
|
||||
import { useI18n } from '../shared/i18n'
|
||||
|
||||
const searchStore = useSearchStore()
|
||||
const featureStore = useFeatureFlagStore()
|
||||
const ingestionStore = useIngestionStore()
|
||||
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
|
||||
const { t } = useI18n()
|
||||
|
||||
const searchInput = ref('')
|
||||
|
|
@ -70,7 +73,9 @@ function runSearch() {
|
|||
}
|
||||
|
||||
onMounted(() => {
|
||||
if (ingestionEnabled.value) {
|
||||
ingestionStore.checkAvailability()
|
||||
}
|
||||
})
|
||||
</script>
|
||||
|
||||
|
|
|
|||
|
|
@ -67,7 +67,7 @@
|
|||
{{ t('studio.prepare') }}
|
||||
</button>
|
||||
<button
|
||||
v-if="chunkingEnabled && ingestionStore.available"
|
||||
v-if="chunkingEnabled && ingestionEnabled && ingestionStore.available"
|
||||
class="toggle-btn"
|
||||
data-e2e="toggle-btn"
|
||||
:class="{ active: mode === 'ingest' }"
|
||||
|
|
@ -504,6 +504,7 @@ const analysisStore = useAnalysisStore()
|
|||
const ingestionStore = useIngestionStore()
|
||||
const { t } = useI18n()
|
||||
const chunkingEnabled = useFeatureFlag('chunking')
|
||||
const ingestionEnabled = useFeatureFlag('ingestion')
|
||||
|
||||
const mode = ref('configure')
|
||||
const currentPage = ref(1)
|
||||
|
|
@ -636,7 +637,9 @@ watch(
|
|||
onMounted(async () => {
|
||||
await documentStore.load()
|
||||
analysisStore.load()
|
||||
if (ingestionEnabled.value) {
|
||||
ingestionStore.checkAvailability()
|
||||
}
|
||||
|
||||
// Restore analysis from history via query param
|
||||
const analysisId = route.query.analysisId
|
||||
|
|
|
|||
|
|
@ -46,6 +46,7 @@
|
|||
</RouterLink>
|
||||
|
||||
<RouterLink
|
||||
v-if="ingestionEnabled"
|
||||
to="/search"
|
||||
class="nav-item"
|
||||
data-e2e="nav-search"
|
||||
|
|
@ -96,7 +97,7 @@
|
|||
|
||||
<div class="sidebar-footer">
|
||||
<div
|
||||
v-if="ingestionStore.available"
|
||||
v-if="ingestionEnabled && ingestionStore.available"
|
||||
class="opensearch-status"
|
||||
:title="
|
||||
ingestionStore.opensearchConnected
|
||||
|
|
@ -136,6 +137,7 @@ import { useIngestionStore } from '../../features/ingestion/store'
|
|||
|
||||
const featureStore = useFeatureFlagStore()
|
||||
const ingestionStore = useIngestionStore()
|
||||
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
|
||||
const version = computed(() => featureStore.appVersion)
|
||||
const route = useRoute()
|
||||
const { t } = useI18n()
|
||||
|
|
@ -145,8 +147,10 @@ defineProps({
|
|||
})
|
||||
|
||||
onMounted(() => {
|
||||
if (ingestionEnabled.value) {
|
||||
ingestionStore.checkAvailability()
|
||||
ingestionStore.startPolling(30_000)
|
||||
}
|
||||
})
|
||||
|
||||
onBeforeUnmount(() => {
|
||||
|
|
|
|||
Loading…
Reference in a new issue