feat(#180): feature-flag ingestion pipeline and add brainless one-liner Quick Start

- Conditionally mount ingestion router only when OpenSearch + embedding are configured
- Add `ingestionAvailable` field to /api/health response
- Add `ingestion` feature flag to frontend (hides Search nav, Ingest button,
  OpenSearch badge, indexed badges/filters when disabled)
- Skip ingestion polling when flag is off
- Make OpenSearch + embedding optional in docker-compose via profiles
- Add docker-compose.ingestion.yml override for full-stack ingestion
- Set BATCH_PAGE_SIZE=5 default in Docker local image
- Lead Quick Start with one-liner docker run command
- Document ingestion as opt-in with dedicated section
- Add BATCH_PAGE_SIZE, MAX_FILE_SIZE_MB, MAX_PAGE_COUNT, RATE_LIMIT_RPM to config tables
- Update test counts (380 backend, 159 frontend)
- Date CHANGELOG 0.4.0, bump frontend version to 0.4.0
- Sync CONTRIBUTING.md with E2E Karate test sections

Closes #180
This commit is contained in:
Pier-Jean Malandrino 2026-04-13 11:18:56 +02:00
parent 6f008ec262
commit ba54427445
21 changed files with 310 additions and 83 deletions

View file

@ -4,7 +4,7 @@ All notable changes to Docling Studio will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/). The format is based on [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/).
## [0.4.0] - Unreleased ## [0.4.0] - 2026-04-13
### Added ### Added

View file

@ -96,15 +96,43 @@ npx prettier --write src/ # auto-format
## Running Tests ## Running Tests
```bash ```bash
# Backend (199 tests) # Backend (377 tests)
cd document-parser cd document-parser
pytest tests/ -v pytest tests/ -v
# Frontend (129 tests) # Frontend (156 tests)
cd frontend cd frontend
npm run test:run npm run test:run
``` ```
### E2E API (Karate)
```bash
# Generate test PDFs + start stack
python e2e/generate-test-data.py
docker compose up -d --wait
# Run all API tests
mvn test -f e2e/api/pom.xml
# Or by tag: @smoke, @regression, @e2e
mvn test -f e2e/api/pom.xml -Dkarate.options="--tags @smoke"
```
### E2E UI (Karate UI)
```bash
# Generate test PDFs + start stack (if not already running)
python e2e/generate-test-data.py
docker compose up -d --wait
# Run critical UI tests (CI scope)
mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @critical"
# Run all UI tests (local scope)
mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @ui"
```
All tests must pass before submitting a PR. All tests must pass before submitting a PR.
## Submitting Changes ## Submitting Changes

View file

@ -76,3 +76,4 @@ RUN pip install --no-cache-dir torch torchvision --index-url https://download.py
RUN chown -R appuser:appuser /app \ RUN chown -R appuser:appuser /app \
&& chown -R appuser:appuser /usr/local/lib/python3.12/site-packages/rapidocr/models && chown -R appuser:appuser /usr/local/lib/python3.12/site-packages/rapidocr/models
ENV CONVERSION_ENGINE=local ENV CONVERSION_ENGINE=local
ENV BATCH_PAGE_SIZE=5

View file

@ -31,9 +31,13 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t
- **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description, image generation - **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description, image generation
- **Bounding box visualization** — color-coded element overlay directly on the PDF - **Bounding box visualization** — color-coded element overlay directly on the PDF
- **Per-page results** — right panel syncs with the current PDF page - **Per-page results** — right panel syncs with the current PDF page
- **Chunking** — split extracted content into semantic chunks (hierarchical, hybrid, or page-based) with configurable token limits and inline editing
- **Ingestion pipeline** — Docling → chunking → embedding → OpenSearch vector indexing (one-click from Studio)
- **Markdown & HTML export** of extracted content - **Markdown & HTML export** of extracted content
- **Document management** — upload, list, delete - **Document management** — upload, list, delete, search, filter by indexing status
- **Analysis history** — re-visit and open past analyses - **Analysis history** — re-visit and open past analyses
- **Upload limits** — configurable max file size and max page count per document
- **Rate limiting** — configurable requests per minute per IP
- **Dark / Light theme** and **FR / EN** localization - **Dark / Light theme** and **FR / EN** localization
@ -74,7 +78,7 @@ document-parser/
├── services/ # Use case orchestration ├── services/ # Use case orchestration
│ ├── document_service.py # Upload, delete, preview │ ├── document_service.py # Upload, delete, preview
│ └── analysis_service.py # Async Docling processing │ └── analysis_service.py # Async Docling processing
└── tests/ # 199 tests (pytest) └── tests/ # 377 tests (pytest)
``` ```
### Frontend structure (feature-based) ### Frontend structure (feature-based)
@ -98,14 +102,24 @@ frontend/src/
## Quick Start ## Quick Start
Docling Studio ships two Docker image variants: One command, nothing else to install:
```bash
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
```
Open [http://localhost:3000](http://localhost:3000), upload a PDF, and get results. That's it.
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
### Image variants
| Variant | Image tag | Size | Description | | Variant | Image tag | Size | Description |
|---------|-----------|------|-------------| |---------|-----------|------|-------------|
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only |
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance | | **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only (downloads ML models on first run) |
### Docker — remote mode (fastest) For remote mode:
```bash ```bash
docker run -p 3000:3000 \ docker run -p 3000:3000 \
@ -113,27 +127,17 @@ docker run -p 3000:3000 \
ghcr.io/scub-france/docling-studio:latest-remote ghcr.io/scub-france/docling-studio:latest-remote
``` ```
### Docker — local mode (self-contained) ### Docker Compose
```bash
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
```
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
Open [http://localhost:3000](http://localhost:3000)
### Docker Compose (for development)
```bash ```bash
git clone https://github.com/scub-france/Docling-Studio.git git clone https://github.com/scub-france/Docling-Studio.git
cd Docling-Studio cd Docling-Studio
# Local mode (default) # Simple mode (backend + frontend only)
docker compose up --build docker compose up --build
# Remote mode # With ingestion pipeline (OpenSearch + embeddings)
CONVERSION_MODE=remote DOCLING_SERVE_URL=http://your-docling-serve:5001 docker compose up --build docker compose --profile ingestion -f docker-compose.yml -f docker-compose.ingestion.yml up --build
``` ```
### Local Development ### Local Development
@ -162,12 +166,12 @@ npm run dev
### Running Tests ### Running Tests
```bash ```bash
# Backend (199 tests) # Backend (377 tests)
cd document-parser cd document-parser
pip install pytest pytest-asyncio httpx pip install pytest pytest-asyncio httpx
pytest tests/ -v pytest tests/ -v
# Frontend (129 tests) # Frontend (156 tests)
cd frontend cd frontend
npm run test:run npm run test:run
``` ```
@ -202,6 +206,43 @@ All configuration is done via environment variables. See [`.env.example`](.env.e
| `UPLOAD_DIR` | `./uploads` | File storage directory | | `UPLOAD_DIR` | `./uploads` | File storage directory |
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path | | `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
| `CONVERSION_TIMEOUT` | `600` | Max seconds for a single Docling conversion | | `CONVERSION_TIMEOUT` | `600` | Max seconds for a single Docling conversion |
| `BATCH_PAGE_SIZE` | `5` (Docker) / `0` | Pages per batch (`0` = process all at once) |
| `MAX_FILE_SIZE_MB` | `50` | Maximum upload file size in MB (`0` = unlimited) |
| `MAX_PAGE_COUNT` | `0` | Maximum number of pages per document (`0` = unlimited) |
| `RATE_LIMIT_RPM` | `100` | Max requests per minute per IP (`0` = disabled) |
## Upload Limits
Docling Studio enforces configurable limits on uploaded documents to protect the server against oversized files and long-running analyses:
- **`MAX_FILE_SIZE_MB`** (default `50`) — rejects uploads exceeding this size. Validated at two levels: early `Content-Length` check and streaming byte count.
- **`MAX_PAGE_COUNT`** (default `0` = unlimited) — rejects documents with more pages than allowed. Useful on shared instances or Hugging Face Spaces to cap processing time.
Both limits are exposed in the `/api/health` endpoint so the frontend can display them to the user before upload. Set either to `0` to disable the corresponding check.
## Ingestion Pipeline (opt-in)
Docling Studio can optionally index extracted chunks into [OpenSearch](https://opensearch.org/) for vector and full-text search. This requires two additional services (OpenSearch + embedding) and is **disabled by default**.
To enable ingestion with Docker Compose:
```bash
docker compose --profile ingestion \
-f docker-compose.yml -f docker-compose.ingestion.yml \
up --build
```
When ingestion is enabled, the UI shows:
- An **Ingest** button in Studio to push chunks to OpenSearch
- An **OpenSearch** connection status badge in the sidebar
- **Indexed / Not indexed** filters on the Documents page
- A **Search** page for full-text and vector search across indexed documents
| Variable | Default | Description |
|----------|---------|-------------|
| `OPENSEARCH_URL` | — | OpenSearch endpoint (empty = ingestion disabled) |
| `EMBEDDING_URL` | — | Embedding service endpoint (empty = ingestion disabled) |
| `EMBEDDING_DIMENSION` | `384` | Vector dimension (must match embedding model) |
## CI / Release ## CI / Release

View file

@ -0,0 +1,19 @@
# Override to enable the ingestion pipeline (OpenSearch + embeddings).
#
# Usage:
# docker compose --profile ingestion -f docker-compose.yml -f docker-compose.ingestion.yml up --build
#
# This wires the backend to the OpenSearch and embedding services started
# by the "ingestion" profile and ensures they are healthy before the
# backend starts.
services:
document-parser:
environment:
OPENSEARCH_URL: http://opensearch:9200
EMBEDDING_URL: http://embedding:8001
depends_on:
opensearch:
condition: service_healthy
embedding:
condition: service_healthy

View file

@ -1,6 +1,7 @@
services: services:
# --- OpenSearch (single-node, security disabled) --- # --- OpenSearch (single-node, security disabled) ---
opensearch: opensearch:
profiles: ["ingestion"]
image: opensearchproject/opensearch:2 image: opensearchproject/opensearch:2
environment: environment:
discovery.type: single-node discovery.type: single-node
@ -16,6 +17,7 @@ services:
# --- Embedding service (sentence-transformers) --- # --- Embedding service (sentence-transformers) ---
embedding: embedding:
profiles: ["ingestion"]
build: build:
context: ./embedding-service context: ./embedding-service
environment: environment:
@ -49,13 +51,8 @@ services:
RATE_LIMIT_RPM: ${RATE_LIMIT_RPM:-100} RATE_LIMIT_RPM: ${RATE_LIMIT_RPM:-100}
MAX_FILE_SIZE_MB: ${MAX_FILE_SIZE_MB:-50} MAX_FILE_SIZE_MB: ${MAX_FILE_SIZE_MB:-50}
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-0} BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-0}
OPENSEARCH_URL: http://opensearch:9200 OPENSEARCH_URL: ${OPENSEARCH_URL:-}
EMBEDDING_URL: http://embedding:8001 EMBEDDING_URL: ${EMBEDDING_URL:-}
depends_on:
opensearch:
condition: service_healthy
embedding:
condition: service_healthy
deploy: deploy:
resources: resources:
limits: limits:

View file

@ -1,15 +1,28 @@
# Getting Started # Getting Started
Docling Studio ships two Docker image variants: ## Quick Start
| Variant | Image tag | Size | Description | One command, nothing else to install:
|---------|-----------|------|-------------|
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance | ```bash
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only (downloads ML models on first run) | docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
```
Open [http://localhost:3000](http://localhost:3000), upload a PDF, and get results. That's it.
!!! note
The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
![Docker architecture](images/docker.png){ width="600" } ![Docker architecture](images/docker.png){ width="600" }
## Docker — remote mode (fastest) ## Image Variants
| Variant | Image tag | Size | Description |
|---------|-----------|------|-------------|
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only |
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
For remote mode:
```bash ```bash
docker run -p 3000:3000 \ docker run -p 3000:3000 \
@ -17,27 +30,19 @@ docker run -p 3000:3000 \
ghcr.io/scub-france/docling-studio:latest-remote ghcr.io/scub-france/docling-studio:latest-remote
``` ```
## Docker — local mode (self-contained) ## Docker Compose
```bash
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
```
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
Open [http://localhost:3000](http://localhost:3000).
## Docker Compose (recommended for development)
```bash ```bash
git clone https://github.com/scub-france/Docling-Studio.git git clone https://github.com/scub-france/Docling-Studio.git
cd Docling-Studio cd Docling-Studio
# Local mode (default) # Simple mode (backend + frontend only)
docker compose up --build docker compose up --build
# Remote mode # With ingestion pipeline (OpenSearch + embeddings)
CONVERSION_MODE=remote DOCLING_SERVE_URL=http://your-docling-serve:5001 docker compose up --build docker compose --profile ingestion \
-f docker-compose.yml -f docker-compose.ingestion.yml \
up --build
``` ```
## Local Development ## Local Development
@ -136,10 +141,48 @@ All configuration is done via environment variables:
| `UPLOAD_DIR` | `./uploads` | File storage directory | | `UPLOAD_DIR` | `./uploads` | File storage directory |
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path | | `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
| `CONVERSION_TIMEOUT` | `600` | Max seconds per Docling conversion | | `CONVERSION_TIMEOUT` | `600` | Max seconds per Docling conversion |
| `BATCH_PAGE_SIZE` | `5` (Docker) / `0` | Pages per batch (`0` = process all at once) |
| `MAX_CONCURRENT_ANALYSES` | `3` | Maximum parallel analysis jobs | | `MAX_CONCURRENT_ANALYSES` | `3` | Maximum parallel analysis jobs |
| `DEPLOYMENT_MODE` | `self-hosted` | `self-hosted` or `huggingface` (shows disclaimer banner) | | `DEPLOYMENT_MODE` | `self-hosted` | `self-hosted` or `huggingface` (shows disclaimer banner) |
| `MAX_FILE_SIZE_MB` | `50` | Maximum upload file size in MB (`0` = unlimited) |
| `MAX_PAGE_COUNT` | `0` | Maximum number of pages per document (`0` = unlimited) |
| `RATE_LIMIT_RPM` | `100` | Max requests per minute per IP (`0` = disabled) |
| `APP_VERSION` | `dev` | Application version (set automatically by CI/Docker) | | `APP_VERSION` | `dev` | Application version (set automatically by CI/Docker) |
## Upload Limits
Docling Studio enforces configurable limits on uploaded documents to protect the server against oversized files and long-running analyses:
- **`MAX_FILE_SIZE_MB`** (default `50`) — rejects uploads exceeding this size. Validated at two levels: early `Content-Length` check and streaming byte count.
- **`MAX_PAGE_COUNT`** (default `0` = unlimited) — rejects documents with more pages than allowed. Useful on shared instances or Hugging Face Spaces to cap processing time.
Both limits are exposed in the `/api/health` endpoint so the frontend can display them to the user before upload. Set either to `0` to disable the corresponding check.
## Ingestion Pipeline (opt-in)
Docling Studio can optionally index extracted chunks into [OpenSearch](https://opensearch.org/) for vector and full-text search. This requires two additional services (OpenSearch + embedding) and is **disabled by default**.
To enable ingestion with Docker Compose:
```bash
docker compose --profile ingestion \
-f docker-compose.yml -f docker-compose.ingestion.yml \
up --build
```
When ingestion is enabled, the UI shows:
- An **Ingest** button in Studio to push chunks to OpenSearch
- An **OpenSearch** connection status badge in the sidebar
- **Indexed / Not indexed** filters on the Documents page
- A **Search** page for full-text and vector search across indexed documents
| Variable | Default | Description |
|----------|---------|-------------|
| `OPENSEARCH_URL` | — | OpenSearch endpoint (empty = ingestion disabled) |
| `EMBEDDING_URL` | — | Embedding service endpoint (empty = ingestion disabled) |
| `EMBEDDING_DIMENSION` | `384` | Vector dimension (must match embedding model) |
## System Requirements ## System Requirements
| | Remote image | Local image | | | Remote image | Local image |

View file

@ -18,7 +18,8 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t
- **Document management** — upload, list, delete - **Document management** — upload, list, delete
- **Analysis history** — re-visit and open past analyses - **Analysis history** — re-visit and open past analyses
- **Feature flags** — capabilities adapt to the conversion engine (local vs remote) - **Feature flags** — capabilities adapt to the conversion engine (local vs remote)
- **Rate limiting** — 60 requests per minute per IP to protect the backend - **Upload limits** — configurable max file size (`MAX_FILE_SIZE_MB`) and max page count (`MAX_PAGE_COUNT`) per document
- **Rate limiting** — configurable requests per minute per IP (`RATE_LIMIT_RPM`)
- **Deployment modes** — self-hosted (default) or HuggingFace Spaces (with disclaimer banner) - **Deployment modes** — self-hosted (default) or HuggingFace Spaces (with disclaimer banner)
- **Health endpoint**`/api/health` reports engine type, deployment mode, and database status - **Health endpoint**`/api/health` reports engine type, deployment mode, and database status
- **Dark / Light theme** and **FR / EN** localization - **Dark / Light theme** and **FR / EN** localization

View file

@ -34,6 +34,7 @@ class HealthResponse(_CamelModel):
database: str database: str
max_page_count: int | None = None max_page_count: int | None = None
max_file_size_mb: int | None = None max_file_size_mb: int | None = None
ingestion_available: bool = False
class DocumentResponse(_CamelModel): class DocumentResponse(_CamelModel):

View file

@ -138,7 +138,11 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
document_repo, analysis_repo = _build_repos() document_repo, analysis_repo = _build_repos()
app.state.analysis_service = _build_analysis_service(document_repo, analysis_repo) app.state.analysis_service = _build_analysis_service(document_repo, analysis_repo)
app.state.document_service = _build_document_service(document_repo, analysis_repo) app.state.document_service = _build_document_service(document_repo, analysis_repo)
app.state.ingestion_service = _build_ingestion_service() ingestion_service = _build_ingestion_service()
app.state.ingestion_service = ingestion_service
if ingestion_service is not None:
app.include_router(ingestion_router)
logger.info("Ingestion router mounted")
logger.info("Docling Studio backend ready (engine=%s)", settings.conversion_engine) logger.info("Docling Studio backend ready (engine=%s)", settings.conversion_engine)
yield yield
@ -165,7 +169,6 @@ if settings.rate_limit_rpm > 0:
app.include_router(documents_router) app.include_router(documents_router)
app.include_router(analyses_router) app.include_router(analyses_router)
app.include_router(ingestion_router)
@app.get("/api/health", response_model=HealthResponse) @app.get("/api/health", response_model=HealthResponse)
@ -188,4 +191,5 @@ async def health() -> HealthResponse:
database=db_status, database=db_status,
max_page_count=settings.max_page_count if settings.max_page_count > 0 else None, max_page_count=settings.max_page_count if settings.max_page_count > 0 else None,
max_file_size_mb=settings.max_file_size_mb if settings.max_file_size_mb > 0 else None, max_file_size_mb=settings.max_file_size_mb if settings.max_file_size_mb > 0 else None,
ingestion_available=getattr(app.state, "ingestion_service", None) is not None,
) )

View file

@ -51,6 +51,23 @@ class TestHealthEndpoint:
assert "maxFileSizeMb" in data assert "maxFileSizeMb" in data
assert data["maxFileSizeMb"] == 50 assert data["maxFileSizeMb"] == 50
def test_health_exposes_ingestion_available_false(self, client):
original = getattr(app.state, "ingestion_service", None)
app.state.ingestion_service = None
resp = client.get("/api/health")
app.state.ingestion_service = original
data = resp.json()
assert "ingestionAvailable" in data
assert data["ingestionAvailable"] is False
def test_health_exposes_ingestion_available_true(self, client):
original = getattr(app.state, "ingestion_service", None)
app.state.ingestion_service = MagicMock()
resp = client.get("/api/health")
app.state.ingestion_service = original
data = resp.json()
assert data["ingestionAvailable"] is True
class TestDocumentEndpoints: class TestDocumentEndpoints:
def test_list_documents(self, client, mock_document_service): def test_list_documents(self, client, mock_document_service):

View file

@ -104,7 +104,18 @@ class TestIngestionStatus:
class TestIngestionDisabled: class TestIngestionDisabled:
def test_returns_503_when_disabled(self) -> None: def test_router_not_mounted_returns_404(self) -> None:
"""When ingestion service is None, router should not be mounted → 404."""
app = FastAPI()
# Do NOT include ingestion router — simulates main.py conditional mount
app.state.ingestion_service = None
app.state.analysis_service = AsyncMock()
tc = TestClient(app)
resp = tc.post("/api/ingestion/job-1")
assert resp.status_code == 404
def test_status_still_returns_503_when_router_mounted_but_service_none(self) -> None:
"""If router is mounted but service is None, endpoints return 503."""
app = FastAPI() app = FastAPI()
app.include_router(router) app.include_router(router)
app.state.ingestion_service = None app.state.ingestion_service = None

View file

@ -1,12 +1,12 @@
{ {
"name": "docling-studio", "name": "docling-studio",
"version": "0.3.0", "version": "0.3.1",
"lockfileVersion": 3, "lockfileVersion": 3,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "docling-studio", "name": "docling-studio",
"version": "0.3.0", "version": "0.3.1",
"dependencies": { "dependencies": {
"dompurify": "^3.3.3", "dompurify": "^3.3.3",
"marked": "^17.0.4", "marked": "^17.0.4",

View file

@ -1,6 +1,6 @@
{ {
"name": "docling-studio", "name": "docling-studio",
"version": "0.3.1", "version": "0.4.0",
"private": true, "private": true,
"type": "module", "type": "module",
"scripts": { "scripts": {

View file

@ -202,7 +202,10 @@
</div> </div>
</div> </div>
<div class="chunk-empty" v-else-if="!chunkingStore.rechunking"> <div
class="chunk-empty"
v-if="!pageChunks.length && !chunkingStore.rechunking && deleteConfirmIdx === -1"
>
<p> <p>
{{ chunks.length ? t('chunking.noChunksOnPage') : t('chunking.noChunks') }} {{ chunks.length ? t('chunking.noChunksOnPage') : t('chunking.noChunks') }}
</p> </p>

View file

@ -82,6 +82,38 @@ describe('useFeatureFlagStore', () => {
expect(store.maxFileSizeMb).toBe(0) expect(store.maxFileSizeMb).toBe(0)
}) })
it('enables ingestion when ingestionAvailable is true', async () => {
mockApiFetch.mockResolvedValue({
status: 'ok',
engine: 'local',
ingestionAvailable: true,
})
const store = useFeatureFlagStore()
await store.load()
expect(store.ingestionAvailable).toBe(true)
expect(store.isEnabled('ingestion')).toBe(true)
})
it('disables ingestion when ingestionAvailable is false', async () => {
mockApiFetch.mockResolvedValue({
status: 'ok',
engine: 'local',
ingestionAvailable: false,
})
const store = useFeatureFlagStore()
await store.load()
expect(store.ingestionAvailable).toBe(false)
expect(store.isEnabled('ingestion')).toBe(false)
})
it('defaults ingestionAvailable to false when missing', async () => {
mockApiFetch.mockResolvedValue({ status: 'ok', engine: 'local' })
const store = useFeatureFlagStore()
await store.load()
expect(store.ingestionAvailable).toBe(false)
expect(store.isEnabled('ingestion')).toBe(false)
})
it('handles health endpoint failure gracefully', async () => { it('handles health endpoint failure gracefully', async () => {
mockApiFetch.mockRejectedValue(new Error('Network error')) mockApiFetch.mockRejectedValue(new Error('Network error'))
const store = useFeatureFlagStore() const store = useFeatureFlagStore()

View file

@ -13,9 +13,10 @@ interface HealthResponse {
deploymentMode?: DeploymentMode deploymentMode?: DeploymentMode
maxPageCount?: number maxPageCount?: number
maxFileSizeMb?: number maxFileSizeMb?: number
ingestionAvailable?: boolean
} }
export type FeatureFlag = 'chunking' | 'disclaimer' export type FeatureFlag = 'chunking' | 'disclaimer' | 'ingestion'
interface FeatureFlagDef { interface FeatureFlagDef {
description: string description: string
@ -25,6 +26,7 @@ interface FeatureFlagDef {
interface FeatureFlagContext { interface FeatureFlagContext {
engine: ConversionEngine | null engine: ConversionEngine | null
deploymentMode: DeploymentMode | null deploymentMode: DeploymentMode | null
ingestionAvailable: boolean
} }
const featureRegistry: Record<FeatureFlag, FeatureFlagDef> = { const featureRegistry: Record<FeatureFlag, FeatureFlagDef> = {
@ -36,6 +38,10 @@ const featureRegistry: Record<FeatureFlag, FeatureFlagDef> = {
description: 'Show shared-instance disclaimer banner', description: 'Show shared-instance disclaimer banner',
isEnabled: (ctx) => ctx.deploymentMode === 'huggingface', isEnabled: (ctx) => ctx.deploymentMode === 'huggingface',
}, },
ingestion: {
description: 'OpenSearch ingestion pipeline (embedding + vector indexing)',
isEnabled: (ctx) => ctx.ingestionAvailable,
},
} }
export const useFeatureFlagStore = defineStore('feature-flags', () => { export const useFeatureFlagStore = defineStore('feature-flags', () => {
@ -43,6 +49,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
const deploymentMode = ref<DeploymentMode | null>(null) const deploymentMode = ref<DeploymentMode | null>(null)
const maxPageCount = ref<number>(0) const maxPageCount = ref<number>(0)
const maxFileSizeMb = ref<number>(0) const maxFileSizeMb = ref<number>(0)
const ingestionAvailable = ref(false)
const appVersion = ref<string>(__APP_VERSION__) const appVersion = ref<string>(__APP_VERSION__)
const loaded = ref(false) const loaded = ref(false)
const error = ref<string | null>(null) const error = ref<string | null>(null)
@ -50,6 +57,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
const context = computed<FeatureFlagContext>(() => ({ const context = computed<FeatureFlagContext>(() => ({
engine: engine.value, engine: engine.value,
deploymentMode: deploymentMode.value, deploymentMode: deploymentMode.value,
ingestionAvailable: ingestionAvailable.value,
})) }))
function isEnabled(flag: FeatureFlag): boolean { function isEnabled(flag: FeatureFlag): boolean {
@ -65,6 +73,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
deploymentMode.value = data.deploymentMode ?? 'self-hosted' deploymentMode.value = data.deploymentMode ?? 'self-hosted'
maxPageCount.value = data.maxPageCount ?? 0 maxPageCount.value = data.maxPageCount ?? 0
maxFileSizeMb.value = data.maxFileSizeMb ?? 0 maxFileSizeMb.value = data.maxFileSizeMb ?? 0
ingestionAvailable.value = data.ingestionAvailable ?? false
appMaxFileSizeMb.value = maxFileSizeMb.value appMaxFileSizeMb.value = maxFileSizeMb.value
appMaxPageCount.value = maxPageCount.value appMaxPageCount.value = maxPageCount.value
if (data.version) appVersion.value = data.version if (data.version) appVersion.value = data.version
@ -81,6 +90,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
deploymentMode, deploymentMode,
maxPageCount, maxPageCount,
maxFileSizeMb, maxFileSizeMb,
ingestionAvailable,
appVersion, appVersion,
loaded, loaded,
error, error,

View file

@ -9,7 +9,7 @@
class="search-input" class="search-input"
:placeholder="t('ingestion.search')" :placeholder="t('ingestion.search')"
/> />
<div class="filter-group"> <div v-if="ingestionEnabled" class="filter-group">
<button <button
v-for="f in filters" v-for="f in filters"
:key="f.value" :key="f.value"
@ -54,17 +54,19 @@
</div> </div>
</div> </div>
<div class="doc-row-actions"> <div class="doc-row-actions">
<span <template v-if="ingestionEnabled">
v-if="ingestionStore.ingestedDocs[doc.id]" <span
class="status-badge indexed" v-if="ingestionStore.ingestedDocs[doc.id]"
:title="t('ingestion.chunksIndexed', { n: ingestionStore.ingestedDocs[doc.id] })" class="status-badge indexed"
> :title="t('ingestion.chunksIndexed', { n: ingestionStore.ingestedDocs[doc.id] })"
{{ t('ingestion.indexed') }} >
<span class="badge-count">{{ ingestionStore.ingestedDocs[doc.id] }}</span> {{ t('ingestion.indexed') }}
</span> <span class="badge-count">{{ ingestionStore.ingestedDocs[doc.id] }}</span>
<span v-else class="status-badge not-indexed"> </span>
{{ t('ingestion.notIndexed') }} <span v-else class="status-badge not-indexed">
</span> {{ t('ingestion.notIndexed') }}
</span>
</template>
<button <button
class="action-btn" class="action-btn"
:title="t('ingestion.openInStudio')" :title="t('ingestion.openInStudio')"
@ -77,7 +79,7 @@
</svg> </svg>
</button> </button>
<button <button
v-if="ingestionStore.ingestedDocs[doc.id]" v-if="ingestionEnabled && ingestionStore.ingestedDocs[doc.id]"
class="action-btn unindex" class="action-btn unindex"
:title="t('ingestion.deleteIndex')" :title="t('ingestion.deleteIndex')"
@click="confirmRemoveFromIndex(doc.id)" @click="confirmRemoveFromIndex(doc.id)"
@ -110,13 +112,16 @@
import { computed, onMounted, ref } from 'vue' import { computed, onMounted, ref } from 'vue'
import { useRouter } from 'vue-router' import { useRouter } from 'vue-router'
import { useDocumentStore } from '../features/document/store' import { useDocumentStore } from '../features/document/store'
import { useFeatureFlagStore } from '../features/feature-flags/store'
import { useIngestionStore } from '../features/ingestion/store' import { useIngestionStore } from '../features/ingestion/store'
import { useI18n } from '../shared/i18n' import { useI18n } from '../shared/i18n'
import { formatSize } from '../shared/format' import { formatSize } from '../shared/format'
import type { Document } from '../shared/types' import type { Document } from '../shared/types'
const docStore = useDocumentStore() const docStore = useDocumentStore()
const featureStore = useFeatureFlagStore()
const ingestionStore = useIngestionStore() const ingestionStore = useIngestionStore()
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
const router = useRouter() const router = useRouter()
const { t } = useI18n() const { t } = useI18n()
@ -173,7 +178,7 @@ function confirmRemoveFromIndex(docId: string) {
} }
async function handleDelete(docId: string) { async function handleDelete(docId: string) {
if (ingestionStore.ingestedDocs[docId]) { if (ingestionEnabled.value && ingestionStore.ingestedDocs[docId]) {
await ingestionStore.deleteIngested(docId) await ingestionStore.deleteIngested(docId)
} }
await docStore.remove(docId) await docStore.remove(docId)
@ -181,7 +186,9 @@ async function handleDelete(docId: string) {
onMounted(() => { onMounted(() => {
docStore.load() docStore.load()
ingestionStore.checkAvailability() if (ingestionEnabled.value) {
ingestionStore.checkAvailability()
}
}) })
</script> </script>

View file

@ -4,7 +4,7 @@
<h1 class="page-title">{{ t('nav.search') }}</h1> <h1 class="page-title">{{ t('nav.search') }}</h1>
</div> </div>
<div v-if="!ingestionStore.available" class="tab-empty"> <div v-if="!ingestionEnabled || !ingestionStore.available" class="tab-empty">
{{ t('ingestion.unavailable') }} {{ t('ingestion.unavailable') }}
</div> </div>
@ -50,13 +50,16 @@
</template> </template>
<script setup lang="ts"> <script setup lang="ts">
import { onMounted, ref } from 'vue' import { computed, onMounted, ref } from 'vue'
import { useSearchStore } from '../features/search/store' import { useSearchStore } from '../features/search/store'
import { useFeatureFlagStore } from '../features/feature-flags/store'
import { useIngestionStore } from '../features/ingestion/store' import { useIngestionStore } from '../features/ingestion/store'
import { useI18n } from '../shared/i18n' import { useI18n } from '../shared/i18n'
const searchStore = useSearchStore() const searchStore = useSearchStore()
const featureStore = useFeatureFlagStore()
const ingestionStore = useIngestionStore() const ingestionStore = useIngestionStore()
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
const { t } = useI18n() const { t } = useI18n()
const searchInput = ref('') const searchInput = ref('')
@ -70,7 +73,9 @@ function runSearch() {
} }
onMounted(() => { onMounted(() => {
ingestionStore.checkAvailability() if (ingestionEnabled.value) {
ingestionStore.checkAvailability()
}
}) })
</script> </script>

View file

@ -67,7 +67,7 @@
{{ t('studio.prepare') }} {{ t('studio.prepare') }}
</button> </button>
<button <button
v-if="chunkingEnabled && ingestionStore.available" v-if="chunkingEnabled && ingestionEnabled && ingestionStore.available"
class="toggle-btn" class="toggle-btn"
data-e2e="toggle-btn" data-e2e="toggle-btn"
:class="{ active: mode === 'ingest' }" :class="{ active: mode === 'ingest' }"
@ -504,6 +504,7 @@ const analysisStore = useAnalysisStore()
const ingestionStore = useIngestionStore() const ingestionStore = useIngestionStore()
const { t } = useI18n() const { t } = useI18n()
const chunkingEnabled = useFeatureFlag('chunking') const chunkingEnabled = useFeatureFlag('chunking')
const ingestionEnabled = useFeatureFlag('ingestion')
const mode = ref('configure') const mode = ref('configure')
const currentPage = ref(1) const currentPage = ref(1)
@ -636,7 +637,9 @@ watch(
onMounted(async () => { onMounted(async () => {
await documentStore.load() await documentStore.load()
analysisStore.load() analysisStore.load()
ingestionStore.checkAvailability() if (ingestionEnabled.value) {
ingestionStore.checkAvailability()
}
// Restore analysis from history via query param // Restore analysis from history via query param
const analysisId = route.query.analysisId const analysisId = route.query.analysisId

View file

@ -46,6 +46,7 @@
</RouterLink> </RouterLink>
<RouterLink <RouterLink
v-if="ingestionEnabled"
to="/search" to="/search"
class="nav-item" class="nav-item"
data-e2e="nav-search" data-e2e="nav-search"
@ -96,7 +97,7 @@
<div class="sidebar-footer"> <div class="sidebar-footer">
<div <div
v-if="ingestionStore.available" v-if="ingestionEnabled && ingestionStore.available"
class="opensearch-status" class="opensearch-status"
:title=" :title="
ingestionStore.opensearchConnected ingestionStore.opensearchConnected
@ -136,6 +137,7 @@ import { useIngestionStore } from '../../features/ingestion/store'
const featureStore = useFeatureFlagStore() const featureStore = useFeatureFlagStore()
const ingestionStore = useIngestionStore() const ingestionStore = useIngestionStore()
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
const version = computed(() => featureStore.appVersion) const version = computed(() => featureStore.appVersion)
const route = useRoute() const route = useRoute()
const { t } = useI18n() const { t } = useI18n()
@ -145,8 +147,10 @@ defineProps({
}) })
onMounted(() => { onMounted(() => {
ingestionStore.checkAvailability() if (ingestionEnabled.value) {
ingestionStore.startPolling(30_000) ingestionStore.checkAvailability()
ingestionStore.startPolling(30_000)
}
}) })
onBeforeUnmount(() => { onBeforeUnmount(() => {