danvics/docling-studio
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Merge pull request #163 from scub-france/release/0.4.0

Browse source 
Release/0.4.0
This commit is contained in:
Pier-Jean Malandrino 2026-04-14 17:22:13 +02:00 committed by GitHub
commit e7c27a6706
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
70 changed files with 5023 additions and 443 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
.env.example
View file

@ -36,3 +36,15 @@
# Database path (inside container)
# DB_PATH=./data/docling_studio.db
# OpenSearch URL (used by docker-compose.dev.yml, auto-set to service name)
# OPENSEARCH_URL=http://opensearch:9200
# Embedding service URL (used by docker-compose.dev.yml, auto-set to service name)
# EMBEDDING_URL=http://embedding:8001
# Embedding model (default: all-MiniLM-L6-v2, used by the embedding service)
# EMBEDDING_MODEL=all-MiniLM-L6-v2
# Embedding vector dimension (default: 384 for Granite Embedding 30M / all-MiniLM-L6-v2)
# EMBEDDING_DIMENSION=384

3
.gitignore vendored
View file

@ -44,5 +44,8 @@ hs_err_pid*
# Docker
docker-compose.override.yml
# Audit profiles (internal tooling)
profiles/
# E2E tests — Maven build outputs & Chrome user data
e2e/**/target/

23
CHANGELOG.md
View file

@ -4,6 +4,29 @@ All notable changes to Docling Studio will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/).
## [0.4.0] - 2026-04-13
### Added
- Inline chunk text editing: double-click or edit button to modify chunk text, with save/cancel and "modified" badge
- Docker Compose dev stack (`docker-compose.dev.yml`) with OpenSearch, Dashboards, hot-reload backend and Vite frontend
- Soft-delete chunks: delete button with confirmation dialog, chunks hidden from UI but preserved in data
- Vector index metadata schema: `IndexedChunk` domain model, OpenSearch mapping builder, configurable embedding dimension
- `VectorStore` port (Protocol): `ensure_index`, `index_chunks`, `search_similar`, `get_chunks`, `delete_document`
- OpenSearch adapter (`OpenSearchStore`): kNN vector search, full-text search, bulk indexing, document CRUD
- Embedding microservice (`embedding-service/`): sentence-transformers REST API with batch processing and Dockerfile
- `EmbeddingService` port and `EmbeddingClient` HTTP adapter for remote embedding generation
- Orchestrated ingestion pipeline: Docling → chunking → embedding → OpenSearch indexing (idempotent)
- Ingestion REST API: `POST /api/ingestion/{jobId}`, `DELETE /api/ingestion/{docId}`, `GET /api/ingestion/status`
- Production docker-compose with OpenSearch and embedding service
- E2E Karate test for full ingestion workflow (PDF → chunks in OpenSearch)
- My Documents screen: search, filter (all/indexed/not indexed), sort (name/date), ingestion status badges
- Ingest button in Studio: one-click ingestion from completed analysis with progress feedback
### Fixed
### Changed
## [0.3.1] - 2026-04-09
### Added

61
CONTRIBUTING.md
View file

@ -17,6 +17,35 @@ Thank you for your interest in contributing to Docling Studio! This guide will h
## Development Setup
### Docker Dev Stack (recommended)
The fastest way to get the full stack running (backend + frontend + OpenSearch):
```bash
docker compose -f docker-compose.dev.yml up
```
This starts:
| Service | URL | Notes |
|---------|-----|-------|
| Frontend (Vite) | http://localhost:3000 | HMR enabled |
| Backend (FastAPI) | http://localhost:8000 | Auto-reload on file changes |
| OpenSearch | http://localhost:9200 | Single-node, security disabled |
| OpenSearch Dashboards | http://localhost:5601 | Index inspection UI |
Source code is bind-mounted — edits on your host are reflected immediately.
To use remote conversion mode instead of local:
```bash
CONVERSION_MODE=remote docker compose -f docker-compose.dev.yml up
```
### Manual Setup
If you prefer running services directly on your machine:
### Backend (Python 3.12+)
```bash
@ -67,15 +96,43 @@ npx prettier --write src/ # auto-format
## Running Tests
```bash
# Backend (199 tests)
# Backend (377 tests)
cd document-parser
pytest tests/ -v
# Frontend (129 tests)
# Frontend (156 tests)
cd frontend
npm run test:run
```
### E2E API (Karate)
```bash
# Generate test PDFs + start stack
python e2e/generate-test-data.py
docker compose up -d --wait
# Run all API tests
mvn test -f e2e/api/pom.xml
# Or by tag: @smoke, @regression, @e2e
mvn test -f e2e/api/pom.xml -Dkarate.options="--tags @smoke"
```
### E2E UI (Karate UI)
```bash
# Generate test PDFs + start stack (if not already running)
python e2e/generate-test-data.py
docker compose up -d --wait
# Run critical UI tests (CI scope)
mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @critical"
# Run all UI tests (local scope)
mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @ui"
```
All tests must pass before submitting a PR.
## Submitting Changes

83
README.md
View file

@ -31,9 +31,13 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t
- **Configurable Docling pipeline** — OCR, table extraction, code/formula enrichment, picture classification & description, image generation
- **Bounding box visualization** — color-coded element overlay directly on the PDF
- **Per-page results** — right panel syncs with the current PDF page
- **Chunking** — split extracted content into semantic chunks (hierarchical, hybrid, or page-based) with configurable token limits and inline editing
- **Ingestion pipeline** — Docling → chunking → embedding → OpenSearch vector indexing (one-click from Studio)
- **Markdown & HTML export** of extracted content
- **Document management** — upload, list, delete
- **Document management** — upload, list, delete, search, filter by indexing status
- **Analysis history** — re-visit and open past analyses
- **Upload limits** — configurable max file size and max page count per document
- **Rate limiting** — configurable requests per minute per IP
- **Dark / Light theme** and **FR / EN** localization
@ -74,7 +78,7 @@ document-parser/
├── services/ # Use case orchestration
│ ├── document_service.py # Upload, delete, preview
│ └── analysis_service.py # Async Docling processing
└── tests/ # 199 tests (pytest)
└── tests/ # 377 tests (pytest)
```
### Frontend structure (feature-based)
@ -98,14 +102,24 @@ frontend/src/
## Quick Start
Docling Studio ships two Docker image variants:
One command, nothing else to install:
```bash
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
```
Open [http://localhost:3000](http://localhost:3000), upload a PDF, and get results. That's it.
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
### Image variants
| Variant | Image tag | Size | Description |
|---------|-----------|------|-------------|
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only |
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only (downloads ML models on first run) |
### Docker — remote mode (fastest)
For remote mode:
```bash
docker run -p 3000:3000 \
@ -113,27 +127,17 @@ docker run -p 3000:3000 \
ghcr.io/scub-france/docling-studio:latest-remote
```
### Docker — local mode (self-contained)
```bash
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
```
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
Open [http://localhost:3000](http://localhost:3000)
### Docker Compose (for development)
### Docker Compose
```bash
git clone https://github.com/scub-france/Docling-Studio.git
cd Docling-Studio
# Local mode (default)
# Simple mode (backend + frontend only)
docker compose up --build
# Remote mode
CONVERSION_MODE=remote DOCLING_SERVE_URL=http://your-docling-serve:5001 docker compose up --build
# With ingestion pipeline (OpenSearch + embeddings)
docker compose --profile ingestion -f docker-compose.yml -f docker-compose.ingestion.yml up --build
```
### Local Development
@ -162,12 +166,12 @@ npm run dev
### Running Tests
```bash
# Backend (199 tests)
# Backend (377 tests)
cd document-parser
pip install pytest pytest-asyncio httpx
pytest tests/ -v
# Frontend (129 tests)
# Frontend (156 tests)
cd frontend
npm run test:run
```
@ -202,6 +206,43 @@ All configuration is done via environment variables. See [`.env.example`](.env.e
| `UPLOAD_DIR` | `./uploads` | File storage directory |
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
| `CONVERSION_TIMEOUT` | `600` | Max seconds for a single Docling conversion |
| `BATCH_PAGE_SIZE` | `10` | Pages per batch (`0` = process all at once) |
| `MAX_FILE_SIZE_MB` | `50` | Maximum upload file size in MB (`0` = unlimited) |
| `MAX_PAGE_COUNT` | `0` | Maximum number of pages per document (`0` = unlimited) |
| `RATE_LIMIT_RPM` | `100` | Max requests per minute per IP (`0` = disabled) |
## Upload Limits
Docling Studio enforces configurable limits on uploaded documents to protect the server against oversized files and long-running analyses:
- **`MAX_FILE_SIZE_MB`** (default `50`) — rejects uploads exceeding this size. Validated at two levels: early `Content-Length` check and streaming byte count.
- **`MAX_PAGE_COUNT`** (default `0` = unlimited) — rejects documents with more pages than allowed. Useful on shared instances or Hugging Face Spaces to cap processing time.
Both limits are exposed in the `/api/health` endpoint so the frontend can display them to the user before upload. Set either to `0` to disable the corresponding check.
## Ingestion Pipeline (opt-in)
Docling Studio can optionally index extracted chunks into [OpenSearch](https://opensearch.org/) for vector and full-text search. This requires two additional services (OpenSearch + embedding) and is **disabled by default**.
To enable ingestion with Docker Compose:
```bash
docker compose --profile ingestion \
-f docker-compose.yml -f docker-compose.ingestion.yml \
up --build
```
When ingestion is enabled, the UI shows:
- An **Ingest** button in Studio to push chunks to OpenSearch
- An **OpenSearch** connection status badge in the sidebar
- **Indexed / Not indexed** filters on the Documents page
- A **Search** page for full-text and vector search across indexed documents
| Variable | Default | Description |
|----------|---------|-------------|
| `OPENSEARCH_URL` | — | OpenSearch endpoint (empty = ingestion disabled) |
| `EMBEDDING_URL` | — | Embedding service endpoint (empty = ingestion disabled) |
| `EMBEDDING_DIMENSION` | `384` | Vector dimension (must match embedding model) |
## CI / Release

110
docker-compose.dev.yml Normal file
View file

@ -0,0 +1,110 @@
# =============================================================================
# Docling Studio — Development stack
#
# Usage:
# docker compose -f docker-compose.dev.yml up
#
# Includes OpenSearch single-node + Dashboards for search/sync features.
# Frontend runs Vite dev server with HMR, backend runs with --reload.
# =============================================================================
services:
# --- OpenSearch (single-node, security disabled for local dev) ---
opensearch:
image: opensearchproject/opensearch:2
environment:
discovery.type: single-node
DISABLE_SECURITY_PLUGIN: "true"
OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"
ports:
- "9200:9200"
volumes:
- opensearch_data:/usr/share/opensearch/data
healthcheck:
test: ["CMD-SHELL", "curl -sf http://localhost:9200/_cluster/health || exit 1"]
interval: 10s
timeout: 5s
retries: 10
# --- OpenSearch Dashboards (index inspection UI) ---
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:2
environment:
OPENSEARCH_HOSTS: '["http://opensearch:9200"]'
DISABLE_SECURITY_DASHBOARDS_PLUGIN: "true"
ports:
- "5601:5601"
depends_on:
opensearch:
condition: service_healthy
# --- Embedding service (sentence-transformers) ---
embedding:
build:
context: ./embedding-service
ports:
- "8001:8001"
environment:
EMBEDDING_MODEL: ${EMBEDDING_MODEL:-all-MiniLM-L6-v2}
EMBEDDING_BATCH_SIZE: ${EMBEDDING_BATCH_SIZE:-64}
healthcheck:
test: ["CMD-SHELL", "curl -sf http://localhost:8001/health || exit 1"]
interval: 10s
timeout: 5s
retries: 10
deploy:
resources:
limits:
memory: 2g
# --- Backend (FastAPI with hot-reload) ---
document-parser:
build:
context: ./document-parser
target: ${CONVERSION_MODE:-local}
ports:
- "8000:8000"
volumes:
- ./document-parser:/app
- uploads_data:/app/uploads
- db_data:/app/data
environment:
CORS_ORIGINS: ${CORS_ORIGINS:-http://localhost:3000,http://localhost:5173}
DOCLING_SERVE_URL: ${DOCLING_SERVE_URL:-}
DOCLING_SERVE_API_KEY: ${DOCLING_SERVE_API_KEY:-}
RATE_LIMIT_RPM: ${RATE_LIMIT_RPM:-100}
MAX_FILE_SIZE_MB: ${MAX_FILE_SIZE_MB:-50}
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-10}
OPENSEARCH_URL: http://opensearch:9200
EMBEDDING_URL: http://embedding:8001
command: ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
depends_on:
opensearch:
condition: service_healthy
embedding:
condition: service_healthy
deploy:
resources:
limits:
memory: 4g
# --- Frontend (Vite dev server with HMR) ---
frontend:
image: node:20-alpine
working_dir: /app
ports:
- "3000:3000"
volumes:
- ./frontend:/app
- frontend_node_modules:/app/node_modules
environment:
VITE_APP_VERSION: dev
command: ["sh", "-c", "npm install && npm run dev -- --host"]
depends_on:
- document-parser
volumes:
opensearch_data:
uploads_data:
db_data:
frontend_node_modules:

19
docker-compose.ingestion.yml Normal file
View file

@ -0,0 +1,19 @@
# Override to enable the ingestion pipeline (OpenSearch + embeddings).
#
# Usage:
# docker compose --profile ingestion -f docker-compose.yml -f docker-compose.ingestion.yml up --build
#
# This wires the backend to the OpenSearch and embedding services started
# by the "ingestion" profile and ensures they are healthy before the
# backend starts.
services:
document-parser:
environment:
OPENSEARCH_URL: http://opensearch:9200
EMBEDDING_URL: http://embedding:8001
depends_on:
opensearch:
condition: service_healthy
embedding:
condition: service_healthy

42
docker-compose.yml
View file

@ -1,4 +1,40 @@
services:
# --- OpenSearch (single-node, security disabled) ---
opensearch:
profiles: ["ingestion"]
image: opensearchproject/opensearch:2
environment:
discovery.type: single-node
DISABLE_SECURITY_PLUGIN: "true"
OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"
volumes:
- opensearch_data:/usr/share/opensearch/data
healthcheck:
test: ["CMD-SHELL", "curl -sf http://localhost:9200/_cluster/health || exit 1"]
interval: 10s
timeout: 5s
retries: 10
# --- Embedding service (sentence-transformers) ---
embedding:
profiles: ["ingestion"]
build:
context: ./embedding-service
environment:
EMBEDDING_MODEL: ${EMBEDDING_MODEL:-all-MiniLM-L6-v2}
EMBEDDING_BATCH_SIZE: ${EMBEDDING_BATCH_SIZE:-64}
healthcheck:
test: ["CMD-SHELL", "curl -sf http://localhost:8001/health || exit 1"]
interval: 15s
timeout: 10s
retries: 20
start_period: 120s
deploy:
resources:
limits:
memory: 2g
# --- Backend (FastAPI) ---
document-parser:
build:
context: ./document-parser
@ -14,12 +50,15 @@ services:
DOCLING_SERVE_API_KEY: ${DOCLING_SERVE_API_KEY:-}
RATE_LIMIT_RPM: ${RATE_LIMIT_RPM:-100}
MAX_FILE_SIZE_MB: ${MAX_FILE_SIZE_MB:-50}
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-0}
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-10}
OPENSEARCH_URL: ${OPENSEARCH_URL:-}
EMBEDDING_URL: ${EMBEDDING_URL:-}
deploy:
resources:
limits:
memory: 4g
# --- Frontend (nginx) ---
frontend:
build:
context: ./frontend
@ -29,5 +68,6 @@ services:
- document-parser
volumes:
opensearch_data:
uploads_data:
db_data:

83
docs/getting-started.md
View file

@ -1,15 +1,28 @@
# Getting Started
Docling Studio ships two Docker image variants:
## Quick Start
| Variant | Image tag | Size | Description |
|---------|-----------|------|-------------|
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only (downloads ML models on first run) |
One command, nothing else to install:
```bash
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
```
Open [http://localhost:3000](http://localhost:3000), upload a PDF, and get results. That's it.
!!! note
The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
![Docker architecture](images/docker.png){ width="600" }
## Docker — remote mode (fastest)
## Image Variants
| Variant | Image tag | Size | Description |
|---------|-----------|------|-------------|
| **local** | `latest-local` | ~1.9 GB | Full — runs Docling in-process, CPU-only |
| **remote** | `latest-remote` | ~270 MB | Lightweight — delegates to an external [Docling Serve](https://github.com/DS4SD/docling-serve) instance |
For remote mode:
```bash
docker run -p 3000:3000 \
@ -17,27 +30,19 @@ docker run -p 3000:3000 \
ghcr.io/scub-france/docling-studio:latest-remote
```
## Docker — local mode (self-contained)
```bash
docker run -p 3000:3000 ghcr.io/scub-france/docling-studio:latest-local
```
> **Note:** The first analysis takes longer as Docling downloads its ML models (~400 MB). Subsequent runs are fast.
Open [http://localhost:3000](http://localhost:3000).
## Docker Compose (recommended for development)
## Docker Compose
```bash
git clone https://github.com/scub-france/Docling-Studio.git
cd Docling-Studio
# Local mode (default)
# Simple mode (backend + frontend only)
docker compose up --build
# Remote mode
CONVERSION_MODE=remote DOCLING_SERVE_URL=http://your-docling-serve:5001 docker compose up --build
# With ingestion pipeline (OpenSearch + embeddings)
docker compose --profile ingestion \
-f docker-compose.yml -f docker-compose.ingestion.yml \
up --build
```
## Local Development
@ -136,10 +141,48 @@ All configuration is done via environment variables:
| `UPLOAD_DIR` | `./uploads` | File storage directory |
| `DB_PATH` | `./data/docling_studio.db` | SQLite database path |
| `CONVERSION_TIMEOUT` | `600` | Max seconds per Docling conversion |
| `BATCH_PAGE_SIZE` | `10` | Pages per batch (`0` = process all at once) |
| `MAX_CONCURRENT_ANALYSES` | `3` | Maximum parallel analysis jobs |
| `DEPLOYMENT_MODE` | `self-hosted` | `self-hosted` or `huggingface` (shows disclaimer banner) |
| `MAX_FILE_SIZE_MB` | `50` | Maximum upload file size in MB (`0` = unlimited) |
| `MAX_PAGE_COUNT` | `0` | Maximum number of pages per document (`0` = unlimited) |
| `RATE_LIMIT_RPM` | `100` | Max requests per minute per IP (`0` = disabled) |
| `APP_VERSION` | `dev` | Application version (set automatically by CI/Docker) |
## Upload Limits
Docling Studio enforces configurable limits on uploaded documents to protect the server against oversized files and long-running analyses:
- **`MAX_FILE_SIZE_MB`** (default `50`) — rejects uploads exceeding this size. Validated at two levels: early `Content-Length` check and streaming byte count.
- **`MAX_PAGE_COUNT`** (default `0` = unlimited) — rejects documents with more pages than allowed. Useful on shared instances or Hugging Face Spaces to cap processing time.
Both limits are exposed in the `/api/health` endpoint so the frontend can display them to the user before upload. Set either to `0` to disable the corresponding check.
## Ingestion Pipeline (opt-in)
Docling Studio can optionally index extracted chunks into [OpenSearch](https://opensearch.org/) for vector and full-text search. This requires two additional services (OpenSearch + embedding) and is **disabled by default**.
To enable ingestion with Docker Compose:
```bash
docker compose --profile ingestion \
-f docker-compose.yml -f docker-compose.ingestion.yml \
up --build
```
When ingestion is enabled, the UI shows:
- An **Ingest** button in Studio to push chunks to OpenSearch
- An **OpenSearch** connection status badge in the sidebar
- **Indexed / Not indexed** filters on the Documents page
- A **Search** page for full-text and vector search across indexed documents
| Variable | Default | Description |
|----------|---------|-------------|
| `OPENSEARCH_URL` | — | OpenSearch endpoint (empty = ingestion disabled) |
| `EMBEDDING_URL` | — | Embedding service endpoint (empty = ingestion disabled) |
| `EMBEDDING_DIMENSION` | `384` | Vector dimension (must match embedding model) |
## System Requirements
| | Remote image | Local image |

3
docs/index.md
View file

@ -18,7 +18,8 @@ Upload a PDF, configure the extraction pipeline, and visualize the results — t
- **Document management** — upload, list, delete
- **Analysis history** — re-visit and open past analyses
- **Feature flags** — capabilities adapt to the conversion engine (local vs remote)
- **Rate limiting** — 60 requests per minute per IP to protect the backend
- **Upload limits** — configurable max file size (`MAX_FILE_SIZE_MB`) and max page count (`MAX_PAGE_COUNT`) per document
- **Rate limiting** — configurable requests per minute per IP (`RATE_LIMIT_RPM`)
- **Deployment modes** — self-hosted (default) or HuggingFace Spaces (with disclaimer banner)
- **Health endpoint** — `/api/health` reports engine type, deployment mode, and database status
- **Dark / Light theme** and **FR / EN** localization

45
document-parser/api/analyses.py
View file

@ -13,6 +13,7 @@ from api.schemas import (
ChunkResponse,
CreateAnalysisRequest,
RechunkRequest,
UpdateChunkTextRequest,
)
from services.analysis_service import AnalysisService
@ -110,6 +111,50 @@ async def rechunk_analysis(
]
@router.patch("/{job_id}/chunks/{chunk_index}", response_model=list[ChunkResponse])
async def update_chunk_text(
job_id: str, chunk_index: int, body: UpdateChunkTextRequest, service: ServiceDep
) -> list[ChunkResponse]:
"""Update the text of a single chunk by index."""
try:
chunks = await service.update_chunk_text(job_id, chunk_index, body.text)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e)) from e
return [
ChunkResponse(
text=c["text"],
headings=c.get("headings", []),
source_page=c.get("sourcePage"),
token_count=c.get("tokenCount", 0),
bboxes=[ChunkBboxResponse(page=b["page"], bbox=b["bbox"]) for b in c.get("bboxes", [])],
modified=c.get("modified", False),
deleted=c.get("deleted", False),
)
for c in chunks
]
@router.delete("/{job_id}/chunks/{chunk_index}", response_model=list[ChunkResponse])
async def delete_chunk(job_id: str, chunk_index: int, service: ServiceDep) -> list[ChunkResponse]:
"""Soft-delete a chunk by index (marks it as deleted)."""
try:
chunks = await service.delete_chunk(job_id, chunk_index)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e)) from e
return [
ChunkResponse(
text=c["text"],
headings=c.get("headings", []),
source_page=c.get("sourcePage"),
token_count=c.get("tokenCount", 0),
bboxes=[ChunkBboxResponse(page=b["page"], bbox=b["bbox"]) for b in c.get("bboxes", [])],
modified=c.get("modified", False),
deleted=c.get("deleted", False),
)
for c in chunks
]
@router.delete("/{job_id}", status_code=204)
async def delete_analysis(job_id: str, service: ServiceDep) -> None:
"""Delete an analysis job."""

119
document-parser/api/ingestion.py Normal file
View file

@ -0,0 +1,119 @@
"""Ingestion API router — trigger and manage vector ingestion pipeline."""
from __future__ import annotations
import logging
from typing import Annotated
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from api.schemas import (
IngestionResponse,
IngestionStatusResponse,
SearchResponse,
SearchResultItem,
)
from services.analysis_service import AnalysisService
from services.ingestion_service import IngestionService
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/ingestion", tags=["ingestion"])
def _get_ingestion_service(request: Request) -> IngestionService:
svc = request.app.state.ingestion_service
if svc is None:
raise HTTPException(
status_code=503,
detail="Ingestion not available (EMBEDDING_URL and OPENSEARCH_URL required)",
)
return svc
def _get_analysis_service(request: Request) -> AnalysisService:
return request.app.state.analysis_service
IngestionDep = Annotated[IngestionService, Depends(_get_ingestion_service)]
AnalysisDep = Annotated[AnalysisService, Depends(_get_analysis_service)]
@router.post("/{job_id}", response_model=IngestionResponse)
async def ingest_analysis(
job_id: str,
ingestion: IngestionDep,
analysis: AnalysisDep,
) -> IngestionResponse:
"""Ingest a completed analysis into the vector index.
Takes the chunks from an existing analysis job, embeds them,
and indexes them into OpenSearch.
"""
job = await analysis.find_by_id(job_id)
if not job:
raise HTTPException(status_code=404, detail="Analysis not found")
if job.status.value != "COMPLETED":
raise HTTPException(status_code=400, detail="Analysis is not completed")
if not job.chunks_json:
raise HTTPException(status_code=400, detail="Analysis has no chunks — run chunking first")
try:
result = await ingestion.ingest(
doc_id=job.document_id,
filename=job.document_filename or "unknown",
chunks_json=job.chunks_json,
)
except Exception as e:
logger.exception("Ingestion failed for job %s", job_id)
raise HTTPException(status_code=500, detail=f"Ingestion failed: {e}") from e
return IngestionResponse(
doc_id=result.doc_id,
chunks_indexed=result.chunks_indexed,
embedding_dimension=result.embedding_dimension,
)
@router.delete("/{doc_id}", status_code=204)
async def delete_ingested_document(doc_id: str, ingestion: IngestionDep) -> None:
"""Delete all indexed chunks for a document."""
await ingestion.delete_document(doc_id)
@router.get("/status", response_model=IngestionStatusResponse)
async def ingestion_status(request: Request) -> IngestionStatusResponse:
"""Check if the ingestion pipeline is available and OpenSearch is connected."""
svc = request.app.state.ingestion_service
if svc is None:
return IngestionStatusResponse(available=False, opensearch_connected=False)
connected = await svc.ping()
return IngestionStatusResponse(available=True, opensearch_connected=connected)
@router.get("/search", response_model=SearchResponse)
async def search_chunks(
ingestion: IngestionDep,
q: str = Query(..., min_length=1, description="Search query"),
doc_id: str | None = Query(None, description="Filter by document ID"),
k: int = Query(20, ge=1, le=100, description="Max results"),
) -> SearchResponse:
"""Full-text search across indexed chunks.
Returns matching chunks with content and metadata.
Optionally filter by document ID.
"""
results = await ingestion.search_fulltext(q, k=k, doc_id=doc_id)
items = [
SearchResultItem(
doc_id=r.chunk.doc_id,
filename=r.chunk.filename,
content=r.chunk.content,
chunk_index=r.chunk.chunk_index,
page_number=r.chunk.page_number,
score=r.score,
headings=r.chunk.headings,
)
for r in results
]
return SearchResponse(results=items, total=len(items), query=q)

37
document-parser/api/schemas.py
View file

@ -34,6 +34,7 @@ class HealthResponse(_CamelModel):
database: str
max_page_count: int | None = None
max_file_size_mb: int | None = None
ingestion_available: bool = False
class DocumentResponse(_CamelModel):
@ -158,6 +159,12 @@ class ChunkResponse(_CamelModel):
source_page: int | None = None
token_count: int = 0
bboxes: list[ChunkBboxResponse] = []
modified: bool = False
deleted: bool = False
class UpdateChunkTextRequest(BaseModel):
text: str
class CreateAnalysisRequest(BaseModel):
@ -174,3 +181,33 @@ class RechunkRequest(BaseModel):
chunkingOptions: ChunkingOptionsRequest = Field(
validation_alias=AliasChoices("chunkingOptions", "chunking_options")
)
class IngestionResponse(_CamelModel):
doc_id: str
chunks_indexed: int
embedding_dimension: int
class IngestionStatusResponse(_CamelModel):
available: bool
opensearch_connected: bool = False
class SearchResultItem(_CamelModel):
"""A single search result with content and metadata."""
doc_id: str
filename: str
content: str
chunk_index: int
page_number: int
score: float
headings: list[str] = []
highlights: list[str] = []
class SearchResponse(_CamelModel):
results: list[SearchResultItem]
total: int
query: str

65
document-parser/domain/ports.py
View file

@ -6,7 +6,7 @@ Infrastructure adapters (local Docling, Docling Serve, etc.) implement these.
from __future__ import annotations
from typing import TYPE_CHECKING, Protocol
from typing import TYPE_CHECKING, Protocol, runtime_checkable
if TYPE_CHECKING:
from domain.models import AnalysisJob, Document
@ -16,6 +16,7 @@ if TYPE_CHECKING:
ConversionOptions,
ConversionResult,
)
from domain.vector_schema import IndexedChunk, SearchResult
class DocumentConverter(Protocol):
@ -79,3 +80,65 @@ class AnalysisRepository(Protocol):
async def delete(self, job_id: str) -> bool: ...
async def delete_by_document(self, document_id: str) -> int: ...
@runtime_checkable
class EmbeddingService(Protocol):
"""Port for text-to-vector embedding.
Implementations may call a local model, a remote microservice, etc.
"""
async def embed(self, texts: list[str]) -> list[list[float]]:
"""Generate embedding vectors for a batch of texts."""
...
@runtime_checkable
class VectorStore(Protocol):
"""Port for vector storage and retrieval.
Implementations (OpenSearch, pgvector, Qdrant, etc.) must satisfy this
contract. The port uses domain types from vector_schema — no infrastructure
details leak into the domain.
"""
async def ensure_index(self, index_name: str, mapping: dict) -> None:
"""Create the index if it does not exist. No-op if it already exists."""
...
async def index_chunks(self, index_name: str, chunks: list[IndexedChunk]) -> int:
"""Bulk-index a list of chunks. Returns the number of successfully indexed chunks."""
...
async def search_similar(
self,
index_name: str,
embedding: list[float],
*,
k: int = 10,
doc_id: str | None = None,
) -> list[SearchResult]:
"""Find the k nearest chunks by embedding similarity.
Args:
index_name: Target index.
embedding: Query vector.
k: Number of results to return.
doc_id: If provided, restrict search to chunks from this document.
"""
...
async def get_chunks(
self,
index_name: str,
doc_id: str,
*,
limit: int = 1000,
) -> list[SearchResult]:
"""Retrieve all indexed chunks for a given document, ordered by chunk_index."""
...
async def delete_document(self, index_name: str, doc_id: str) -> int:
"""Delete all chunks for a document from the index. Returns count deleted."""
...

177
document-parser/domain/vector_schema.py Normal file
View file

@ -0,0 +1,177 @@
"""Vector index schema — data contract for OpenSearch ingestion and inspection.
This module defines the standard metadata schema for the vector index used by
the ingestion pipeline (0.4.0) and the inspection UI (0.5.0).
Field usage by milestone:
┌────────────┬────────────────────────┬───────────────────────────────┬──────────────┐
│ Field │ 0.4.0 (write) │ 0.5.0 (read) │ Source │
├────────────┼────────────────────────┼───────────────────────────────┼──────────────┤
│ content │ Full-text search │ Chunk panel display │ Docling std │
│ embedding │ Indexed │ kNN semantic search │ Docling std │
│ doc_items │ Indexed │ Element type filtering │ Docling std │
│ headings │ Indexed │ Section hierarchy display │ Docling std │
│ origin │ Indexed │ Document provenance │ Docling std │
│ bboxes │ Written at ingestion │ Chunk↔bbox highlight │ Studio │
│ page_number│ Written at ingestion │ Split view navigation │ Studio │
│ chunk_index│ Written at ingestion │ Chunk ordering in panel │ Studio │
│ chunk_type │ Written at ingestion │ Metadata panel │ Studio │
│ doc_id │ Document linking │ Document list navigation │ Studio │
│ filename │ "My Documents" list │ Display │ Studio │
└────────────┴────────────────────────┴───────────────────────────────┴──────────────┘
"""
from __future__ import annotations
from dataclasses import dataclass, field
# -- Value objects for a single indexed chunk ----------------------------------
DEFAULT_EMBEDDING_DIMENSION = 384 # Granite Embedding 30M (sentence-transformers)
DEFAULT_INDEX_NAME = "docling-studio-chunks"
@dataclass(frozen=True)
class ChunkBboxEntry:
"""Bounding box for a chunk region on a specific page."""
page: int
x: float
y: float
w: float
h: float
@dataclass(frozen=True)
class DocItemRef:
"""Reference to a Docling DocItem (element in the document structure)."""
self_ref: str
label: str # text, table, picture, list, etc.
@dataclass(frozen=True)
class ChunkOrigin:
"""Provenance metadata — links a chunk back to its source document binary."""
binary_hash: str
filename: str
@dataclass(frozen=True)
class IndexedChunk:
"""A single chunk ready to be indexed in the vector store.
This is the domain-level representation of a document in the OpenSearch index.
It combines Docling-standard fields (content, embedding, doc_items, headings,
origin) with Docling Studio enriched fields (bboxes, page_number, chunk_index,
chunk_type, doc_id, filename).
"""
doc_id: str
filename: str
content: str
embedding: list[float]
chunk_index: int
chunk_type: str # text, table, picture, list, etc.
page_number: int
bboxes: list[ChunkBboxEntry] = field(default_factory=list)
headings: list[str] = field(default_factory=list)
doc_items: list[DocItemRef] = field(default_factory=list)
origin: ChunkOrigin | None = None
def to_dict(self) -> dict:
"""Serialize to a dict matching the OpenSearch index mapping."""
result: dict = {
"doc_id": self.doc_id,
"filename": self.filename,
"content": self.content,
"embedding": self.embedding,
"chunk_index": self.chunk_index,
"chunk_type": self.chunk_type,
"page_number": self.page_number,
"bboxes": [
{"page": b.page, "x": b.x, "y": b.y, "w": b.w, "h": b.h} for b in self.bboxes
],
"headings": self.headings,
"doc_items": [{"self_ref": d.self_ref, "label": d.label} for d in self.doc_items],
}
if self.origin:
result["origin"] = {
"binary_hash": self.origin.binary_hash,
"filename": self.origin.filename,
}
return result
# -- Search result -------------------------------------------------------------
@dataclass(frozen=True)
class SearchResult:
"""A chunk returned from a vector store query."""
chunk: IndexedChunk
score: float # similarity score (higher = more similar)
# -- Index mapping template ----------------------------------------------------
def build_index_mapping(embedding_dimension: int = DEFAULT_EMBEDDING_DIMENSION) -> dict:
"""Build the OpenSearch index mapping for the chunk index.
Args:
embedding_dimension: Vector dimension for the knn_vector field.
Defaults to 384 (Granite Embedding 30M / all-MiniLM-L6-v2).
"""
return {
"settings": {
"index": {
"knn": True,
},
},
"mappings": {
"properties": {
"doc_id": {"type": "keyword"},
"filename": {"type": "keyword"},
"content": {"type": "text", "analyzer": "standard"},
"embedding": {
"type": "knn_vector",
"dimension": embedding_dimension,
"method": {
"engine": "faiss",
"name": "hnsw",
},
},
"chunk_index": {"type": "integer"},
"chunk_type": {"type": "keyword"},
"page_number": {"type": "integer"},
"bboxes": {
"type": "nested",
"properties": {
"page": {"type": "integer"},
"x": {"type": "float"},
"y": {"type": "float"},
"w": {"type": "float"},
"h": {"type": "float"},
},
},
"headings": {"type": "text"},
"doc_items": {
"type": "nested",
"properties": {
"self_ref": {"type": "keyword"},
"label": {"type": "keyword"},
},
},
"origin": {
"type": "object",
"properties": {
"binary_hash": {"type": "keyword"},
"filename": {"type": "keyword"},
},
},
},
},
}

51
document-parser/infra/embedding_client.py Normal file
View file

@ -0,0 +1,51 @@
"""HTTP client adapter for the embedding microservice.
Satisfies the ``EmbeddingService`` Protocol defined in ``domain.ports``.
Calls the embedding-service REST API (POST /embed).
"""
from __future__ import annotations
import logging
import httpx
logger = logging.getLogger(__name__)
# Maximum texts per request to avoid payload / memory issues on the server.
_MAX_BATCH = 256
class EmbeddingClient:
"""Remote embedding adapter backed by the embedding-service microservice.
Args:
base_url: Embedding service URL (e.g. ``http://localhost:8001``).
timeout: HTTP request timeout in seconds.
"""
def __init__(self, base_url: str, *, timeout: float = 120.0) -> None:
self._base_url = base_url.rstrip("/")
self._timeout = timeout
async def embed(self, texts: list[str]) -> list[list[float]]:
"""Generate embeddings by calling the remote service.
Automatically splits large batches into sub-batches of ``_MAX_BATCH``.
"""
if not texts:
return []
all_embeddings: list[list[float]] = []
async with httpx.AsyncClient(timeout=self._timeout) as client:
for start in range(0, len(texts), _MAX_BATCH):
batch = texts[start : start + _MAX_BATCH]
resp = await client.post(
f"{self._base_url}/embed",
json={"texts": batch},
)
resp.raise_for_status()
data = resp.json()
all_embeddings.extend(data["embeddings"])
return all_embeddings

204
document-parser/infra/opensearch_store.py Normal file
View file

@ -0,0 +1,204 @@
"""OpenSearch adapter implementing the VectorStore port.
Uses the opensearch-py client for kNN vector search, full-text search,
and document CRUD against an OpenSearch cluster.
"""
from __future__ import annotations
import logging
from typing import Any
from opensearchpy import AsyncOpenSearch, NotFoundError
from domain.vector_schema import (
ChunkBboxEntry,
ChunkOrigin,
DocItemRef,
IndexedChunk,
SearchResult,
)
logger = logging.getLogger(__name__)
def _hit_to_indexed_chunk(hit: dict[str, Any]) -> IndexedChunk:
"""Reconstruct an IndexedChunk from an OpenSearch _source document."""
src = hit["_source"]
origin_raw = src.get("origin")
origin = (
ChunkOrigin(binary_hash=origin_raw["binary_hash"], filename=origin_raw["filename"])
if origin_raw
else None
)
return IndexedChunk(
doc_id=src["doc_id"],
filename=src["filename"],
content=src["content"],
embedding=src.get("embedding", []),
chunk_index=src["chunk_index"],
chunk_type=src["chunk_type"],
page_number=src["page_number"],
bboxes=[
ChunkBboxEntry(page=b["page"], x=b["x"], y=b["y"], w=b["w"], h=b["h"])
for b in src.get("bboxes", [])
],
headings=src.get("headings", []),
doc_items=[
DocItemRef(self_ref=d["self_ref"], label=d["label"]) for d in src.get("doc_items", [])
],
origin=origin,
)
def _hit_to_result(hit: dict[str, Any]) -> SearchResult:
"""Convert an OpenSearch hit to a SearchResult."""
return SearchResult(
chunk=_hit_to_indexed_chunk(hit),
score=hit.get("_score", 0.0),
)
class OpenSearchStore:
"""Concrete VectorStore adapter backed by OpenSearch.
Satisfies the ``VectorStore`` Protocol defined in ``domain.ports``.
Args:
url: OpenSearch cluster URL (e.g. ``http://localhost:9200``).
verify_certs: Whether to verify TLS certificates.
"""
def __init__(self, url: str, *, verify_certs: bool = False) -> None:
self._client = AsyncOpenSearch(
hosts=[url],
use_ssl=url.startswith("https"),
verify_certs=verify_certs,
ssl_show_warn=False,
)
# -- lifecycle -------------------------------------------------------------
async def close(self) -> None:
"""Close the underlying HTTP connection pool."""
await self._client.close()
# -- VectorStore protocol methods ------------------------------------------
async def ensure_index(self, index_name: str, mapping: dict) -> None:
"""Create the index if it does not exist. No-op if it already exists."""
exists = await self._client.indices.exists(index=index_name)
if not exists:
await self._client.indices.create(index=index_name, body=mapping)
logger.info("Created OpenSearch index '%s'", index_name)
else:
logger.debug("Index '%s' already exists — skipping creation", index_name)
async def index_chunks(self, index_name: str, chunks: list[IndexedChunk]) -> int:
"""Bulk-index a list of chunks. Returns the number successfully indexed."""
if not chunks:
return 0
body: list[dict[str, Any]] = []
for chunk in chunks:
doc_id = f"{chunk.doc_id}_{chunk.chunk_index}"
body.append({"index": {"_index": index_name, "_id": doc_id}})
body.append(chunk.to_dict())
resp = await self._client.bulk(body=body, refresh="wait_for")
errors = sum(1 for item in resp["items"] if item["index"].get("error"))
indexed = len(chunks) - errors
if errors:
logger.warning("Bulk index to '%s': %d/%d failed", index_name, errors, len(chunks))
return indexed
async def search_similar(
self,
index_name: str,
embedding: list[float],
*,
k: int = 10,
doc_id: str | None = None,
) -> list[SearchResult]:
"""kNN search for the k nearest chunks by embedding similarity."""
knn_query: dict[str, Any] = {
"knn": {
"embedding": {
"vector": embedding,
"k": k,
},
},
}
if doc_id:
knn_query["knn"]["embedding"]["filter"] = {
"term": {"doc_id": doc_id},
}
resp = await self._client.search(
index=index_name,
body={"size": k, "query": knn_query},
_source_excludes=["embedding"],
)
return [_hit_to_result(hit) for hit in resp["hits"]["hits"]]
async def get_chunks(
self,
index_name: str,
doc_id: str,
*,
limit: int = 1000,
) -> list[SearchResult]:
"""Retrieve all indexed chunks for a document, ordered by chunk_index."""
resp = await self._client.search(
index=index_name,
body={
"size": limit,
"query": {"term": {"doc_id": doc_id}},
"sort": [{"chunk_index": {"order": "asc"}}],
},
_source_excludes=["embedding"],
)
return [_hit_to_result(hit) for hit in resp["hits"]["hits"]]
async def delete_document(self, index_name: str, doc_id: str) -> int:
"""Delete all chunks for a document. Returns the number deleted."""
try:
resp = await self._client.delete_by_query(
index=index_name,
body={"query": {"term": {"doc_id": doc_id}}},
refresh=True,
)
deleted: int = resp.get("deleted", 0)
return deleted
except NotFoundError:
return 0
# -- full-text search (bonus from spec) ------------------------------------
async def search_fulltext(
self,
index_name: str,
query_text: str,
*,
k: int = 10,
doc_id: str | None = None,
) -> list[SearchResult]:
"""Full-text search on the content field.
This method is not part of the VectorStore protocol but is specified
in the issue acceptance criteria.
"""
must: list[dict[str, Any]] = [{"match": {"content": query_text}}]
if doc_id:
must.append({"term": {"doc_id": doc_id}})
resp = await self._client.search(
index=index_name,
body={
"size": k,
"query": {"bool": {"must": must}},
},
_source_excludes=["embedding"],
)
return [_hit_to_result(hit) for hit in resp["hits"]["hits"]]

10
document-parser/infra/settings.py
View file

@ -23,6 +23,9 @@ class Settings:
max_file_size_mb: int = 50 # upload limit in MB (0 = unlimited)
rate_limit_rpm: int = 100 # requests per minute per IP (0 = disabled)
batch_page_size: int = 0 # 0 = disabled, > 0 = pages per batch
opensearch_url: str = "" # empty = disabled
embedding_url: str = "" # empty = disabled (e.g. http://localhost:8001)
embedding_dimension: int = 384 # Granite Embedding 30M / all-MiniLM-L6-v2
upload_dir: str = "./uploads"
db_path: str = "./data/docling_studio.db"
cors_origins: list[str] = field(
@ -51,6 +54,8 @@ class Settings:
errors.append(f"rate_limit_rpm must be >= 0 (got {self.rate_limit_rpm})")
if self.batch_page_size < 0:
errors.append(f"batch_page_size must be >= 0 (got {self.batch_page_size})")
if self.embedding_dimension < 1:
errors.append(f"embedding_dimension must be >= 1 (got {self.embedding_dimension})")
if self.default_table_mode not in ("accurate", "fast"):
errors.append(
f"default_table_mode must be 'accurate' or 'fast' (got '{self.default_table_mode}')"
@ -89,7 +94,10 @@ class Settings:
max_file_size=int(os.environ.get("MAX_FILE_SIZE", "0")),
max_file_size_mb=int(os.environ.get("MAX_FILE_SIZE_MB", "50")),
rate_limit_rpm=int(os.environ.get("RATE_LIMIT_RPM", "100")),
batch_page_size=int(os.environ.get("BATCH_PAGE_SIZE", "0")),
batch_page_size=int(os.environ.get("BATCH_PAGE_SIZE", "10")),
opensearch_url=os.environ.get("OPENSEARCH_URL", ""),
embedding_url=os.environ.get("EMBEDDING_URL", ""),
embedding_dimension=int(os.environ.get("EMBEDDING_DIMENSION", "384")),
upload_dir=os.environ.get("UPLOAD_DIR", "./uploads"),
db_path=os.environ.get("DB_PATH", "./data/docling_studio.db"),
cors_origins=[o.strip() for o in cors_raw.split(",")],

32
document-parser/main.py
View file

@ -20,6 +20,7 @@ from fastapi.middleware.cors import CORSMiddleware
from api.analyses import router as analyses_router
from api.documents import router as documents_router
from api.ingestion import router as ingestion_router
from api.schemas import HealthResponse
from infra.rate_limiter import RateLimiterMiddleware
from infra.settings import settings
@ -28,6 +29,7 @@ from persistence.database import get_connection, init_db
from persistence.document_repo import SqliteDocumentRepository
from services.analysis_service import AnalysisConfig, AnalysisService
from services.document_service import DocumentConfig, DocumentService
from services.ingestion_service import IngestionConfig, IngestionService
logging.basicConfig(
level=logging.INFO,
@ -87,6 +89,28 @@ def _build_analysis_service(
)
def _build_ingestion_service() -> IngestionService | None:
"""Build the ingestion service — only if embedding + opensearch are configured."""
if not settings.embedding_url or not settings.opensearch_url:
logger.info("Ingestion disabled (EMBEDDING_URL or OPENSEARCH_URL not set)")
return None
from infra.embedding_client import EmbeddingClient
from infra.opensearch_store import OpenSearchStore
embedding = EmbeddingClient(settings.embedding_url)
vector_store = OpenSearchStore(settings.opensearch_url)
config = IngestionConfig(
embedding_dimension=settings.embedding_dimension,
)
logger.info(
"Ingestion enabled (embedding=%s, opensearch=%s)",
settings.embedding_url,
settings.opensearch_url,
)
return IngestionService(embedding, vector_store, config)
def _build_document_service(
document_repo: SqliteDocumentRepository,
analysis_repo: SqliteAnalysisRepository,
@ -114,6 +138,11 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
document_repo, analysis_repo = _build_repos()
app.state.analysis_service = _build_analysis_service(document_repo, analysis_repo)
app.state.document_service = _build_document_service(document_repo, analysis_repo)
ingestion_service = _build_ingestion_service()
app.state.ingestion_service = ingestion_service
if ingestion_service is not None:
app.include_router(ingestion_router)
logger.info("Ingestion router mounted")
logger.info("Docling Studio backend ready (engine=%s)", settings.conversion_engine)
yield
@ -128,7 +157,7 @@ app.add_middleware(
CORSMiddleware,
allow_origins=settings.cors_origins,
allow_credentials=True,
allow_methods=["GET", "POST", "DELETE", "OPTIONS"],
allow_methods=["GET", "POST", "PATCH", "DELETE", "OPTIONS"],
allow_headers=["Content-Type", "Authorization"],
)
if settings.rate_limit_rpm > 0:
@ -162,4 +191,5 @@ async def health() -> HealthResponse:
database=db_status,
max_page_count=settings.max_page_count if settings.max_page_count > 0 else None,
max_file_size_mb=settings.max_file_size_mb if settings.max_file_size_mb > 0 else None,
ingestion_available=getattr(app.state, "ingestion_service", None) is not None,
)

1
document-parser/requirements.txt
View file

@ -7,3 +7,4 @@ pillow>=10.0.0,<11.0.0
aiosqlite>=0.20.0,<1.0.0
httpx>=0.27.0,<1.0.0
pypdfium2>=4.0.0,<5.0.0
opensearch-py[async]>=2.6.0,<3.0.0

43
document-parser/services/analysis_service.py
View file

@ -160,6 +160,49 @@ class AnalysisService:
return chunks
async def update_chunk_text(self, job_id: str, chunk_index: int, text: str) -> list[dict]:
"""Update the text of a single chunk by index. Returns the full updated chunks list."""
job = await self._analysis_repo.find_by_id(job_id)
if not job:
raise ValueError(f"Analysis not found: {job_id}")
if job.status != AnalysisStatus.COMPLETED:
raise ValueError(f"Analysis is not completed: {job_id}")
if not job.chunks_json:
raise ValueError(f"No chunks available: {job_id}")
chunks = json.loads(job.chunks_json)
if chunk_index < 0 or chunk_index >= len(chunks):
raise ValueError(f"Chunk index out of range: {chunk_index}")
chunks[chunk_index]["text"] = text
chunks[chunk_index]["modified"] = True
chunks_json = json.dumps(chunks)
await self._analysis_repo.update_chunks(job_id, chunks_json)
return chunks
async def delete_chunk(self, job_id: str, chunk_index: int) -> list[dict]:
"""Soft-delete a chunk by index. Returns the full updated chunks list."""
job = await self._analysis_repo.find_by_id(job_id)
if not job:
raise ValueError(f"Analysis not found: {job_id}")
if job.status != AnalysisStatus.COMPLETED:
raise ValueError(f"Analysis is not completed: {job_id}")
if not job.chunks_json:
raise ValueError(f"No chunks available: {job_id}")
chunks = json.loads(job.chunks_json)
if chunk_index < 0 or chunk_index >= len(chunks):
raise ValueError(f"Chunk index out of range: {chunk_index}")
chunks[chunk_index]["deleted"] = True
chunks_json = json.dumps(chunks)
await self._analysis_repo.update_chunks(job_id, chunks_json)
return chunks
async def _run_batched_conversion(
self,
job_id: str,

190
document-parser/services/ingestion_service.py Normal file
View file

@ -0,0 +1,190 @@
"""Ingestion service — orchestrates Docling → embedding → OpenSearch.
Chains the full ingestion pipeline:
1. Convert document via Docling (reuse existing analysis)
2. Chunk with selected strategy
3. Embed all chunk texts via EmbeddingService
4. Index into OpenSearch via VectorStore
Idempotent: re-ingesting a document deletes old chunks before re-indexing.
"""
from __future__ import annotations
import json
import logging
from dataclasses import dataclass
from typing import TYPE_CHECKING
from domain.vector_schema import (
ChunkBboxEntry,
ChunkOrigin,
IndexedChunk,
build_index_mapping,
)
if TYPE_CHECKING:
from domain.ports import EmbeddingService, VectorStore
logger = logging.getLogger(__name__)
@dataclass
class IngestionConfig:
"""Configuration for the ingestion pipeline."""
index_name: str = "docling-studio-chunks"
embedding_dimension: int = 384
@dataclass
class IngestionResult:
"""Result of an ingestion pipeline run."""
doc_id: str
chunks_indexed: int
embedding_dimension: int
class IngestionService:
"""Orchestrates the embedding + indexing pipeline."""
def __init__(
self,
embedding_service: EmbeddingService,
vector_store: VectorStore,
config: IngestionConfig | None = None,
) -> None:
self._embedding = embedding_service
self._vector_store = vector_store
self._config = config or IngestionConfig()
async def ensure_index(self) -> None:
"""Ensure the vector index exists with the correct mapping."""
mapping = build_index_mapping(self._config.embedding_dimension)
await self._vector_store.ensure_index(self._config.index_name, mapping)
async def ingest(
self,
doc_id: str,
filename: str,
chunks_json: str,
*,
binary_hash: str | None = None,
) -> IngestionResult:
"""Run the embedding + indexing pipeline on pre-chunked data.
This method is idempotent: it deletes any existing chunks for the
document before re-indexing.
Args:
doc_id: Unique document identifier.
filename: Original filename.
chunks_json: JSON-serialized list of chunk dicts (from analysis).
binary_hash: Optional hash of the source file for provenance.
Returns:
IngestionResult with the number of chunks indexed.
"""
await self.ensure_index()
chunks_data: list[dict] = json.loads(chunks_json)
active_chunks = [c for c in chunks_data if not c.get("deleted")]
if not active_chunks:
logger.info("No active chunks for doc %s — skipping ingestion", doc_id)
return IngestionResult(doc_id=doc_id, chunks_indexed=0, embedding_dimension=0)
# 1. Embed all chunk texts
texts = [c["text"] for c in active_chunks]
logger.info("Embedding %d chunks for doc %s", len(texts), doc_id)
embeddings = await self._embedding.embed(texts)
# 2. Build IndexedChunk domain objects
origin = (
ChunkOrigin(binary_hash=binary_hash or "", filename=filename) if binary_hash else None
)
indexed_chunks: list[IndexedChunk] = []
for i, (chunk_data, embedding) in enumerate(zip(active_chunks, embeddings, strict=True)):
bboxes = [
ChunkBboxEntry(
page=b["page"],
x=b["bbox"][0] if b.get("bbox") else 0,
y=b["bbox"][1] if b.get("bbox") else 0,
w=(b["bbox"][2] - b["bbox"][0]) if b.get("bbox") and len(b["bbox"]) >= 4 else 0,
h=(b["bbox"][3] - b["bbox"][1]) if b.get("bbox") and len(b["bbox"]) >= 4 else 0,
)
for b in chunk_data.get("bboxes", [])
]
indexed_chunks.append(
IndexedChunk(
doc_id=doc_id,
filename=filename,
content=chunk_data["text"],
embedding=embedding,
chunk_index=i,
chunk_type=chunk_data.get("chunkType", "text"),
page_number=chunk_data.get("sourcePage", 0) or 0,
bboxes=bboxes,
headings=chunk_data.get("headings", []),
origin=origin,
)
)
# 3. Delete old chunks (idempotent re-indexing)
deleted = await self._vector_store.delete_document(self._config.index_name, doc_id)
if deleted:
logger.info("Deleted %d old chunks for doc %s", deleted, doc_id)
# 4. Index new chunks
indexed = await self._vector_store.index_chunks(self._config.index_name, indexed_chunks)
logger.info("Indexed %d/%d chunks for doc %s", indexed, len(indexed_chunks), doc_id)
return IngestionResult(
doc_id=doc_id,
chunks_indexed=indexed,
embedding_dimension=len(embeddings[0]) if embeddings else 0,
)
async def delete_document(self, doc_id: str) -> int:
"""Remove all indexed chunks for a document."""
return await self._vector_store.delete_document(self._config.index_name, doc_id)
async def search(
self,
query: str,
*,
k: int = 10,
doc_id: str | None = None,
) -> list:
"""Semantic search: embed the query then find nearest chunks."""
embeddings = await self._embedding.embed([query])
return await self._vector_store.search_similar(
self._config.index_name,
embeddings[0],
k=k,
doc_id=doc_id,
)
async def search_fulltext(
self,
query: str,
*,
k: int = 20,
doc_id: str | None = None,
) -> list:
"""Full-text keyword search in indexed chunks."""
return await self._vector_store.search_fulltext(
self._config.index_name,
query,
k=k,
doc_id=doc_id,
)
async def ping(self) -> bool:
"""Check if the OpenSearch cluster is reachable."""
try:
info = await self._vector_store._client.info()
return bool(info)
except Exception:
logger.debug("OpenSearch ping failed", exc_info=True)
return False

140
document-parser/tests/test_analysis_service.py
View file

@ -4,10 +4,12 @@ from __future__ import annotations
import asyncio
import functools
import json
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from domain.models import AnalysisStatus
from domain.services import extract_html_body, merge_results
from domain.value_objects import ConversionResult, PageDetail
from services.analysis_service import AnalysisConfig, AnalysisService, _count_pdf_pages
@ -515,3 +517,141 @@ class TestBatchedConversion:
assert result is None
# Only first batch should have been converted
assert converter.convert.call_count == 1
class TestUpdateChunkText:
"""Tests for AnalysisService.update_chunk_text."""
@pytest.mark.asyncio
async def test_update_chunk_text_success(self):
chunks = [
{"text": "original", "headings": [], "sourcePage": 1, "tokenCount": 5, "bboxes": []},
{"text": "second", "headings": [], "sourcePage": 2, "tokenCount": 3, "bboxes": []},
]
job = MagicMock()
job.status = AnalysisStatus.COMPLETED
job.chunks_json = json.dumps(chunks)
repo = MagicMock()
repo.find_by_id = AsyncMock(return_value=job)
repo.update_chunks = AsyncMock(return_value=True)
service = _make_service(analysis_repo=repo)
result = await service.update_chunk_text("j1", 0, "updated text")
assert result[0]["text"] == "updated text"
assert result[0]["modified"] is True
assert result[1]["text"] == "second"
assert result[1].get("modified", False) is False
repo.update_chunks.assert_called_once()
@pytest.mark.asyncio
async def test_update_chunk_text_not_found(self):
repo = MagicMock()
repo.find_by_id = AsyncMock(return_value=None)
service = _make_service(analysis_repo=repo)
with pytest.raises(ValueError, match="Analysis not found"):
await service.update_chunk_text("missing", 0, "text")
@pytest.mark.asyncio
async def test_update_chunk_text_not_completed(self):
job = MagicMock()
job.status = AnalysisStatus.RUNNING
repo = MagicMock()
repo.find_by_id = AsyncMock(return_value=job)
service = _make_service(analysis_repo=repo)
with pytest.raises(ValueError, match="not completed"):
await service.update_chunk_text("j1", 0, "text")
@pytest.mark.asyncio
async def test_update_chunk_text_index_out_of_range(self):
chunks = [
{"text": "only one", "headings": [], "sourcePage": 1, "tokenCount": 5, "bboxes": []}
]
job = MagicMock()
job.status = AnalysisStatus.COMPLETED
job.chunks_json = json.dumps(chunks)
repo = MagicMock()
repo.find_by_id = AsyncMock(return_value=job)
service = _make_service(analysis_repo=repo)
with pytest.raises(ValueError, match="out of range"):
await service.update_chunk_text("j1", 5, "text")
@pytest.mark.asyncio
async def test_update_chunk_text_no_chunks(self):
job = MagicMock()
job.status = AnalysisStatus.COMPLETED
job.chunks_json = None
repo = MagicMock()
repo.find_by_id = AsyncMock(return_value=job)
service = _make_service(analysis_repo=repo)
with pytest.raises(ValueError, match="No chunks available"):
await service.update_chunk_text("j1", 0, "text")
class TestDeleteChunk:
"""Tests for AnalysisService.delete_chunk."""
@pytest.mark.asyncio
async def test_delete_chunk_success(self):
chunks = [
{"text": "chunk1", "headings": [], "sourcePage": 1, "tokenCount": 5, "bboxes": []},
{"text": "chunk2", "headings": [], "sourcePage": 2, "tokenCount": 3, "bboxes": []},
]
job = MagicMock()
job.status = AnalysisStatus.COMPLETED
job.chunks_json = json.dumps(chunks)
repo = MagicMock()
repo.find_by_id = AsyncMock(return_value=job)
repo.update_chunks = AsyncMock(return_value=True)
service = _make_service(analysis_repo=repo)
result = await service.delete_chunk("j1", 0)
assert result[0]["deleted"] is True
assert result[1].get("deleted", False) is False
repo.update_chunks.assert_called_once()
@pytest.mark.asyncio
async def test_delete_chunk_not_found(self):
repo = MagicMock()
repo.find_by_id = AsyncMock(return_value=None)
service = _make_service(analysis_repo=repo)
with pytest.raises(ValueError, match="Analysis not found"):
await service.delete_chunk("missing", 0)
@pytest.mark.asyncio
async def test_delete_chunk_index_out_of_range(self):
chunks = [{"text": "only", "headings": [], "sourcePage": 1, "tokenCount": 5, "bboxes": []}]
job = MagicMock()
job.status = AnalysisStatus.COMPLETED
job.chunks_json = json.dumps(chunks)
repo = MagicMock()
repo.find_by_id = AsyncMock(return_value=job)
service = _make_service(analysis_repo=repo)
with pytest.raises(ValueError, match="out of range"):
await service.delete_chunk("j1", 5)
@pytest.mark.asyncio
async def test_delete_chunk_no_chunks(self):
job = MagicMock()
job.status = AnalysisStatus.COMPLETED
job.chunks_json = None
repo = MagicMock()
repo.find_by_id = AsyncMock(return_value=job)
service = _make_service(analysis_repo=repo)
with pytest.raises(ValueError, match="No chunks available"):
await service.delete_chunk("j1", 0)

17
document-parser/tests/test_api_endpoints.py
View file

@ -51,6 +51,23 @@ class TestHealthEndpoint:
assert "maxFileSizeMb" in data
assert data["maxFileSizeMb"] == 50
def test_health_exposes_ingestion_available_false(self, client):
original = getattr(app.state, "ingestion_service", None)
app.state.ingestion_service = None
resp = client.get("/api/health")
app.state.ingestion_service = original
data = resp.json()
assert "ingestionAvailable" in data
assert data["ingestionAvailable"] is False
def test_health_exposes_ingestion_available_true(self, client):
original = getattr(app.state, "ingestion_service", None)
app.state.ingestion_service = MagicMock()
resp = client.get("/api/health")
app.state.ingestion_service = original
data = resp.json()
assert data["ingestionAvailable"] is True
class TestDocumentEndpoints:
def test_list_documents(self, client, mock_document_service):

108
document-parser/tests/test_chunking.py
View file

@ -278,6 +278,114 @@ class TestCreateAnalysisWithChunking:
assert chunks[0]["text"] == "chunk1"
class TestUpdateChunkTextEndpoint:
def test_update_chunk_text_success(self, client, mock_analysis_service):
updated_chunks = [
{
"text": "updated text",
"headings": ["H1"],
"sourcePage": 1,
"tokenCount": 10,
"bboxes": [],
"modified": True,
},
{
"text": "chunk2",
"headings": [],
"sourcePage": 2,
"tokenCount": 20,
"bboxes": [],
"modified": False,
},
]
mock_analysis_service.update_chunk_text = AsyncMock(return_value=updated_chunks)
resp = client.patch(
"/api/analyses/j1/chunks/0",
json={"text": "updated text"},
)
assert resp.status_code == 200
data = resp.json()
assert len(data) == 2
assert data[0]["text"] == "updated text"
assert data[0]["modified"] is True
assert data[1]["modified"] is False
def test_update_chunk_text_invalid_index(self, client, mock_analysis_service):
mock_analysis_service.update_chunk_text = AsyncMock(
side_effect=ValueError("Chunk index out of range: 99"),
)
resp = client.patch(
"/api/analyses/j1/chunks/99",
json={"text": "new"},
)
assert resp.status_code == 400
def test_update_chunk_text_not_completed(self, client, mock_analysis_service):
mock_analysis_service.update_chunk_text = AsyncMock(
side_effect=ValueError("Analysis is not completed: j1"),
)
resp = client.patch(
"/api/analyses/j1/chunks/0",
json={"text": "new"},
)
assert resp.status_code == 400
def test_update_chunk_text_not_found(self, client, mock_analysis_service):
mock_analysis_service.update_chunk_text = AsyncMock(
side_effect=ValueError("Analysis not found: j1"),
)
resp = client.patch(
"/api/analyses/j1/chunks/0",
json={"text": "new"},
)
assert resp.status_code == 400
class TestDeleteChunkEndpoint:
def test_delete_chunk_success(self, client, mock_analysis_service):
updated_chunks = [
{
"text": "chunk1",
"headings": [],
"sourcePage": 1,
"tokenCount": 10,
"bboxes": [],
"deleted": True,
},
{
"text": "chunk2",
"headings": [],
"sourcePage": 2,
"tokenCount": 20,
"bboxes": [],
"deleted": False,
},
]
mock_analysis_service.delete_chunk = AsyncMock(return_value=updated_chunks)
resp = client.delete("/api/analyses/j1/chunks/0")
assert resp.status_code == 200
data = resp.json()
assert len(data) == 2
assert data[0]["deleted"] is True
assert data[1]["deleted"] is False
def test_delete_chunk_invalid_index(self, client, mock_analysis_service):
mock_analysis_service.delete_chunk = AsyncMock(
side_effect=ValueError("Chunk index out of range: 99"),
)
resp = client.delete("/api/analyses/j1/chunks/99")
assert resp.status_code == 400
def test_delete_chunk_not_completed(self, client, mock_analysis_service):
mock_analysis_service.delete_chunk = AsyncMock(
side_effect=ValueError("Analysis is not completed: j1"),
)
resp = client.delete("/api/analyses/j1/chunks/0")
assert resp.status_code == 400
class TestRechunkEndpoint:
def test_rechunk_success(self, client, mock_analysis_service):
mock_analysis_service.rechunk = AsyncMock(

112
document-parser/tests/test_embedding_client.py Normal file
View file

@ -0,0 +1,112 @@
"""Tests for the embedding client adapter (infra.embedding_client).
Mock httpx to validate adapter logic without running the embedding service.
"""
from __future__ import annotations
from unittest.mock import AsyncMock, MagicMock, patch
from domain.ports import EmbeddingService
from infra.embedding_client import _MAX_BATCH, EmbeddingClient
# -- Protocol satisfaction -----------------------------------------------------
class TestProtocolSatisfaction:
def test_satisfies_embedding_service_protocol(self) -> None:
client = EmbeddingClient("http://localhost:8001")
assert isinstance(client, EmbeddingService)
# -- embed ---------------------------------------------------------------------
class TestEmbed:
async def test_returns_empty_for_empty_input(self) -> None:
client = EmbeddingClient("http://localhost:8001")
result = await client.embed([])
assert result == []
async def test_calls_service_and_returns_embeddings(self) -> None:
client = EmbeddingClient("http://localhost:8001")
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_response.json.return_value = {
"embeddings": [[0.1, 0.2], [0.3, 0.4]],
"model": "all-MiniLM-L6-v2",
"dimension": 2,
}
mock_http_client = AsyncMock()
mock_http_client.post.return_value = mock_response
mock_http_client.__aenter__ = AsyncMock(return_value=mock_http_client)
mock_http_client.__aexit__ = AsyncMock(return_value=False)
with patch("infra.embedding_client.httpx.AsyncClient", return_value=mock_http_client):
result = await client.embed(["hello", "world"])
assert result == [[0.1, 0.2], [0.3, 0.4]]
mock_http_client.post.assert_awaited_once_with(
"http://localhost:8001/embed",
json={"texts": ["hello", "world"]},
)
async def test_strips_trailing_slash_from_base_url(self) -> None:
client = EmbeddingClient("http://localhost:8001/")
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_response.json.return_value = {"embeddings": [[0.1]], "model": "m", "dimension": 1}
mock_http_client = AsyncMock()
mock_http_client.post.return_value = mock_response
mock_http_client.__aenter__ = AsyncMock(return_value=mock_http_client)
mock_http_client.__aexit__ = AsyncMock(return_value=False)
with patch("infra.embedding_client.httpx.AsyncClient", return_value=mock_http_client):
await client.embed(["test"])
mock_http_client.post.assert_awaited_once_with(
"http://localhost:8001/embed",
json={"texts": ["test"]},
)
async def test_splits_large_batches(self) -> None:
client = EmbeddingClient("http://localhost:8001")
texts = [f"text_{i}" for i in range(_MAX_BATCH + 10)]
call_count = 0
def make_response(batch_size: int) -> MagicMock:
resp = MagicMock()
resp.raise_for_status = MagicMock()
resp.json.return_value = {
"embeddings": [[0.1]] * batch_size,
"model": "m",
"dimension": 1,
}
return resp
async def mock_post(url: str, json: dict) -> MagicMock:
nonlocal call_count
call_count += 1
return make_response(len(json["texts"]))
mock_http_client = AsyncMock()
mock_http_client.post = mock_post
mock_http_client.__aenter__ = AsyncMock(return_value=mock_http_client)
mock_http_client.__aexit__ = AsyncMock(return_value=False)
with patch("infra.embedding_client.httpx.AsyncClient", return_value=mock_http_client):
result = await client.embed(texts)
assert len(result) == _MAX_BATCH + 10
assert call_count == 2 # _MAX_BATCH + 10 remaining
# -- max batch constant --------------------------------------------------------
class TestMaxBatch:
def test_max_batch_is_256(self) -> None:
assert _MAX_BATCH == 256

187
document-parser/tests/test_ingestion_api.py Normal file
View file

@ -0,0 +1,187 @@
"""Tests for the ingestion API endpoints (api.ingestion)."""
from __future__ import annotations
from unittest.mock import AsyncMock
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from api.ingestion import router
from domain.models import AnalysisJob
from services.ingestion_service import IngestionResult
@pytest.fixture
def mock_ingestion_service() -> AsyncMock:
svc = AsyncMock()
svc.ingest.return_value = IngestionResult(
doc_id="doc-1", chunks_indexed=5, embedding_dimension=384
)
svc.delete_document.return_value = 3
return svc
@pytest.fixture
def mock_analysis_service() -> AsyncMock:
svc = AsyncMock()
job = AnalysisJob(document_id="doc-1")
job.document_filename = "test.pdf"
job.mark_running()
job.mark_completed(
markdown="# Test",
html="<h1>Test</h1>",
pages_json="[]",
document_json='{"doc": true}',
chunks_json='[{"text": "hello"}]',
)
svc.find_by_id.return_value = job
return svc
@pytest.fixture
def client(mock_ingestion_service: AsyncMock, mock_analysis_service: AsyncMock) -> TestClient:
app = FastAPI()
app.include_router(router)
app.state.ingestion_service = mock_ingestion_service
app.state.analysis_service = mock_analysis_service
return TestClient(app)
class TestIngestAnalysis:
def test_ingest_success(self, client: TestClient) -> None:
resp = client.post("/api/ingestion/job-1")
assert resp.status_code == 200
data = resp.json()
assert data["docId"] == "doc-1"
assert data["chunksIndexed"] == 5
assert data["embeddingDimension"] == 384
def test_ingest_not_found(self, client: TestClient, mock_analysis_service: AsyncMock) -> None:
mock_analysis_service.find_by_id.return_value = None
resp = client.post("/api/ingestion/missing")
assert resp.status_code == 404
def test_ingest_not_completed(
self, client: TestClient, mock_analysis_service: AsyncMock
) -> None:
job = AnalysisJob(document_id="doc-1")
mock_analysis_service.find_by_id.return_value = job
resp = client.post("/api/ingestion/job-1")
assert resp.status_code == 400
def test_ingest_no_chunks(self, client: TestClient, mock_analysis_service: AsyncMock) -> None:
job = AnalysisJob(document_id="doc-1")
job.mark_running()
job.mark_completed(markdown="x", html="x", pages_json="[]")
mock_analysis_service.find_by_id.return_value = job
resp = client.post("/api/ingestion/job-1")
assert resp.status_code == 400
class TestDeleteIngested:
def test_delete_success(self, client: TestClient) -> None:
resp = client.delete("/api/ingestion/doc-1")
assert resp.status_code == 204
class TestIngestionStatus:
def test_available(self, client: TestClient) -> None:
resp = client.get("/api/ingestion/status")
assert resp.status_code == 200
assert resp.json()["available"] is True
def test_not_available(self) -> None:
app = FastAPI()
app.include_router(router)
app.state.ingestion_service = None
app.state.analysis_service = AsyncMock()
tc = TestClient(app)
resp = tc.get("/api/ingestion/status")
assert resp.status_code == 200
assert resp.json()["available"] is False
class TestIngestionDisabled:
def test_router_not_mounted_returns_404(self) -> None:
"""When ingestion service is None, router should not be mounted → 404."""
app = FastAPI()
# Do NOT include ingestion router — simulates main.py conditional mount
app.state.ingestion_service = None
app.state.analysis_service = AsyncMock()
tc = TestClient(app)
resp = tc.post("/api/ingestion/job-1")
assert resp.status_code == 404
def test_status_still_returns_503_when_router_mounted_but_service_none(self) -> None:
"""If router is mounted but service is None, endpoints return 503."""
app = FastAPI()
app.include_router(router)
app.state.ingestion_service = None
app.state.analysis_service = AsyncMock()
tc = TestClient(app)
resp = tc.post("/api/ingestion/job-1")
assert resp.status_code == 503
class TestStatusOpenSearch:
def test_status_with_opensearch_connected(
self, client: TestClient, mock_ingestion_service: AsyncMock
) -> None:
mock_ingestion_service.ping.return_value = True
resp = client.get("/api/ingestion/status")
assert resp.status_code == 200
data = resp.json()
assert data["available"] is True
assert data["opensearchConnected"] is True
def test_status_with_opensearch_disconnected(
self, client: TestClient, mock_ingestion_service: AsyncMock
) -> None:
mock_ingestion_service.ping.return_value = False
resp = client.get("/api/ingestion/status")
assert resp.status_code == 200
data = resp.json()
assert data["available"] is True
assert data["opensearchConnected"] is False
class TestSearchEndpoint:
def test_search_success(self, client: TestClient, mock_ingestion_service: AsyncMock) -> None:
from domain.vector_schema import IndexedChunk, SearchResult
chunk = IndexedChunk(
doc_id="doc-1",
filename="test.pdf",
content="hello world",
embedding=[],
chunk_index=0,
chunk_type="text",
page_number=1,
headings=["Intro"],
)
mock_ingestion_service.search_fulltext.return_value = [
SearchResult(chunk=chunk, score=0.95)
]
resp = client.get("/api/ingestion/search", params={"q": "hello"})
assert resp.status_code == 200
data = resp.json()
assert data["total"] == 1
assert data["query"] == "hello"
assert data["results"][0]["content"] == "hello world"
assert data["results"][0]["score"] == 0.95
def test_search_empty_query(self, client: TestClient) -> None:
resp = client.get("/api/ingestion/search", params={"q": ""})
assert resp.status_code == 422
def test_search_with_doc_filter(
self, client: TestClient, mock_ingestion_service: AsyncMock
) -> None:
mock_ingestion_service.search_fulltext.return_value = []
resp = client.get("/api/ingestion/search", params={"q": "test", "doc_id": "doc-1"})
assert resp.status_code == 200
mock_ingestion_service.search_fulltext.assert_awaited_once_with(
"test", k=20, doc_id="doc-1"
)

188
document-parser/tests/test_ingestion_service.py Normal file
View file

@ -0,0 +1,188 @@
"""Tests for the ingestion service (services.ingestion_service)."""
from __future__ import annotations
import json
from unittest.mock import AsyncMock
import pytest
from services.ingestion_service import IngestionConfig, IngestionService
def _make_chunks_json(count: int = 3, *, with_deleted: bool = False) -> str:
chunks = []
for i in range(count):
chunk = {
"text": f"chunk text {i}",
"headings": [f"Heading {i}"],
"sourcePage": i + 1,
"tokenCount": 10,
"bboxes": [{"page": i + 1, "bbox": [0.0, 0.0, 100.0, 50.0]}],
}
if with_deleted and i == count - 1:
chunk["deleted"] = True
chunks.append(chunk)
return json.dumps(chunks)
@pytest.fixture
def mock_embedding() -> AsyncMock:
svc = AsyncMock()
svc.embed.return_value = [[0.1, 0.2, 0.3]] * 3
return svc
@pytest.fixture
def mock_vector_store() -> AsyncMock:
store = AsyncMock()
store.ensure_index.return_value = None
store.delete_document.return_value = 0
store.index_chunks.return_value = 3
return store
@pytest.fixture
def service(mock_embedding: AsyncMock, mock_vector_store: AsyncMock) -> IngestionService:
return IngestionService(
embedding_service=mock_embedding,
vector_store=mock_vector_store,
config=IngestionConfig(index_name="test-idx", embedding_dimension=3),
)
class TestIngest:
async def test_full_pipeline(
self, service: IngestionService, mock_embedding: AsyncMock, mock_vector_store: AsyncMock
) -> None:
result = await service.ingest("doc-1", "test.pdf", _make_chunks_json(3))
assert result.doc_id == "doc-1"
assert result.chunks_indexed == 3
mock_embedding.embed.assert_awaited_once()
texts = mock_embedding.embed.call_args[0][0]
assert len(texts) == 3
mock_vector_store.ensure_index.assert_awaited_once()
mock_vector_store.delete_document.assert_awaited_once_with("test-idx", "doc-1")
mock_vector_store.index_chunks.assert_awaited_once()
indexed = mock_vector_store.index_chunks.call_args[0][1]
assert len(indexed) == 3
assert indexed[0].doc_id == "doc-1"
assert indexed[0].filename == "test.pdf"
assert indexed[0].embedding == [0.1, 0.2, 0.3]
async def test_skips_deleted_chunks(
self, service: IngestionService, mock_embedding: AsyncMock, mock_vector_store: AsyncMock
) -> None:
mock_embedding.embed.return_value = [[0.1, 0.2, 0.3]] * 2
mock_vector_store.index_chunks.return_value = 2
result = await service.ingest("doc-1", "test.pdf", _make_chunks_json(3, with_deleted=True))
assert result.chunks_indexed == 2
texts = mock_embedding.embed.call_args[0][0]
assert len(texts) == 2
async def test_empty_chunks(
self, service: IngestionService, mock_embedding: AsyncMock, mock_vector_store: AsyncMock
) -> None:
result = await service.ingest("doc-1", "test.pdf", json.dumps([]))
assert result.chunks_indexed == 0
mock_embedding.embed.assert_not_awaited()
async def test_idempotent_deletes_old(
self, service: IngestionService, mock_vector_store: AsyncMock
) -> None:
mock_vector_store.delete_document.return_value = 5
await service.ingest("doc-1", "test.pdf", _make_chunks_json(3))
mock_vector_store.delete_document.assert_awaited_once_with("test-idx", "doc-1")
async def test_bbox_conversion(
self, service: IngestionService, mock_embedding: AsyncMock, mock_vector_store: AsyncMock
) -> None:
mock_embedding.embed.return_value = [[0.1, 0.2, 0.3]]
mock_vector_store.index_chunks.return_value = 1
await service.ingest("doc-1", "test.pdf", _make_chunks_json(1))
indexed = mock_vector_store.index_chunks.call_args[0][1]
bbox = indexed[0].bboxes[0]
assert bbox.x == 0.0
assert bbox.y == 0.0
assert bbox.w == 100.0
assert bbox.h == 50.0
async def test_with_binary_hash(
self, service: IngestionService, mock_vector_store: AsyncMock
) -> None:
mock_embedding = service._embedding
mock_embedding.embed.return_value = [[0.1]] * 1
await service.ingest("doc-1", "test.pdf", _make_chunks_json(1), binary_hash="abc123")
indexed = mock_vector_store.index_chunks.call_args[0][1]
assert indexed[0].origin is not None
assert indexed[0].origin.binary_hash == "abc123"
class TestDeleteDocument:
async def test_delegates_to_vector_store(
self, service: IngestionService, mock_vector_store: AsyncMock
) -> None:
mock_vector_store.delete_document.return_value = 3
result = await service.delete_document("doc-1")
assert result == 3
class TestSearch:
async def test_embeds_and_searches(
self, service: IngestionService, mock_embedding: AsyncMock, mock_vector_store: AsyncMock
) -> None:
mock_embedding.embed.return_value = [[0.5, 0.6, 0.7]]
mock_vector_store.search_similar.return_value = []
await service.search("test query", k=5)
mock_embedding.embed.assert_awaited_once_with(["test query"])
mock_vector_store.search_similar.assert_awaited_once()
class TestSearchFulltext:
async def test_delegates_to_vector_store(
self, service: IngestionService, mock_vector_store: AsyncMock
) -> None:
mock_vector_store.search_fulltext.return_value = []
await service.search_fulltext("hello world", k=5)
mock_vector_store.search_fulltext.assert_awaited_once_with(
"test-idx", "hello world", k=5, doc_id=None
)
async def test_filters_by_doc_id(
self, service: IngestionService, mock_vector_store: AsyncMock
) -> None:
mock_vector_store.search_fulltext.return_value = []
await service.search_fulltext("hello", doc_id="doc-1")
mock_vector_store.search_fulltext.assert_awaited_once_with(
"test-idx", "hello", k=20, doc_id="doc-1"
)
class TestPing:
async def test_ping_success(
self, service: IngestionService, mock_vector_store: AsyncMock
) -> None:
mock_vector_store._client = AsyncMock()
mock_vector_store._client.info.return_value = {"cluster_name": "test"}
result = await service.ping()
assert result is True
async def test_ping_failure(
self, service: IngestionService, mock_vector_store: AsyncMock
) -> None:
mock_vector_store._client = AsyncMock()
mock_vector_store._client.info.side_effect = ConnectionError("down")
result = await service.ping()
assert result is False
class TestEnsureIndex:
async def test_calls_vector_store(
self, service: IngestionService, mock_vector_store: AsyncMock
) -> None:
await service.ensure_index()
mock_vector_store.ensure_index.assert_awaited_once()
call_args = mock_vector_store.ensure_index.call_args
assert call_args[0][0] == "test-idx"

320
document-parser/tests/test_opensearch_store.py Normal file
View file

@ -0,0 +1,320 @@
"""Tests for the OpenSearch adapter (infra.opensearch_store).
These tests mock the AsyncOpenSearch client to validate adapter logic
without requiring a running OpenSearch instance.
"""
from __future__ import annotations
from unittest.mock import AsyncMock
import pytest
from domain.ports import VectorStore
from domain.vector_schema import (
ChunkBboxEntry,
ChunkOrigin,
DocItemRef,
IndexedChunk,
SearchResult,
build_index_mapping,
)
from infra.opensearch_store import OpenSearchStore, _hit_to_indexed_chunk, _hit_to_result
# -- Fixtures -----------------------------------------------------------------
def _make_chunk(
doc_id: str = "doc-1",
chunk_index: int = 0,
content: str = "hello world",
embedding: list[float] | None = None,
) -> IndexedChunk:
return IndexedChunk(
doc_id=doc_id,
filename="test.pdf",
content=content,
embedding=embedding or [0.1, 0.2, 0.3],
chunk_index=chunk_index,
chunk_type="text",
page_number=1,
bboxes=[ChunkBboxEntry(page=1, x=0.0, y=0.0, w=100.0, h=50.0)],
headings=["Chapter 1"],
doc_items=[DocItemRef(self_ref="#/texts/0", label="text")],
origin=ChunkOrigin(binary_hash="abc123", filename="test.pdf"),
)
def _make_hit(
doc_id: str = "doc-1",
chunk_index: int = 0,
score: float = 0.95,
content: str = "hello world",
) -> dict:
return {
"_id": f"{doc_id}_{chunk_index}",
"_score": score,
"_source": {
"doc_id": doc_id,
"filename": "test.pdf",
"content": content,
"chunk_index": chunk_index,
"chunk_type": "text",
"page_number": 1,
"bboxes": [{"page": 1, "x": 0.0, "y": 0.0, "w": 100.0, "h": 50.0}],
"headings": ["Chapter 1"],
"doc_items": [{"self_ref": "#/texts/0", "label": "text"}],
"origin": {"binary_hash": "abc123", "filename": "test.pdf"},
},
}
@pytest.fixture
def store() -> OpenSearchStore:
return OpenSearchStore("http://localhost:9200")
@pytest.fixture
def mock_client(store: OpenSearchStore) -> AsyncMock:
client = AsyncMock()
store._client = client
return client
# -- Protocol satisfaction -----------------------------------------------------
class TestProtocolSatisfaction:
def test_satisfies_vector_store_protocol(self) -> None:
"""OpenSearchStore structurally satisfies VectorStore Protocol."""
store = OpenSearchStore("http://localhost:9200")
assert isinstance(store, VectorStore)
# -- Hit deserialization -------------------------------------------------------
class TestHitDeserialization:
def test_hit_to_indexed_chunk(self) -> None:
hit = _make_hit()
chunk = _hit_to_indexed_chunk(hit)
assert isinstance(chunk, IndexedChunk)
assert chunk.doc_id == "doc-1"
assert chunk.content == "hello world"
assert chunk.chunk_index == 0
assert chunk.page_number == 1
assert len(chunk.bboxes) == 1
assert chunk.bboxes[0].w == 100.0
assert len(chunk.doc_items) == 1
assert chunk.doc_items[0].label == "text"
assert chunk.origin is not None
assert chunk.origin.binary_hash == "abc123"
def test_hit_to_indexed_chunk_no_origin(self) -> None:
hit = _make_hit()
hit["_source"]["origin"] = None
chunk = _hit_to_indexed_chunk(hit)
assert chunk.origin is None
def test_hit_to_indexed_chunk_missing_optional_fields(self) -> None:
hit = _make_hit()
del hit["_source"]["bboxes"]
del hit["_source"]["headings"]
del hit["_source"]["doc_items"]
del hit["_source"]["origin"]
chunk = _hit_to_indexed_chunk(hit)
assert chunk.bboxes == []
assert chunk.headings == []
assert chunk.doc_items == []
assert chunk.origin is None
def test_hit_to_result(self) -> None:
hit = _make_hit(score=0.87)
result = _hit_to_result(hit)
assert isinstance(result, SearchResult)
assert result.score == 0.87
assert result.chunk.doc_id == "doc-1"
# -- ensure_index --------------------------------------------------------------
class TestEnsureIndex:
async def test_creates_index_when_not_exists(
self, store: OpenSearchStore, mock_client: AsyncMock
) -> None:
mock_client.indices.exists.return_value = False
mapping = build_index_mapping()
await store.ensure_index("test-index", mapping)
mock_client.indices.create.assert_awaited_once_with(index="test-index", body=mapping)
async def test_noop_when_index_exists(
self, store: OpenSearchStore, mock_client: AsyncMock
) -> None:
mock_client.indices.exists.return_value = True
await store.ensure_index("test-index", {})
mock_client.indices.create.assert_not_awaited()
# -- index_chunks --------------------------------------------------------------
class TestIndexChunks:
async def test_bulk_indexes_chunks(
self, store: OpenSearchStore, mock_client: AsyncMock
) -> None:
chunks = [_make_chunk(chunk_index=0), _make_chunk(chunk_index=1)]
mock_client.bulk.return_value = {
"errors": False,
"items": [
{"index": {"_id": "doc-1_0", "status": 201}},
{"index": {"_id": "doc-1_1", "status": 201}},
],
}
count = await store.index_chunks("test-index", chunks)
assert count == 2
mock_client.bulk.assert_awaited_once()
async def test_returns_zero_for_empty_list(
self, store: OpenSearchStore, mock_client: AsyncMock
) -> None:
count = await store.index_chunks("test-index", [])
assert count == 0
mock_client.bulk.assert_not_awaited()
async def test_counts_partial_failures(
self, store: OpenSearchStore, mock_client: AsyncMock
) -> None:
chunks = [_make_chunk(chunk_index=0), _make_chunk(chunk_index=1)]
mock_client.bulk.return_value = {
"errors": True,
"items": [
{"index": {"_id": "doc-1_0", "status": 201}},
{"index": {"_id": "doc-1_1", "error": {"reason": "mapping"}}},
],
}
count = await store.index_chunks("test-index", chunks)
assert count == 1
async def test_bulk_body_structure(
self, store: OpenSearchStore, mock_client: AsyncMock
) -> None:
chunk = _make_chunk(doc_id="d1", chunk_index=3)
mock_client.bulk.return_value = {
"errors": False,
"items": [{"index": {"_id": "d1_3", "status": 201}}],
}
await store.index_chunks("idx", [chunk])
call_body = mock_client.bulk.call_args[1]["body"]
assert call_body[0] == {"index": {"_index": "idx", "_id": "d1_3"}}
assert call_body[1]["doc_id"] == "d1"
assert call_body[1]["chunk_index"] == 3
# -- search_similar ------------------------------------------------------------
class TestSearchSimilar:
async def test_knn_search(self, store: OpenSearchStore, mock_client: AsyncMock) -> None:
mock_client.search.return_value = {"hits": {"hits": [_make_hit(score=0.99)]}}
results = await store.search_similar("idx", [0.1, 0.2, 0.3], k=5)
assert len(results) == 1
assert results[0].score == 0.99
call_body = mock_client.search.call_args[1]["body"]
assert call_body["query"]["knn"]["embedding"]["vector"] == [0.1, 0.2, 0.3]
assert call_body["query"]["knn"]["embedding"]["k"] == 5
async def test_knn_search_with_doc_filter(
self, store: OpenSearchStore, mock_client: AsyncMock
) -> None:
mock_client.search.return_value = {"hits": {"hits": []}}
await store.search_similar("idx", [0.1], doc_id="doc-42")
call_body = mock_client.search.call_args[1]["body"]
assert call_body["query"]["knn"]["embedding"]["filter"] == {"term": {"doc_id": "doc-42"}}
async def test_knn_search_no_filter_by_default(
self, store: OpenSearchStore, mock_client: AsyncMock
) -> None:
mock_client.search.return_value = {"hits": {"hits": []}}
await store.search_similar("idx", [0.1])
call_body = mock_client.search.call_args[1]["body"]
assert "filter" not in call_body["query"]["knn"]["embedding"]
# -- get_chunks ----------------------------------------------------------------
class TestGetChunks:
async def test_retrieves_by_doc_id(
self, store: OpenSearchStore, mock_client: AsyncMock
) -> None:
mock_client.search.return_value = {
"hits": {"hits": [_make_hit(chunk_index=0), _make_hit(chunk_index=1)]}
}
results = await store.get_chunks("idx", "doc-1")
assert len(results) == 2
call_body = mock_client.search.call_args[1]["body"]
assert call_body["query"] == {"term": {"doc_id": "doc-1"}}
assert call_body["sort"] == [{"chunk_index": {"order": "asc"}}]
async def test_respects_limit(self, store: OpenSearchStore, mock_client: AsyncMock) -> None:
mock_client.search.return_value = {"hits": {"hits": []}}
await store.get_chunks("idx", "doc-1", limit=50)
call_body = mock_client.search.call_args[1]["body"]
assert call_body["size"] == 50
# -- delete_document -----------------------------------------------------------
class TestDeleteDocument:
async def test_deletes_by_doc_id(self, store: OpenSearchStore, mock_client: AsyncMock) -> None:
mock_client.delete_by_query.return_value = {"deleted": 5}
count = await store.delete_document("idx", "doc-1")
assert count == 5
call_body = mock_client.delete_by_query.call_args[1]["body"]
assert call_body["query"] == {"term": {"doc_id": "doc-1"}}
async def test_returns_zero_on_not_found(
self, store: OpenSearchStore, mock_client: AsyncMock
) -> None:
from opensearchpy import NotFoundError
mock_client.delete_by_query.side_effect = NotFoundError(404, "index_not_found")
count = await store.delete_document("idx", "doc-1")
assert count == 0
# -- search_fulltext -----------------------------------------------------------
class TestSearchFulltext:
async def test_fulltext_search(self, store: OpenSearchStore, mock_client: AsyncMock) -> None:
mock_client.search.return_value = {
"hits": {"hits": [_make_hit(content="matching text", score=1.5)]}
}
results = await store.search_fulltext("idx", "matching")
assert len(results) == 1
assert results[0].chunk.content == "matching text"
call_body = mock_client.search.call_args[1]["body"]
assert {"match": {"content": "matching"}} in call_body["query"]["bool"]["must"]
async def test_fulltext_search_with_doc_filter(
self, store: OpenSearchStore, mock_client: AsyncMock
) -> None:
mock_client.search.return_value = {"hits": {"hits": []}}
await store.search_fulltext("idx", "query", doc_id="doc-5")
call_body = mock_client.search.call_args[1]["body"]
must_clauses = call_body["query"]["bool"]["must"]
assert {"term": {"doc_id": "doc-5"}} in must_clauses
# -- close ---------------------------------------------------------------------
class TestClose:
async def test_close_delegates_to_client(
self, store: OpenSearchStore, mock_client: AsyncMock
) -> None:
await store.close()
mock_client.close.assert_awaited_once()

228
document-parser/tests/test_vector_schema.py Normal file
View file

@ -0,0 +1,228 @@
"""Tests for vector index schema — value objects and OpenSearch mapping."""
from __future__ import annotations
import pytest
from domain.vector_schema import (
DEFAULT_EMBEDDING_DIMENSION,
DEFAULT_INDEX_NAME,
ChunkBboxEntry,
ChunkOrigin,
DocItemRef,
IndexedChunk,
SearchResult,
build_index_mapping,
)
class TestChunkBboxEntry:
def test_construction(self):
bbox = ChunkBboxEntry(page=1, x=10.0, y=20.0, w=100.0, h=50.0)
assert bbox.page == 1
assert bbox.x == 10.0
assert bbox.w == 100.0
def test_frozen(self):
bbox = ChunkBboxEntry(page=1, x=0, y=0, w=10, h=10)
with pytest.raises(AttributeError):
bbox.page = 2 # type: ignore[misc]
class TestDocItemRef:
def test_construction(self):
ref = DocItemRef(self_ref="#/texts/0", label="text")
assert ref.self_ref == "#/texts/0"
assert ref.label == "text"
class TestChunkOrigin:
def test_construction(self):
origin = ChunkOrigin(binary_hash="abc123", filename="doc.pdf")
assert origin.binary_hash == "abc123"
assert origin.filename == "doc.pdf"
class TestIndexedChunk:
def _make_chunk(self, **overrides) -> IndexedChunk:
defaults = {
"doc_id": "doc-1",
"filename": "test.pdf",
"content": "Hello world",
"embedding": [0.1] * 384,
"chunk_index": 0,
"chunk_type": "text",
"page_number": 1,
}
defaults.update(overrides)
return IndexedChunk(**defaults)
def test_minimal_chunk(self):
chunk = self._make_chunk()
assert chunk.doc_id == "doc-1"
assert chunk.content == "Hello world"
assert chunk.bboxes == []
assert chunk.headings == []
assert chunk.doc_items == []
assert chunk.origin is None
def test_full_chunk(self):
chunk = self._make_chunk(
bboxes=[ChunkBboxEntry(page=1, x=10, y=20, w=100, h=50)],
headings=["Chapter 1", "Section A"],
doc_items=[DocItemRef(self_ref="#/texts/0", label="text")],
origin=ChunkOrigin(binary_hash="abc", filename="test.pdf"),
)
assert len(chunk.bboxes) == 1
assert chunk.headings == ["Chapter 1", "Section A"]
assert chunk.doc_items[0].label == "text"
assert chunk.origin.binary_hash == "abc"
def test_to_dict_minimal(self):
chunk = self._make_chunk()
d = chunk.to_dict()
assert d["doc_id"] == "doc-1"
assert d["filename"] == "test.pdf"
assert d["content"] == "Hello world"
assert d["embedding"] == [0.1] * 384
assert d["chunk_index"] == 0
assert d["chunk_type"] == "text"
assert d["page_number"] == 1
assert d["bboxes"] == []
assert d["headings"] == []
assert d["doc_items"] == []
assert "origin" not in d
def test_to_dict_full(self):
chunk = self._make_chunk(
bboxes=[ChunkBboxEntry(page=1, x=10.5, y=20.0, w=100.0, h=50.0)],
headings=["H1"],
doc_items=[DocItemRef(self_ref="#/texts/0", label="text")],
origin=ChunkOrigin(binary_hash="abc", filename="test.pdf"),
)
d = chunk.to_dict()
assert d["bboxes"] == [{"page": 1, "x": 10.5, "y": 20.0, "w": 100.0, "h": 50.0}]
assert d["headings"] == ["H1"]
assert d["doc_items"] == [{"self_ref": "#/texts/0", "label": "text"}]
assert d["origin"] == {"binary_hash": "abc", "filename": "test.pdf"}
def test_frozen(self):
chunk = self._make_chunk()
with pytest.raises(AttributeError):
chunk.content = "modified" # type: ignore[misc]
class TestBuildIndexMapping:
def test_default_dimension(self):
mapping = build_index_mapping()
props = mapping["mappings"]["properties"]
assert props["embedding"]["dimension"] == 384
assert props["embedding"]["type"] == "knn_vector"
assert props["embedding"]["method"]["engine"] == "faiss"
assert props["embedding"]["method"]["name"] == "hnsw"
def test_custom_dimension(self):
mapping = build_index_mapping(embedding_dimension=768)
assert mapping["mappings"]["properties"]["embedding"]["dimension"] == 768
def test_knn_enabled(self):
mapping = build_index_mapping()
assert mapping["settings"]["index"]["knn"] is True
def test_all_fields_present(self):
mapping = build_index_mapping()
props = mapping["mappings"]["properties"]
expected_fields = {
"doc_id",
"filename",
"content",
"embedding",
"chunk_index",
"chunk_type",
"page_number",
"bboxes",
"headings",
"doc_items",
"origin",
}
assert set(props.keys()) == expected_fields
def test_bboxes_nested_type(self):
mapping = build_index_mapping()
bboxes = mapping["mappings"]["properties"]["bboxes"]
assert bboxes["type"] == "nested"
assert "page" in bboxes["properties"]
assert "x" in bboxes["properties"]
assert "y" in bboxes["properties"]
assert "w" in bboxes["properties"]
assert "h" in bboxes["properties"]
def test_doc_items_nested_type(self):
mapping = build_index_mapping()
doc_items = mapping["mappings"]["properties"]["doc_items"]
assert doc_items["type"] == "nested"
assert "self_ref" in doc_items["properties"]
assert "label" in doc_items["properties"]
def test_origin_object_type(self):
mapping = build_index_mapping()
origin = mapping["mappings"]["properties"]["origin"]
assert origin["type"] == "object"
assert "binary_hash" in origin["properties"]
assert "filename" in origin["properties"]
def test_content_uses_standard_analyzer(self):
mapping = build_index_mapping()
content = mapping["mappings"]["properties"]["content"]
assert content["type"] == "text"
assert content["analyzer"] == "standard"
def test_keyword_fields(self):
mapping = build_index_mapping()
props = mapping["mappings"]["properties"]
for field_name in ("doc_id", "filename", "chunk_type"):
assert props[field_name]["type"] == "keyword", f"{field_name} should be keyword"
def test_integer_fields(self):
mapping = build_index_mapping()
props = mapping["mappings"]["properties"]
for field_name in ("chunk_index", "page_number"):
assert props[field_name]["type"] == "integer", f"{field_name} should be integer"
class TestSearchResult:
def test_construction(self):
chunk = IndexedChunk(
doc_id="doc-1",
filename="test.pdf",
content="Hello",
embedding=[0.1] * 384,
chunk_index=0,
chunk_type="text",
page_number=1,
)
result = SearchResult(chunk=chunk, score=0.95)
assert result.chunk.content == "Hello"
assert result.score == 0.95
def test_frozen(self):
chunk = IndexedChunk(
doc_id="d",
filename="f",
content="c",
embedding=[],
chunk_index=0,
chunk_type="text",
page_number=1,
)
result = SearchResult(chunk=chunk, score=0.5)
with pytest.raises(AttributeError):
result.score = 0.9 # type: ignore[misc]
class TestConstants:
def test_default_embedding_dimension(self):
assert DEFAULT_EMBEDDING_DIMENSION == 384
def test_default_index_name(self):
assert DEFAULT_INDEX_NAME == "docling-studio-chunks"

113
document-parser/tests/test_vector_store_port.py Normal file
View file

@ -0,0 +1,113 @@
"""Tests for VectorStore port — verify the protocol contract is implementable."""
from __future__ import annotations
import pytest
from domain.ports import VectorStore
from domain.vector_schema import IndexedChunk, SearchResult
class FakeVectorStore:
"""Minimal concrete implementation to verify the protocol is implementable."""
async def ensure_index(self, index_name: str, mapping: dict) -> None:
pass
async def index_chunks(self, index_name: str, chunks: list[IndexedChunk]) -> int:
return len(chunks)
async def search_similar(
self,
index_name: str,
embedding: list[float],
*,
k: int = 10,
doc_id: str | None = None,
) -> list[SearchResult]:
return []
async def get_chunks(
self,
index_name: str,
doc_id: str,
*,
limit: int = 1000,
) -> list[SearchResult]:
return []
async def delete_document(self, index_name: str, doc_id: str) -> int:
return 0
class TestVectorStorePort:
def test_fake_satisfies_protocol(self):
"""A class implementing all methods is accepted as a VectorStore."""
store: VectorStore = FakeVectorStore()
assert store is not None
@pytest.mark.asyncio
async def test_ensure_index(self):
store = FakeVectorStore()
await store.ensure_index("test-index", {"mappings": {}})
@pytest.mark.asyncio
async def test_index_chunks(self):
store = FakeVectorStore()
chunk = IndexedChunk(
doc_id="d1",
filename="test.pdf",
content="Hello",
embedding=[0.1] * 384,
chunk_index=0,
chunk_type="text",
page_number=1,
)
count = await store.index_chunks("test-index", [chunk])
assert count == 1
@pytest.mark.asyncio
async def test_search_similar(self):
store = FakeVectorStore()
results = await store.search_similar("test-index", [0.1] * 384, k=5)
assert results == []
@pytest.mark.asyncio
async def test_search_similar_with_doc_filter(self):
store = FakeVectorStore()
results = await store.search_similar("test-index", [0.1] * 384, k=5, doc_id="d1")
assert results == []
@pytest.mark.asyncio
async def test_get_chunks(self):
store = FakeVectorStore()
results = await store.get_chunks("test-index", "d1")
assert results == []
@pytest.mark.asyncio
async def test_get_chunks_with_limit(self):
store = FakeVectorStore()
results = await store.get_chunks("test-index", "d1", limit=50)
assert results == []
@pytest.mark.asyncio
async def test_delete_document(self):
store = FakeVectorStore()
count = await store.delete_document("test-index", "d1")
assert count == 0
def test_protocol_methods_list(self):
"""Verify the protocol exposes the expected methods."""
expected = {
"ensure_index",
"index_chunks",
"search_similar",
"get_chunks",
"delete_document",
}
protocol_methods = {
name
for name in dir(VectorStore)
if not name.startswith("_") and callable(getattr(VectorStore, name, None))
}
assert expected.issubset(protocol_methods)

59
e2e/api/src/test/resources/ingestion/ingest-and-verify.feature Normal file
View file

@ -0,0 +1,59 @@
@e2e @ingestion
Feature: Ingestion pipeline — PDF → chunks → embeddings → OpenSearch
Background:
* url baseUrl
Scenario: Upload PDF, analyze with chunking, ingest into OpenSearch, verify
# Step 1: Check ingestion is available
Given path '/api/ingestion/status'
When method GET
Then status 200
And match response.available == true
# Step 2: Upload a PDF
Given path '/api/documents/upload'
And multipart file file = { read: 'classpath:common/data/generated/medium.pdf', filename: 'medium.pdf', contentType: 'application/pdf' }
When method POST
Then status 200
* def docId = response.id
# Step 3: Create analysis with chunking
Given path '/api/analyses'
And request { documentId: '#(docId)', pipelineOptions: { doOcr: true, tableMode: 'fast' }, chunkingOptions: { chunkerType: 'hybrid', maxTokens: 256 } }
When method POST
Then status 200
* def jobId = response.id
# Step 4: Poll until completed
Given path '/api/analyses', jobId
And retry until response.status == 'COMPLETED' || response.status == 'FAILED'
When method GET
Then status 200
And match response.status == 'COMPLETED'
And match response.chunksJson == '#string'
# Step 5: Trigger ingestion
Given path '/api/ingestion', jobId
When method POST
Then status 200
And match response.docId == docId
And match response.chunksIndexed == '#number'
And assert response.chunksIndexed > 0
And match response.embeddingDimension == '#number'
And assert response.embeddingDimension > 0
# Step 6: Cleanup — delete ingested data
Given path '/api/ingestion', docId
When method DELETE
Then status 204
# Step 7: Cleanup — delete analysis and document
Given path '/api/analyses', jobId
When method DELETE
Then status 204
Given path '/api/documents', docId
When method DELETE
Then status 204

2
e2e/ui/src/test/resources/analyses/analysis.feature
View file

@ -18,7 +18,7 @@ Feature: UI — Launch an analysis and verify results
* waitFor('[data-e2e=doc-item].selected')
# Verify we are in Configure mode (first toggle button is active)
* waitFor('[data-e2e=toggle-btn].active')
* waitFor('[data-e2e~=configure-btn].active')
# Click Run / Exécuter
* waitFor('[data-e2e=run-btn]')

7
e2e/ui/src/test/resources/analyses/pipeline-options.feature
View file

@ -51,10 +51,9 @@ Feature: UI — Pipeline configuration options
* select('[data-e2e=config-select]', 'fast')
# Switch to Verify mode and back
* def toggleBtns = locateAll('[data-e2e=toggle-btn]')
* toggleBtns[1].click()
* waitFor('[data-e2e=toggle-btn].active')
* toggleBtns[0].click()
* click('[data-e2e~=verify-btn]')
* waitFor('[data-e2e~=verify-btn].active')
* click('[data-e2e~=configure-btn]')
* waitFor('[data-e2e=config-select]')
# Verify table mode is still fast

6
e2e/ui/src/test/resources/analyses/rechunk.feature
View file

@ -25,9 +25,9 @@ Feature: UI — Rechunk an analysis with different parameters
* call read('classpath:common/helpers/ui-wait-analysis.feature')
* waitFor('[data-e2e=result-tabs]')
# Now Préparer toggle should be enabled — click the last one
* def toggleBtns = locateAll('[data-e2e=toggle-btn]')
* toggleBtns[karate.sizeOf(toggleBtns) - 1].click()
# Switch to Prepare tab — use dedicated selector (avoids race with feature flag load)
* waitFor('[data-e2e~=prepare-btn]')
* click('[data-e2e~=prepare-btn]')
# Wait for chunk panel to load
* waitFor('[data-e2e=chunk-panel]')

9
e2e/ui/src/test/resources/workflows/full-ui-path.feature
View file

@ -22,7 +22,7 @@ Feature: UI — Full happy path via browser
* waitFor('[data-e2e=doc-item].selected')
# Step 5: Verify Configure mode is active
* waitFor('[data-e2e=toggle-btn].active')
* waitFor('[data-e2e~=configure-btn].active')
# Step 6: Run the analysis
* click('[data-e2e=run-btn]')
@ -46,8 +46,8 @@ Feature: UI — Full happy path via browser
* match text('[data-e2e=raw-content]') != ''
# Step 9: Switch to Préparer mode and rechunk
* def toggleBtns = locateAll('[data-e2e=toggle-btn]')
* toggleBtns[karate.sizeOf(toggleBtns) - 1].click()
* waitFor('[data-e2e~=prepare-btn]')
* click('[data-e2e~=prepare-btn]')
* waitFor('[data-e2e=chunk-panel]')
# Expand config if needed
@ -66,8 +66,7 @@ Feature: UI — Full happy path via browser
* assert karate.sizeOf(locateAll('[data-e2e=chunk-card]')) > 0
# Step 10: Delete the document via UI
* def toggleBtns2 = locateAll('[data-e2e=toggle-btn]')
* toggleBtns2[0].click()
* click('[data-e2e~=configure-btn]')
* waitFor('[data-e2e=doc-item]')
# Hover and delete

19
embedding-service/Dockerfile Normal file
View file

@ -0,0 +1,19 @@
FROM python:3.12-slim
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
# Install dependencies first (cache layer)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Pre-download default model into the image
ARG EMBEDDING_MODEL=all-MiniLM-L6-v2
RUN python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('${EMBEDDING_MODEL}')"
COPY main.py .
EXPOSE 8001
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8001"]

89
embedding-service/main.py Normal file
View file

@ -0,0 +1,89 @@
"""Embedding microservice — exposes sentence-transformers models via REST API.
POST /embed {"texts": ["...", "..."]} → {"embeddings": [[...], [...]], "model": "...", "dimension": N}
GET /health → {"status": "ok", "model": "...", "dimension": N}
"""
from __future__ import annotations
import logging
import os
import time
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from sentence_transformers import SentenceTransformer
logger = logging.getLogger(__name__)
MODEL_NAME = os.environ.get("EMBEDDING_MODEL", "all-MiniLM-L6-v2")
BATCH_SIZE = int(os.environ.get("EMBEDDING_BATCH_SIZE", "64"))
app = FastAPI(title="Docling Studio — Embedding Service", version="0.4.0")
# Load model at startup (downloaded / cached in HF cache dir)
model: SentenceTransformer | None = None
@app.on_event("startup")
async def _load_model() -> None:
global model # noqa: PLW0603
logger.info("Loading sentence-transformers model '%s' …", MODEL_NAME)
t0 = time.monotonic()
model = SentenceTransformer(MODEL_NAME)
elapsed = time.monotonic() - t0
dim = model.get_sentence_embedding_dimension()
logger.info("Model loaded in %.1fs — dimension=%d", elapsed, dim)
# -- Schemas -------------------------------------------------------------------
class EmbedRequest(BaseModel):
texts: list[str] = Field(..., min_length=1, description="Texts to embed")
class EmbedResponse(BaseModel):
embeddings: list[list[float]]
model: str
dimension: int
class HealthResponse(BaseModel):
status: str
model: str
dimension: int
# -- Endpoints -----------------------------------------------------------------
@app.post("/embed", response_model=EmbedResponse)
async def embed(request: EmbedRequest) -> EmbedResponse:
"""Generate embeddings for a batch of texts."""
if model is None:
raise HTTPException(status_code=503, detail="Model not loaded yet")
vectors = model.encode(
request.texts,
batch_size=BATCH_SIZE,
show_progress_bar=False,
normalize_embeddings=True,
)
return EmbedResponse(
embeddings=vectors.tolist(),
model=MODEL_NAME,
dimension=model.get_sentence_embedding_dimension(),
)
@app.get("/health", response_model=HealthResponse)
async def health() -> HealthResponse:
"""Health check — verifies the model is loaded."""
if model is None:
raise HTTPException(status_code=503, detail="Model not loaded yet")
return HealthResponse(
status="ok",
model=MODEL_NAME,
dimension=model.get_sentence_embedding_dimension(),
)

3
embedding-service/requirements.txt Normal file
View file

@ -0,0 +1,3 @@
fastapi>=0.115.0,<1.0.0
uvicorn[standard]>=0.32.0,<1.0.0
sentence-transformers>=3.0.0,<4.0.0

64
embedding-service/test_main.py Normal file
View file

@ -0,0 +1,64 @@
"""Tests for the embedding microservice API."""
from __future__ import annotations
from unittest.mock import MagicMock, patch
import numpy as np
import pytest
from fastapi.testclient import TestClient
import main
@pytest.fixture(autouse=True)
def _mock_model() -> None:
"""Inject a mock SentenceTransformer model for all tests."""
mock = MagicMock()
mock.get_sentence_embedding_dimension.return_value = 3
mock.encode.return_value = np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]])
main.model = mock
yield
main.model = None
@pytest.fixture
def client() -> TestClient:
return TestClient(main.app)
class TestEmbed:
def test_embed_returns_vectors(self, client: TestClient) -> None:
resp = client.post("/embed", json={"texts": ["hello", "world"]})
assert resp.status_code == 200
data = resp.json()
assert len(data["embeddings"]) == 2
assert data["dimension"] == 3
assert data["model"] == main.MODEL_NAME
def test_embed_empty_texts_rejected(self, client: TestClient) -> None:
resp = client.post("/embed", json={"texts": []})
assert resp.status_code == 422
def test_embed_missing_texts(self, client: TestClient) -> None:
resp = client.post("/embed", json={})
assert resp.status_code == 422
def test_embed_model_not_loaded(self, client: TestClient) -> None:
main.model = None
resp = client.post("/embed", json={"texts": ["test"]})
assert resp.status_code == 503
class TestHealth:
def test_health_ok(self, client: TestClient) -> None:
resp = client.get("/health")
assert resp.status_code == 200
data = resp.json()
assert data["status"] == "ok"
assert data["dimension"] == 3
def test_health_model_not_loaded(self, client: TestClient) -> None:
main.model = None
resp = client.get("/health")
assert resp.status_code == 503

4
frontend/package-lock.json generated
View file

@ -1,12 +1,12 @@
{
"name": "docling-studio",
"version": "0.3.0",
"version": "0.3.1",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "docling-studio",
"version": "0.3.0",
"version": "0.3.1",
"dependencies": {
"dompurify": "^3.3.3",
"marked": "^17.0.4",

2
frontend/package.json
View file

@ -1,6 +1,6 @@
{
"name": "docling-studio",
"version": "0.3.1",
"version": "0.4.0",
"private": true,
"type": "module",
"scripts": {

5
frontend/src/app/router/index.ts
View file

@ -22,6 +22,11 @@ const routes: RouteRecordRaw[] = [
name: 'documents',
component: () => import('../../pages/DocumentsPage.vue'),
},
{
path: '/search',
name: 'search',
component: () => import('../../pages/SearchPage.vue'),
},
{
path: '/settings',
name: 'settings',

10
frontend/src/features/analysis/store.ts
View file

@ -111,6 +111,15 @@ export const useAnalysisStore = defineStore('analysis', () => {
}
}
function updateChunks(chunks: Chunk[]): void {
if (currentAnalysis.value) {
currentAnalysis.value = {
...currentAnalysis.value,
chunksJson: JSON.stringify(chunks),
}
}
}
async function select(id: string): Promise<void> {
try {
currentAnalysis.value = await api.fetchAnalysis(id)
@ -142,6 +151,7 @@ export const useAnalysisStore = defineStore('analysis', () => {
load,
run,
select,
updateChunks,
remove,
stopPolling,
}

31
frontend/src/features/chunking/api.test.ts
View file

@ -1,5 +1,5 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { rechunkAnalysis } from './api'
import { rechunkAnalysis, updateChunkText, deleteChunk } from './api'
vi.mock('../../shared/api/http', () => ({
apiFetch: vi.fn(),
@ -25,4 +25,33 @@ describe('chunking API', () => {
})
expect(result).toEqual(chunks)
})
it('updateChunkText sends PATCH to chunk endpoint', async () => {
const chunks = [
{ text: 'updated', headings: [], sourcePage: 1, tokenCount: 10, bboxes: [], modified: true },
]
apiFetch.mockResolvedValue(chunks)
const result = await updateChunkText('job-1', 0, 'updated')
expect(apiFetch).toHaveBeenCalledWith('/api/analyses/job-1/chunks/0', {
method: 'PATCH',
body: JSON.stringify({ text: 'updated' }),
})
expect(result).toEqual(chunks)
})
it('deleteChunk sends DELETE to chunk endpoint', async () => {
const chunks = [
{ text: 'chunk1', headings: [], sourcePage: 1, tokenCount: 10, bboxes: [], deleted: true },
]
apiFetch.mockResolvedValue(chunks)
const result = await deleteChunk('job-1', 0)
expect(apiFetch).toHaveBeenCalledWith('/api/analyses/job-1/chunks/0', {
method: 'DELETE',
})
expect(result).toEqual(chunks)
})
})

13
frontend/src/features/chunking/api.ts
View file

@ -7,3 +7,16 @@ export function rechunkAnalysis(jobId: string, chunkingOptions: ChunkingOptions)
body: JSON.stringify({ chunkingOptions }),
})
}
export function updateChunkText(jobId: string, chunkIndex: number, text: string): Promise<Chunk[]> {
return apiFetch<Chunk[]>(`/api/analyses/${jobId}/chunks/${chunkIndex}`, {
method: 'PATCH',
body: JSON.stringify({ text }),
})
}
export function deleteChunk(jobId: string, chunkIndex: number): Promise<Chunk[]> {
return apiFetch<Chunk[]>(`/api/analyses/${jobId}/chunks/${chunkIndex}`, {
method: 'DELETE',
})
}

88
frontend/src/features/chunking/store.test.ts
View file

@ -4,6 +4,8 @@ import { useChunkingStore } from './store'
vi.mock('./api', () => ({
rechunkAnalysis: vi.fn(),
updateChunkText: vi.fn(),
deleteChunk: vi.fn(),
}))
import * as api from './api'
@ -17,6 +19,8 @@ describe('useChunkingStore', () => {
it('starts with default state', () => {
const store = useChunkingStore()
expect(store.rechunking).toBe(false)
expect(store.saving).toBe(false)
expect(store.deleting).toBe(false)
expect(store.error).toBeNull()
})
@ -62,4 +66,88 @@ describe('useChunkingStore', () => {
expect(store.rechunking).toBe(false)
expect(store.error).toBe('fail')
})
it('updateChunkText calls API and returns chunks', async () => {
const chunks = [
{ text: 'updated', headings: [], sourcePage: 1, tokenCount: 5, bboxes: [], modified: true },
]
vi.mocked(api.updateChunkText).mockResolvedValue(chunks)
const store = useChunkingStore()
const result = await store.updateChunkText('j1', 0, 'updated')
expect(api.updateChunkText).toHaveBeenCalledWith('j1', 0, 'updated')
expect(result).toEqual(chunks)
expect(store.saving).toBe(false)
})
it('updateChunkText sets saving during execution', async () => {
let resolve: (v: any) => void
vi.mocked(api.updateChunkText).mockImplementation(
() =>
new Promise((r) => {
resolve = r
}),
)
const store = useChunkingStore()
const promise = store.updateChunkText('j1', 0, 'updated')
expect(store.saving).toBe(true)
resolve!([])
await promise
expect(store.saving).toBe(false)
})
it('updateChunkText handles errors', async () => {
vi.mocked(api.updateChunkText).mockRejectedValue(new Error('save failed'))
vi.spyOn(console, 'error').mockImplementation(() => {})
const store = useChunkingStore()
await expect(store.updateChunkText('j1', 0, 'text')).rejects.toThrow('save failed')
expect(store.saving).toBe(false)
expect(store.error).toBe('save failed')
})
it('deleteChunk calls API and returns chunks', async () => {
const chunks = [
{ text: 'chunk1', headings: [], sourcePage: 1, tokenCount: 5, bboxes: [], deleted: true },
]
vi.mocked(api.deleteChunk).mockResolvedValue(chunks)
const store = useChunkingStore()
const result = await store.deleteChunk('j1', 0)
expect(api.deleteChunk).toHaveBeenCalledWith('j1', 0)
expect(result).toEqual(chunks)
expect(store.deleting).toBe(false)
})
it('deleteChunk sets deleting during execution', async () => {
let resolve: (v: any) => void
vi.mocked(api.deleteChunk).mockImplementation(
() =>
new Promise((r) => {
resolve = r
}),
)
const store = useChunkingStore()
const promise = store.deleteChunk('j1', 0)
expect(store.deleting).toBe(true)
resolve!([])
await promise
expect(store.deleting).toBe(false)
})
it('deleteChunk handles errors', async () => {
vi.mocked(api.deleteChunk).mockRejectedValue(new Error('delete failed'))
vi.spyOn(console, 'error').mockImplementation(() => {})
const store = useChunkingStore()
await expect(store.deleteChunk('j1', 0)).rejects.toThrow('delete failed')
expect(store.deleting).toBe(false)
expect(store.error).toBe('delete failed')
})
})

36
frontend/src/features/chunking/store.ts
View file

@ -5,6 +5,8 @@ import * as api from './api'
export const useChunkingStore = defineStore('chunking', () => {
const rechunking = ref(false)
const saving = ref(false)
const deleting = ref(false)
const error = ref<string | null>(null)
async function rechunk(jobId: string, chunkingOptions: ChunkingOptions): Promise<Chunk[]> {
@ -21,5 +23,37 @@ export const useChunkingStore = defineStore('chunking', () => {
}
}
return { rechunking, error, rechunk }
async function updateChunkText(
jobId: string,
chunkIndex: number,
text: string,
): Promise<Chunk[]> {
saving.value = true
error.value = null
try {
return await api.updateChunkText(jobId, chunkIndex, text)
} catch (e) {
error.value = (e as Error).message || 'Failed to update chunk'
console.error('Failed to update chunk', e)
throw e
} finally {
saving.value = false
}
}
async function deleteChunk(jobId: string, chunkIndex: number): Promise<Chunk[]> {
deleting.value = true
error.value = null
try {
return await api.deleteChunk(jobId, chunkIndex)
} catch (e) {
error.value = (e as Error).message || 'Failed to delete chunk'
console.error('Failed to delete chunk', e)
throw e
} finally {
deleting.value = false
}
}
return { rechunking, saving, deleting, error, rechunk, updateChunkText, deleteChunk }
})

334
frontend/src/features/chunking/ui/ChunkPanel.vue
View file

@ -84,7 +84,7 @@
<!-- Chunks list -->
<div class="chunk-results" data-e2e="chunk-results" v-if="pageChunks.length">
<div class="chunk-summary" data-e2e="chunk-summary">
{{ pagination.totalItems.value }} {{ t('chunking.chunks') }}
{{ activeChunks.length }} {{ t('chunking.chunks') }}
</div>
<div class="chunk-list">
<div
@ -102,16 +102,110 @@
{{ chunk.tokenCount }} tokens
</span>
<span class="chunk-page" v-if="chunk.sourcePage"> p.{{ chunk.sourcePage }} </span>
<span v-if="chunk.modified" class="chunk-modified" data-e2e="chunk-modified">
{{ t('chunking.modified') }}
</span>
<button
v-if="editingIdx !== globalIndex(localIdx)"
class="chunk-edit-icon"
data-e2e="chunk-edit-btn"
:title="t('chunking.edit')"
@click.stop="startEdit(globalIndex(localIdx), chunk.text)"
>
<svg viewBox="0 0 20 20" fill="currentColor" width="14" height="14">
<path
d="M13.586 3.586a2 2 0 112.828 2.828l-.793.793-2.828-2.828.793-.793zM11.379 5.793L3 14.172V17h2.828l8.38-8.379-2.83-2.828z"
/>
</svg>
</button>
<button
class="chunk-delete-icon"
data-e2e="chunk-delete-btn"
:title="t('chunking.delete')"
@click.stop="confirmDelete(globalIndex(localIdx))"
>
<svg viewBox="0 0 20 20" fill="currentColor" width="14" height="14">
<path
fill-rule="evenodd"
d="M9 2a1 1 0 00-.894.553L7.382 4H4a1 1 0 000 2v10a2 2 0 002 2h8a2 2 0 002-2V6a1 1 0 100-2h-3.382l-.724-1.447A1 1 0 0011 2H9zM7 8a1 1 0 012 0v6a1 1 0 11-2 0V8zm5-1a1 1 0 00-1 1v6a1 1 0 102 0V8a1 1 0 00-1-1z"
clip-rule="evenodd"
/>
</svg>
</button>
</div>
<div class="chunk-headings" v-if="chunk.headings.length">
<span class="chunk-heading" v-for="h in chunk.headings" :key="h">{{ h }}</span>
</div>
<div class="chunk-text" data-e2e="chunk-text">{{ chunk.text }}</div>
<!-- Edit mode -->
<div v-if="editingIdx === globalIndex(localIdx)" class="chunk-edit">
<textarea
ref="editTextarea"
class="chunk-edit-textarea"
data-e2e="chunk-edit-textarea"
v-model="editText"
rows="6"
/>
<div class="chunk-edit-actions">
<button
class="chunk-edit-btn save"
data-e2e="chunk-edit-save"
:disabled="chunkingStore.saving"
@click="saveEdit(globalIndex(localIdx))"
>
{{ chunkingStore.saving ? t('chunking.saving') : t('chunking.save') }}
</button>
<button
class="chunk-edit-btn cancel"
data-e2e="chunk-edit-cancel"
@click="cancelEdit"
>
{{ t('chunking.cancel') }}
</button>
</div>
</div>
<!-- Read mode -->
<div
v-else
class="chunk-text"
data-e2e="chunk-text"
@dblclick="startEdit(globalIndex(localIdx), chunk.text)"
>
{{ chunk.text }}
</div>
</div>
</div>
</div>
<div class="chunk-empty" v-else-if="!chunkingStore.rechunking">
<!-- Delete confirmation dialog -->
<div v-if="deleteConfirmIdx !== -1" class="chunk-confirm-overlay" data-e2e="chunk-confirm">
<div class="chunk-confirm-dialog">
<p class="chunk-confirm-text">{{ t('chunking.deleteConfirm') }}</p>
<div class="chunk-confirm-actions">
<button
class="chunk-confirm-btn danger"
data-e2e="chunk-confirm-yes"
:disabled="chunkingStore.deleting"
@click="doDelete"
>
{{ chunkingStore.deleting ? t('chunking.deleting') : t('chunking.delete') }}
</button>
<button
class="chunk-confirm-btn cancel"
data-e2e="chunk-confirm-no"
@click="deleteConfirmIdx = -1"
>
{{ t('chunking.cancel') }}
</button>
</div>
</div>
</div>
<div
class="chunk-empty"
v-if="!pageChunks.length && !chunkingStore.rechunking && deleteConfirmIdx === -1"
>
<p>
{{ chunks.length ? t('chunking.noChunksOnPage') : t('chunking.noChunks') }}
</p>
@ -131,6 +225,7 @@
<script setup lang="ts">
import { ref, reactive, computed } from 'vue'
import { useChunkingStore } from '../store'
import { useAnalysisStore } from '../../analysis/store'
import { useI18n } from '../../../shared/i18n'
import { usePagination } from '../../../shared/composables/usePagination'
import { PaginationBar } from '../../../shared/ui'
@ -146,10 +241,10 @@ const props = defineProps<{
const emit = defineEmits<{
'highlight-bboxes': [bboxes: ChunkBbox[]]
rechunked: []
}>()
const chunkingStore = useChunkingStore()
const analysisStore = useAnalysisStore()
const { t } = useI18n()
const configOpen = ref(true)
@ -170,11 +265,55 @@ const isBatchedAnalysis = computed(() => {
return props.analysisStatus === 'COMPLETED' && !props.hasDocumentJson
})
const pageChunks = computed(() => props.chunks.filter((c) => c.sourcePage === props.currentPage))
const activeChunks = computed(() => props.chunks.filter((c) => !c.deleted))
const pageChunks = computed(() =>
activeChunks.value.filter((c) => c.sourcePage === props.currentPage),
)
const pagination = usePagination(pageChunks, { pageSize: 20 })
function globalIndex(localIdx: number): number {
return (pagination.page.value - 1) * pagination.pageSize.value + localIdx
const pageLocalIdx = (pagination.page.value - 1) * pagination.pageSize.value + localIdx
const pageChunk = pageChunks.value[pageLocalIdx]
return props.chunks.indexOf(pageChunk)
}
const deleteConfirmIdx = ref(-1)
function confirmDelete(chunkIndex: number) {
deleteConfirmIdx.value = chunkIndex
}
async function doDelete() {
if (!props.analysisId || deleteConfirmIdx.value === -1) return
const chunks = await chunkingStore.deleteChunk(props.analysisId, deleteConfirmIdx.value)
deleteConfirmIdx.value = -1
analysisStore.updateChunks(chunks)
}
const editingIdx = ref(-1)
const editText = ref('')
function startEdit(idx: number, text: string) {
editingIdx.value = idx
editText.value = text
}
function cancelEdit() {
editingIdx.value = -1
editText.value = ''
}
async function saveEdit(chunkIndex: number) {
if (!props.analysisId) return
const allChunks = props.chunks
const originalText = allChunks[chunkIndex]?.text
if (editText.value === originalText) {
cancelEdit()
return
}
const chunks = await chunkingStore.updateChunkText(props.analysisId, chunkIndex, editText.value)
analysisStore.updateChunks(chunks)
cancelEdit()
}
const hoveredChunkIdx = ref(-1)
@ -192,8 +331,8 @@ function onChunkLeave() {
async function doRechunk() {
if (!props.analysisId) return
await chunkingStore.rechunk(props.analysisId, { ...options })
emit('rechunked')
const chunks = await chunkingStore.rechunk(props.analysisId, { ...options })
analysisStore.updateChunks(chunks)
}
</script>
@ -203,6 +342,7 @@ async function doRechunk() {
flex-direction: column;
height: 100%;
overflow: hidden;
position: relative;
}
.chunk-config {
@ -425,6 +565,183 @@ async function doRechunk() {
border-radius: 4px;
}
.chunk-modified {
font-size: 10px;
font-weight: 600;
color: #f59e0b;
background: rgba(245, 158, 11, 0.12);
padding: 1px 6px;
border-radius: 4px;
text-transform: uppercase;
letter-spacing: 0.3px;
}
.chunk-edit-icon {
margin-left: auto;
background: none;
border: none;
color: var(--text-secondary);
cursor: pointer;
padding: 2px;
border-radius: 4px;
display: flex;
align-items: center;
opacity: 0;
transition: opacity 0.15s;
}
.chunk-delete-icon {
background: none;
border: none;
color: var(--text-secondary);
cursor: pointer;
padding: 2px;
border-radius: 4px;
display: flex;
align-items: center;
opacity: 0;
transition: opacity 0.15s;
}
.chunk-card:hover .chunk-edit-icon {
opacity: 1;
}
.chunk-edit-icon:hover {
color: var(--accent);
background: var(--bg-tertiary);
}
.chunk-card:hover .chunk-delete-icon {
opacity: 1;
}
.chunk-delete-icon:hover {
color: #ef4444;
background: rgba(239, 68, 68, 0.1);
}
.chunk-edit {
display: flex;
flex-direction: column;
gap: 8px;
}
.chunk-edit-textarea {
width: 100%;
font-size: 12px;
font-family: inherit;
line-height: 1.5;
color: var(--text);
background: var(--bg);
border: 1px solid var(--accent);
border-radius: var(--radius-sm, 4px);
padding: 8px;
resize: vertical;
box-sizing: border-box;
}
.chunk-edit-textarea:focus {
outline: none;
border-color: var(--accent);
box-shadow: 0 0 0 2px rgba(99, 102, 241, 0.2);
}
.chunk-edit-actions {
display: flex;
gap: 6px;
justify-content: flex-end;
}
.chunk-edit-btn {
padding: 4px 12px;
border: none;
border-radius: var(--radius-sm, 4px);
font-size: 12px;
font-weight: 500;
cursor: pointer;
}
.chunk-edit-btn.save {
background: var(--accent);
color: white;
}
.chunk-edit-btn.save:disabled {
opacity: 0.5;
cursor: not-allowed;
}
.chunk-edit-btn.cancel {
background: var(--bg-tertiary);
color: var(--text-secondary);
}
.chunk-edit-btn.cancel:hover {
color: var(--text);
}
.chunk-confirm-overlay {
position: absolute;
inset: 0;
background: rgba(0, 0, 0, 0.4);
display: flex;
align-items: center;
justify-content: center;
z-index: 10;
}
.chunk-confirm-dialog {
background: var(--bg);
border: 1px solid var(--border);
border-radius: var(--radius);
padding: 20px;
max-width: 300px;
width: 90%;
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.15);
}
.chunk-confirm-text {
font-size: 13px;
color: var(--text);
margin: 0 0 16px;
line-height: 1.5;
}
.chunk-confirm-actions {
display: flex;
gap: 8px;
justify-content: flex-end;
}
.chunk-confirm-btn {
padding: 6px 14px;
border: none;
border-radius: var(--radius-sm, 4px);
font-size: 12px;
font-weight: 500;
cursor: pointer;
}
.chunk-confirm-btn.danger {
background: #ef4444;
color: white;
}
.chunk-confirm-btn.danger:disabled {
opacity: 0.5;
cursor: not-allowed;
}
.chunk-confirm-btn.cancel {
background: var(--bg-tertiary);
color: var(--text-secondary);
}
.chunk-confirm-btn.cancel:hover {
color: var(--text);
}
.chunk-text {
font-size: 12px;
color: var(--text);
@ -433,6 +750,7 @@ async function doRechunk() {
word-break: break-word;
max-height: 120px;
overflow-y: auto;
cursor: text;
}
.chunk-empty {

32
frontend/src/features/feature-flags/store.test.ts
View file

@ -82,6 +82,38 @@ describe('useFeatureFlagStore', () => {
expect(store.maxFileSizeMb).toBe(0)
})
it('enables ingestion when ingestionAvailable is true', async () => {
mockApiFetch.mockResolvedValue({
status: 'ok',
engine: 'local',
ingestionAvailable: true,
})
const store = useFeatureFlagStore()
await store.load()
expect(store.ingestionAvailable).toBe(true)
expect(store.isEnabled('ingestion')).toBe(true)
})
it('disables ingestion when ingestionAvailable is false', async () => {
mockApiFetch.mockResolvedValue({
status: 'ok',
engine: 'local',
ingestionAvailable: false,
})
const store = useFeatureFlagStore()
await store.load()
expect(store.ingestionAvailable).toBe(false)
expect(store.isEnabled('ingestion')).toBe(false)
})
it('defaults ingestionAvailable to false when missing', async () => {
mockApiFetch.mockResolvedValue({ status: 'ok', engine: 'local' })
const store = useFeatureFlagStore()
await store.load()
expect(store.ingestionAvailable).toBe(false)
expect(store.isEnabled('ingestion')).toBe(false)
})
it('handles health endpoint failure gracefully', async () => {
mockApiFetch.mockRejectedValue(new Error('Network error'))
const store = useFeatureFlagStore()

12
frontend/src/features/feature-flags/store.ts
View file

@ -13,9 +13,10 @@ interface HealthResponse {
deploymentMode?: DeploymentMode
maxPageCount?: number
maxFileSizeMb?: number
ingestionAvailable?: boolean
}
export type FeatureFlag = 'chunking' | 'disclaimer'
export type FeatureFlag = 'chunking' | 'disclaimer' | 'ingestion'
interface FeatureFlagDef {
description: string
@ -25,6 +26,7 @@ interface FeatureFlagDef {
interface FeatureFlagContext {
engine: ConversionEngine | null
deploymentMode: DeploymentMode | null
ingestionAvailable: boolean
}
const featureRegistry: Record<FeatureFlag, FeatureFlagDef> = {
@ -36,6 +38,10 @@ const featureRegistry: Record<FeatureFlag, FeatureFlagDef> = {
description: 'Show shared-instance disclaimer banner',
isEnabled: (ctx) => ctx.deploymentMode === 'huggingface',
},
ingestion: {
description: 'OpenSearch ingestion pipeline (embedding + vector indexing)',
isEnabled: (ctx) => ctx.ingestionAvailable,
},
}
export const useFeatureFlagStore = defineStore('feature-flags', () => {
@ -43,6 +49,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
const deploymentMode = ref<DeploymentMode | null>(null)
const maxPageCount = ref<number>(0)
const maxFileSizeMb = ref<number>(0)
const ingestionAvailable = ref(false)
const appVersion = ref<string>(__APP_VERSION__)
const loaded = ref(false)
const error = ref<string | null>(null)
@ -50,6 +57,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
const context = computed<FeatureFlagContext>(() => ({
engine: engine.value,
deploymentMode: deploymentMode.value,
ingestionAvailable: ingestionAvailable.value,
}))
function isEnabled(flag: FeatureFlag): boolean {
@ -65,6 +73,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
deploymentMode.value = data.deploymentMode ?? 'self-hosted'
maxPageCount.value = data.maxPageCount ?? 0
maxFileSizeMb.value = data.maxFileSizeMb ?? 0
ingestionAvailable.value = data.ingestionAvailable ?? false
appMaxFileSizeMb.value = maxFileSizeMb.value
appMaxPageCount.value = maxPageCount.value
if (data.version) appVersion.value = data.version
@ -81,6 +90,7 @@ export const useFeatureFlagStore = defineStore('feature-flags', () => {
deploymentMode,
maxPageCount,
maxFileSizeMb,
ingestionAvailable,
appVersion,
loaded,
error,

48
frontend/src/features/ingestion/api.test.ts Normal file
View file

@ -0,0 +1,48 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { ingestAnalysis, deleteIngested, fetchIngestionStatus } from './api'
const mockFetch = vi.fn()
vi.stubGlobal('fetch', mockFetch)
beforeEach(() => {
mockFetch.mockReset()
})
describe('ingestAnalysis', () => {
it('posts to /api/ingestion/:jobId', async () => {
mockFetch.mockResolvedValue({
ok: true,
status: 200,
json: () => Promise.resolve({ docId: 'doc-1', chunksIndexed: 5, embeddingDimension: 384 }),
})
const result = await ingestAnalysis('job-1')
expect(mockFetch).toHaveBeenCalledWith(
'/api/ingestion/job-1',
expect.objectContaining({ method: 'POST' }),
)
expect(result.chunksIndexed).toBe(5)
})
})
describe('deleteIngested', () => {
it('deletes /api/ingestion/:docId', async () => {
mockFetch.mockResolvedValue({ ok: true, status: 204, json: () => Promise.resolve(null) })
await deleteIngested('doc-1')
expect(mockFetch).toHaveBeenCalledWith(
'/api/ingestion/doc-1',
expect.objectContaining({ method: 'DELETE' }),
)
})
})
describe('fetchIngestionStatus', () => {
it('gets /api/ingestion/status', async () => {
mockFetch.mockResolvedValue({
ok: true,
status: 200,
json: () => Promise.resolve({ available: true }),
})
const result = await fetchIngestionStatus()
expect(result.available).toBe(true)
})
})

26
frontend/src/features/ingestion/api.ts Normal file
View file

@ -0,0 +1,26 @@
import { apiFetch } from '../../shared/api/http'
export interface IngestionResult {
docId: string
chunksIndexed: number
embeddingDimension: number
}
export interface IngestionStatus {
available: boolean
opensearchConnected: boolean
}
export function ingestAnalysis(jobId: string): Promise<IngestionResult> {
return apiFetch<IngestionResult>(`/api/ingestion/${jobId}`, {
method: 'POST',
})
}
export function deleteIngested(docId: string): Promise<unknown> {
return apiFetch(`/api/ingestion/${docId}`, { method: 'DELETE' })
}
export function fetchIngestionStatus(): Promise<IngestionStatus> {
return apiFetch<IngestionStatus>('/api/ingestion/status')
}

2
frontend/src/features/ingestion/index.ts Normal file
View file

@ -0,0 +1,2 @@
export { useIngestionStore } from './store'
export { default as IngestPanel } from './ui/IngestPanel.vue'

66
frontend/src/features/ingestion/store.test.ts Normal file
View file

@ -0,0 +1,66 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { setActivePinia, createPinia } from 'pinia'
import { useIngestionStore } from './store'
import * as api from './api'
vi.mock('./api', () => ({
fetchIngestionStatus: vi.fn(),
ingestAnalysis: vi.fn(),
deleteIngested: vi.fn(),
}))
beforeEach(() => {
setActivePinia(createPinia())
vi.clearAllMocks()
})
describe('useIngestionStore', () => {
describe('checkAvailability', () => {
it('sets available to true when API responds', async () => {
vi.mocked(api.fetchIngestionStatus).mockResolvedValue({ available: true })
const store = useIngestionStore()
await store.checkAvailability()
expect(store.available).toBe(true)
})
it('sets available to false on error', async () => {
vi.mocked(api.fetchIngestionStatus).mockRejectedValue(new Error('fail'))
const store = useIngestionStore()
await store.checkAvailability()
expect(store.available).toBe(false)
})
})
describe('ingest', () => {
it('calls API and tracks ingested doc', async () => {
vi.mocked(api.ingestAnalysis).mockResolvedValue({
docId: 'doc-1',
chunksIndexed: 5,
embeddingDimension: 384,
})
const store = useIngestionStore()
const result = await store.ingest('job-1')
expect(result?.chunksIndexed).toBe(5)
expect(store.ingestedDocs['doc-1']).toBe(5)
expect(store.ingesting).toBe(false)
})
it('sets error on failure', async () => {
vi.mocked(api.ingestAnalysis).mockRejectedValue(new Error('fail'))
const store = useIngestionStore()
const result = await store.ingest('job-1')
expect(result).toBeNull()
expect(store.error).toBe('fail')
})
})
describe('deleteIngested', () => {
it('removes doc from tracked map', async () => {
vi.mocked(api.deleteIngested).mockResolvedValue(null)
const store = useIngestionStore()
store.ingestedDocs['doc-1'] = 5
await store.deleteIngested('doc-1')
expect(store.ingestedDocs['doc-1']).toBeUndefined()
})
})
})

88
frontend/src/features/ingestion/store.ts Normal file
View file

@ -0,0 +1,88 @@
import { defineStore } from 'pinia'
import { ref } from 'vue'
import * as api from './api'
export type IngestionStep = 'embedding' | 'indexing' | 'done'
export const useIngestionStore = defineStore('ingestion', () => {
const available = ref(false)
const opensearchConnected = ref(false)
const ingesting = ref(false)
const error = ref<string | null>(null)
/** Map of docId → chunks indexed count (tracks which docs are ingested) */
const ingestedDocs = ref<Record<string, number>>({})
/** Current step of the ingestion pipeline (null when idle) */
const currentStep = ref<IngestionStep | null>(null)
let _pollTimer: ReturnType<typeof setInterval> | null = null
async function checkAvailability(): Promise<void> {
try {
const status = await api.fetchIngestionStatus()
available.value = status.available
opensearchConnected.value = status.opensearchConnected
} catch {
available.value = false
opensearchConnected.value = false
}
}
function startPolling(intervalMs = 30_000): void {
stopPolling()
_pollTimer = setInterval(checkAvailability, intervalMs)
}
function stopPolling(): void {
if (_pollTimer) {
clearInterval(_pollTimer)
_pollTimer = null
}
}
async function ingest(jobId: string): Promise<api.IngestionResult | null> {
ingesting.value = true
error.value = null
currentStep.value = 'embedding'
try {
currentStep.value = 'indexing'
const result = await api.ingestAnalysis(jobId)
currentStep.value = 'done'
ingestedDocs.value[result.docId] = result.chunksIndexed
return result
} catch (e) {
error.value = (e as Error).message || 'Ingestion failed'
console.error('Ingestion failed', e)
currentStep.value = null
return null
} finally {
ingesting.value = false
// Reset step after a short delay so the user sees the "done" state
setTimeout(() => {
currentStep.value = null
}, 2000)
}
}
async function deleteIngested(docId: string): Promise<void> {
try {
await api.deleteIngested(docId)
delete ingestedDocs.value[docId]
} catch (e) {
error.value = (e as Error).message || 'Failed to delete ingested data'
console.error('Failed to delete ingested data', e)
}
}
return {
available,
opensearchConnected,
ingesting,
error,
ingestedDocs,
currentStep,
checkAvailability,
startPolling,
stopPolling,
ingest,
deleteIngested,
}
})

346
frontend/src/features/ingestion/ui/IngestPanel.vue Normal file
View file

@ -0,0 +1,346 @@
<template>
<div class="ingest-panel" data-e2e="ingest-panel">
<!-- Unavailable state -->
<div v-if="!ingestionStore.available" class="ingest-empty">
<svg class="empty-icon" viewBox="0 0 20 20" fill="currentColor">
<path
fill-rule="evenodd"
d="M18 10a8 8 0 11-16 0 8 8 0 0116 0zm-7 4a1 1 0 11-2 0 1 1 0 012 0zm-1-9a1 1 0 00-1 1v4a1 1 0 102 0V6a1 1 0 00-1-1z"
clip-rule="evenodd"
/>
</svg>
<p class="empty-text">{{ t('ingestion.unavailable') }}</p>
</div>
<!-- Ready to ingest -->
<template v-else>
<!-- Summary -->
<div class="ingest-summary">
<div class="summary-row">
<span class="summary-label">{{ t('ingestion.document') }}</span>
<span class="summary-value">{{ documentName }}</span>
</div>
<div class="summary-row">
<span class="summary-label">{{ t('ingestion.chunkCount') }}</span>
<span class="summary-value summary-mono">{{ chunkCount }}</span>
</div>
</div>
<!-- Stepper -->
<div v-if="ingestionStore.currentStep" class="ingestion-stepper">
<div
class="step"
:class="{
active: ingestionStore.currentStep === 'embedding',
done:
ingestionStore.currentStep === 'indexing' || ingestionStore.currentStep === 'done',
}"
>
<span class="step-dot" />
<span class="step-label">{{ t('ingestion.stepEmbedding') }}</span>
</div>
<div
class="step-line"
:class="{
done:
ingestionStore.currentStep === 'indexing' || ingestionStore.currentStep === 'done',
}"
/>
<div
class="step"
:class="{
active: ingestionStore.currentStep === 'indexing',
done: ingestionStore.currentStep === 'done',
}"
>
<span class="step-dot" />
<span class="step-label">{{ t('ingestion.stepIndexing') }}</span>
</div>
<div class="step-line" :class="{ done: ingestionStore.currentStep === 'done' }" />
<div
class="step"
:class="{
active: ingestionStore.currentStep === 'done',
done: ingestionStore.currentStep === 'done',
}"
>
<span class="step-dot" />
<span class="step-label">{{ t('ingestion.stepDone') }}</span>
</div>
</div>
<!-- Error -->
<div v-if="ingestionStore.error" class="ingest-error">
{{ ingestionStore.error }}
</div>
<!-- Success -->
<div v-if="ingestionStore.currentStep === 'done'" class="ingest-success">
<svg class="success-icon" viewBox="0 0 20 20" fill="currentColor">
<path
fill-rule="evenodd"
d="M10 18a8 8 0 100-16 8 8 0 000 16zm3.707-9.293a1 1 0 00-1.414-1.414L9 10.586 7.707 9.293a1 1 0 00-1.414 1.414l2 2a1 1 0 001.414 0l4-4z"
clip-rule="evenodd"
/>
</svg>
<span>{{ t('ingestion.successMessage') }}</span>
</div>
<!-- Action -->
<button
class="ingest-btn"
data-e2e="ingest-btn"
:disabled="ingestionStore.ingesting || !analysisId"
@click="runIngestion"
>
<div v-if="ingestionStore.ingesting" class="spinner-sm" />
<svg v-else viewBox="0 0 20 20" fill="currentColor" class="btn-icon">
<path
fill-rule="evenodd"
d="M3 17a1 1 0 011-1h12a1 1 0 110 2H4a1 1 0 01-1-1zM6.293 6.707a1 1 0 010-1.414l3-3a1 1 0 011.414 0l3 3a1 1 0 01-1.414 1.414L11 5.414V13a1 1 0 11-2 0V5.414L7.707 6.707a1 1 0 01-1.414 0z"
clip-rule="evenodd"
/>
</svg>
{{ ingestionStore.ingesting ? t('ingestion.ingesting') : t('ingestion.ingest') }}
</button>
</template>
</div>
</template>
<script setup lang="ts">
import { useIngestionStore } from '../store'
import { useI18n } from '../../../shared/i18n'
const props = defineProps<{
analysisId: string | null
documentName: string
chunkCount: number
}>()
const ingestionStore = useIngestionStore()
const { t } = useI18n()
async function runIngestion() {
if (!props.analysisId) return
await ingestionStore.ingest(props.analysisId)
}
</script>
<style scoped>
.ingest-panel {
padding: 20px;
display: flex;
flex-direction: column;
gap: 20px;
height: 100%;
}
.ingest-empty {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
gap: 12px;
padding: 40px 20px;
flex: 1;
}
.empty-icon {
width: 32px;
height: 32px;
color: var(--text-muted);
opacity: 0.5;
}
.empty-text {
font-size: 14px;
color: var(--text-muted);
text-align: center;
}
/* Summary */
.ingest-summary {
display: flex;
flex-direction: column;
gap: 8px;
padding: 12px 16px;
background: var(--bg-surface);
border: 1px solid var(--border);
border-radius: var(--radius-sm);
}
.summary-row {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
}
.summary-label {
font-size: 12px;
color: var(--text-muted);
font-weight: 500;
}
.summary-value {
font-size: 13px;
color: var(--text);
font-weight: 500;
text-align: right;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
min-width: 0;
}
.summary-mono {
font-family: 'IBM Plex Mono', monospace;
}
/* Stepper */
.ingestion-stepper {
display: flex;
align-items: center;
justify-content: center;
gap: 0;
padding: 12px 0;
}
.step {
display: flex;
align-items: center;
gap: 6px;
padding: 0 8px;
}
.step-dot {
width: 10px;
height: 10px;
border-radius: 50%;
background: var(--border);
transition: all 0.3s ease;
}
.step.active .step-dot {
background: var(--accent);
box-shadow: 0 0 6px var(--accent);
animation: pulse-dot 1s ease-in-out infinite;
}
.step.done .step-dot {
background: var(--success, #22c55e);
}
.step-label {
font-size: 12px;
color: var(--text-muted);
font-weight: 500;
}
.step.active .step-label {
color: var(--accent);
}
.step.done .step-label {
color: var(--success, #22c55e);
}
.step-line {
width: 40px;
height: 2px;
background: var(--border);
transition: background 0.3s ease;
}
.step-line.done {
background: var(--success, #22c55e);
}
@keyframes pulse-dot {
0%,
100% {
transform: scale(1);
opacity: 1;
}
50% {
transform: scale(1.3);
opacity: 0.7;
}
}
/* Error */
.ingest-error {
padding: 10px 14px;
background: rgba(239, 68, 68, 0.1);
border: 1px solid rgba(239, 68, 68, 0.3);
border-radius: var(--radius-sm);
color: var(--error);
font-size: 13px;
}
/* Success */
.ingest-success {
display: flex;
align-items: center;
gap: 8px;
padding: 10px 14px;
background: rgba(34, 197, 94, 0.1);
border: 1px solid rgba(34, 197, 94, 0.3);
border-radius: var(--radius-sm);
color: var(--success, #22c55e);
font-size: 13px;
font-weight: 500;
}
.success-icon {
width: 18px;
height: 18px;
flex-shrink: 0;
}
/* Action button */
.ingest-btn {
display: flex;
align-items: center;
justify-content: center;
gap: 8px;
padding: 10px 20px;
font-size: 14px;
font-weight: 600;
color: white;
background: var(--success, #22c55e);
border: none;
border-radius: var(--radius-sm);
cursor: pointer;
transition: all var(--transition);
}
.ingest-btn:hover:not(:disabled) {
filter: brightness(1.1);
}
.ingest-btn:disabled {
opacity: 0.6;
cursor: not-allowed;
}
.ingest-btn .btn-icon {
width: 16px;
height: 16px;
}
.spinner-sm {
width: 14px;
height: 14px;
border: 2px solid rgba(255, 255, 255, 0.3);
border-top-color: white;
border-radius: 50%;
animation: spin 0.6s linear infinite;
}
@keyframes spin {
to {
transform: rotate(360deg);
}
}
</style>

54
frontend/src/features/search/api.test.ts Normal file
View file

@ -0,0 +1,54 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { searchChunks } from './api'
const mockFetch = vi.fn()
vi.stubGlobal('fetch', mockFetch)
beforeEach(() => {
mockFetch.mockReset()
})
describe('searchChunks', () => {
it('calls /api/ingestion/search with query', async () => {
mockFetch.mockResolvedValue({
ok: true,
status: 200,
json: () =>
Promise.resolve({
results: [
{
docId: 'doc-1',
filename: 'test.pdf',
content: 'hello',
chunkIndex: 0,
pageNumber: 1,
score: 0.95,
headings: [],
},
],
total: 1,
query: 'hello',
}),
})
const result = await searchChunks('hello')
expect(mockFetch).toHaveBeenCalledWith(
'/api/ingestion/search?q=hello',
expect.objectContaining({ headers: { 'Content-Type': 'application/json' } }),
)
expect(result.results).toHaveLength(1)
expect(result.results[0].score).toBe(0.95)
})
it('passes docId and k options', async () => {
mockFetch.mockResolvedValue({
ok: true,
status: 200,
json: () => Promise.resolve({ results: [], total: 0, query: 'test' }),
})
await searchChunks('test', { docId: 'doc-1', k: 5 })
expect(mockFetch).toHaveBeenCalledWith(
'/api/ingestion/search?q=test&doc_id=doc-1&k=5',
expect.objectContaining({ headers: { 'Content-Type': 'application/json' } }),
)
})
})

27
frontend/src/features/search/api.ts Normal file
View file

@ -0,0 +1,27 @@
import { apiFetch } from '../../shared/api/http'
export interface SearchResultItem {
docId: string
filename: string
content: string
chunkIndex: number
pageNumber: number
score: number
headings: string[]
}
export interface SearchResponse {
results: SearchResultItem[]
total: number
query: string
}
export function searchChunks(
query: string,
options: { docId?: string; k?: number } = {},
): Promise<SearchResponse> {
const params = new URLSearchParams({ q: query })
if (options.docId) params.set('doc_id', options.docId)
if (options.k) params.set('k', String(options.k))
return apiFetch<SearchResponse>(`/api/ingestion/search?${params}`)
}

1
frontend/src/features/search/index.ts Normal file
View file

@ -0,0 +1 @@
export { useSearchStore } from './store'

87
frontend/src/features/search/store.test.ts Normal file
View file

@ -0,0 +1,87 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { setActivePinia, createPinia } from 'pinia'
import { useSearchStore } from './store'
import * as api from './api'
vi.mock('./api', () => ({
searchChunks: vi.fn(),
}))
beforeEach(() => {
setActivePinia(createPinia())
vi.clearAllMocks()
})
describe('useSearchStore', () => {
describe('search', () => {
it('stores results on success', async () => {
vi.mocked(api.searchChunks).mockResolvedValue({
results: [
{
docId: 'doc-1',
filename: 'test.pdf',
content: 'hello world',
chunkIndex: 0,
pageNumber: 1,
score: 0.9,
headings: [],
},
],
total: 1,
query: 'hello',
})
const store = useSearchStore()
await store.search('hello')
expect(store.results).toHaveLength(1)
expect(store.query).toBe('hello')
expect(store.searching).toBe(false)
})
it('clears results on empty query', async () => {
const store = useSearchStore()
store.results = [
{
docId: 'doc-1',
filename: 'test.pdf',
content: 'hello',
chunkIndex: 0,
pageNumber: 1,
score: 0.9,
headings: [],
},
]
await store.search('')
expect(store.results).toHaveLength(0)
expect(store.query).toBe('')
})
it('clears results on error', async () => {
vi.mocked(api.searchChunks).mockRejectedValue(new Error('fail'))
const store = useSearchStore()
await store.search('hello')
expect(store.results).toHaveLength(0)
expect(store.searching).toBe(false)
})
})
describe('clear', () => {
it('resets state', () => {
const store = useSearchStore()
store.query = 'test'
store.results = [
{
docId: 'doc-1',
filename: 'test.pdf',
content: 'hello',
chunkIndex: 0,
pageNumber: 1,
score: 0.9,
headings: [],
},
]
store.clear()
expect(store.query).toBe('')
expect(store.results).toHaveLength(0)
})
})
})

41
frontend/src/features/search/store.ts Normal file
View file

@ -0,0 +1,41 @@
import { defineStore } from 'pinia'
import { ref } from 'vue'
import * as api from './api'
export const useSearchStore = defineStore('search', () => {
const results = ref<api.SearchResultItem[]>([])
const query = ref('')
const searching = ref(false)
async function search(q: string, docId?: string): Promise<void> {
if (!q.trim()) {
results.value = []
query.value = ''
return
}
searching.value = true
query.value = q
try {
const resp = await api.searchChunks(q, { docId })
results.value = resp.results
} catch (e) {
console.error('Search failed', e)
results.value = []
} finally {
searching.value = false
}
}
function clear(): void {
results.value = []
query.value = ''
}
return {
results,
query,
searching,
search,
clear,
}
})

266
frontend/src/pages/DocumentsPage.vue
View file

@ -2,13 +2,40 @@
<div class="documents-page">
<div class="page-header">
<h1 class="page-title">{{ t('nav.documents') }}</h1>
<div class="header-actions">
<input
v-model="searchQuery"
type="text"
class="search-input"
:placeholder="t('ingestion.search')"
/>
<div v-if="ingestionEnabled" class="filter-group">
<button
v-for="f in filters"
:key="f.value"
class="filter-btn"
:class="{ active: activeFilter === f.value }"
@click="activeFilter = f.value"
>
{{ f.label }}
</button>
</div>
<div class="sort-group">
<button class="sort-btn" :class="{ active: sortBy === 'name' }" @click="sortBy = 'name'">
{{ t('ingestion.sortName') }}
</button>
<button class="sort-btn" :class="{ active: sortBy === 'date' }" @click="sortBy = 'date'">
{{ t('ingestion.sortDate') }}
</button>
</div>
</div>
</div>
<div class="page-content">
<div v-if="docStore.documents.length === 0" class="tab-empty">
<div v-if="filteredDocs.length === 0" class="tab-empty">
{{ t('history.emptyDocs') }}
</div>
<div v-else class="doc-items">
<div v-for="doc in docStore.documents" :key="doc.id" class="doc-row">
<div v-for="doc in filteredDocs" :key="doc.id" class="doc-row">
<div class="doc-row-info">
<svg class="doc-row-icon" viewBox="0 0 20 20" fill="currentColor">
<path
@ -26,15 +53,55 @@
</span>
</div>
</div>
<button class="doc-row-delete" @click="docStore.remove(doc.id)">
<svg viewBox="0 0 20 20" fill="currentColor">
<path
fill-rule="evenodd"
d="M9 2a1 1 0 00-.894.553L7.382 4H4a1 1 0 000 2v10a2 2 0 002 2h8a2 2 0 002-2V6a1 1 0 100-2h-3.382l-.724-1.447A1 1 0 0011 2H9zM7 8a1 1 0 012 0v6a1 1 0 11-2 0V8zm5-1a1 1 0 00-1 1v6a1 1 0 102 0V8a1 1 0 00-1-1z"
clip-rule="evenodd"
/>
</svg>
</button>
<div class="doc-row-actions">
<template v-if="ingestionEnabled">
<span
v-if="ingestionStore.ingestedDocs[doc.id]"
class="status-badge indexed"
:title="t('ingestion.chunksIndexed', { n: ingestionStore.ingestedDocs[doc.id] })"
>
{{ t('ingestion.indexed') }}
<span class="badge-count">{{ ingestionStore.ingestedDocs[doc.id] }}</span>
</span>
<span v-else class="status-badge not-indexed">
{{ t('ingestion.notIndexed') }}
</span>
</template>
<button
class="action-btn"
:title="t('ingestion.openInStudio')"
@click="openInStudio(doc)"
>
<svg viewBox="0 0 20 20" fill="currentColor">
<path
d="M10.394 2.08a1 1 0 00-.788 0l-7 3a1 1 0 000 1.84L5.25 8.051a.999.999 0 01.356-.257l4-1.714a1 1 0 11.788 1.838l-2.727 1.17 1.94.831a1 1 0 00.787 0l7-3a1 1 0 000-1.838l-7-3z"
/>
</svg>
</button>
<button
v-if="ingestionEnabled && ingestionStore.ingestedDocs[doc.id]"
class="action-btn unindex"
:title="t('ingestion.deleteIndex')"
@click="confirmRemoveFromIndex(doc.id)"
>
<svg viewBox="0 0 20 20" fill="currentColor">
<path
fill-rule="evenodd"
d="M3 17a1 1 0 011-1h12a1 1 0 110 2H4a1 1 0 01-1-1zM6.293 6.707a1 1 0 010-1.414l3-3a1 1 0 011.414 0l3 3a1 1 0 01-1.414 1.414L11 5.414V13a1 1 0 11-2 0V5.414L7.707 6.707a1 1 0 01-1.414 0z"
clip-rule="evenodd"
/>
</svg>
</button>
<button class="action-btn delete" @click="handleDelete(doc.id)">
<svg viewBox="0 0 20 20" fill="currentColor">
<path
fill-rule="evenodd"
d="M9 2a1 1 0 00-.894.553L7.382 4H4a1 1 0 000 2v10a2 2 0 002 2h8a2 2 0 002-2V6a1 1 0 100-2h-3.382l-.724-1.447A1 1 0 0011 2H9zM7 8a1 1 0 012 0v6a1 1 0 11-2 0V8zm5-1a1 1 0 00-1 1v6a1 1 0 102 0V8a1 1 0 00-1-1z"
clip-rule="evenodd"
/>
</svg>
</button>
</div>
</div>
</div>
</div>
@ -42,21 +109,86 @@
</template>
<script setup lang="ts">
import { onMounted } from 'vue'
import { computed, onMounted, ref } from 'vue'
import { useRouter } from 'vue-router'
import { useDocumentStore } from '../features/document/store'
import { useFeatureFlagStore } from '../features/feature-flags/store'
import { useIngestionStore } from '../features/ingestion/store'
import { useI18n } from '../shared/i18n'
import { formatSize } from '../shared/format'
import type { Document } from '../shared/types'
const docStore = useDocumentStore()
const featureStore = useFeatureFlagStore()
const ingestionStore = useIngestionStore()
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
const router = useRouter()
const { t } = useI18n()
const searchQuery = ref('')
const activeFilter = ref<'all' | 'indexed' | 'not-indexed'>('all')
const sortBy = ref<'name' | 'date'>('date')
const filters = computed(() => [
{ value: 'all' as const, label: t('ingestion.filterAll') },
{ value: 'indexed' as const, label: t('ingestion.filterIndexed') },
{ value: 'not-indexed' as const, label: t('ingestion.filterNotIndexed') },
])
const filteredDocs = computed(() => {
let docs = [...docStore.documents]
// Search filter
if (searchQuery.value.trim()) {
const q = searchQuery.value.toLowerCase()
docs = docs.filter((d) => d.filename.toLowerCase().includes(q))
}
// Status filter
if (activeFilter.value === 'indexed') {
docs = docs.filter((d) => ingestionStore.ingestedDocs[d.id])
} else if (activeFilter.value === 'not-indexed') {
docs = docs.filter((d) => !ingestionStore.ingestedDocs[d.id])
}
// Sort
if (sortBy.value === 'name') {
docs.sort((a, b) => a.filename.localeCompare(b.filename))
} else {
docs.sort((a, b) => new Date(b.createdAt).getTime() - new Date(a.createdAt).getTime())
}
return docs
})
function formatDate(iso: string) {
if (!iso) return ''
return new Date(iso).toLocaleString()
}
function openInStudio(doc: Document) {
docStore.select(doc.id)
router.push('/studio')
}
function confirmRemoveFromIndex(docId: string) {
if (confirm(t('ingestion.deleteConfirm'))) {
ingestionStore.deleteIngested(docId)
}
}
async function handleDelete(docId: string) {
if (ingestionEnabled.value && ingestionStore.ingestedDocs[docId]) {
await ingestionStore.deleteIngested(docId)
}
await docStore.remove(docId)
}
onMounted(() => {
docStore.load()
if (ingestionEnabled.value) {
ingestionStore.checkAvailability()
}
})
</script>
@ -72,6 +204,11 @@ onMounted(() => {
padding: 16px 24px;
border-bottom: 1px solid var(--border);
flex-shrink: 0;
display: flex;
align-items: center;
justify-content: space-between;
gap: 16px;
flex-wrap: wrap;
}
.page-title {
@ -80,6 +217,57 @@ onMounted(() => {
color: var(--text);
}
.header-actions {
display: flex;
align-items: center;
gap: 12px;
}
.search-input {
padding: 6px 12px;
border: 1px solid var(--border);
border-radius: var(--radius-sm);
background: var(--bg);
color: var(--text);
font-size: 13px;
width: 180px;
outline: none;
transition: border-color var(--transition);
}
.search-input:focus {
border-color: var(--accent);
}
.filter-group,
.sort-group {
display: flex;
gap: 2px;
background: var(--bg-surface);
border-radius: var(--radius-sm);
padding: 2px;
border: 1px solid var(--border);
}
.filter-btn,
.sort-btn {
padding: 4px 10px;
border: none;
background: none;
color: var(--text-secondary);
font-size: 12px;
font-weight: 500;
border-radius: 4px;
cursor: pointer;
transition: all var(--transition);
}
.filter-btn.active,
.sort-btn.active {
background: var(--accent);
color: white;
}
.page-content {
flex: 1;
overflow-y: auto;
@ -152,7 +340,41 @@ onMounted(() => {
font-family: 'IBM Plex Mono', monospace;
}
.doc-row-delete {
.doc-row-actions {
display: flex;
align-items: center;
gap: 8px;
flex-shrink: 0;
}
.status-badge {
display: inline-flex;
align-items: center;
gap: 4px;
padding: 3px 8px;
border-radius: 10px;
font-size: 11px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.03em;
}
.status-badge.indexed {
background: rgba(34, 197, 94, 0.15);
color: var(--success);
}
.status-badge.not-indexed {
background: rgba(156, 163, 175, 0.15);
color: var(--text-muted);
}
.badge-count {
font-family: 'IBM Plex Mono', monospace;
font-size: 10px;
}
.action-btn {
background: none;
border: none;
padding: 6px;
@ -164,14 +386,26 @@ onMounted(() => {
transition: all var(--transition);
}
.doc-row:hover .doc-row-delete {
.doc-row:hover .action-btn {
opacity: 1;
}
.doc-row-delete:hover {
.action-btn:hover {
color: var(--accent);
background: rgba(249, 115, 22, 0.1);
}
.action-btn.delete:hover {
color: var(--error);
background: rgba(239, 68, 68, 0.1);
}
.doc-row-delete svg {
.action-btn.unindex:hover {
color: var(--warning, #f59e0b);
background: rgba(245, 158, 11, 0.1);
}
.action-btn svg {
width: 16px;
height: 16px;
}

197
frontend/src/pages/SearchPage.vue Normal file
View file

@ -0,0 +1,197 @@
<template>
<div class="search-page">
<div class="page-header">
<h1 class="page-title">{{ t('nav.search') }}</h1>
</div>
<div v-if="!ingestionEnabled || !ingestionStore.available" class="tab-empty">
{{ t('ingestion.unavailable') }}
</div>
<template v-else>
<div class="chunk-search-bar">
<input
v-model="searchInput"
type="text"
class="search-input"
:placeholder="t('ingestion.searchChunks')"
@keyup.enter="runSearch"
/>
<div v-if="searchStore.searching" class="spinner-xs" />
</div>
<div v-if="searchStore.results.length > 0" class="search-results">
<div v-for="(result, idx) in searchStore.results" :key="idx" class="search-result-item">
<div class="result-header">
<span class="result-filename">{{ result.filename }}</span>
<span class="result-meta"
>p.{{ result.pageNumber }} — chunk #{{ result.chunkIndex }}</span
>
<span class="result-score">{{ result.score.toFixed(1) }}</span>
</div>
<p class="result-content">
{{ result.content.slice(0, 200) }}{{ result.content.length > 200 ? '…' : '' }}
</p>
</div>
</div>
<div
v-if="searchStore.query && !searchStore.searching && searchStore.results.length === 0"
class="tab-empty"
>
{{ t('ingestion.noResults', { q: searchStore.query }) }}
</div>
<div v-if="!searchStore.query" class="tab-empty">
{{ t('search.hint') }}
</div>
</template>
</div>
</template>
<script setup lang="ts">
import { computed, onMounted, ref } from 'vue'
import { useSearchStore } from '../features/search/store'
import { useFeatureFlagStore } from '../features/feature-flags/store'
import { useIngestionStore } from '../features/ingestion/store'
import { useI18n } from '../shared/i18n'
const searchStore = useSearchStore()
const featureStore = useFeatureFlagStore()
const ingestionStore = useIngestionStore()
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
const { t } = useI18n()
const searchInput = ref('')
function runSearch() {
if (searchInput.value.trim()) {
searchStore.search(searchInput.value)
} else {
searchStore.clear()
}
}
onMounted(() => {
if (ingestionEnabled.value) {
ingestionStore.checkAvailability()
}
})
</script>
<style scoped>
.search-page {
display: flex;
flex-direction: column;
height: 100%;
overflow: hidden;
}
.page-header {
padding: 16px 24px;
border-bottom: 1px solid var(--border);
flex-shrink: 0;
}
.page-title {
font-size: 18px;
font-weight: 600;
color: var(--text);
}
.tab-empty {
text-align: center;
color: var(--text-muted);
padding: 60px 20px;
font-size: 14px;
}
/* Chunk search */
.chunk-search-bar {
display: flex;
align-items: center;
gap: 8px;
padding: 12px 24px;
border-bottom: 1px solid var(--border);
}
.search-input {
flex: 1;
padding: 8px 12px;
border: 1px solid var(--border);
border-radius: var(--radius-sm);
background: var(--bg);
color: var(--text);
font-size: 13px;
outline: none;
transition: border-color var(--transition);
}
.search-input:focus {
border-color: var(--accent);
}
.spinner-xs {
width: 14px;
height: 14px;
border: 2px solid var(--border);
border-top-color: var(--accent);
border-radius: 50%;
animation: spin 0.6s linear infinite;
}
@keyframes spin {
to {
transform: rotate(360deg);
}
}
/* Search results */
.search-results {
flex: 1;
overflow-y: auto;
}
.search-result-item {
padding: 10px 24px;
border-bottom: 1px solid var(--border);
}
.search-result-item:last-child {
border-bottom: none;
}
.result-header {
display: flex;
align-items: center;
gap: 8px;
margin-bottom: 4px;
}
.result-filename {
font-weight: 600;
font-size: 13px;
color: var(--text);
}
.result-meta {
font-size: 11px;
color: var(--text-muted);
font-family: 'IBM Plex Mono', monospace;
}
.result-score {
margin-left: auto;
font-size: 11px;
font-weight: 600;
color: var(--accent);
font-family: 'IBM Plex Mono', monospace;
}
.result-content {
font-size: 13px;
color: var(--text-muted);
line-height: 1.5;
margin: 0;
}
</style>

60
frontend/src/pages/StudioPage.vue
View file

@ -22,7 +22,7 @@
<div class="mode-toggle">
<button
class="toggle-btn"
data-e2e="toggle-btn"
data-e2e="toggle-btn configure-btn"
:class="{ active: mode === 'configure' }"
@click="mode = 'configure'"
>
@ -37,7 +37,7 @@
</button>
<button
class="toggle-btn"
data-e2e="toggle-btn"
data-e2e="toggle-btn verify-btn"
:class="{ active: mode === 'verify' }"
@click="mode = 'verify'"
:disabled="!analysisStore.currentAnalysis"
@ -54,7 +54,7 @@
<button
v-if="chunkingEnabled"
class="toggle-btn"
data-e2e="toggle-btn"
data-e2e="toggle-btn prepare-btn"
:class="{ active: mode === 'prepare' }"
@click="mode = 'prepare'"
:disabled="!analysisStore.currentAnalysis"
@ -66,6 +66,24 @@
</svg>
{{ t('studio.prepare') }}
</button>
<button
v-if="chunkingEnabled && ingestionEnabled && ingestionStore.available"
class="toggle-btn"
data-e2e="toggle-btn"
:class="{ active: mode === 'ingest' }"
@click="mode = 'ingest'"
:disabled="!canIngest"
:title="!canIngest ? t('ingestion.unavailable') : ''"
>
<svg class="toggle-icon" viewBox="0 0 20 20" fill="currentColor">
<path
fill-rule="evenodd"
d="M3 17a1 1 0 011-1h12a1 1 0 110 2H4a1 1 0 01-1-1zM6.293 6.707a1 1 0 010-1.414l3-3a1 1 0 011.414 0l3 3a1 1 0 01-1.414 1.414L11 5.414V13a1 1 0 11-2 0V5.414L7.707 6.707a1 1 0 01-1.414 0z"
clip-rule="evenodd"
/>
</svg>
{{ t('studio.ingest') }}
</button>
</div>
</div>
<div class="topbar-actions">
@ -446,7 +464,15 @@
:has-document-json="analysisStore.currentAnalysis?.hasDocumentJson ?? false"
:chunks="analysisStore.currentChunks"
@highlight-bboxes="highlightedChunkBboxes = $event"
@rechunked="onRechunked"
/>
</div>
<!-- INGEST MODE -->
<div v-if="mode === 'ingest'" class="ingest-panel-wrapper">
<IngestPanel
:analysis-id="analysisStore.currentAnalysis?.id ?? null"
:document-name="selectedDoc?.filename ?? ''"
:chunk-count="analysisStore.currentChunks?.length ?? 0"
/>
</div>
</div>
@ -459,10 +485,12 @@ import { ref, computed, watch, nextTick, onMounted, onBeforeUnmount, reactive }
import { useRoute, useRouter } from 'vue-router'
import { useDocumentStore } from '../features/document/store'
import { useAnalysisStore } from '../features/analysis/store'
import { useIngestionStore } from '../features/ingestion/store'
import { DocumentUpload, DocumentList } from '../features/document/index'
import { ResultTabs } from '../features/analysis/index'
import BboxOverlay from '../features/analysis/ui/BboxOverlay.vue'
import { ChunkPanel } from '../features/chunking'
import { IngestPanel } from '../features/ingestion'
import { useFeatureFlag } from '../features/feature-flags'
import { getPreviewUrl } from '../features/document/api'
import { useI18n } from '../shared/i18n'
@ -472,8 +500,10 @@ const route = useRoute()
const router = useRouter()
const documentStore = useDocumentStore()
const analysisStore = useAnalysisStore()
const ingestionStore = useIngestionStore()
const { t } = useI18n()
const chunkingEnabled = useFeatureFlag('chunking')
const ingestionEnabled = useFeatureFlag('ingestion')
const mode = ref('configure')
const currentPage = ref(1)
@ -528,6 +558,14 @@ const pipelineOptions = reactive<PipelineOptions>({
images_scale: 1.0,
})
const canIngest = computed(() => {
return (
ingestionStore.available &&
analysisStore.currentAnalysis?.status === 'COMPLETED' &&
analysisStore.currentAnalysis?.chunksJson != null
)
})
const hasAnalysisResults = computed(() => {
return (
analysisStore.currentAnalysis?.status === 'COMPLETED' && analysisStore.currentPages?.length > 0
@ -568,12 +606,6 @@ function addMore() {
documentStore.selectedId = null
}
async function onRechunked() {
if (analysisStore.currentAnalysis?.id) {
await analysisStore.select(analysisStore.currentAnalysis.id)
}
}
// Clear highlights when switching modes or pages
watch(mode, () => {
highlightedElementIndex.value = -1
@ -598,6 +630,9 @@ watch(
onMounted(async () => {
await documentStore.load()
analysisStore.load()
if (ingestionEnabled.value) {
ingestionStore.checkAvailability()
}
// Restore analysis from history via query param
const analysisId = route.query.analysisId
@ -1369,9 +1404,10 @@ onBeforeUnmount(() => {
padding-top: 16px;
}
/* Verify panel */
/* Verify / Prepare / Ingest panels */
.verify-panel,
.prepare-panel {
.prepare-panel,
.ingest-panel-wrapper {
height: 100%;
overflow: hidden;
display: flex;

88
frontend/src/shared/i18n.ts
View file

@ -109,6 +109,7 @@ const messages: Messages = {
// Chunking
'studio.prepare': 'Préparer',
'studio.ingest': 'Ingérer',
'chunking.settings': 'Chunking',
'chunking.chunkerType': 'Type de chunker',
'chunking.maxTokens': 'Tokens max',
@ -119,8 +120,50 @@ const messages: Messages = {
'chunking.chunks': 'chunks',
'chunking.noChunks': 'Lancez le chunking pour préparer les segments.',
'chunking.noChunksOnPage': 'Aucun chunk sur cette page.',
'chunking.edit': 'Modifier',
'chunking.save': 'Enregistrer',
'chunking.saving': 'Enregistrement...',
'chunking.cancel': 'Annuler',
'chunking.modified': 'modifié',
'chunking.delete': 'Supprimer',
'chunking.deleting': 'Suppression...',
'chunking.deleteConfirm':
'Supprimer ce chunk ? Il sera marqué comme supprimé jusqu\u2019à la prochaine synchronisation.',
'chunking.batchNotice':
'Le chunking n\u2019est pas disponible pour cette analyse. Les documents volumineux trait\u00e9s par batch ne g\u00e9n\u00e8rent pas la structure interne n\u00e9cessaire au d\u00e9coupage.',
'Le chunking n\u2019est pas disponible pour cette analyse. Les documents volumineux trait\u00e9s par batch ne g\u00e9n\u00e8rent pas la structure interne n\u00e9cessaire au d\u00e9coupage. Coming soon !',
// Search
'nav.search': 'Recherche',
'search.hint': 'Saisissez un terme pour rechercher dans les chunks indexés.',
// Ingestion / My Documents
'ingestion.ingest': 'Ingérer',
'ingestion.document': 'Document',
'ingestion.chunkCount': 'Chunks prêts',
'ingestion.successMessage': 'Indexation terminée avec succès !',
'ingestion.ingesting': 'Ingestion...',
'ingestion.reindex': 'Ré-indexer',
'ingestion.indexed': 'Indexé',
'ingestion.notIndexed': 'Non indexé',
'ingestion.chunksIndexed': '{n} chunks indexés',
'ingestion.openInStudio': 'Ouvrir dans le Studio',
'ingestion.deleteIndex': "Supprimer de l'index",
'ingestion.deleteConfirm':
'Retirer ce document de l\u2019index ? Les chunks seront supprimés mais le document source restera.',
'ingestion.unavailable': 'Ingestion non disponible',
'ingestion.filterAll': 'Tous',
'ingestion.filterIndexed': 'Indexés',
'ingestion.filterNotIndexed': 'Non indexés',
'ingestion.sortName': 'Nom',
'ingestion.sortDate': 'Date',
'ingestion.search': 'Rechercher...',
'ingestion.searchChunks': 'Rechercher dans les chunks…',
'ingestion.noResults': 'Aucun résultat pour « {q} ».',
'ingestion.stepEmbedding': 'Embedding…',
'ingestion.stepIndexing': 'Indexation…',
'ingestion.stepDone': 'Terminé',
'ingestion.opensearchConnected': 'OpenSearch connecté',
'ingestion.opensearchDisconnected': 'OpenSearch déconnecté',
// Pagination
'pagination.pageOf': 'Page {current} sur {total}',
@ -235,6 +278,7 @@ const messages: Messages = {
'history.open': 'Open',
'studio.prepare': 'Prepare',
'studio.ingest': 'Ingest',
'chunking.settings': 'Chunking',
'chunking.chunkerType': 'Chunker type',
'chunking.maxTokens': 'Max tokens',
@ -245,8 +289,48 @@ const messages: Messages = {
'chunking.chunks': 'chunks',
'chunking.noChunks': 'Run chunking to prepare segments.',
'chunking.noChunksOnPage': 'No chunks on this page.',
'chunking.edit': 'Edit',
'chunking.save': 'Save',
'chunking.saving': 'Saving...',
'chunking.cancel': 'Cancel',
'chunking.modified': 'modified',
'chunking.delete': 'Delete',
'chunking.deleting': 'Deleting...',
'chunking.deleteConfirm':
'Delete this chunk? It will be marked as deleted until the next sync.',
'chunking.batchNotice':
'Chunking is not available for this analysis. Large documents processed in batch mode do not generate the internal structure required for chunking.',
'Chunking is not available for this analysis. Large documents processed in batch mode do not generate the internal structure required for chunking. Coming soon!',
'nav.search': 'Search',
'search.hint': 'Enter a term to search through indexed chunks.',
'ingestion.ingest': 'Ingest',
'ingestion.document': 'Document',
'ingestion.chunkCount': 'Chunks ready',
'ingestion.successMessage': 'Indexing completed successfully!',
'ingestion.ingesting': 'Ingesting...',
'ingestion.reindex': 'Re-index',
'ingestion.indexed': 'Indexed',
'ingestion.notIndexed': 'Not indexed',
'ingestion.chunksIndexed': '{n} chunks indexed',
'ingestion.openInStudio': 'Open in Studio',
'ingestion.deleteIndex': 'Remove from index',
'ingestion.deleteConfirm':
'Remove this document from the index? Chunks will be deleted but the source document will remain.',
'ingestion.unavailable': 'Ingestion unavailable',
'ingestion.filterAll': 'All',
'ingestion.filterIndexed': 'Indexed',
'ingestion.filterNotIndexed': 'Not indexed',
'ingestion.sortName': 'Name',
'ingestion.sortDate': 'Date',
'ingestion.search': 'Search...',
'ingestion.searchChunks': 'Search indexed chunks…',
'ingestion.noResults': 'No results for "{q}".',
'ingestion.stepEmbedding': 'Embedding…',
'ingestion.stepIndexing': 'Indexing…',
'ingestion.stepDone': 'Done',
'ingestion.opensearchConnected': 'OpenSearch connected',
'ingestion.opensearchDisconnected': 'OpenSearch unreachable',
'pagination.pageOf': 'Page {current} of {total}',
'pagination.perPage': '/ page',

2
frontend/src/shared/types.ts
View file

@ -58,6 +58,8 @@ export interface Chunk {
sourcePage: number | null
tokenCount: number
bboxes: ChunkBbox[]
modified?: boolean
deleted?: boolean
}
export interface PageElement {

79
frontend/src/shared/ui/AppSidebar.vue
View file

@ -45,6 +45,23 @@
<span class="nav-label">{{ t('nav.documents') }}</span>
</RouterLink>
<RouterLink
v-if="ingestionEnabled"
to="/search"
class="nav-item"
data-e2e="nav-search"
:class="{ active: route.name === 'search' }"
>
<svg class="nav-icon" viewBox="0 0 20 20" fill="currentColor">
<path
fill-rule="evenodd"
d="M8 4a4 4 0 100 8 4 4 0 000-8zM2 8a6 6 0 1110.89 3.476l4.817 4.817a1 1 0 01-1.414 1.414l-4.816-4.816A6 6 0 012 8z"
clip-rule="evenodd"
/>
</svg>
<span class="nav-label">{{ t('nav.search') }}</span>
</RouterLink>
<RouterLink
to="/history"
class="nav-item"
@ -79,6 +96,21 @@
</nav>
<div class="sidebar-footer">
<div
v-if="ingestionEnabled && ingestionStore.available"
class="opensearch-status"
:title="
ingestionStore.opensearchConnected
? t('ingestion.opensearchConnected')
: t('ingestion.opensearchDisconnected')
"
>
<span
class="status-dot"
:class="ingestionStore.opensearchConnected ? 'connected' : 'disconnected'"
/>
<span class="status-label">OpenSearch</span>
</div>
<a
class="github-badge"
href="https://github.com/scub-france/Docling-Studio"
@ -97,12 +129,15 @@
</template>
<script setup lang="ts">
import { computed } from 'vue'
import { computed, onMounted, onBeforeUnmount } from 'vue'
import { RouterLink, useRoute } from 'vue-router'
import { useI18n } from '../i18n'
import { useFeatureFlagStore } from '../../features/feature-flags/store'
import { useIngestionStore } from '../../features/ingestion/store'
const featureStore = useFeatureFlagStore()
const ingestionStore = useIngestionStore()
const ingestionEnabled = computed(() => featureStore.isEnabled('ingestion'))
const version = computed(() => featureStore.appVersion)
const route = useRoute()
const { t } = useI18n()
@ -110,6 +145,17 @@ const { t } = useI18n()
defineProps({
open: { type: Boolean, default: false },
})
onMounted(() => {
if (ingestionEnabled.value) {
ingestionStore.checkAvailability()
ingestionStore.startPolling(30_000)
}
})
onBeforeUnmount(() => {
ingestionStore.stopPolling()
})
</script>
<style scoped>
@ -196,4 +242,35 @@ defineProps({
color: var(--text-muted);
font-family: 'IBM Plex Mono', monospace;
}
.opensearch-status {
display: flex;
align-items: center;
gap: 6px;
margin-bottom: 10px;
cursor: default;
}
.status-dot {
width: 8px;
height: 8px;
border-radius: 50%;
flex-shrink: 0;
}
.status-dot.connected {
background: var(--success, #22c55e);
box-shadow: 0 0 4px var(--success, #22c55e);
}
.status-dot.disconnected {
background: var(--error, #ef4444);
box-shadow: 0 0 4px var(--error, #ef4444);
}
.status-label {
font-size: 11px;
color: var(--text-muted);
font-family: 'IBM Plex Mono', monospace;
}
</style>

267
profiles/fastapi-vue/commands.sh
View file

@ -1,267 +0,0 @@
#!/usr/bin/env bash
# ============================================================================
# Docling Studio — Automated Audit Checks (FastAPI + Vue 3 profile)
# ============================================================================
# Runs verification commands for each of the 12 release audits.
# Usage: bash profiles/fastapi-vue/commands.sh
# Run from the repository root.
# ============================================================================
set -uo pipefail
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
PASS=0
WARN=0
FAIL=0
pass() { echo -e " ${GREEN}PASS${NC} $1"; ((PASS++)); }
warn() { echo -e " ${YELLOW}WARN${NC} $1"; ((WARN++)); }
fail() { echo -e " ${RED}FAIL${NC} $1"; ((FAIL++)); }
# ── 01. Clean Architecture ─────────────────────────────────────────────────
echo ""
echo "== 01. Clean Architecture =="
# Domain must not import from api, persistence, infra
if grep -rn "from api\|from persistence\|from infra\|import fastapi\|import aiosqlite" document-parser/domain/ 2>/dev/null; then
fail "Domain layer imports forbidden modules"
else
pass "Domain layer has no forbidden imports"
fi
# API must not import from persistence directly
if grep -rn "from persistence" document-parser/api/ 2>/dev/null; then
fail "API layer imports directly from persistence"
else
pass "API layer does not import from persistence"
fi
# ── 02. DDD ────────────────────────────────────────────────────────────────
echo ""
echo "== 02. DDD =="
# Domain models exist
if [ -f document-parser/domain/models.py ] && [ -f document-parser/domain/ports.py ]; then
pass "Domain models and ports exist"
else
fail "Missing domain/models.py or domain/ports.py"
fi
# Value objects exist
if [ -f document-parser/domain/value_objects.py ]; then
pass "Value objects defined"
else
warn "No value_objects.py in domain"
fi
# ── 03. Clean Code ─────────────────────────────────────────────────────────
echo ""
echo "== 03. Clean Code =="
# Check for files > 300 lines (backend)
LARGE_FILES=$(find document-parser -name "*.py" ! -path "*/.venv/*" ! -path "*/__pycache__/*" ! -path "*/tests/*" -exec awk 'END{if(NR>300) print FILENAME": "NR" lines"}' {} \;)
if [ -n "$LARGE_FILES" ]; then
warn "Large Python files (>300 lines):"
echo "$LARGE_FILES"
else
pass "No Python files exceed 300 lines"
fi
# Check for files > 300 lines (frontend)
LARGE_VUE=$(find frontend/src -name "*.vue" -o -name "*.ts" | xargs awk 'END{if(NR>300) print FILENAME": "NR" lines"}' 2>/dev/null)
if [ -n "$LARGE_VUE" ]; then
warn "Large frontend files (>300 lines):"
echo "$LARGE_VUE"
else
pass "No frontend files exceed 300 lines"
fi
# ── 04. KISS ───────────────────────────────────────────────────────────────
echo ""
echo "== 04. KISS =="
# Check for overly complex patterns
if grep -rn "type:\s*ignore" document-parser/ --include="*.py" ! -path "*/.venv/*" 2>/dev/null; then
warn "Found type: ignore comments (review if justified)"
else
pass "No unjustified type: ignore"
fi
# ── 05. DRY ────────────────────────────────────────────────────────────────
echo ""
echo "== 05. DRY =="
# Check for magic numbers in backend
if grep -rn "[^a-zA-Z_][0-9]\{3,\}[^0-9]" document-parser/ --include="*.py" ! -path "*/.venv/*" ! -path "*/tests/*" 2>/dev/null | grep -v "port\|version\|status\|#\|MAX_\|DEFAULT_\|LIMIT_" | head -5; then
warn "Possible magic numbers found (review above)"
else
pass "No obvious magic numbers"
fi
# ── 06. SOLID ──────────────────────────────────────────────────────────────
echo ""
echo "== 06. SOLID =="
# Check that ports (interfaces) exist
if grep -l "Protocol\|ABC\|abstractmethod" document-parser/domain/ports.py 2>/dev/null; then
pass "Domain ports use Protocol/ABC (Dependency Inversion)"
else
fail "No abstract ports found in domain"
fi
# ── 07. Decoupling ─────────────────────────────────────────────────────────
echo ""
echo "== 07. Decoupling =="
# Frontend should not hardcode backend URLs (except in config)
if grep -rn "localhost:8000\|127.0.0.1:8000" frontend/src/ --include="*.ts" --include="*.vue" 2>/dev/null | grep -v "config\|env\|http.ts"; then
fail "Frontend hardcodes backend URL outside config"
else
pass "Frontend backend URL is configurable"
fi
# ── 08. Security ───────────────────────────────────────────────────────────
echo ""
echo "== 08. Security =="
# Check for hardcoded secrets
if grep -rni "password\s*=\s*['\"].\+['\"\|secret\s*=\s*['\"].\+['\"\|api_key\s*=\s*['\"].\+['\"]" document-parser/ --include="*.py" ! -path "*/.venv/*" ! -path "*/tests/*" 2>/dev/null; then
fail "Possible hardcoded secrets found"
else
pass "No hardcoded secrets detected"
fi
# Check for eval/exec
if grep -rn "\beval(\|exec(" document-parser/ --include="*.py" ! -path "*/.venv/*" 2>/dev/null; then
fail "eval() or exec() found in backend"
else
pass "No eval/exec in backend"
fi
# Check CORS configuration exists
if grep -rn "CORSMiddleware" document-parser/ --include="*.py" ! -path "*/.venv/*" 2>/dev/null > /dev/null; then
pass "CORS middleware is configured"
else
warn "No CORS middleware found"
fi
# ── 09. Tests ──────────────────────────────────────────────────────────────
echo ""
echo "== 09. Tests =="
# Backend tests exist
BACKEND_TESTS=$(find document-parser/tests -name "test_*.py" 2>/dev/null | wc -l)
if [ "$BACKEND_TESTS" -gt 0 ]; then
pass "Backend: $BACKEND_TESTS test files found"
else
fail "No backend test files found"
fi
# Frontend tests exist
FRONTEND_TESTS=$(find frontend/src -name "*.test.*" 2>/dev/null | wc -l)
if [ "$FRONTEND_TESTS" -gt 0 ]; then
pass "Frontend: $FRONTEND_TESTS test files found"
else
fail "No frontend test files found"
fi
# E2E tests exist
E2E_TESTS=$(find e2e -name "*.feature" 2>/dev/null | wc -l)
if [ "$E2E_TESTS" -gt 0 ]; then
pass "E2E: $E2E_TESTS feature files found"
else
warn "No e2e feature files found"
fi
# Check for skipped tests
if grep -rn "@skip\|@ignore\|xit(\|xdescribe(\|pytest.mark.skip" document-parser/tests/ frontend/src/ 2>/dev/null | grep -v "helpers"; then
warn "Skipped tests found (review if intentional)"
else
pass "No skipped tests"
fi
# ── 10. CI / Build ────────────────────────────────────────────────────────
echo ""
echo "== 10. CI / Build =="
# CI workflow exists
if [ -f .github/workflows/ci.yml ]; then
pass "CI workflow exists"
else
fail "No CI workflow found"
fi
# Dockerfile exists
if [ -f Dockerfile ]; then
pass "Dockerfile exists"
else
fail "No Dockerfile found"
fi
# Health check in docker-compose
if grep -q "healthcheck" docker-compose.yml 2>/dev/null; then
pass "Docker Compose has health check"
else
warn "No health check in docker-compose.yml"
fi
# ── 11. Documentation ─────────────────────────────────────────────────────
echo ""
echo "== 11. Documentation =="
# CHANGELOG exists and has content
if [ -f CHANGELOG.md ] && [ -s CHANGELOG.md ]; then
pass "CHANGELOG.md exists and is not empty"
else
fail "CHANGELOG.md missing or empty"
fi
# README exists
if [ -f README.md ]; then
pass "README.md exists"
else
fail "README.md missing"
fi
# Check for TODO/FIXME without issue reference
TODOS=$(grep -rn "TODO\|FIXME" document-parser/ frontend/src/ --include="*.py" --include="*.ts" --include="*.vue" ! -path "*/.venv/*" ! -path "*/node_modules/*" 2>/dev/null | grep -v "#[0-9]" | head -5)
if [ -n "$TODOS" ]; then
warn "TODO/FIXME without issue reference:"
echo "$TODOS"
else
pass "No orphaned TODO/FIXME"
fi
# ── 12. Performance ───────────────────────────────────────────────────────
echo ""
echo "== 12. Performance =="
# Check for synchronous file I/O in async context
if grep -rn "open(" document-parser/api/ document-parser/services/ --include="*.py" 2>/dev/null | grep -v "aiofiles\|async\|#"; then
warn "Synchronous file I/O in async code (review above)"
else
pass "No synchronous file I/O in async endpoints"
fi
# Check for N+1 patterns (loop with DB call)
if grep -rn "for.*in.*:" document-parser/services/ --include="*.py" -A5 2>/dev/null | grep "await.*repo\|await.*db"; then
warn "Possible N+1 query pattern (review above)"
else
pass "No obvious N+1 patterns"
fi
# ── Summary ────────────────────────────────────────────────────────────────
echo ""
echo "============================================"
echo -e " ${GREEN}PASS${NC}: $PASS"
echo -e " ${YELLOW}WARN${NC}: $WARN"
echo -e " ${RED}FAIL${NC}: $FAIL"
echo "============================================"
if [ "$FAIL" -gt 0 ]; then
exit 1
fi

70
profiles/fastapi-vue/profile.md
View file

@ -1,70 +0,0 @@
# Stack Profile — FastAPI + Vue 3
Profile for running the 12 release audits on Docling Studio.
## Layer Mapping
| Generic Layer | Docling Studio Path | Language |
|---------------|---------------------|----------|
| **Domain** | `document-parser/domain/` | Python |
| **Services** | `document-parser/services/` | Python |
| **API** | `document-parser/api/` | Python |
| **Infrastructure** | `document-parser/infra/` | Python |
| **Persistence** | `document-parser/persistence/` | Python |
| **Frontend** | `frontend/src/` | TypeScript / Vue |
| **Tests (backend)** | `document-parser/tests/` | Python |
| **Tests (frontend)** | `frontend/src/**/*.test.*` | TypeScript |
| **Tests (e2e API)** | `e2e/api/` | Karate (Gherkin) |
| **Tests (e2e UI)** | `e2e/ui/` | Karate UI (Gherkin) |
| **CI/CD** | `.github/workflows/` | YAML |
| **Docker** | `Dockerfile`, `docker-compose.yml`, `nginx.conf` | Docker / Nginx |
## Excluded Paths
These paths are excluded from audits:
- `document-parser/.venv/`
- `document-parser/__pycache__/`
- `frontend/node_modules/`
- `frontend/dist/`
- `e2e/**/target/`
## Framework Detection
Imports that should NOT appear in the domain layer:
```python
# Forbidden in document-parser/domain/
from fastapi import ...
from pydantic import ... # except BaseModel for value objects
import aiosqlite
from infra import ...
from persistence import ...
from api import ...
```
Imports that should NOT cross feature boundaries in the frontend:
```typescript
// features/analysis/ should NOT import from features/chunking/store
// features/document/ should NOT import from features/analysis/store
// Cross-feature communication goes through shared/ or events
```
## Tools & Commands
| Task | Command |
|------|---------|
| Backend lint | `cd document-parser && ruff check .` |
| Backend format | `cd document-parser && ruff format --check .` |
| Backend tests | `cd document-parser && pytest tests/ -v` |
| Frontend lint | `cd frontend && npx eslint src/` |
| Frontend type-check | `cd frontend && npm run type-check` |
| Frontend format | `cd frontend && npx prettier --check src/` |
| Frontend tests | `cd frontend && npm run test:run` |
| E2E API tests | `mvn test -f e2e/api/pom.xml -Dkarate.options="--tags @smoke"` |
| E2E UI tests | `mvn test -f e2e/ui/pom.xml -Dkarate.options="--tags @critical"` |
| Docker build | `docker compose build` |
| Docker health | `curl -s http://localhost:3000/api/health` |
| Dependency audit (Python) | `cd document-parser && pip audit` |
| Dependency audit (Node) | `cd frontend && npm audit` |