Commit graph

30 commits

Author SHA1 Message Date
Pier-Jean Malandrino
621a9e3fce docs(design): E2 design docs for 0.6.0 navigation refactor
Adds technical design docs for the navigation epic of the doc-centric
pivot:

- 207 — Document-centric routing (/docs, /docs/:id?mode=, /index/:store, /runs)
- 208 — Doc workspace breadcrumb (Studio > <doc> > <mode>)
- 209 — Sidebar nav rework (Home / Docs / Stores / Runs / Settings)
- 210 — Feature flag mode gating (deep-link redirect + flag exposure)

Status: Accepted on all four. Each doc spells out the contract,
alternatives considered, risks per audit dimension, and testing
strategy. Backwards-compatibility is preserved throughout: legacy
routes and pages keep working until E3/E4/E5 explicitly replace them.

Refs #207 #208 #209 #210
2026-04-29 17:31:41 +02:00
Pier-Jean Malandrino
f439d5e579 docs(design): E1 design docs for 0.6.0 doc-centric ingest
Adds technical design docs for the foundation of the doc-centric ingest pivot:

- 202 — Document lifecycle state machine
- 203 — Per (document, store) ingestion state
- 204 — Auto-detect Stale state via chunk content hash
- 205 — Audit trail for chunk edits (chunks → first-class entity)
- 206 — Migration of existing documents to the new model

Status: Accepted on all five. Each doc spells out the domain entities,
persistence schema, services orchestration, API contract, alternatives
considered, risks, and testing strategy. ADR placeholders called out
where load-bearing decisions warrant a follow-up document.

Refs #202 #203 #204 #205 #206
2026-04-29 15:24:16 +02:00
Pier-Jean Malandrino
efc27932dd refactor(audit): remediate 0.5.0 audit findings — clean architecture, security, DRY, SOLID, perf
Closes the 12 MAJ raised by the release/0.5.0 audit pipeline (cf.
docs/audit/reports/release-0.5.0/summary.md → summary-reaudit.md).

Volet 1 — Reasoning architecture (audits 01/02/06/07 strengthening)
  * Domain ports: LLMProvider, ReasoningRunner, ReasoningParseError
  * Domain DTOs: LLMProviderType, ReasoningResult, ReasoningIteration
  * infra/llm/ollama_provider.py — OllamaProvider with health_check
  * infra/docling_agent_reasoning.py — runner adapter, encapsulates the
    private _rag_loop call (tracked at docling-project/docling-agent#26),
    commits OLLAMA_HOST once at boot (eliminates the per-request env race),
    translates upstream IndexError into ReasoningParseError
  * api/reasoning.py — zero coupling to docling-agent / mellea / docling-core,
    consumes app.state.reasoning_runner via the port
  * main.py — DI wires OllamaProvider + DoclingAgentReasoningRunner at boot
    when REASONING_ENABLED=true and deps are importable
  * Rename RAG_* env vars → REASONING_*, endpoint /rag → /reasoning,
    type RAGResult → ReasoningResult, frontend feature flag wiring,
    i18n strings, tests, docs (BREAKING — pre-1.0 surface, no external
    consumers in production)
  * 17 new tests: adapter unit tests with sys.modules stubs, OllamaProvider
    httpx tests, R3 concurrent-host isolation, R6 multi-iteration trace
    serialization, R13 Protocol conformance via isinstance
  * E2E Karate scenario: nav-reasoning hidden when REASONING_ENABLED=false
  * README — Live Reasoning section (env vars, archi, link to issue #26)

Bloc B — Security (audit 08, dev-only context)
  * docker-compose.yml — DEV DEFAULTS header, OpenSearch DISABLE_SECURITY_PLUGIN
    flagged as dev-only with link to OpenSearch security docs
  * main.py — boot warning if NEO4J_URI is set with the default 'changeme'
    password, so prod operators can't silently inherit it

Bloc C — DRY frontend (audit 05)
  * shared/storage/keys.ts — STORAGE_KEYS centralised (theme, locale)
  * features/settings/store.ts — dead apiUrl ref + orphan i18n keys removed
  * api/schemas.py — DOCUMENT_STATUS_UPLOADED constant

Bloc D — Quality (audits 02/06/07/09/10/12)
  * domain/ports.py — DocumentConverter.supports_page_batching property
    (LSP fix, replaces isinstance(ServeConverter) check)
  * domain/ports.py — VectorStore.ping() (encapsulation, replaces
    _vector_store._client.info() reach-around)
  * api/analyses.py + api/ingestion.py — path params {job_id} → {analysis_id}
    aligned with the user-facing terminology (URLs unchanged)
  * api/documents.py — Path.read_bytes() + generate_preview() wrapped in
    asyncio.to_thread, unblocks the FastAPI event loop on /preview
  * infra/docling_tree.py — PEP 604 union for isinstance (Ruff UP038)
  * src/__tests__/integration/ — cross-feature integration test relocated
    out of features/history/ so feature folders stay self-contained
  * Tightened terminal `assert X is not None` checks (isinstance(.., datetime),
    exact value comparisons)

Validation
  * 446 backend pytest, 202 frontend vitest — all green
  * ruff + ruff format + ESLint + Prettier + vue-tsc clean
  * Re-audit verdict: 0 CRIT / 0 MAJ, score ~94/100, GO

Closes #200
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
fe83dcdf79 feat(settings): paste-image size/type limits for #195 (#196)
* docs: rename Clean Architecture → Hexagonal Architecture (ports & adapters)

Le backend suit le pattern ports & adapters (ports dans domain/ports.py,
adaptateurs dans infra/), pas Clean Architecture au sens Uncle Bob.
Aligne la terminologie dans README, docs/architecture.md, ADR guide,
audit master, fiche audit 01, et la nav mkdocs.

Les noms de fichiers et la commande /audit:clean-architecture restent
stables pour preserver les liens croises et les skills existants.

* feat(settings): add paste-image size/type limits surfaced via /api/health

Introduces MAX_PASTE_IMAGE_SIZE_MB (default 10) and
PASTE_ALLOWED_IMAGE_TYPES (default image/png,image/jpeg,image/webp)
env vars so the upcoming Verify-mode clipboard-paste handler can
validate client-side against the same limits the backend enforces.

Follows the existing MAX_FILE_SIZE_MB pattern. Ships the accepted
design doc at docs/design/195-copy-paste-image-verify-mode.md.

Refs #195
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
8103460e9c feat(reasoning): reasoning-trace viewer v1 with SQLite-backed graph
Adds the `docling-agent` reasoning-trace viewer as a Studio tunnel, per
`docs/design/reasoning-trace.md`. Users pick an analyzed document, import
a RAGResult JSON, and the iterations are overlaid on the document graph.

Graph source is decoupled from Neo4j: a new pure builder
(`infra/docling_graph.build_graph_payload`) reads `document_json` from
SQLite and emits the same Cytoscape-shaped payload that `fetch_graph`
returns from Neo4j. Neo4j stays exclusive to the Maintain ingestion
pipeline. Shared DoclingDocument helpers live in `infra/docling_tree.py`
so TreeWriter and the builder can't drift on label taxonomy or tree walks.

Also removes the Cytoscape minimap (cytoscape-navigator) from GraphView:
second render instance hurt perf on large documents for no UX win.

Backend
- new `GET /api/documents/:id/reasoning-graph` (SQLite-only)
- new `infra/docling_tree.py`, `infra/docling_graph.py`
- `analysis_repo.find_latest_completed_by_document`
- tests: `test_docling_graph.py` (builder), `test_graph_api.py` (endpoint)

Frontend
- `features/reasoning/` — store, overlay, types, panel, import dialog,
  workspace, doc picker
- new `ReasoningPage` + `/reasoning` and `/reasoning/:docId` routes
- `GraphView` gains a `fetcher` prop so reasoning can inject the
  SQLite-backed fetcher while Maintain keeps using the Neo4j one
- drops minimap (nav container, dep, CSS)
- legend filters + section parenting extracted for reuse
- i18n base strings (FR + EN)
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
25a8794a0f docs(neo4j): ADR-001 graph viz lib + 200-page endpoint cap
Document the Cytoscape.js vs vis-network / Neovis / D3 / Reagraph analysis
for the graph view, and make the 200-page cap on /api/documents/{id}/graph
explicit (HTTP 413 + truncated flag beyond the cap).

Refs #186
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
712fc3f1cd feat(neo4j): Day 1 — compose service, driver, schema bootstrap
Add Neo4j as an optional graph-native storage layer (ingestion profile).
Introduces infra/neo4j with a singleton async driver wrapper and an
idempotent bootstrap of constraints + indexes, wired into the FastAPI
lifespan. Integration tests skip when no live Neo4j is reachable.

Refs #186
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
845425af62 chore: add Coming soon to batch chunking notice, set BATCH_PAGE_SIZE default to 10 2026-04-13 13:23:27 +02:00
Pier-Jean Malandrino
9602ae6d94 refactor: move BATCH_PAGE_SIZE default from Dockerfile to settings.py
Default value of 5 is now in the application code (settings.py) instead
of only in the Docker image ENV. Consistent across all deployment modes
(dev local, Docker, tests). Aligned docker-compose files and docs.
2026-04-13 11:32:33 +02:00
Pier-Jean Malandrino
ba54427445 feat(#180): feature-flag ingestion pipeline and add brainless one-liner Quick Start
- Conditionally mount ingestion router only when OpenSearch + embedding are configured
- Add `ingestionAvailable` field to /api/health response
- Add `ingestion` feature flag to frontend (hides Search nav, Ingest button,
  OpenSearch badge, indexed badges/filters when disabled)
- Skip ingestion polling when flag is off
- Make OpenSearch + embedding optional in docker-compose via profiles
- Add docker-compose.ingestion.yml override for full-stack ingestion
- Set BATCH_PAGE_SIZE=5 default in Docker local image
- Lead Quick Start with one-liner docker run command
- Document ingestion as opt-in with dedicated section
- Add BATCH_PAGE_SIZE, MAX_FILE_SIZE_MB, MAX_PAGE_COUNT, RATE_LIMIT_RPM to config tables
- Update test counts (380 backend, 159 frontend)
- Date CHANGELOG 0.4.0, bump frontend version to 0.4.0
- Sync CONTRIBUTING.md with E2E Karate test sections

Closes #180
2026-04-13 11:18:56 +02:00
Pier-Jean Malandrino
f6030bb2f1 fix(docs): resolve mkdocs strict mode build failure (#145)
- Add all process docs to mkdocs.yml nav (15 missing pages)
- Replace relative links to files outside docs/ with absolute GitHub URLs
- Add docs/audit/ directory (templates + reports)
- Add validation config to allow unlisted audit reports as info-level
2026-04-10 15:11:29 +02:00
Pier-Jean Malandrino
941028e705 ci: add release-gate pipeline, fix CI duplication, add OSS playbook docs
- Add release-gate.yml: 11-job pipeline (lint, tests, Docker build/smoke,
  Trivy scan, dep audit, audit checks, e2e API+UI, PR summary comment)
- Fix CI double-trigger on release branches (push + PR)
- Add CODE_OF_CONDUCT.md, SECURITY.md, PR template
- Add docs: git-workflow, architecture (ADR), release, operations, community
- Add profiles/fastapi-vue for automated audit checks
- Add docs/PROCESSES.md: index of 19 available processes
2026-04-09 13:20:43 +02:00
Pier-Jean Malandrino
fdbab2f49e feat: add Karate UI e2e tests with data-e2e selectors (#124)
- Add e2e/ui/ as peer project to e2e/api/ (own pom.xml, runner, config)
- 5 critical UI journeys: upload, delete, analysis, batch-progress, rechunk
- 4 local-only tests: sidebar, i18n, error-states, pipeline-options
- 1 full happy path workflow covering all modes
- Add data-e2e attributes on all tested Vue components (decoupled from CSS)
- Add CONVENTIONS.md with 7 golden rules for writing Karate UI tests
- Update CI with dedicated e2e-ui job (Chrome headless, --no-sandbox)
- Update docs/contributing.md with UI test instructions

Closes #124
2026-04-08 17:53:52 +02:00
Pier-Jean Malandrino
09c5e5e89f Replace architecture diagram with new global/docker schemas 2026-04-05 10:53:21 +02:00
Pier-Jean Malandrino
36be9027c1 Document infra layer, feature flags, rate limiting, analysis lifecycle, local vs remote modes 2026-04-05 10:45:25 +02:00
Pier-Jean Malandrino
3bbff10d1a Add chunking options and missing env vars to getting-started docs 2026-04-05 10:45:21 +02:00
Pier-Jean Malandrino
ff7010bd6b Add missing features to docs index (chunking, feature flags, rate limiting, deployment modes) 2026-04-05 10:45:17 +02:00
Pier-Jean Malandrino
18316f513e Fix Docker image tag in docs quick start (latest → latest-local) 2026-04-05 10:25:57 +02:00
Pier-Jean Malandrino
ab6d42aecd Move bbox.py from domain/ to infra/ and unify coordinate logic
domain/ must be pure with no external dependencies. bbox.py imports
docling_core and belongs in infra/. Also refactor ServeConverter to
use the canonical to_topleft_list via BoundingBox instead of
duplicated manual coordinate conversion. Move docling-core to base
requirements since it is now needed in both modes.
2026-04-03 13:17:26 +02:00
Pier-Jean Malandrino
48b7d5d3e8 Use CPU-only torch in local image to reduce size from 5.8GB to 1.9GB
Install torch and torchvision from the CPU-only index before docling
to avoid pulling CUDA/nvidia/triton dependencies. Update documentation
with measured image sizes (270MB remote, 1.9GB local).
2026-04-03 09:52:22 +02:00
Pier-Jean Malandrino
aa7c539cc3 Update documentation for remote/local image variants
Reflect the two Docker targets across README, getting-started,
contributing guides, and .env.example with new configuration
variables (CONVERSION_ENGINE, DOCLING_SERVE_URL, DOCLING_SERVE_API_KEY).
2026-04-03 09:24:41 +02:00
Pier-Jean Malandrino
1331582e22 Update logo with final version 2026-04-02 17:28:22 +02:00
Pier-Jean Malandrino
d22f2e973e Add GIF in README 2026-03-23 10:55:54 +01:00
Pier-Jean Malandrino
c2aeec3092 Add live documentaiton and change architecture diagram 2026-03-23 09:51:03 +01:00
Pier-Jean Malandrino
e2fe935b63 Build true detailed documentaiton 2026-03-22 08:56:00 +01:00
pjmalandrino
7bb4f8e298 Split the results of OCR 2026-03-20 21:45:26 +01:00
pjmalandrino
2e1518d26d Update documentation 2026-03-20 17:41:17 +01:00
pjmalandrino
ac1b68ac18 Final fixes before first release 2026-03-17 17:09:34 +01:00
pjmalandrino
50ba1d43dd Make a nex project base, managing new Docling version 2026-03-17 08:43:00 +01:00
pjmalandrino
4d998251a9 Adapt gitignore 2025-05-30 10:17:47 +02:00