Commit graph

354 commits

Author SHA1 Message Date
Pier-Jean Malandrino
f10cd01594 feat(#207): document-centric routing skeleton
Adds the routing scaffold for the doc-centric pivot. Each new route
renders a placeholder page until E3/E4/E5 implement them; legacy
routes (/studio, /documents, /history, /search, /reasoning) keep
working in parallel.

New routes (Vue Router, history mode):
  /docs                   library (placeholder, #211)
  /docs/new               import (placeholder, #214)
  /docs/:id?mode=         workspace (placeholder, #216)
  /index                  stores list (placeholder, 0.7.0)
  /index/:store           store detail (placeholder)
  /index/:store/query     RAG playground (placeholder)
  /runs                   run history (placeholder)
  /runs/:id               run detail (placeholder)

Mode parsing
- shared/routing/modes.ts: DocMode union ('ask'|'inspect'|'chunks'),
  parseMode() returns the default ('ask') for missing or unknown
  values. #210 will layer feature-flag-aware redirection on top.

Route names
- shared/routing/names.ts: ROUTES typed const so callers do
  router.push({ name: ROUTES.DOC_WORKSPACE, ... }) instead of
  stringly-typed names.

Pages
- ComingSoonShell shared component: card + back-home link, themed
  with existing CSS tokens.
- 8 thin placeholder pages (one per new route) that compose the shell
  and forward route params.
- i18n keys under comingSoon.* added in fr + en.

Tests
- shared/routing/modes.test.ts (10 cases): isDocMode + parseMode +
  ALL_MODES invariants.
- app/router/router.test.ts (5 cases): every doc-centric route
  resolves to a component, legacy routes still work, doc workspace
  receives id and parsed mode as props, unknown mode falls back to
  ask, unknown path redirects to home.

Routes table extracted to routes.ts so tests build a router with
createMemoryHistory() (no window required) instead of needing the
production createWebHistory() router.

Refs #207
2026-04-29 17:38:19 +02:00
Pier-Jean Malandrino
621a9e3fce docs(design): E2 design docs for 0.6.0 navigation refactor
Adds technical design docs for the navigation epic of the doc-centric
pivot:

- 207 — Document-centric routing (/docs, /docs/:id?mode=, /index/:store, /runs)
- 208 — Doc workspace breadcrumb (Studio > <doc> > <mode>)
- 209 — Sidebar nav rework (Home / Docs / Stores / Runs / Settings)
- 210 — Feature flag mode gating (deep-link redirect + flag exposure)

Status: Accepted on all four. Each doc spells out the contract,
alternatives considered, risks per audit dimension, and testing
strategy. Backwards-compatibility is preserved throughout: legacy
routes and pages keep working until E3/E4/E5 explicitly replace them.

Refs #207 #208 #209 #210
2026-04-29 17:31:41 +02:00
Pier-Jean Malandrino
77baa9d01b
Merge pull request #230 from scub-france/chore/206-lifecycle-state-migration
chore(#206): backfill 0.6.0 doc-centric data model
2026-04-29 17:11:10 +02:00
Pier-Jean Malandrino
f85195be0c chore(#206): backfill 0.6.0 doc-centric data model
CLI script to migrate existing tenants to the data model introduced
by #202-205. Idempotent and resumable — re-running is a no-op.

tools/migrate_06.py
- Step 1: backfill_lifecycle — infers documents.lifecycle_state from
  analysis_jobs.status + chunks_json presence
    Failed analyses, no completed       -> Failed
    Completed + chunks_json             -> Chunked
    Completed, no chunks_json           -> Parsed
    Otherwise                           -> Uploaded
- Step 2: materialize_chunks — promotes the latest chunks_json blob
  for each doc into rows in the new chunks table. Stable id derivation
  via uuid5(NAMESPACE_OID, '<doc>|<seq>|<text>') so re-running lands
  on the same ids
- Step 3: backfill_links — for any doc that has chunks materialized,
  creates a (doc, default-store) link in Ingested state with a freshly
  computed chunkset_hash. Treats 'chunks exist' as proxy for 'doc was
  ingested into the legacy single index' — sufficient for the typical
  pre-0.6.0 deployment, with OpenSearch reindex documented separately
- Step 4: reaggregate — applies #203's aggregate_lifecycle rule so
  doc-chunked transitions to Ingested

Persistence
- migration_progress table for resumability + per-step idempotency

CLI
    python -m tools.migrate_06              # full migration
    python -m tools.migrate_06 --dry-run    # plan only, no writes
    python -m tools.migrate_06 --only-step <name>   # rerun a phase

Tests
- 9 tests: inference rule per (completed, failed, chunks_json) tuple,
  end-to-end migration on a hand-built three-doc snapshot, idempotency
  (second run = no writes), --dry-run writes nothing, deterministic
  chunk ids across reruns

Refs #206
2026-04-29 17:10:44 +02:00
Pier-Jean Malandrino
25dc4bffe4
Merge pull request #229 from scub-france/feature/205-chunk-edits-audit-trail
feat(#205): promote chunks to first-class entities + audit trail
2026-04-29 17:10:32 +02:00
Pier-Jean Malandrino
e34ed03d05 feat(#205): promote chunks to first-class entities + audit trail
The data and domain layers for the chunks editor (#219-224 in 0.6.0).
Chunks were previously stored as a JSON blob in analysis_jobs.chunks_json;
this commit makes them first-class persisted entities with stable IDs,
soft-delete, and an immutable audit log.

Domain
- Chunk: persistent entity with id, document_id, sequence, text,
  headings, source_page, bboxes, doc_items, token_count, timestamps,
  deleted_at (soft delete)
- ChunkEdit: immutable audit row (action, actor, at, before, after,
  parents, children, reason)
- ChunkPush: snapshot of which chunk_ids landed in which store at push
- ChunkEditAction enum: insert/update/delete/merge/split
- domain/chunk_editing.py: pure operations on a chunkset (insert,
  update, delete, merge, split). Each returns a new chunkset and the
  affected chunk(s); errors raise ChunkEditingError.

Persistence
- Three new tables: chunks, chunk_edits, chunk_pushes (FK + indexes)
- SqliteChunkRepository (insert, insert_many, update, soft_delete,
  find_for_document, find_by_id; respects deleted_at)
- SqliteChunkEditRepository (append-only audit log; paginated reads
  ordered newest-first; per-chunk history)
- SqliteChunkPushRepository (per-(doc, store) latest snapshot)

Ports
- ChunkRepository, ChunkEditRepository, ChunkPushRepository protocols
  added to domain/ports.py

Tests
- 17 tests for the pure chunk-editing operations covering insert /
  update / delete / merge / split, sequence shifts, lineage, error
  paths (out-of-range, missing id, deleted target, cross-document)
- 11 tests for the three repositories: round-trips, soft-delete
  filtering, history ordering, lineage round-trip, cascade-delete with
  document, find_latest semantics

Service orchestration (ChunkEditingService — atomic chunk + audit
write) and the API endpoints land in a follow-up commit on the same
feature branch / next release. The data + domain foundation here is
what unblocks #219-224.

Refs #205
2026-04-29 17:09:59 +02:00
Pier-Jean Malandrino
1064c9899c
Merge pull request #228 from scub-france/feature/204-auto-stale-chunk-hash
feat(#204): deterministic chunkset hash for auto-stale detection
2026-04-29 17:09:47 +02:00
Pier-Jean Malandrino
45a823b973 feat(#204): deterministic chunkset hash for auto-stale detection
Adds the pure-domain hash function that summarises a chunkset for
stale-detection purposes. Recorded on each DocumentStoreLink at push
time (#203 ships the column slot); compared against the recomputed
current hash to flip a link to Stale when the source has drifted.

domain/hashing.py
- chunkset_hash(chunks: Iterable[ChunkResult]) -> str
- SHA-256 over (text, source_page, headings) per chunk
- Excludes bboxes / doc_items / token_count by design
- 0x1F separator between chunks defends against the join-attack
  (split A+B vs concat AB)

Tests
- 9 tests: determinism, sensitivity per included field, exclusion of
  rendering-only fields, join-attack resistance, order sensitivity,
  empty-input invariant
- Locked fixture: a hand-built 3-chunk input has a fixed expected hash;
  CI fails loud if anyone changes the canonical inputs without updating
  the fixture deliberately (and the release notes)

Service integration (recompute on chunk write, set on push) lands with
#205 once chunks are first-class — direct integration on the legacy
chunks_json path is deliberately deferred to keep #204 focused.

Refs #204
2026-04-29 17:09:25 +02:00
Pier-Jean Malandrino
6e691efc65
Merge pull request #227 from scub-france/feature/203-per-store-ingestion-state
feat(#203): per (document, store) ingestion state
2026-04-29 17:09:05 +02:00
Pier-Jean Malandrino
4c30e5eb8f feat(#203): per (document, store) ingestion state
Introduces the data layer for multi-store ingestion. Documents can now
live in multiple stores, each with its own Ingested/Stale/Failed state.
The doc-level lifecycle (#202) becomes the aggregate over all per-store
links, computed by a pure domain function.

Domain
- Store entity (name, slug, kind, embedder, config, is_default)
- DocumentStoreLink entity with mark_ingested / mark_stale / mark_failed
  helpers
- StoreKind and DocumentStoreLinkState enums
- aggregate_lifecycle(): pure function — Failed > Stale > Ingested
  > fallback (the doc's pre-link Uploaded/Parsed/Chunked state)

Persistence
- New tables 'stores' and 'document_store_links' with the right indexes
  (doc_id, store_id, state) and a UNIQUE (doc, store) on the link
- Default 'opensearch' store seeded idempotently in init_db, embedder
  pulled from DEFAULT_EMBEDDER (fallback bge-m3)
- SqliteStoreRepository (find_by_slug, find_by_id, get_default, …)
- SqliteDocumentStoreLinkRepository with ON CONFLICT … DO UPDATE upsert

Ports
- StoreRepository and DocumentStoreLinkRepository protocols added

Tests
- 14 new tests: seed idempotency, insert/find round-trips, UNIQUE
  constraint, cascade delete with the document, every link state
  round-trips, aggregation rule with all branches

Refs #203
2026-04-29 17:08:34 +02:00
Pier-Jean Malandrino
16861c50cf
Merge pull request #226 from scub-france/feature/202-document-lifecycle-state-machine
feat(#202): introduce Document lifecycle state machine
2026-04-29 17:08:11 +02:00
Pier-Jean Malandrino
c878ee5f3b feat(#202): introduce Document lifecycle state machine
Adds a first-class lifecycle state to every document, distinct from
AnalysisJob.status. The lifecycle describes the document as a whole and
is the foundation for the doc-centric pivot in 0.6.0.

Domain
- DocumentLifecycleState enum (Uploaded/Parsed/Chunked/Ingested/Stale/Failed)
- Document.lifecycle_state and lifecycle_state_at fields
- Document.transition_to() validates against a transition table
  (domain/lifecycle.py) and returns a DocumentLifecycleChanged event
- InvalidLifecycleTransitionError on disallowed transitions

Persistence
- ALTER TABLE documents to add the two columns (default 'Uploaded')
- New index idx_documents_lifecycle_state for filter perf
- _COLUMN_MIGRATIONS refactored to support multiple tables
- _POST_MIGRATION_DDL list for indexes on freshly-added columns
- SqliteDocumentRepository.update_lifecycle()

Services
- AnalysisService drives transitions on parse / chunk / re-chunk / fail
  via _transition_document(); idempotent and resilient (logs WARN and
  continues if a stale state is somehow encountered)

API
- DocumentResponse exposes lifecycleState + lifecycleStateAt
  (additive — existing 'status' field kept for backwards compat)

Frontend
- Document type extended with lifecycleState and lifecycleStateAt
- DocumentLifecycleState union literal mirroring the backend enum

Tests
- 24 new tests in test_lifecycle.py covering transitions, idempotency,
  invariant preservation, and event emission
- test_repos.py: round-trip + every-enum-value check + update_lifecycle
- test_chunking.py: rechunk path now mocks document_repo correctly

Refs #202
2026-04-29 15:34:25 +02:00
Pier-Jean Malandrino
f439d5e579 docs(design): E1 design docs for 0.6.0 doc-centric ingest
Adds technical design docs for the foundation of the doc-centric ingest pivot:

- 202 — Document lifecycle state machine
- 203 — Per (document, store) ingestion state
- 204 — Auto-detect Stale state via chunk content hash
- 205 — Audit trail for chunk edits (chunks → first-class entity)
- 206 — Migration of existing documents to the new model

Status: Accepted on all five. Each doc spells out the domain entities,
persistence schema, services orchestration, API contract, alternatives
considered, risks, and testing strategy. ADR placeholders called out
where load-bearing decisions warrant a follow-up document.

Refs #202 #203 #204 #205 #206
2026-04-29 15:24:16 +02:00
Pier-Jean Malandrino
17f59d4cee chore: gitignore local-only artifacts on release/0.6.0
Bootstrap commit for the doc-centric ingest release:
- .github/ISSUE_TEMPLATE/issue.md (local copy)
- docs/claude-code-commands.md (personal notes)
- docs/design/TEMPLATE.md (local design template)
- document-parser/tests/test_local_converter.py (local test scratch)
2026-04-29 15:13:43 +02:00
Pier-Jean Malandrino
2f811d41c8 fix(docs): also tolerate links.not_found for cross-tree audit links
The previous tweak only set unrecognized_links: info but the actual
warnings are 'target X is not found among documentation files' which
is the not_found validator. Setting both to info so strict-mode build
passes on the audit reports' source-file references.
2026-04-29 14:12:01 +02:00
Pier-Jean Malandrino
8825292146 fix(docs): tolerate cross-tree links from audit reports in strict mode
The audit reports under docs/audit/reports/release-*/ reference repo
source files (e.g. `[file.py:line](file.py:line)`) on purpose — they
are read with the repo open, not as standalone published pages. MkDocs
in strict mode treats those as unrecognized links and aborts the build.

Set `validation.links.unrecognized_links: info` so the warning still
prints but doesn't fail the build. Real broken doc links (anchors,
absolute paths) keep their existing levels.

Refs the doc deploy run that broke just after the v0.5.0 tag was pushed.
2026-04-29 14:04:42 +02:00
Pier-Jean Malandrino
7577a98415 fix(ci): pin Trivy to latest in release-gate (v0.69.3 was yanked)
Some checks failed
Release Docker Images / Build & push — local (push) Has been cancelled
Release Docker Images / Build & push — remote (push) Has been cancelled
aquasecurity/trivy-action@v0.35.0 defaults to Trivy CLI v0.69.3, but
that tag was removed from GitHub releases mid-run on 2026-04-29 — the
HIGH step on Security scan — local started failing at setup time:

  aquasecurity/trivy info checking GitHub for tag 'v0.69.3'
  aquasecurity/trivy crit unable to find 'v0.69.3'
  ##[error]Process completed with exit code 1.

The CRITICAL step still passed (binary was in cache from a prior run),
so this only surfaced as a HIGH-step failure — but the job exit code
still propagates and breaks the gate.

Following `latest` rather than chasing a specific tag that upstream
can yank without notice.

Refs #189
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
628027d861 fix(ci): drop paths constraint from CVE-2026-40393 trivy ignore
Trivy reports OS-package CVEs against the package name (libgbm1,
libgl1-mesa-dri, libglx-mesa0, mesa-libgallium) — not against
installed file paths. The previous `paths:` filter silently failed
to match, so the ignore was a no-op and the gate kept failing on a
CVE we explicitly chose to defer.

Trace from the failing run (#25097385670):

  Using YAML ignorefile '.trivyignore.yaml':
    - id: CVE-2026-40393
  ...
  libgbm1  CVE-2026-40393  CRITICAL  affected  25.0.7-2
  ...
  ##[error]Process completed with exit code 1.

Removing `paths:` lets the ID-only match apply across all 4 affected
Mesa packages until 2026-06-30.

Refs #189
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
a7fbdf7b1b fix(ci): wire .trivyignore.yaml into release-gate Trivy scans
The CRITICAL + HIGH Trivy steps in release-gate.yml were invoking
aquasecurity/trivy-action without the `trivyignores` input, so the
.trivyignore.yaml at the repo root (committed in #190 to mitigate
CVE-2026-40393 / Mesa) was silently ignored — the gate kept failing
on a CVE we explicitly chose to defer.

Pass `trivyignores: .trivyignore.yaml` to both Trivy steps so the file
takes effect. The HIGH step also gets it for consistency (it doesn't
fail the gate, but reporting a CVE we ignore as HIGH would be noise).

Refs #189
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
efc27932dd refactor(audit): remediate 0.5.0 audit findings — clean architecture, security, DRY, SOLID, perf
Closes the 12 MAJ raised by the release/0.5.0 audit pipeline (cf.
docs/audit/reports/release-0.5.0/summary.md → summary-reaudit.md).

Volet 1 — Reasoning architecture (audits 01/02/06/07 strengthening)
  * Domain ports: LLMProvider, ReasoningRunner, ReasoningParseError
  * Domain DTOs: LLMProviderType, ReasoningResult, ReasoningIteration
  * infra/llm/ollama_provider.py — OllamaProvider with health_check
  * infra/docling_agent_reasoning.py — runner adapter, encapsulates the
    private _rag_loop call (tracked at docling-project/docling-agent#26),
    commits OLLAMA_HOST once at boot (eliminates the per-request env race),
    translates upstream IndexError into ReasoningParseError
  * api/reasoning.py — zero coupling to docling-agent / mellea / docling-core,
    consumes app.state.reasoning_runner via the port
  * main.py — DI wires OllamaProvider + DoclingAgentReasoningRunner at boot
    when REASONING_ENABLED=true and deps are importable
  * Rename RAG_* env vars → REASONING_*, endpoint /rag → /reasoning,
    type RAGResult → ReasoningResult, frontend feature flag wiring,
    i18n strings, tests, docs (BREAKING — pre-1.0 surface, no external
    consumers in production)
  * 17 new tests: adapter unit tests with sys.modules stubs, OllamaProvider
    httpx tests, R3 concurrent-host isolation, R6 multi-iteration trace
    serialization, R13 Protocol conformance via isinstance
  * E2E Karate scenario: nav-reasoning hidden when REASONING_ENABLED=false
  * README — Live Reasoning section (env vars, archi, link to issue #26)

Bloc B — Security (audit 08, dev-only context)
  * docker-compose.yml — DEV DEFAULTS header, OpenSearch DISABLE_SECURITY_PLUGIN
    flagged as dev-only with link to OpenSearch security docs
  * main.py — boot warning if NEO4J_URI is set with the default 'changeme'
    password, so prod operators can't silently inherit it

Bloc C — DRY frontend (audit 05)
  * shared/storage/keys.ts — STORAGE_KEYS centralised (theme, locale)
  * features/settings/store.ts — dead apiUrl ref + orphan i18n keys removed
  * api/schemas.py — DOCUMENT_STATUS_UPLOADED constant

Bloc D — Quality (audits 02/06/07/09/10/12)
  * domain/ports.py — DocumentConverter.supports_page_batching property
    (LSP fix, replaces isinstance(ServeConverter) check)
  * domain/ports.py — VectorStore.ping() (encapsulation, replaces
    _vector_store._client.info() reach-around)
  * api/analyses.py + api/ingestion.py — path params {job_id} → {analysis_id}
    aligned with the user-facing terminology (URLs unchanged)
  * api/documents.py — Path.read_bytes() + generate_preview() wrapped in
    asyncio.to_thread, unblocks the FastAPI event loop on /preview
  * infra/docling_tree.py — PEP 604 union for isinstance (Ruff UP038)
  * src/__tests__/integration/ — cross-feature integration test relocated
    out of features/history/ so feature folders stay self-contained
  * Tightened terminal `assert X is not None` checks (isinstance(.., datetime),
    exact value comparisons)

Validation
  * 446 backend pytest, 202 frontend vitest — all green
  * ruff + ruff format + ESLint + Prettier + vue-tsc clean
  * Re-audit verdict: 0 CRIT / 0 MAJ, score ~94/100, GO

Closes #200
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
e203fe8b71 chore(release): cut 0.5.0 — bump frontend version + complete CHANGELOG
Lifts the audit 11 CRIT blocker (missing CHANGELOG [0.5.0] section) and
the matching MAJ on the frontend version pin.

Refs #200
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
157a779e20 fix(graph): collapse Docling InlineGroup and Picture children (#197)
Two patterns in Docling's serialization were mirrored 1:1 by the graph
projection and produced node explosions on real documents:

- An InlineGroup (paragraph of mixed style runs) emits one `groups[]`
  entry plus N `texts[]` runs. Naive iteration created one Paragraph
  node per run.
- A Picture's `children` carry internal text labels extracted by the
  layout model (flowchart boxes, chart axis labels, diagram callouts).
  Each child became its own Paragraph node, drowning the figure.

`build_collapse_index` (in the shared `infra.docling_tree` helper) now
returns the `skip_refs` set + `inline_meta` overrides for both cases.
The Neo4j `tree_writer` and the in-memory `docling_graph` consume the
same index, so both projections stay in sync.

InlineGroups are projected as a single :Paragraph carrying the
concatenated text and the union of children's provs (re-indexed).
Pictures keep their :Figure node and prov; their descendants are
dropped. Captions live in the picture's separate `captions` field, not
in `children`, so they are unaffected.
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
fe83dcdf79 feat(settings): paste-image size/type limits for #195 (#196)
* docs: rename Clean Architecture → Hexagonal Architecture (ports & adapters)

Le backend suit le pattern ports & adapters (ports dans domain/ports.py,
adaptateurs dans infra/), pas Clean Architecture au sens Uncle Bob.
Aligne la terminologie dans README, docs/architecture.md, ADR guide,
audit master, fiche audit 01, et la nav mkdocs.

Les noms de fichiers et la commande /audit:clean-architecture restent
stables pour preserver les liens croises et les skills existants.

* feat(settings): add paste-image size/type limits surfaced via /api/health

Introduces MAX_PASTE_IMAGE_SIZE_MB (default 10) and
PASTE_ALLOWED_IMAGE_TYPES (default image/png,image/jpeg,image/webp)
env vars so the upcoming Verify-mode clipboard-paste handler can
validate client-side against the same limits the backend enforces.

Follows the existing MAX_FILE_SIZE_MB pattern. Ships the accepted
design doc at docs/design/195-copy-paste-image-verify-mode.md.

Refs #195
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
cc535a982d fix(ci): ignore CVE-2026-40393 (Mesa) with expiry — Debian has no backport (#190)
Refs #189
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
84a10c472b fix(reasoning): re-scroll PDF when re-clicking the active iteration
The watch-based plumbing from iteration click to PDF scroll relied on a
"flip via null" pattern (assign null then the value) to coerce Vue into
re-firing the watcher. Vue 3 collapses synchronous mutations of the same
ref and only delivers the final value, so the trick was a no-op: a second
click on the same iteration left the document view stuck on the previous
page. The bug only showed when the trace had a single iteration — with
several, the user naturally clicks different ones and the value really
changes.

Replace the watch chain with imperative dispatch. ReasoningPanel now just
emits iterationFocus; ReasoningWorkspace handles it by calling the graph
focus and the new StructureViewer.scrollToFocused method directly. Both
side effects fire on every click regardless of state.
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
8694353d8b feat(reasoning): bidirectional PDF ↔ graph focus + DocumentView mode
Propagate Docling `self_ref` through PageElement so bboxes and graph nodes
share a stable identity. Add a Document/Graph mode switch to the reasoning
workspace; selecting a node highlights its bbox (numbered badge, focus ring,
optional dim of non-visited) and clicking a bbox re-centers the graph.
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
5b7700df83 feat(reasoning): live docling-agent runner + UX polish
Backend — live runner
- New `POST /api/documents/:id/rag` endpoint. Loads `document_json` from
  SQLite, reconstructs the DoclingDocument, wraps the model id in
  `ModelIdentifier(ollama_name=...)`, and calls `agent._rag_loop`
  off-thread (blocking sync call). Returns a `RAGResult` in the shape
  the existing v1 import path already consumes, so the frontend overlay
  is fully reused.
- `_rag_loop` is private upstream; we call it because `run()` wraps the
  answer in a synthetic DoclingDocument and drops the iteration trace.
- Settings: `RAG_ENABLED`, `OLLAMA_HOST`, `RAG_MODEL_ID`. Router mounts
  unconditionally; handler 503s when the flag is off or deps aren't
  installed. `rag_available` surfaced in `/api/health`.
- Maps known docling-agent bugs to readable HTTP errors: 502 with
  "the model couldn't produce a parseable answer" when `_rag_loop`
  raises `IndexError` from `find_json_dicts([])[0]` after 3 + 3
  rejection-sampling retries (model-dependent).
- Tests: 11 cases (flag off, query empty, no analysis, happy path,
  model_id wrap, Ollama env, IndexError → 502, other errors → 500,
  deps missing → 503).

Backend — bug fix
- Default `BATCH_PAGE_SIZE` flipped from `10` to `0` to match the
  dataclass default. The old default silently dropped `document_json`
  (see `domain/services.merge_results`) for any doc > 10 pages, which
  broke the reasoning tunnel. Set `BATCH_PAGE_SIZE>0` explicitly on
  memory-constrained deploys if batching is wanted.

Frontend — runner UX
- `features/reasoning/api.ts:runReasoning()` — POST wrapper.
- `RunReasoningDialog.vue` — query textarea + optional model_id
  override. Blocks close while running, 20-40s loading state,
  synthesises a sidecar-shaped envelope so the panel surfaces query +
  model the same way an imported trace would.
- `ReasoningWorkspace.vue` — primary "Run reasoning" button; "Import
  trace" relegated to ghost secondary.
- Store: `runDialogOpen`, `running`, `setRunning`.

Frontend — answer polish
- Answer rendered through `marked` + DOMPurify (models emit markdown
  lists; `pre-wrap` rendered them as plain "1. …" strings).
- Dedicated answer block with orange border, "ANSWER" label, "Copy"
  button (clipboard + "Copied ✓" feedback).
- IterationCard: drop the duplicate `response` block (the main answer
  is authoritative); style reasons equal to `"fallback"` (docling-agent
  `select_from_failure` placeholder) as italic muted "— no structured
  rationale".

Frontend — node details contents
- Clicking a SectionHeader (or any node with compound children) lists
  its contained elements in `NodeDetailsPanel` under a new "Contents"
  block. Children come from the same `parentMap` used for Cytoscape
  compound parenting (explicit PARENT_OF + synthetic section scope),
  inverted once and cached as a computed.
- Click a child row → pan the viewport to it + swap the selection.

Housekeeping
- `cytoscape-navigator` removed from `package-lock.json` (follow-up
  from the minimap removal in the previous commit).
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
8103460e9c feat(reasoning): reasoning-trace viewer v1 with SQLite-backed graph
Adds the `docling-agent` reasoning-trace viewer as a Studio tunnel, per
`docs/design/reasoning-trace.md`. Users pick an analyzed document, import
a RAGResult JSON, and the iterations are overlaid on the document graph.

Graph source is decoupled from Neo4j: a new pure builder
(`infra/docling_graph.build_graph_payload`) reads `document_json` from
SQLite and emits the same Cytoscape-shaped payload that `fetch_graph`
returns from Neo4j. Neo4j stays exclusive to the Maintain ingestion
pipeline. Shared DoclingDocument helpers live in `infra/docling_tree.py`
so TreeWriter and the builder can't drift on label taxonomy or tree walks.

Also removes the Cytoscape minimap (cytoscape-navigator) from GraphView:
second render instance hurt perf on large documents for no UX win.

Backend
- new `GET /api/documents/:id/reasoning-graph` (SQLite-only)
- new `infra/docling_tree.py`, `infra/docling_graph.py`
- `analysis_repo.find_latest_completed_by_document`
- tests: `test_docling_graph.py` (builder), `test_graph_api.py` (endpoint)

Frontend
- `features/reasoning/` — store, overlay, types, panel, import dialog,
  workspace, doc picker
- new `ReasoningPage` + `/reasoning` and `/reasoning/:docId` routes
- `GraphView` gains a `fetcher` prop so reasoning can inject the
  SQLite-backed fetcher while Maintain keeps using the Neo4j one
- drops minimap (nav container, dep, CSS)
- legend filters + section parenting extracted for reuse
- i18n base strings (FR + EN)
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
f8675aea83 test(chunking): add doc_items field to ChunkResult serialization test
The `doc_items` field was added to `ChunkResult` earlier in the
feature branch (used by ChunkWriter for DERIVED_FROM edges), but the
test fixture was never updated. CI caught it now that the PR is open.

Fixes: tests/test_chunking.py::TestChunkResult::test_serializable
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
bd73c4bbfd chore(lint): fix ruff violations on Neo4j files
Pre-existing ruff violations surfaced by CI on PR #188:

- TC001: move runtime `Neo4jDriver` imports into `TYPE_CHECKING`
  blocks (queries.py, chunk_writer.py, schema.py, tree_reader.py,
  tree_writer.py)
- SIM117: combine nested `async with` in chunk_writer.write_chunks
  and tree_writer.write_document
- SIM105: replace try/except/pass with `contextlib.suppress` in
  tree_writer._element_props
- F401: remove unused `Neo4jDriver` import in main._init_neo4j
- RUF100: remove unused `# noqa: E402` in tests/neo4j/conftest.py
- I001: sort imports in tests/neo4j/test_chunk_writer.py and
  test_tree_writer.py

Zero behaviour change. `ruff check .` now passes cleanly.
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
03e5dfac17 feat(ui): add Maintain step with graph visualization
Move the graph view from a Verify-tab (where it sat post-analysis, off
the main flow) to a dedicated Maintain step after Ingest, so the graph
result is visible at the natural end of the Configure → Verify →
Prepare → Ingest → Maintain pipeline.

- StudioPage: new 'maintain' mode toggle + panel rendering GraphView
- ResultTabs: remove obsolete graph tab
- i18n: add studio.maintain (fr + en)
- GraphView: fix init order — flip loading off and await nextTick before
  renderGraph so the canvas <div> is mounted when cytoscape reads its
  container ref (previous code bailed silently on null ref)
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
45cecc1115 fix(neo4j): rewrite fetch_graph with CALL subqueries
Previous query chained 6 OPTIONAL MATCH clauses for edges with no
intervening WITH collect(), producing a cartesian product. At 6 pages
(~60 elements, ~300 edges) Neo4j hit 102% CPU and hung > 5min.

Rewritten with one CALL {} subquery per node/edge type: each block
returns a single row with its collected list — no multiplication across
types. 6-page doc now returns in 213ms (was: no return).

Python reshape code (queries.py:137-210) untouched — record keys and
edge map shape preserved.

Refs: https://neo4j.com/developer/kb/using-subqueries-to-control-the-scope-of-aggregations/
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
c2550867b7 feat(neo4j): Day 3 — ChunkWriter, graph API, GraphView, README
ChunkWriter mirrors chunks into Neo4j after OpenSearch indexing, creating
HAS_CHUNK edges and DERIVED_FROM back-references to the source Elements
(via doc_items propagated from the local chunker).

Graph API: GET /api/documents/{id}/graph returns a cytoscape-shaped
payload with nodes + edges for Document / Element / Page / Chunk.
Hard cap at 200 pages returns HTTP 413 per design §8.4.

Frontend: new Graph tab in Studio results, rendered with Cytoscape.js +
dagre layout (lazy-loaded, ~175 KB gz). Legend, node styling per element
label, directional edges styled per edge type.

README gains a Neo4j section with the schema, three demo Cypher
queries, and env vars. Backend tests skip cleanly when the neo4j python
package is not installed locally.

Refs #186
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
c1d3a687ac feat(neo4j): Day 2 — TreeWriter, TreeReader, pipeline wiring
Serialize a DoclingDocument to a Neo4j graph: Document + Page + Element
nodes with dynamic specific labels (SectionHeader, Paragraph, Table,
Figure, …), plus HAS_ROOT / PARENT_OF / NEXT / ON_PAGE edges. Replace-on-
write for idempotent re-ingestion.

The reader returns the verbatim document_json stored on the Document
node — reconstruction from graph nodes is deferred to v0.6.

Wired into AnalysisService._finalize_analysis: runs after conversion,
degrades gracefully by default, fails fast when neo4j_required is set.

Refs #186
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
25a8794a0f docs(neo4j): ADR-001 graph viz lib + 200-page endpoint cap
Document the Cytoscape.js vs vis-network / Neovis / D3 / Reagraph analysis
for the graph view, and make the 200-page cap on /api/documents/{id}/graph
explicit (HTTP 413 + truncated flag beyond the cap).

Refs #186
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
712fc3f1cd feat(neo4j): Day 1 — compose service, driver, schema bootstrap
Add Neo4j as an optional graph-native storage layer (ingestion profile).
Introduces infra/neo4j with a singleton async driver wrapper and an
idempotent bootstrap of constraints + indexes, wired into the FastAPI
lifespan. Integration tests skip when no live Neo4j is reachable.

Refs #186
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
e0f8e81b63 feat: enable chunking in remote (Docling Serve) mode
Hybrid approach: reuse LocalChunker to chunk the DoclingDocument JSON
returned by Serve, so chunking works identically in both local and
remote modes without calling Serve's chunk endpoint.

Backend:
- _build_chunker() always returns LocalChunker (remove engine guard)
- Use docling-core[chunking] extra for required dependencies
- Skip client-side batching in remote mode (Serve manages its own
  resources, and batching discards document_json needed for chunking)
- Fix Serve form fields: remove generate_page_images (not a Serve
  field), use repeated form keys for to_formats and page_range
- Log Serve error response body on 4xx/5xx for diagnosis
- Fix FastAPI 204 DELETE routes missing response_model=None

Frontend:
- Update chunking feature flag to enable Prepare UI in remote mode

Closes #51
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
d0c77b7618 fix(ci): install pytestarch in backend tests job (#177)
CI was missing pytestarch dependency, causing test_architecture.py to fail
at collection time. Switch to requirements-test.txt which includes all
test dependencies.
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
97a8160821 feat: add hexagonal architecture tests with pytestarch (#177)
- Create tests/test_architecture.py with 20 automated rules:
  inter-layer dependency checks (domain, services, api, infra, persistence),
  external dependency constraints (fastapi, sqlalchemy, httpx, opensearchpy),
  and port convention enforcement (Protocol only in domain.ports)
- Add requirements-test.txt with pytestarch dependency
- Fix persistence.database importing infra.settings (read DB_PATH from env directly)
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
d62b4ad32e refactor: centralize magic numbers for page dimensions, limits, and timeout (#168)
- Move DEFAULT_PAGE_WIDTH/HEIGHT to domain/value_objects.py and import in both converters
- Add opensearch_default_limit to Settings (configurable via OPENSEARCH_DEFAULT_LIMIT env var)
- Pass settings.conversion_timeout to ServeConverter, removing independent _DEFAULT_TIMEOUT
- Update OpenSearchStore to accept default_limit from Settings via constructor
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
e7c27a6706
Merge pull request #163 from scub-france/release/0.4.0
Release/0.4.0
2026-04-14 17:22:13 +02:00
Pier-Jean Malandrino
c4057bb010 fix(e2e): update chunks synchronously in store to prevent rechunk timeout
Some checks failed
Auto-close issues on release branch merge / Close referenced issues (push) Has been cancelled
Release Docker Images / Build & push — local (push) Has been cancelled
Release Docker Images / Build & push — remote (push) Has been cancelled
ChunkPanel emitted 'rechunked' which triggered an async re-fetch via
analysisStore.select() — but Vue does not await async emit handlers,
so chunk-cards never appeared before the E2E 30s timeout.

Now rechunk/edit/delete write returned chunks directly into the
analysis store via updateChunks(), removing the async round-trip.
2026-04-14 17:06:34 +02:00
Pier-Jean Malandrino
2477a2af4b
Merge pull request #181 from scub-france/feature/feature-flag-ingestion
feat(#180): feature-flag ingestion pipeline and brainless one-liner Quick Start
2026-04-14 16:30:36 +02:00
Pier-Jean Malandrino
845425af62 chore: add Coming soon to batch chunking notice, set BATCH_PAGE_SIZE default to 10 2026-04-13 13:23:27 +02:00
Pier-Jean Malandrino
5f8b3559e3 chore: gitignore profiles/ (internal audit tooling) 2026-04-13 11:36:20 +02:00
Pier-Jean Malandrino
9602ae6d94 refactor: move BATCH_PAGE_SIZE default from Dockerfile to settings.py
Default value of 5 is now in the application code (settings.py) instead
of only in the Docker image ENV. Consistent across all deployment modes
(dev local, Docker, tests). Aligned docker-compose files and docs.
2026-04-13 11:32:33 +02:00
Pier-Jean Malandrino
ba54427445 feat(#180): feature-flag ingestion pipeline and add brainless one-liner Quick Start
- Conditionally mount ingestion router only when OpenSearch + embedding are configured
- Add `ingestionAvailable` field to /api/health response
- Add `ingestion` feature flag to frontend (hides Search nav, Ingest button,
  OpenSearch badge, indexed badges/filters when disabled)
- Skip ingestion polling when flag is off
- Make OpenSearch + embedding optional in docker-compose via profiles
- Add docker-compose.ingestion.yml override for full-stack ingestion
- Set BATCH_PAGE_SIZE=5 default in Docker local image
- Lead Quick Start with one-liner docker run command
- Document ingestion as opt-in with dedicated section
- Add BATCH_PAGE_SIZE, MAX_FILE_SIZE_MB, MAX_PAGE_COUNT, RATE_LIMIT_RPM to config tables
- Update test counts (380 backend, 159 frontend)
- Date CHANGELOG 0.4.0, bump frontend version to 0.4.0
- Sync CONTRIBUTING.md with E2E Karate test sections

Closes #180
2026-04-13 11:18:56 +02:00
Pier-Jean Malandrino
c7f9468991
Merge pull request #179 from scub-france/fix/e2e-toggle-selector-remaining
fix(e2e): use word-match selector for remaining toggle-btn references
2026-04-12 20:36:13 +02:00
Pier-Jean Malandrino
39091358ae fix(e2e): use word-match selector for remaining toggle-btn references
analysis.feature and full-ui-path.feature still used exact-match
[data-e2e=toggle-btn] which broke after #178 changed the attribute to
compound values. Switch to [data-e2e~=configure-btn].
2026-04-12 20:35:28 +02:00
Pier-Jean Malandrino
6f008ec262
Merge pull request #178 from scub-france/fix/e2e-rechunk-feature-flag
fix(e2e): replace fragile index-based toggle selection with dedicated selectors
2026-04-12 10:16:46 +02:00