Commit graph

378 commits

Author SHA1 Message Date
Pier-Jean Malandrino
8ac81b9461 test(#256): Karate UI regression for Doc tab chunk mode
Setup via API (upload + analyse + wait COMPLETED), then drive the UI
to the chunks tab and assert the chunk list renders. Includes a direct
HTTP probe on /api/documents/{id}/chunks so an API-only regression
fails the test even if the UI somehow swallows it.

Tagged @critical so it runs in the smoke suite.
2026-05-07 11:09:19 +02:00
Pier-Jean Malandrino
4a422a8464 fix(#256): drop pushDocumentToStore duplicate, align rechunk return shape
The frontend had two endpoints pointing at the same operation:
  - features/document/api.ts:pushDocumentToStore  → POST /push
  - features/chunks/api.ts:pushChunksToStore      → POST /chunks/push

Backend now exposes a single /chunks/push route (semantically more
accurate — it's chunks that get pushed, not the doc itself), so the
document-side variant is removed. document/store.ts:pushToStore now
delegates to chunks/api.

rechunkDocument returned {jobId} based on the old async-job model.
The new backend rechunk runs synchronously and returns the chunk
list directly. The DocsLibraryPage caller now reports a count
(via new docs.rechunkDone i18n key) instead of a fake job id.

Adds data-e2e=tab-{mode} on the doc workspace tab strip so the
new Karate scenario can target the chunks tab without text matching.

Extends chunks/api.test.ts with HTTP error-propagation guards on
every call — the original 404 bug would have surfaced in CI.
2026-05-07 11:09:13 +02:00
Pier-Jean Malandrino
70972e807a feat(#256): add /api/documents/{id}/chunks routes + ChunkService
The frontend was wired against /api/documents/{id}/chunks/* (canonical
doc-centric chunkset) but the backend never exposed those routes — the
chunk tab in the doc workspace 404'd. The domain entities (Chunk,
ChunkEdit, ChunkPush) and persistence repos already existed since #205;
what was missing was the service + API layer that connects them.

ChunkService owns all canonical chunkset invariants (sequence ordering,
soft-delete + audit log atomicity) and shares the chunker port with
AnalysisService so chunking strategy stays a single implementation.

AnalysisService grew a duck-typed promoter hook that copies the chunks
of the first successful analysis into the canonical chunkset. The hook
is idempotent so subsequent ad-hoc analyses (Studio / OCR Debug) never
overwrite hand-edited state.

Routes added (all additive, /api/documents prefix):
  GET    /{id}/chunks
  POST   /{id}/chunks
  PATCH  /{id}/chunks/{chunkId}
  DELETE /{id}/chunks/{chunkId}
  POST   /{id}/chunks/{chunkId}/split
  POST   /{id}/chunks/merge
  POST   /{id}/rechunk
  GET    /{id}/tree
  GET    /{id}/diff?store=...
  POST   /{id}/chunks/push
2026-05-07 11:09:04 +02:00
Pier-Jean Malandrino
43a434c854 docs(#256): flesh out design doc with chunk service + 9 routes plan 2026-05-07 10:49:23 +02:00
Pier-Jean Malandrino
7fe9a8a823 docs(#256): scaffold design doc for chunk-mode 404 2026-05-07 10:22:45 +02:00
Pier-Jean Malandrino
df1bbef143 feat(#251): expose backend error detail in apiFetch errors
Le wrapper apiFetch jetait `API error: 422` sans le body. Désormais lit
detail (string ou liste Pydantic) et le concatène au message :
`422: Neo4j config requires a non-empty 'index_name'`.

Permet aux pages StoreCreatePage / StoreEditPage de surfacer la cause
réelle du refus serveur dans l'errorMessage du form.
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
151505cdbf feat(#251): add Neo4j as a StoreKind
- domain : StoreKind.NEO4J ajouté à l'enum
- service : validation config par kind étendue (Neo4j requires non-empty index_name)
- frontend : Neo4jConfigForm (index_name + database optionnelle), dispatcher StoreConfigForm, dropdown kind, i18n FR/EN
- tests service : 2 cas Neo4j (create OK + missing index_name 422)

Auth (URI/user/password) reste pilotée par les variables d'environnement
globales (cf. infra/neo4j.py) — la config par store ne porte que les
paramètres routables (index, database).
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
d55066005d feat(#251): nav CRUD sur StoresListPage + StoreDetailPage
- StoresListPage : bouton "Nouveau store" header + ligne action Edit/Delete (delete bloqué sur 'default'), navigation par slug, colonne "Défaut", confirm window.confirm
- StoreDetailPage : actions Edit / Delete dans le header, redirection vers la liste après suppression
- api.test.ts : couvre la nouvelle shape StoreInfo (slug + isDefault + embedder)
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
7ad25aeb2a feat(#251): API client typé create/update/delete + StoreForm + Create/Edit pages
Frontend store API client :
- Types StoreInfo (avec slug + isDefault + embedder), StoreDetail, StoreCreatePayload, StoreUpdatePayload
- Fonctions createStore (POST), updateStore (PATCH), deleteStore (DELETE), fetchStore (GET /:slug)
- Tests Vitest étendus pour les 4 nouveaux endpoints

Composants formulaire (features/store/ui/) :
- StoreForm : champs communs (name/slug/kind/embedder/isDefault) avec validation slug locale, lock du slug pour 'default'
- StoreConfigForm : dispatcher dynamique par kind (extensible)
- OpenSearchConfigForm : champ indexName, validation requise

Pages CRUD :
- StoreCreatePage (/index/new) : formulaire + redirect liste
- StoreEditPage (/index/:store/edit) : préremplit via fetchStore, redirige vers la fiche

Routes + i18n :
- Routes ROUTES.STORE_CREATE et ROUTES.STORE_EDIT ajoutées (names + routes.ts)
- Clés i18n FR/EN sous stores.* et storeForm.*
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
245ef40a24 feat(#251): /api/stores router + wiring + schemas Pydantic + tests API
- Schemas camelCase : StoreInfoResponse, StoreResponse, StoreCreate/UpdateRequest, StoreDocEntryResponse
- Router /api/stores : GET (list), POST (create 201), GET/{slug}, PATCH/{slug}, DELETE/{slug} (204)
- Endpoints documents-in-store : GET /{slug}/documents, DELETE /{slug}/documents/{docId}
- Mapping erreurs StoreServiceError → HTTPException via http_status hint
- StoreService étendu : list_documents/remove_document avec injection optionnelle de document_repo (filename humain)
- Wiring lifespan : SqliteStoreRepository + SqliteDocumentStoreLinkRepository + StoreService sur app.state
- include_router(stores_router) après documents/analyses
- Tests API : 18 cas couvrant 200/201/204/404/409/422 + camelCase + cycle complet
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
9d2e9ab0f4 feat(#251): StoreService — CRUD orchestration + per-kind config validation
Couche service entre l'API et les repos SQLite :
- list_stores avec enrichissement document_count via document_store_links
- get/create/update/delete avec validations
- Slug normalisé lowercase, motif kebab-case strict
- Unicité name + slug (409 sur collision)
- Single-default invariant via clear_default_except
- Validation config par kind (OpenSearch require index_name) — extensible
- Delete refusé sur le store seedé "default" et sur tout store avec liens
- Erreurs typées (Validation/NotFound/Conflict) avec hint http_status
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
ce140bab43 feat(#251): SqliteStoreRepository — update/delete/find_by_name/clear_default_except
Étend le repo avec les méthodes manquantes pour le CRUD :
- find_by_name : valider unicité name au create/update
- update : remplace les champs mutables (id et created_at intacts)
- clear_default_except : promotion mono-default lors d'un changement de default
- delete : retourne True/False selon suppression effective

Tests dédiés sur chacune des nouvelles méthodes.
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
53d28c2282 fix(workspace): reset doc state on navigation (bbox / analysis state leak)
Quand l'utilisateur naviguait d'un doc à un autre via /docs/:id, Vue
réutilisait l'instance du composant — onMounted ne se redéclenchait pas,
laissant l'analyse précédente (et donc les bbox du StructureViewer)
visibles tant que le nouveau fetch n'était pas terminé.

- DocWorkspacePage: watch props.id pour recharger le doc + :key sur les
  onglets (forçe un remount propre, reset l'état interne du
  StructureViewer comme selectedPage/hiddenTypes).
- DocInspectTab / DocAskTab: watch props.docId avec immediate:true,
  reset l'état au début du load + guard de race condition (ignore les
  réponses pour un docId obsolète).
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
ed2f47fdc8 chore: re-track frontend/package-lock.json (CI requires it)
Le commit précédent l'avait untracké. Or .github/workflows/ci.yml et
release-gate.yml référencent explicitement le lockfile pour le cache npm
et la step `npm ci` exige sa présence. Resultat : CI cassé.

On garde docs/git-workflow/autonomy-mode.md ignoré (artefact local).
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
bb24098c21 chore: gitignore local autonomy-mode doc + untrack frontend lockfile
- Ajoute docs/git-workflow/autonomy-mode.md aux artefacts locaux ignorés.
- Retire frontend/package-lock.json du suivi git (reste sur disque).
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
1e9e57ab2d feat(#243,#244,#245): E8 — Stores pages (list, detail, query)
- features/store: api + types (StoreInfo, StoreDocEntry, QueryResult)
- StoresListPage: table with connection status, click → StoreDetail
- StoreDetailPage: doc table with StatusBadge, remove, bulk remove, link to query
- StoreQueryPage: query form (Ctrl+Enter), topK, scored results with doc links
- i18n: stores.* / storeDetail.* / storeQuery.* keys (FR + EN)

Closes #243
Closes #244
Closes #245
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
2718d86262 feat(#242): DocAskTab — Reasoning intégré au workspace (doc structure + question + trace cliquable) 2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
0062890830 feat(#240,#241): DocInspectTab — Markdown/Elements/Images from Docling analysis 2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
e40b5ba0a5 feat(#225,#224): vocabulary rename index→ingest + stale stores strip
- #225: rename all push/index/pousser/indexé labels to ingest/ingérer/ingéré
  across status tooltips, docs/chunks push modals, and legacy ingestion panel
- #224: add DocStoreLink type + storeLinks on Document; StaleStoresStrip
  component shows per-store state badges with one-click re-ingest buttons
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
88f109ed4c feat(#216): E4/E5 — Doc workspace shell, DocTreeRail, DocWorkspaceHeader, editable chunks editor
Closes #216
Closes #217
Closes #218
Closes #219
Closes #220
Closes #221
Closes #222
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
6527307553 feat(#211): E3 — Document library (/docs), filters, bulk actions, multi-file import, StatusBadge
- StatusBadge component with 6 lifecycle states, compact/full variants, tooltip (#215)
- DocsLibraryPage: table (Name/Status/Stores/Updated), empty state (#211)
- Filter bar: status pills, store chips, debounced search, URL sync (#212)
- Multi-select with sticky bulk bar: Re-chunk, Push to store modal, Delete (#213)
- DocsNewPage: multi-file drop zone with per-file progress, sequential upload (#214)
- Document.stores?: string[] field, formatRelativeTime(iso, locale) helper
- rechunkDocument / pushDocumentToStore API + store actions (rechunk / pushToStore)
- i18n keys status.*, docs.*, docsNew.* in FR + EN

Closes #211, Closes #212, Closes #213, Closes #214, Closes #215
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
7b6d64c434 feat(#210): feature-flag mode gating + deep-link redirect
Backend
- HealthResponse exposes inspectModeEnabled / chunksModeEnabled /
  askModeEnabled (additive; defaults true). main.py /api/health
  populates them from settings.
- infra/settings.py: three new env-var-driven booleans (defaults true)
  parsed in from_env() like the existing reasoning_enabled flag.
- tests/test_api_endpoints.py: extra assertion that /api/health
  surfaces the three new fields with their defaults.

Frontend — flag store
- features/feature-flags/store.ts: FeatureFlag union extended with
  inspectMode / chunksMode / askMode. New entries in featureRegistry
  are gated on context fields populated from health. Missing fields
  fall back to true so a frontend pointed at an older backend keeps
  every mode visible.
- store gains a modeFlags() helper returning Record<DocMode, boolean>
  so the routing guard does not need to know the FeatureFlag union.

Frontend — routing
- shared/routing/resolveMode.ts: pure resolver. If the requested mode
  is enabled, return it; else first enabled in priority ask > chunks
  > inspect; else null.
- app/router/index.ts: beforeEach guard scoped to ROUTES.DOC_WORKSPACE.
  Disabled mode → rewrite ?mode= to the first enabled one. All three
  off → redirect to /docs?reason=no-mode-enabled.

Frontend — flash
- pages/DocsLibraryPage.vue: shows a banner when ?reason=no-mode-enabled
  is set. #211 will move this into the proper library page banner.
- i18n flags.allModesDisabled added in fr + en.

Tests
- shared/routing/resolveMode.test.ts (6 cases): every (requested,
  enabled) combination including all-disabled, priority order,
  missing requested.
- features/feature-flags/store.test.ts: three new cases covering the
  new fields in /api/health, fall-back-to-true on missing fields, and
  modeFlags() shape.

Refs #210
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
058d718116 feat(#209): rework sidebar nav (Home / Docs / Stores / Runs / Settings)
The sidebar was the project's actual nav (the design doc called it
'top nav' but the existing app is sidebar-driven). Five entries
reflecting the doc-centric IA replace the seven analysis-centric ones.

Sidebar
- shared/ui/AppSidebar.vue: data-driven over a NavItem[] table
  instead of seven hard-coded RouterLinks. Inline SVG icon
  components keep the visual weight close to the previous version
  with no new dependency.
- Five entries: Home / Docs (★ primary, bolder label) / Stores /
  Runs / Settings.
- Active state via matchesActive(path, prefixes): exact match for
  Home, prefix match (with segment / query boundary) for the others.
  Pure helper in shared/ui/navActive.ts, unit tested.

Removed from the sidebar (legacy pages still reachable by URL):
  Studio, Documents, Search, Reasoning, History.

i18n
- nav.docs / nav.stores / nav.runs added in fr + en, grouped under
  the 0.6.0 nav block.
- Legacy nav labels (nav.studio / nav.documents / nav.history /
  nav.reasoning / nav.search) kept — the legacy pages still render
  headings using them. Duplicate nav.search entries removed (the key
  now lives once per locale, in the nav block).

Tests
- shared/ui/navActive.test.ts (5 cases): exact-match for Home,
  prefix boundary semantics, multi-prefix matching, empty list edge.

Refs #209
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
ab5de2aac3 feat(#208): doc workspace breadcrumb (Studio > <doc> > <mode>)
Adds the breadcrumb anchoring the user across modes on the doc
workspace. Empty / hidden on routes that don't opt in.

Components
- shared/breadcrumb/AppBreadcrumb.vue: data-driven, accessible
  (<nav aria-label>, <ol>, aria-current=page on the leaf).
  Renders nothing when crumbs.length === 0.
- shared/breadcrumb/types.ts: Crumb = LinkCrumb | LeafCrumb
  discriminated union.
- shared/breadcrumb/store.ts: Pinia store + useCrumbs(source)
  composable that auto-clears on unmount. Accepts a static array
  OR a reactive ref/computed so pages with async doc fetches can
  rebuild crumbs as data lands.
- shared/breadcrumb/text.ts: truncate(text, max) helper.

Wiring
- App.vue main outlet now sits below <AppBreadcrumb>; the shell
  reads from useBreadcrumbStore so pages don't need teleports.
- DocWorkspacePage provides Studio > <id-truncated> > <mode-label>;
  once the doc is fetched (#216 / E4), the id placeholder will
  swap for the truncated filename.

i18n
- breadcrumb.aria, breadcrumb.studio, breadcrumb.mode.{ask,inspect,
  chunks} added in fr + en.

Tests
- shared/breadcrumb/text.test.ts: 5 cases on truncate.
- shared/breadcrumb/store.test.ts: store actions + useCrumbs reactive/
  static seeding (the lifecycle-clear path is covered end-to-end by
  the integration tests landing in #211 / #216 — the project doesn't
  use @vue/test-utils so a unit test for onBeforeUnmount is overkill).

Refs #208
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
47e0d197b3 feat(#207): document-centric routing skeleton
Adds the routing scaffold for the doc-centric pivot. Each new route
renders a placeholder page until E3/E4/E5 implement them; legacy
routes (/studio, /documents, /history, /search, /reasoning) keep
working in parallel.

New routes (Vue Router, history mode):
  /docs                   library (placeholder, #211)
  /docs/new               import (placeholder, #214)
  /docs/:id?mode=         workspace (placeholder, #216)
  /index                  stores list (placeholder, 0.7.0)
  /index/:store           store detail (placeholder)
  /index/:store/query     RAG playground (placeholder)
  /runs                   run history (placeholder)
  /runs/:id               run detail (placeholder)

Mode parsing
- shared/routing/modes.ts: DocMode union ('ask'|'inspect'|'chunks'),
  parseMode() returns the default ('ask') for missing or unknown
  values. #210 will layer feature-flag-aware redirection on top.

Route names
- shared/routing/names.ts: ROUTES typed const so callers do
  router.push({ name: ROUTES.DOC_WORKSPACE, ... }) instead of
  stringly-typed names.

Pages
- ComingSoonShell shared component: card + back-home link, themed
  with existing CSS tokens.
- 8 thin placeholder pages (one per new route) that compose the shell
  and forward route params.
- i18n keys under comingSoon.* added in fr + en.

Tests
- shared/routing/modes.test.ts (10 cases): isDocMode + parseMode +
  ALL_MODES invariants.
- app/router/router.test.ts (5 cases): every doc-centric route
  resolves to a component, legacy routes still work, doc workspace
  receives id and parsed mode as props, unknown mode falls back to
  ask, unknown path redirects to home.

Routes table extracted to routes.ts so tests build a router with
createMemoryHistory() (no window required) instead of needing the
production createWebHistory() router.

Refs #207
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
8234a3c23f docs(design): E2 design docs for 0.6.0 navigation refactor
Adds technical design docs for the navigation epic of the doc-centric
pivot:

- 207 — Document-centric routing (/docs, /docs/:id?mode=, /index/:store, /runs)
- 208 — Doc workspace breadcrumb (Studio > <doc> > <mode>)
- 209 — Sidebar nav rework (Home / Docs / Stores / Runs / Settings)
- 210 — Feature flag mode gating (deep-link redirect + flag exposure)

Status: Accepted on all four. Each doc spells out the contract,
alternatives considered, risks per audit dimension, and testing
strategy. Backwards-compatibility is preserved throughout: legacy
routes and pages keep working until E3/E4/E5 explicitly replace them.

Refs #207 #208 #209 #210
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
0c09274836 chore(#206): backfill 0.6.0 doc-centric data model
CLI script to migrate existing tenants to the data model introduced
by #202-205. Idempotent and resumable — re-running is a no-op.

tools/migrate_06.py
- Step 1: backfill_lifecycle — infers documents.lifecycle_state from
  analysis_jobs.status + chunks_json presence
    Failed analyses, no completed       -> Failed
    Completed + chunks_json             -> Chunked
    Completed, no chunks_json           -> Parsed
    Otherwise                           -> Uploaded
- Step 2: materialize_chunks — promotes the latest chunks_json blob
  for each doc into rows in the new chunks table. Stable id derivation
  via uuid5(NAMESPACE_OID, '<doc>|<seq>|<text>') so re-running lands
  on the same ids
- Step 3: backfill_links — for any doc that has chunks materialized,
  creates a (doc, default-store) link in Ingested state with a freshly
  computed chunkset_hash. Treats 'chunks exist' as proxy for 'doc was
  ingested into the legacy single index' — sufficient for the typical
  pre-0.6.0 deployment, with OpenSearch reindex documented separately
- Step 4: reaggregate — applies #203's aggregate_lifecycle rule so
  doc-chunked transitions to Ingested

Persistence
- migration_progress table for resumability + per-step idempotency

CLI
    python -m tools.migrate_06              # full migration
    python -m tools.migrate_06 --dry-run    # plan only, no writes
    python -m tools.migrate_06 --only-step <name>   # rerun a phase

Tests
- 9 tests: inference rule per (completed, failed, chunks_json) tuple,
  end-to-end migration on a hand-built three-doc snapshot, idempotency
  (second run = no writes), --dry-run writes nothing, deterministic
  chunk ids across reruns

Refs #206
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
648f2a5c0d feat(#205): promote chunks to first-class entities + audit trail
The data and domain layers for the chunks editor (#219-224 in 0.6.0).
Chunks were previously stored as a JSON blob in analysis_jobs.chunks_json;
this commit makes them first-class persisted entities with stable IDs,
soft-delete, and an immutable audit log.

Domain
- Chunk: persistent entity with id, document_id, sequence, text,
  headings, source_page, bboxes, doc_items, token_count, timestamps,
  deleted_at (soft delete)
- ChunkEdit: immutable audit row (action, actor, at, before, after,
  parents, children, reason)
- ChunkPush: snapshot of which chunk_ids landed in which store at push
- ChunkEditAction enum: insert/update/delete/merge/split
- domain/chunk_editing.py: pure operations on a chunkset (insert,
  update, delete, merge, split). Each returns a new chunkset and the
  affected chunk(s); errors raise ChunkEditingError.

Persistence
- Three new tables: chunks, chunk_edits, chunk_pushes (FK + indexes)
- SqliteChunkRepository (insert, insert_many, update, soft_delete,
  find_for_document, find_by_id; respects deleted_at)
- SqliteChunkEditRepository (append-only audit log; paginated reads
  ordered newest-first; per-chunk history)
- SqliteChunkPushRepository (per-(doc, store) latest snapshot)

Ports
- ChunkRepository, ChunkEditRepository, ChunkPushRepository protocols
  added to domain/ports.py

Tests
- 17 tests for the pure chunk-editing operations covering insert /
  update / delete / merge / split, sequence shifts, lineage, error
  paths (out-of-range, missing id, deleted target, cross-document)
- 11 tests for the three repositories: round-trips, soft-delete
  filtering, history ordering, lineage round-trip, cascade-delete with
  document, find_latest semantics

Service orchestration (ChunkEditingService — atomic chunk + audit
write) and the API endpoints land in a follow-up commit on the same
feature branch / next release. The data + domain foundation here is
what unblocks #219-224.

Refs #205
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
4b59ef2330 feat(#204): deterministic chunkset hash for auto-stale detection
Adds the pure-domain hash function that summarises a chunkset for
stale-detection purposes. Recorded on each DocumentStoreLink at push
time (#203 ships the column slot); compared against the recomputed
current hash to flip a link to Stale when the source has drifted.

domain/hashing.py
- chunkset_hash(chunks: Iterable[ChunkResult]) -> str
- SHA-256 over (text, source_page, headings) per chunk
- Excludes bboxes / doc_items / token_count by design
- 0x1F separator between chunks defends against the join-attack
  (split A+B vs concat AB)

Tests
- 9 tests: determinism, sensitivity per included field, exclusion of
  rendering-only fields, join-attack resistance, order sensitivity,
  empty-input invariant
- Locked fixture: a hand-built 3-chunk input has a fixed expected hash;
  CI fails loud if anyone changes the canonical inputs without updating
  the fixture deliberately (and the release notes)

Service integration (recompute on chunk write, set on push) lands with
#205 once chunks are first-class — direct integration on the legacy
chunks_json path is deliberately deferred to keep #204 focused.

Refs #204
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
d81f36d899 feat(#203): per (document, store) ingestion state
Introduces the data layer for multi-store ingestion. Documents can now
live in multiple stores, each with its own Ingested/Stale/Failed state.
The doc-level lifecycle (#202) becomes the aggregate over all per-store
links, computed by a pure domain function.

Domain
- Store entity (name, slug, kind, embedder, config, is_default)
- DocumentStoreLink entity with mark_ingested / mark_stale / mark_failed
  helpers
- StoreKind and DocumentStoreLinkState enums
- aggregate_lifecycle(): pure function — Failed > Stale > Ingested
  > fallback (the doc's pre-link Uploaded/Parsed/Chunked state)

Persistence
- New tables 'stores' and 'document_store_links' with the right indexes
  (doc_id, store_id, state) and a UNIQUE (doc, store) on the link
- Default 'opensearch' store seeded idempotently in init_db, embedder
  pulled from DEFAULT_EMBEDDER (fallback bge-m3)
- SqliteStoreRepository (find_by_slug, find_by_id, get_default, …)
- SqliteDocumentStoreLinkRepository with ON CONFLICT … DO UPDATE upsert

Ports
- StoreRepository and DocumentStoreLinkRepository protocols added

Tests
- 14 new tests: seed idempotency, insert/find round-trips, UNIQUE
  constraint, cascade delete with the document, every link state
  round-trips, aggregation rule with all branches

Refs #203
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
ff0b795b6d feat(#202): introduce Document lifecycle state machine
Adds a first-class lifecycle state to every document, distinct from
AnalysisJob.status. The lifecycle describes the document as a whole and
is the foundation for the doc-centric pivot in 0.6.0.

Domain
- DocumentLifecycleState enum (Uploaded/Parsed/Chunked/Ingested/Stale/Failed)
- Document.lifecycle_state and lifecycle_state_at fields
- Document.transition_to() validates against a transition table
  (domain/lifecycle.py) and returns a DocumentLifecycleChanged event
- InvalidLifecycleTransitionError on disallowed transitions

Persistence
- ALTER TABLE documents to add the two columns (default 'Uploaded')
- New index idx_documents_lifecycle_state for filter perf
- _COLUMN_MIGRATIONS refactored to support multiple tables
- _POST_MIGRATION_DDL list for indexes on freshly-added columns
- SqliteDocumentRepository.update_lifecycle()

Services
- AnalysisService drives transitions on parse / chunk / re-chunk / fail
  via _transition_document(); idempotent and resilient (logs WARN and
  continues if a stale state is somehow encountered)

API
- DocumentResponse exposes lifecycleState + lifecycleStateAt
  (additive — existing 'status' field kept for backwards compat)

Frontend
- Document type extended with lifecycleState and lifecycleStateAt
- DocumentLifecycleState union literal mirroring the backend enum

Tests
- 24 new tests in test_lifecycle.py covering transitions, idempotency,
  invariant preservation, and event emission
- test_repos.py: round-trip + every-enum-value check + update_lifecycle
- test_chunking.py: rechunk path now mocks document_repo correctly

Refs #202
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
805d881d65 docs(design): E1 design docs for 0.6.0 doc-centric ingest
Adds technical design docs for the foundation of the doc-centric ingest pivot:

- 202 — Document lifecycle state machine
- 203 — Per (document, store) ingestion state
- 204 — Auto-detect Stale state via chunk content hash
- 205 — Audit trail for chunk edits (chunks → first-class entity)
- 206 — Migration of existing documents to the new model

Status: Accepted on all five. Each doc spells out the domain entities,
persistence schema, services orchestration, API contract, alternatives
considered, risks, and testing strategy. ADR placeholders called out
where load-bearing decisions warrant a follow-up document.

Refs #202 #203 #204 #205 #206
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
5444838d6c chore: gitignore local-only artifacts on release/0.6.0
Bootstrap commit for the doc-centric ingest release:
- .github/ISSUE_TEMPLATE/issue.md (local copy)
- docs/claude-code-commands.md (personal notes)
- docs/design/TEMPLATE.md (local design template)
- document-parser/tests/test_local_converter.py (local test scratch)
2026-05-05 09:38:39 +02:00
Pier-Jean Malandrino
c3d4b11f48 fix(ci): install pytestarch in docling-compat workflow
Some checks failed
CI / Backend tests (push) Has been cancelled
CI / Frontend tests & build (push) Has been cancelled
CI / E2E API tests (Karate) (push) Has been cancelled
CI / E2E UI tests (Karate UI) (push) Has been cancelled
The daily docling-compat job manually installed only pytest, pytest-asyncio
and httpx, which made test_architecture.py fail with ModuleNotFoundError:
no module named 'pytestarch' — wrongly opening 'Docling compatibility break'
issues. Align with ci.yml by installing requirements-test.txt instead.
2026-05-05 09:38:36 +02:00
Pier-Jean Malandrino
2807cc3aea fix(nginx): move template outside sites-enabled to avoid nginx loading it raw
Some checks failed
Release Docker Images / Build & push — local (push) Has been cancelled
Release Docker Images / Build & push — remote (push) Has been cancelled
2026-04-30 12:15:46 +02:00
Pier-Jean Malandrino
9f2c61839e chore(release): 0.5.1 2026-04-30 11:38:14 +02:00
Pier-Jean Malandrino
2e2c2eaa98 docs(readme): document NGINX_MAX_BODY_SIZE env var and nginx upload layer 2026-04-30 11:38:14 +02:00
Pier-Jean Malandrino
789b07c7d1 fix(nginx): make upload body size configurable via NGINX_MAX_BODY_SIZE env var
nginx.conf was hard-capped at 5M, causing 413 on uploads larger than 5MB
despite the backend accepting up to MAX_FILE_SIZE_MB (default 50).

Both nginx configs become envsubst templates. NGINX_MAX_BODY_SIZE defaults
to 200M so the backend application limit is always the effective arbiter.
gettext-base added to the single-image apt deps to provide envsubst.
2026-04-30 11:38:14 +02:00
Pier-Jean Malandrino
2f811d41c8 fix(docs): also tolerate links.not_found for cross-tree audit links
The previous tweak only set unrecognized_links: info but the actual
warnings are 'target X is not found among documentation files' which
is the not_found validator. Setting both to info so strict-mode build
passes on the audit reports' source-file references.
2026-04-29 14:12:01 +02:00
Pier-Jean Malandrino
8825292146 fix(docs): tolerate cross-tree links from audit reports in strict mode
The audit reports under docs/audit/reports/release-*/ reference repo
source files (e.g. `[file.py:line](file.py:line)`) on purpose — they
are read with the repo open, not as standalone published pages. MkDocs
in strict mode treats those as unrecognized links and aborts the build.

Set `validation.links.unrecognized_links: info` so the warning still
prints but doesn't fail the build. Real broken doc links (anchors,
absolute paths) keep their existing levels.

Refs the doc deploy run that broke just after the v0.5.0 tag was pushed.
2026-04-29 14:04:42 +02:00
Pier-Jean Malandrino
7577a98415 fix(ci): pin Trivy to latest in release-gate (v0.69.3 was yanked)
Some checks failed
Release Docker Images / Build & push — local (push) Has been cancelled
Release Docker Images / Build & push — remote (push) Has been cancelled
aquasecurity/trivy-action@v0.35.0 defaults to Trivy CLI v0.69.3, but
that tag was removed from GitHub releases mid-run on 2026-04-29 — the
HIGH step on Security scan — local started failing at setup time:

  aquasecurity/trivy info checking GitHub for tag 'v0.69.3'
  aquasecurity/trivy crit unable to find 'v0.69.3'
  ##[error]Process completed with exit code 1.

The CRITICAL step still passed (binary was in cache from a prior run),
so this only surfaced as a HIGH-step failure — but the job exit code
still propagates and breaks the gate.

Following `latest` rather than chasing a specific tag that upstream
can yank without notice.

Refs #189
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
628027d861 fix(ci): drop paths constraint from CVE-2026-40393 trivy ignore
Trivy reports OS-package CVEs against the package name (libgbm1,
libgl1-mesa-dri, libglx-mesa0, mesa-libgallium) — not against
installed file paths. The previous `paths:` filter silently failed
to match, so the ignore was a no-op and the gate kept failing on a
CVE we explicitly chose to defer.

Trace from the failing run (#25097385670):

  Using YAML ignorefile '.trivyignore.yaml':
    - id: CVE-2026-40393
  ...
  libgbm1  CVE-2026-40393  CRITICAL  affected  25.0.7-2
  ...
  ##[error]Process completed with exit code 1.

Removing `paths:` lets the ID-only match apply across all 4 affected
Mesa packages until 2026-06-30.

Refs #189
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
a7fbdf7b1b fix(ci): wire .trivyignore.yaml into release-gate Trivy scans
The CRITICAL + HIGH Trivy steps in release-gate.yml were invoking
aquasecurity/trivy-action without the `trivyignores` input, so the
.trivyignore.yaml at the repo root (committed in #190 to mitigate
CVE-2026-40393 / Mesa) was silently ignored — the gate kept failing
on a CVE we explicitly chose to defer.

Pass `trivyignores: .trivyignore.yaml` to both Trivy steps so the file
takes effect. The HIGH step also gets it for consistency (it doesn't
fail the gate, but reporting a CVE we ignore as HIGH would be noise).

Refs #189
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
efc27932dd refactor(audit): remediate 0.5.0 audit findings — clean architecture, security, DRY, SOLID, perf
Closes the 12 MAJ raised by the release/0.5.0 audit pipeline (cf.
docs/audit/reports/release-0.5.0/summary.md → summary-reaudit.md).

Volet 1 — Reasoning architecture (audits 01/02/06/07 strengthening)
  * Domain ports: LLMProvider, ReasoningRunner, ReasoningParseError
  * Domain DTOs: LLMProviderType, ReasoningResult, ReasoningIteration
  * infra/llm/ollama_provider.py — OllamaProvider with health_check
  * infra/docling_agent_reasoning.py — runner adapter, encapsulates the
    private _rag_loop call (tracked at docling-project/docling-agent#26),
    commits OLLAMA_HOST once at boot (eliminates the per-request env race),
    translates upstream IndexError into ReasoningParseError
  * api/reasoning.py — zero coupling to docling-agent / mellea / docling-core,
    consumes app.state.reasoning_runner via the port
  * main.py — DI wires OllamaProvider + DoclingAgentReasoningRunner at boot
    when REASONING_ENABLED=true and deps are importable
  * Rename RAG_* env vars → REASONING_*, endpoint /rag → /reasoning,
    type RAGResult → ReasoningResult, frontend feature flag wiring,
    i18n strings, tests, docs (BREAKING — pre-1.0 surface, no external
    consumers in production)
  * 17 new tests: adapter unit tests with sys.modules stubs, OllamaProvider
    httpx tests, R3 concurrent-host isolation, R6 multi-iteration trace
    serialization, R13 Protocol conformance via isinstance
  * E2E Karate scenario: nav-reasoning hidden when REASONING_ENABLED=false
  * README — Live Reasoning section (env vars, archi, link to issue #26)

Bloc B — Security (audit 08, dev-only context)
  * docker-compose.yml — DEV DEFAULTS header, OpenSearch DISABLE_SECURITY_PLUGIN
    flagged as dev-only with link to OpenSearch security docs
  * main.py — boot warning if NEO4J_URI is set with the default 'changeme'
    password, so prod operators can't silently inherit it

Bloc C — DRY frontend (audit 05)
  * shared/storage/keys.ts — STORAGE_KEYS centralised (theme, locale)
  * features/settings/store.ts — dead apiUrl ref + orphan i18n keys removed
  * api/schemas.py — DOCUMENT_STATUS_UPLOADED constant

Bloc D — Quality (audits 02/06/07/09/10/12)
  * domain/ports.py — DocumentConverter.supports_page_batching property
    (LSP fix, replaces isinstance(ServeConverter) check)
  * domain/ports.py — VectorStore.ping() (encapsulation, replaces
    _vector_store._client.info() reach-around)
  * api/analyses.py + api/ingestion.py — path params {job_id} → {analysis_id}
    aligned with the user-facing terminology (URLs unchanged)
  * api/documents.py — Path.read_bytes() + generate_preview() wrapped in
    asyncio.to_thread, unblocks the FastAPI event loop on /preview
  * infra/docling_tree.py — PEP 604 union for isinstance (Ruff UP038)
  * src/__tests__/integration/ — cross-feature integration test relocated
    out of features/history/ so feature folders stay self-contained
  * Tightened terminal `assert X is not None` checks (isinstance(.., datetime),
    exact value comparisons)

Validation
  * 446 backend pytest, 202 frontend vitest — all green
  * ruff + ruff format + ESLint + Prettier + vue-tsc clean
  * Re-audit verdict: 0 CRIT / 0 MAJ, score ~94/100, GO

Closes #200
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
e203fe8b71 chore(release): cut 0.5.0 — bump frontend version + complete CHANGELOG
Lifts the audit 11 CRIT blocker (missing CHANGELOG [0.5.0] section) and
the matching MAJ on the frontend version pin.

Refs #200
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
157a779e20 fix(graph): collapse Docling InlineGroup and Picture children (#197)
Two patterns in Docling's serialization were mirrored 1:1 by the graph
projection and produced node explosions on real documents:

- An InlineGroup (paragraph of mixed style runs) emits one `groups[]`
  entry plus N `texts[]` runs. Naive iteration created one Paragraph
  node per run.
- A Picture's `children` carry internal text labels extracted by the
  layout model (flowchart boxes, chart axis labels, diagram callouts).
  Each child became its own Paragraph node, drowning the figure.

`build_collapse_index` (in the shared `infra.docling_tree` helper) now
returns the `skip_refs` set + `inline_meta` overrides for both cases.
The Neo4j `tree_writer` and the in-memory `docling_graph` consume the
same index, so both projections stay in sync.

InlineGroups are projected as a single :Paragraph carrying the
concatenated text and the union of children's provs (re-indexed).
Pictures keep their :Figure node and prov; their descendants are
dropped. Captions live in the picture's separate `captions` field, not
in `children`, so they are unaffected.
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
fe83dcdf79 feat(settings): paste-image size/type limits for #195 (#196)
* docs: rename Clean Architecture → Hexagonal Architecture (ports & adapters)

Le backend suit le pattern ports & adapters (ports dans domain/ports.py,
adaptateurs dans infra/), pas Clean Architecture au sens Uncle Bob.
Aligne la terminologie dans README, docs/architecture.md, ADR guide,
audit master, fiche audit 01, et la nav mkdocs.

Les noms de fichiers et la commande /audit:clean-architecture restent
stables pour preserver les liens croises et les skills existants.

* feat(settings): add paste-image size/type limits surfaced via /api/health

Introduces MAX_PASTE_IMAGE_SIZE_MB (default 10) and
PASTE_ALLOWED_IMAGE_TYPES (default image/png,image/jpeg,image/webp)
env vars so the upcoming Verify-mode clipboard-paste handler can
validate client-side against the same limits the backend enforces.

Follows the existing MAX_FILE_SIZE_MB pattern. Ships the accepted
design doc at docs/design/195-copy-paste-image-verify-mode.md.

Refs #195
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
cc535a982d fix(ci): ignore CVE-2026-40393 (Mesa) with expiry — Debian has no backport (#190)
Refs #189
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
84a10c472b fix(reasoning): re-scroll PDF when re-clicking the active iteration
The watch-based plumbing from iteration click to PDF scroll relied on a
"flip via null" pattern (assign null then the value) to coerce Vue into
re-firing the watcher. Vue 3 collapses synchronous mutations of the same
ref and only delivers the final value, so the trick was a no-op: a second
click on the same iteration left the document view stuck on the previous
page. The bug only showed when the trace had a single iteration — with
several, the user naturally clicks different ones and the value really
changes.

Replace the watch chain with imperative dispatch. ReasoningPanel now just
emits iterationFocus; ReasoningWorkspace handles it by calling the graph
focus and the new StructureViewer.scrollToFocused method directly. Both
side effects fire on every click regardless of state.
2026-04-29 14:00:00 +02:00
Pier-Jean Malandrino
8694353d8b feat(reasoning): bidirectional PDF ↔ graph focus + DocumentView mode
Propagate Docling `self_ref` through PageElement so bboxes and graph nodes
share a stable identity. Add a Document/Graph mode switch to the reasoning
workspace; selecting a node highlights its bbox (numbered badge, focus ring,
optional dim of non-visited) and clicking a bbox re-centers the graph.
2026-04-29 14:00:00 +02:00