pediatric-ai-scribe-v3/TODO.md

# TODO

Live state as of 2026-09-11. Everything not listed under **Open** is deployed
and green (674 tests, three consecutive clean runs).

## Open

### Needs your decision
- [ ] **Replace Cloudflare Turnstile.** Used on registration and password reset
      only (`src/routes/auth.js`); login is not gated, it relies on a
      10-per-15-min limit and a constant-time credential check. Recommended
      replacement: **ALTCHA** — open source, self-hosted, proof-of-work, no
      third-party calls and no tracking, which also lets three CSP entries and
      `frameSrc` go away. Alternatives: **mCaptcha** (open source, self-hosted,
      heavier to run) and **Cap** (newer, smaller). hCaptcha is neither Google
      nor open source, so it trades one third party for another.
- [ ] **Kubernetes / CI-CD hardening.** Details under *Deployment readiness*.
- [x] **Audio backups are in MinIO.** Same server, its own `audio-backups`
      bucket. The app key carries a second policy covering only that bucket, so
      the mounted credentials serve both. Verified: storage=object, round trip
      byte-identical, objects removed with their rows.
- [ ] **Basic index has no reader.** `MilvusVectorStore.search()` exists, but no
      tool calls it. Decide where the query path lives: pymilvus inside the
      deliberately-lean `nextcloud-basic-mcp` image, or a query API from the
      indexer container. Nothing can read that index until this is settled.
      Source: `/home/danvics/docker/nextcloud-basic-mcp`
- [ ] **Apply the restored clinical vector-store compose.** Written, committed
      and validated at `/home/danvics/docker/clinical-storage-milvus`, deliberately
      NOT applied — `up -d` recreates the live clinical Milvus.

### Known gaps
- [ ] **Multi-collection, ped-ai half.** The MCP side is deployed
      (`clinical_semantic_search(collection=…)` + `clinical_list_collections`,
      allowlisted by `MILVUS_COLLECTIONS`). ped-ai still searches one collection
      per request. Needs an admin setting for which collections to search, then
      fan-out and merge — `dedupeSources` in `src/utils/clinicalRetrieval.js`
      already merges and renumbers. See `clinical-assist/COLLECTIONS.md`.
- [x] **Mail indexing works.** It was never reached: mail ran last, after nine
      other sources, and Tables alone is thousands of rows at about a second
      each. Mail leads now — it is the only bounded source (identities only,
      capped by `BASIC_INDEXING_MAIL_MAX_MESSAGES`, bodies left to the
      processor), so it cannot starve the others the way they starved it.
      Messages are indexing.
- [x] **Mail attachments were never indexed.** An attachment's id is its index
      within its message, so `/api/attachments/{id}` meant nothing and answered
      500 every time. Fixed to `/api/messages/{id}/attachment/{id}`, verified
      live against a real message.
- [x] **The basic collection is renamed** `personal_assistant_bge_m3_1024`
      (was `basic_bge_m3_1024_v2`), matching `mcp_bge_m3_1024` on the clinical
      side. Milvus grants name the collection, so the rename revoked
      basic_reader/basic_writer; `bootstrap_basic.py` restored them, but it
      must be bind-mounted because the operator image ships an older copy.
- [ ] **The indexed folder is `Personal assistant`,** and the setting now takes
      a comma-separated list (`Personal assistant,Clinical Notes`), each walked
      recursively. Note the file reconciliation removed the chunks of the 68
      `Documents` files, since a complete listing is the deletion authority and
      they are no longer under an indexed root. Entities went 27,930 -> ~9,700.
      Re-add those files under an indexed folder if they are still wanted.

## Deployment readiness (CI/CD and Kubernetes)

What already exists: `ci.yml` (tests on PR and main), `security.yml` (weekly
audit), `docker-publish.yml` and Android/versioning workflows, a Dockerfile
`HEALTHCHECK`, and `/api/health`.

Worth doing before Kubernetes, roughly in order:
- [ ] **Fail CI on vulnerabilities.** `security.yml` runs weekly and does not
      gate merges. `npm audit --audit-level=high` in `ci.yml` would have caught
      the nodemailer advisories at the PR.
- [ ] **Build the image on PRs too.** `docker-publish.yml` only runs on tags, so
      a Dockerfile break is found at release time.
- [ ] **Run the e2e suite in CI.** `scripts/e2e.sh` and
      `docker-compose.e2e.yml` exist but nothing calls them.
- [ ] **Separate liveness from readiness.** `/api/health` is one endpoint;
      Kubernetes wants liveness (process up) apart from readiness (database,
      gateway and MinIO reachable), or rollouts take traffic too early.
- [ ] **Graceful shutdown.** No SIGTERM handler, so a rolling update can cut off
      an in-flight transcription or image job.
- [ ] **Externalise state.** Uploads and audio backups assume local paths and a
      single instance; more than one replica needs them all in MinIO/Postgres.
- [ ] **Config as secrets.** Everything is env vars in compose today, which maps
      to ConfigMap/Secret cleanly, but `JWT_SECRET`, gateway keys and database
      credentials should be a Secret from the start.
- [ ] **Pin the base image by digest** and keep the SBOM the build already has.

## Done since this file was written

### 2026-09-11
- **Live transcription**: proved working end to end against the live gateway —
  `local-kokoro-tts` produced 92KB of speech and
  `mistral-voxtral-mini-transcribe` returned the sentence back verbatim. The
  Settings picker offered six hardcoded ids that do not exist on this gateway
  (`local-whisper-large-v3-turbo` → 400 Invalid model name); it now lists the
  nine the gateway advertises, cached, with the admin default marked.
- **Assistant voice mode** is wired end to end: record → browser recognition,
  falling back to server transcription → send → spoken answer. All four helpers
  it needs exist.
- **Recordings can be exported** (server and local copies) and a recorder that
  dies — an error, or the microphone taken by another app, unplugged or
  revoked — now says so instead of appearing to record silence. No wake lock,
  deliberately: stopping on sleep or sign-out is the behaviour you want.
- **nodemailer 9.0.1 → 9.1.1**, clearing four high advisories, two of them
  delivery bugs that can route mail to an attacker-controlled domain.
- **Admin routers state their own authentication.** `adminMilestones` relied on
  `adminConfig` being mounted first on `/api/admin`; it failed closed, but on
  mount order rather than intent.
- **Test suite made deterministic.** A file failed about one run in four with
  "Unable to deserialize cloned data": node:test parses each child's stdout, and
  page/server logging was landing inside those frames. Every test child's stdout
  is now pure TAP.
- **iOS**: text fields are 16px on phones, so Safari no longer zooms the page on
  focus — which was also why fixed chrome (the menu button) scrolled away.
- **Settings claim corrected**: there is no "use my normal physical exam"
  trigger, and the prompt forbids copying template content.

- Storage stack is under version control (`/home/danvics/docker/personal-assistant-storage-milvus`),
  secrets verified excluded, with a README recording the misleading project names
  and the MinIO/separate-etcd requirements.
- Both Milvus stores keep objects in MinIO; verified end to end on the basic side
  (16 objects in the bucket, rows queryable, collection Loaded).
- The operator image is built from source (`Dockerfile.operator`), so the drifted
  `check.py` that broke image generation can no longer be run by accident.

## Worth knowing

- **Four repos were rescued from container images today**: `nextcloud-basic-mcp`,
  `clinical-assist`, the deleted clinical-storage compose file, and the
  operator's `check.py` drift. Prefer building from a repo over a live container.
- **Two Milvus instances, historically misleading names.**
  Clinical index = `nextcloud-mcp-server-milvus-1` in
  `clinical-storage-milvus` (MinIO-backed, collection `mcp_bge_m3_1024`).
  Personal assistant index = `personal-assistant-storage-milvus-basic-milvus-1`
  (db `basic`, collection `personal_assistant_bge_m3_1024`).
- **Embedded etcd is unusable with authorization on.** Every non-root Milvus user
  failed `etcdserver: invalid auth token`. Both stores now run a separate etcd
  container, matching the profile that always worked.
- **Both Milvus stores now keep objects in MinIO**, matching what clinical always
  did. `COMMON_STORAGETYPE=local` wrote segment files relative to the working
  directory, so a recreate destroyed them while etcd kept referencing them and the
  collection hung at Loading forever. S3 semantics also mean either store can be
  pointed at a managed bucket without touching Milvus — which is what makes a
  Terraform-managed deployment straightforward.
- **Milvus object-store credentials live in `milvus-user.yaml`** in the protected
  secrets dir, not in the compose, because Milvus has no file-based option for
  them and the compose should stay reviewable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

## Sign-in by one-time code, alongside passwords

An option, not a replacement. An account keeps its password; the sign-in screen
offers "email me a code" as a second way in, and an account can use either.

- Reuse the invite machinery rather than inventing a second one: codes are
  hashed at rest, single-use, short-lived, and claimed by one atomic UPDATE
  carrying every condition, so two attempts at the same code cannot both win.
- Much shorter TTL than an invite — minutes, not days — and rate limited per
  account and per IP, because a login code is a credential and an attacker can
  ask for one without knowing the password.
- A code must never reveal whether the address has an account. The response is
  the same either way.
- 2FA still applies on top where enabled: a code replaces the password step, not
  the second factor.

## The signed-out preview, as it should behave

The preview works now, but it is currently all-or-nothing. Intended shape:

- Signed out with preview on, the assistant chat is visible and usable.
- Anything else — saved chats, images, settings — prompts sign-in rather than
  failing. Today those simply 401.
- A cap of three questions per visitor, then an invitation to sign in. The cap
  has to be enforced on the server: a preview visitor has no identity, so it
  cannot be per-account, and a purely client-side count is worth nothing.
- Nothing is persisted for a preview visitor. That is already true and must
  stay true — PREVIEW_USER has id: null precisely so nothing can be owned.
- If the visitor then signs in **in the same browser**, the preview
  conversation is adopted into the new account and saved. This is the
  interesting part: because preview deliberately has no identity, the transcript
  has to be held client-side and replayed on the first authenticated request,
  not looked up server-side. Size-cap it and drop it on sign-out.