36cb742ce7
Three things, one subject: making CI say the truth about this repo.
## The red on every run was ours, not the runners'
Every docker-build run came back success, success, failure — the same
shape for weeks. The failing job was `deploy`, and it was failing to
*not run*:
if: ${{ github.event.inputs.deploy == 'true' }}
On a push there is no github.event.inputs at all. This Forgejo does not
treat that as false and skip; it dispatches the job, the runner cannot
resolve it, and the task ends in "Early termination". The runners were
never at fault, and nothing about them needed changing.
The `'runs-on' key not defined` line is a red herring: the `build` job
prints it too and succeeds. It names the job's *needs* target, not the
job, and the old android-apk workflow used `needs:` happily for months.
Deploy is now its own workflow with only workflow_dispatch — no
condition to evaluate, so nothing can be dispatched by mistake. No job
in either file now carries a job-level `if`. The one conditional left is
a *step* (push to registry), and step conditions are evaluated by the
runner once the job is already running, which is why that one has always
worked.
## dev and main
docker-build now runs on `dev` as well. Both branches prove the same two
things — tests pass, image builds — and only `main` publishes the image,
so nothing on `dev` can be mistaken for something deployable. Deploying
stays a person pressing a button after looking at the change.
CONTRIBUTING.md documents the flow.
## Android
Removed: the mobile/ Capacitor project, docs/mobile-build.md, and the
Android bits of scripts/release.sh. All of it is in git history — 4613a278
is the last commit that had it — for when it is rebuilt.
src/utils/platform.js stays. isMobileClient only decides token lifetime,
it is twelve lines, and it is the contract a future app would come back
to; deleting it would be a change to auth for no gain.
.github/workflows/ went too — all five. There is no GitHub remote on
this repository, so none of them has ever run, and two of them wrote
into mobile/ paths that no longer exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv6sqaY6Vq3ChZHMem3cnU
198 lines
12 KiB
Markdown
198 lines
12 KiB
Markdown
# TODO
|
|
|
|
Live state as of 2026-09-11. Everything not listed under **Open** is deployed
|
|
and green (674 tests, three consecutive clean runs).
|
|
|
|
## Open
|
|
|
|
### Needs your decision
|
|
- [ ] **Replace Cloudflare Turnstile.** Used on registration and password reset
|
|
only (`src/routes/auth.js`); login is not gated, it relies on a
|
|
10-per-15-min limit and a constant-time credential check. Recommended
|
|
replacement: **ALTCHA** — open source, self-hosted, proof-of-work, no
|
|
third-party calls and no tracking, which also lets three CSP entries and
|
|
`frameSrc` go away. Alternatives: **mCaptcha** (open source, self-hosted,
|
|
heavier to run) and **Cap** (newer, smaller). hCaptcha is neither Google
|
|
nor open source, so it trades one third party for another.
|
|
- [ ] **Kubernetes / CI-CD hardening.** Details under *Deployment readiness*.
|
|
- [x] **Audio backups are in MinIO.** Same server, its own `audio-backups`
|
|
bucket. The app key carries a second policy covering only that bucket, so
|
|
the mounted credentials serve both. Verified: storage=object, round trip
|
|
byte-identical, objects removed with their rows.
|
|
- [ ] **Basic index has no reader.** `MilvusVectorStore.search()` exists, but no
|
|
tool calls it. Decide where the query path lives: pymilvus inside the
|
|
deliberately-lean `nextcloud-basic-mcp` image, or a query API from the
|
|
indexer container. Nothing can read that index until this is settled.
|
|
Source: `/home/danvics/docker/nextcloud-basic-mcp`
|
|
- [ ] **Apply the restored clinical vector-store compose.** Written, committed
|
|
and validated at `/home/danvics/docker/clinical-storage-milvus`, deliberately
|
|
NOT applied — `up -d` recreates the live clinical Milvus.
|
|
|
|
### Known gaps
|
|
- [ ] **Multi-collection, ped-ai half.** The MCP side is deployed
|
|
(`clinical_semantic_search(collection=…)` + `clinical_list_collections`,
|
|
allowlisted by `MILVUS_COLLECTIONS`). ped-ai still searches one collection
|
|
per request. Needs an admin setting for which collections to search, then
|
|
fan-out and merge — `dedupeSources` in `src/utils/clinicalRetrieval.js`
|
|
already merges and renumbers. See `clinical-assist/COLLECTIONS.md`.
|
|
- [x] **Mail indexing works.** It was never reached: mail ran last, after nine
|
|
other sources, and Tables alone is thousands of rows at about a second
|
|
each. Mail leads now — it is the only bounded source (identities only,
|
|
capped by `BASIC_INDEXING_MAIL_MAX_MESSAGES`, bodies left to the
|
|
processor), so it cannot starve the others the way they starved it.
|
|
Messages are indexing.
|
|
- [x] **Mail attachments were never indexed.** An attachment's id is its index
|
|
within its message, so `/api/attachments/{id}` meant nothing and answered
|
|
500 every time. Fixed to `/api/messages/{id}/attachment/{id}`, verified
|
|
live against a real message.
|
|
- [x] **The basic collection is renamed** `personal_assistant_bge_m3_1024`
|
|
(was `basic_bge_m3_1024_v2`), matching `mcp_bge_m3_1024` on the clinical
|
|
side. Milvus grants name the collection, so the rename revoked
|
|
basic_reader/basic_writer; `bootstrap_basic.py` restored them, but it
|
|
must be bind-mounted because the operator image ships an older copy.
|
|
- [ ] **The indexed folder is `Personal assistant`,** and the setting now takes
|
|
a comma-separated list (`Personal assistant,Clinical Notes`), each walked
|
|
recursively. Note the file reconciliation removed the chunks of the 68
|
|
`Documents` files, since a complete listing is the deletion authority and
|
|
they are no longer under an indexed root. Entities went 27,930 -> ~9,700.
|
|
Re-add those files under an indexed folder if they are still wanted.
|
|
|
|
## Deployment readiness (CI/CD and Kubernetes)
|
|
|
|
What already exists: `.forgejo/workflows/docker-build.yml` (tests then image,
|
|
on `dev` and `main`; publishes from `main` only), `.forgejo/workflows/deploy.yml`
|
|
(manual dispatch), a Dockerfile `HEALTHCHECK`, and `/api/health`. The GitHub
|
|
workflows were removed — this repository has no GitHub remote, so none of them
|
|
ever ran.
|
|
|
|
Worth doing before Kubernetes, roughly in order:
|
|
- [ ] **Fail CI on vulnerabilities.** Nothing runs `npm audit` any more; the
|
|
weekly GitHub job went with the rest. `npm audit --audit-level=high` as a
|
|
step in `docker-build.yml` would have caught the nodemailer advisories.
|
|
- [ ] **Run the e2e suite in CI.** `scripts/e2e.sh` and
|
|
`docker-compose.e2e.yml` exist but nothing calls them.
|
|
- [ ] **Separate liveness from readiness.** `/api/health` is one endpoint;
|
|
Kubernetes wants liveness (process up) apart from readiness (database,
|
|
gateway and MinIO reachable), or rollouts take traffic too early.
|
|
- [ ] **Graceful shutdown.** No SIGTERM handler, so a rolling update can cut off
|
|
an in-flight transcription or image job.
|
|
- [ ] **Externalise state.** Uploads and audio backups assume local paths and a
|
|
single instance; more than one replica needs them all in MinIO/Postgres.
|
|
- [ ] **Config as secrets.** Everything is env vars in compose today, which maps
|
|
to ConfigMap/Secret cleanly, but `JWT_SECRET`, gateway keys and database
|
|
credentials should be a Secret from the start.
|
|
- [ ] **Pin the base image by digest** and keep the SBOM the build already has.
|
|
|
|
## Done since this file was written
|
|
|
|
### 2026-09-11
|
|
- **Live transcription**: proved working end to end against the live gateway —
|
|
`local-kokoro-tts` produced 92KB of speech and
|
|
`mistral-voxtral-mini-transcribe` returned the sentence back verbatim. The
|
|
Settings picker offered six hardcoded ids that do not exist on this gateway
|
|
(`local-whisper-large-v3-turbo` → 400 Invalid model name); it now lists the
|
|
nine the gateway advertises, cached, with the admin default marked.
|
|
- **Assistant voice mode** is wired end to end: record → browser recognition,
|
|
falling back to server transcription → send → spoken answer. All four helpers
|
|
it needs exist.
|
|
- **Recordings can be exported** (server and local copies) and a recorder that
|
|
dies — an error, or the microphone taken by another app, unplugged or
|
|
revoked — now says so instead of appearing to record silence. No wake lock,
|
|
deliberately: stopping on sleep or sign-out is the behaviour you want.
|
|
- **nodemailer 9.0.1 → 9.1.1**, clearing four high advisories, two of them
|
|
delivery bugs that can route mail to an attacker-controlled domain.
|
|
- **Admin routers state their own authentication.** `adminMilestones` relied on
|
|
`adminConfig` being mounted first on `/api/admin`; it failed closed, but on
|
|
mount order rather than intent.
|
|
- **Test suite made deterministic.** A file failed about one run in four with
|
|
"Unable to deserialize cloned data": node:test parses each child's stdout, and
|
|
page/server logging was landing inside those frames. Every test child's stdout
|
|
is now pure TAP.
|
|
- **iOS**: text fields are 16px on phones, so Safari no longer zooms the page on
|
|
focus — which was also why fixed chrome (the menu button) scrolled away.
|
|
- **Settings claim corrected**: there is no "use my normal physical exam"
|
|
trigger, and the prompt forbids copying template content.
|
|
|
|
- Storage stack is under version control (`/home/danvics/docker/personal-assistant-storage-milvus`),
|
|
secrets verified excluded, with a README recording the misleading project names
|
|
and the MinIO/separate-etcd requirements.
|
|
- Both Milvus stores keep objects in MinIO; verified end to end on the basic side
|
|
(16 objects in the bucket, rows queryable, collection Loaded).
|
|
- The operator image is built from source (`Dockerfile.operator`), so the drifted
|
|
`check.py` that broke image generation can no longer be run by accident.
|
|
|
|
## Worth knowing
|
|
|
|
- **Four repos were rescued from container images today**: `nextcloud-basic-mcp`,
|
|
`clinical-assist`, the deleted clinical-storage compose file, and the
|
|
operator's `check.py` drift. Prefer building from a repo over a live container.
|
|
- **Two Milvus instances, historically misleading names.**
|
|
Clinical index = `nextcloud-mcp-server-milvus-1` in
|
|
`clinical-storage-milvus` (MinIO-backed, collection `mcp_bge_m3_1024`).
|
|
Personal assistant index = `personal-assistant-storage-milvus-basic-milvus-1`
|
|
(db `basic`, collection `personal_assistant_bge_m3_1024`).
|
|
- **Embedded etcd is unusable with authorization on.** Every non-root Milvus user
|
|
failed `etcdserver: invalid auth token`. Both stores now run a separate etcd
|
|
container, matching the profile that always worked.
|
|
- **Both Milvus stores now keep objects in MinIO**, matching what clinical always
|
|
did. `COMMON_STORAGETYPE=local` wrote segment files relative to the working
|
|
directory, so a recreate destroyed them while etcd kept referencing them and the
|
|
collection hung at Loading forever. S3 semantics also mean either store can be
|
|
pointed at a managed bucket without touching Milvus — which is what makes a
|
|
Terraform-managed deployment straightforward.
|
|
- **Milvus object-store credentials live in `milvus-user.yaml`** in the protected
|
|
secrets dir, not in the compose, because Milvus has no file-based option for
|
|
them and the compose should stay reviewable.
|
|
|
|
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
|
|
## Sign-in by one-time code, alongside passwords
|
|
|
|
An option, not a replacement. An account keeps its password; the sign-in screen
|
|
offers "email me a code" as a second way in, and an account can use either.
|
|
|
|
- Reuse the invite machinery rather than inventing a second one: codes are
|
|
hashed at rest, single-use, short-lived, and claimed by one atomic UPDATE
|
|
carrying every condition, so two attempts at the same code cannot both win.
|
|
- Much shorter TTL than an invite — minutes, not days — and rate limited per
|
|
account and per IP, because a login code is a credential and an attacker can
|
|
ask for one without knowing the password.
|
|
- A code must never reveal whether the address has an account. The response is
|
|
the same either way.
|
|
- 2FA still applies on top where enabled: a code replaces the password step, not
|
|
the second factor.
|
|
|
|
## The signed-out preview, as it should behave
|
|
|
|
The preview works now, but it is currently all-or-nothing. Intended shape:
|
|
|
|
- Signed out with preview on, the assistant chat is visible and usable.
|
|
- Anything else — saved chats, images, settings — prompts sign-in rather than
|
|
failing. Today those simply 401.
|
|
- A cap of three questions per visitor, then an invitation to sign in. The cap
|
|
has to be enforced on the server: a preview visitor has no identity, so it
|
|
cannot be per-account, and a purely client-side count is worth nothing.
|
|
- Nothing is persisted for a preview visitor. That is already true and must
|
|
stay true — PREVIEW_USER has id: null precisely so nothing can be owned.
|
|
- If the visitor then signs in **in the same browser**, the preview
|
|
conversation is adopted into the new account and saved. This is the
|
|
interesting part: because preview deliberately has no identity, the transcript
|
|
has to be held client-side and replayed on the first authenticated request,
|
|
not looked up server-side. Size-cap it and drop it on sign-out.
|
|
|
|
## PubMed as a search source of its own
|
|
|
|
Web search reaches the open web through Tavily, Serper, Brave or a self-hosted
|
|
SearXNG. PubMed is a different kind of source and deserves its own tool rather
|
|
than being a site: filter on a search engine.
|
|
|
|
- E-utilities (esearch then efetch) needs no key for modest use, and an
|
|
NCBI_API_KEY raises the rate limit. No commercial third party either way.
|
|
- It returns structured records — title, journal, year, PMID, abstract — so a
|
|
citation can be exact rather than reconstructed from a page title. That is the
|
|
real argument for a separate tool: the References section could carry a PMID.
|
|
- A separate tool, not a provider option, because the model should be able to
|
|
reach for "the literature" distinctly from "the web": one search per resource
|
|
each, and a resource may legitimately want both.
|
|
- Same guardrails as web search: admin-enabled, opt-in per generation, only the
|
|
query leaves, and hidden entirely when unavailable.
|