pediatric-ai-scribe-v3/docs/developer-guide.md
Daniel 67e416c6d9
Some checks failed
Forgejo Android APK / Root app tests (push) Successful in 47s
Forgejo Docker Build / Root app tests (push) Successful in 48s
Forgejo Android APK / Build signed APK (push) Successful in 2m38s
Forgejo Docker Build / Build Docker image (push) Successful in 12s
Forgejo Docker Build / Deploy to the host (push) Failing after 2s
docs: merge the duplicate pairs and correct them against the running app
Three pairs of docs described the same thing twice, and the copies had drifted
apart. Merged each into one file, keeping the unique content from both:

- ARCHITECTURE.md -> architecture.md (its operational map: ownership, request
  flow, runtime boundaries, source of truth, deployment shape)
- DEVELOPMENT.md -> developer-guide.md (change workflow, Clinical Assistant
  high-risk areas, frontend rendering rules, deployment checks)
- transcription-options.md -> speech.md (the clinic setup table, and the list
  of browser-Whisper paths that must stay removed)

Then audited what remained against the code and the live database rather than
against the previous docs. Corrected:

- Google Vertex was still documented as a provider across nine files. The SDK
  is gone; AI_PROVIDER=vertex now logs an advisory and falls back to
  OpenRouter, and Gemini is reached through LiteLLM. Fixed the provider
  selection order to match src/utils/ai.js, which starts from LITELLM_API_BASE.
- promptSafe was documented on 8 routes; it is on 13.
- Node 20 -> 24, "24 vanilla JS modules" -> no fixed count, and
  transcribe.js/tts.js -> sttProvider.js/ttsProvider.js, which is what exists.
- STT/TTS are LiteLLM-only; README listed direct Google, AWS Transcribe and
  ElevenLabs paths that are not in the runtime.
- Learning Hub PPTX export was documented as pptxgenjs, which is not a
  dependency. It is pandoc against a reference deck.
- POST /api/admin/milestones/seed does not exist; it is /bulk-import.
- NEXTCLOUD_URL and NTFY_TOPIC are not read anywhere. Nextcloud is per-user in
  the users table, and the ntfy topic is derived as pedscribe-{userId}.
- A prose paragraph sat inside the Clinical Assistant settings table, so half
  the rows rendered as text.

Filled the gaps the audit exposed:

- database.md was missing 12 of 29 tables, including user_resources,
  personal_notes, login_codes, registration_invites and generated_image_jobs.
- developer-guide.md was missing 11 routers and 10 frontend modules.
- api-reference.md detailed 121 of 244 endpoints and said so, but whole
  features were absent. Added an endpoint index covering Clinical Assistant,
  My Resources, Notes, Diagrams, ED Encounters, invites and sign-in codes.
- configuration.md was missing METRICS_TOKEN, REDIS_URL, API_RATE_LIMIT_MAX,
  the LITELLM_* model variables, the DB_* ones maintenance.js reads, and the
  per-purpose S3 resolution scheme.
- clinical-assistant.md documented 2 of its 17 environment variables.
- features-explained.md had no entry for My Resources or Clinical Assistant.

Renamed the three remaining SHOUTING filenames to kebab-case, which is what the
docs viewer's prettyName() was working around, and rewrote README's index,
which listed architecture.md twice and omitted nine files.

Noted but not changed: the Turnstile site key is hardcoded in index.html rather
than read from TURNSTILE_SITE_KEY, and /api/health/detailed can report
tts: 'elevenlabs' though no ElevenLabs path exists.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv6sqaY6Vq3ChZHMem3cnU
2026-09-12 04:57:35 +02:00

465 lines
18 KiB
Markdown

# Developer guide
How the codebase is organized, how the main subsystems work, and where to
extend them.
## Layout
```
server.js Express entry, middleware stack, route mount
Dockerfile node:24-alpine, argon2 native compile deps
docker-compose.yml app + postgres services
migrations/ node-pg-migrate versioned schema changes
scripts/
maintenance.js REINDEX / drift CLI
release.sh semver bump + tag + push
import-milestones.js one-off seed import
src/
db/
database.js pool, baseline init, query helpers
migrate.js programmatic node-pg-migrate runner
middleware/
auth.js JWT + session-table validation + sliding idle
logging.js request log
utils/
ai.js callAI() multi-provider router + model whitelist
models.js built-in model registry
prompts.js templates (DB-overridable)
promptSafe.js <UNTRUSTED_*> wrapping + INJECTION_GUARD
crypto.js AES-256-GCM (PHI at rest)
passwords.js argon2id + bcrypt fallback + rehash
sessions.js token hash, UA parse, session id
platform.js isMobileClient()
redact.js PHI redactor for audit details
auditQueue.js batched audit/api/access writer
fileType.js magic-byte upload verifier
errors.js generic 500 responder
logger.js audit + api + access + Loki shipper
embeddings.js LiteLLM embeddings
notify.js ntfy push
transcribe.js, tts.js LiteLLM STT / TTS routes
routes/ Express routers for auth, AI workflows, education, logs, and user data
public/
index.html SPA shell, version-stamped asset refs
sw.js cache shell, network-first API
manifest.json PWA
js/ vanilla JS modules (no bundler)
components/ per-tab HTML fragments loaded on demand
css/styles.css
template-guide.md downloadable user template guide
mobile/ Capacitor 6 wrapper (Android + iOS)
.github/workflows/ CI (auto-version, APK, docker)
```
## Backend
### Middleware stack (`server.js`)
```
request
→ helmet CSP, HSTS, X-Content-Type-Options
→ CORS fail-closed in prod if APP_URL/CORS_ORIGINS missing
→ cookieParser
→ express.json 10 MB cap
→ rate limiters 200/min general; 10/15min login; 20/15min sensitive
→ static public/ with per-filetype Cache-Control; ?v=BUILD_ID cache-busts HTML on deploy
→ route handlers
→ 404 fallback serves index.html for SPA routes
```
On boot:
- `APP_VERSION` read from `package.json`, printed + returned by `/api/health/detailed`.
- `BUILD_ID` = full Git HEAD SHA (worktrees and packed refs supported), or the validated image-baked revision. Unversioned development builds report `unknown`; no random SHA is invented.
- `JWT_SECRET` / `DATA_ENCRYPTION_KEY` fail-fast if missing in production.
- `initDatabase()``runMigrations()` → collation drift check.
- SIGTERM / SIGINT handler drains the audit queue and closes the pool.
### DB helpers (`src/db/database.js`)
```js
await db.get(sql, params); // first row or null
await db.all(sql, params); // array of rows
await db.run(sql, params); // { lastInsertRowid, changes }
await db.query(sql, params); // raw pg result
await db.getSetting(key); // app_settings value (2 min cache)
await db.setSetting(key, v); // writes + invalidates cache
db.pool // pg.Pool instance for transactions
```
SQL uses `?` placeholders (auto-converted to `$1, $2, ...`) OR native `$N`.
INSERTs without explicit RETURNING get `RETURNING id` appended.
### Auth middleware (`src/middleware/auth.js`)
```js
var { authMiddleware, adminMiddleware, moderatorMiddleware } = require('../middleware/auth');
router.post('/thing', authMiddleware, handler); // requires auth
router.post('/admin-thing', authMiddleware, adminMiddleware, handler);
```
Sets `req.user` with `{ id, email, name, role, totp_enabled, disabled }` and
`req.sessionId`.
### AI (`src/utils/ai.js`)
```js
var { callAI } = require('../utils/ai');
var result = await callAI([
{ role: 'system', content: PROMPTS.hpiEncounter + INJECTION_GUARD },
{ role: 'user', content: wrapUserText('transcript', transcript) }
], { model, maxTokens: 4000 });
// result = { content, model, usage }
```
Throws `{ code: 'model_not_permitted' }` if the requested model isn't in the
active allowlist.
### Prompts (`src/utils/prompts.js`)
Flat `PROMPTS` object. Any template can be overridden live by writing a
`prompt.{name}` row in `app_settings`. Admin Panel's Prompt Editor is the UX
for that.
### Settings (`src/utils/config.js`)
```js
var v = await config.get('feature.read_aloud', 'false'); // key, default
await config.set('registration_enabled', 'true');
```
2-minute in-memory cache. Writes invalidate immediately.
### Logger (`src/utils/logger.js`)
```js
logger.audit(userId, 'action', 'details', req, { category: 'auth' });
logger.apiCall(userId, endpoint, { model, tokensInput, tokensOutput, duration });
logger.access(userId, 'login', req, true);
logger.error('scope', err.message);
```
Writes go to the database (batched), the daily JSONL file, and Loki (if
configured).
## Frontend
No framework, no bundler, no build step. `public/index.html` is the only
document. Modules communicate through `window.*` globals and `CustomEvent` on
`document`.
### Tab system
`app.js` owns `activateTab(name)`:
1. Fetch `/components/{name}.html` (browser-cached for 1 h, bust by
`?v=BUILD_ID` that the server injects).
2. Inject into `.app-body`.
3. Dispatch `CustomEvent('tabChanged', { detail: { tab: name } })`.
4. Feature modules (`soap.js`, `encounters.js`, etc.) listen for their own
tab name and initialize DOM references inside the just-injected fragment.
### Auth model (client)
`auth.js` branches on `isNativeApp()`:
- **Web** — no localStorage token; relies on the `ped_auth` httpOnly cookie
set by the server. `/api/auth/me` on boot verifies the session; failed
verification shows the login screen.
- **Mobile** — token lives in `capacitor-secure-storage-plugin` (iOS
Keychain / Android EncryptedSharedPreferences). `getAuthHeaders()` emits
`Authorization: Bearer <token>`.
`authFetch.js` wraps `window.fetch` to catch 401 on authenticated `/api/*`
requests and force re-login. `BroadcastChannel('pedscribe-auth')` propagates
logout to sibling tabs.
### Module load order
Fixed in `index.html`:
```
secureStorage → authFetch → auth → (feature modules)
```
All `<script defer>`. Dependencies enforced by declaration order.
## Adding things
### A new AI endpoint
1. Create `src/routes/myFeature.js`:
```js
var express = require('express');
var router = express.Router();
var { callAI } = require('../utils/ai');
var { authMiddleware } = require('../middleware/auth');
var PROMPTS = require('../utils/prompts');
var { wrapUserText, INJECTION_GUARD } = require('../utils/promptSafe');
var logger = require('../utils/logger');
router.post('/my-feature', authMiddleware, async (req, res) => {
try {
var { transcript, model } = req.body;
var result = await callAI([
{ role: 'system', content: PROMPTS.myFeature + INJECTION_GUARD },
{ role: 'user', content: wrapUserText('transcript', transcript) }
], { model });
res.json({ success: true, text: result.content });
logger.audit(req.user.id, 'generate_my_feature', 'Generated', req, { category: 'clinical' });
} catch (err) {
res.status(500).json({ error: 'Request failed' });
}
});
module.exports = router;
```
2. Mount in `server.js`:
```js
app.use('/api', require('./src/routes/myFeature'));
```
3. Add the template to `src/utils/prompts.js`.
4. Add a component under `public/components/myfeature.html`.
5. Add `public/js/myFeature.js` with a `tabChanged` listener.
6. Register the tab button in `public/index.html`.
### A new table / column
New changes go in a migration file. See `docs/migrations.md`.
```bash
docker exec -w /app pediatric-ai-scribe npm run migrate:new -- add_my_table
# edit the generated file
```
### A new setting
1. Optional — add a default in the `defaults` array inside `initDatabase()`
(only needed if the app should seed it on fresh installs).
2. Read at runtime: `await db.getSetting('my_key')`.
3. Admin-editable automatically through `PUT /api/admin/config` which accepts
arbitrary keys.
## Physician Templates And Preferences
1. Settings saves user templates/preferences through `/api/memories` into
`user_memories`.
2. New rows encrypt `name` and `content` with the shared `enc1:` string format.
3. `GET /api/memories/context` decrypts rows and returns only AI-context
categories: `physical_exam`, `ros`, `encounter_format`, `family_history`,
`assessment_plan`, `template_soap`, `template_hpi`, `template_wellvisit`,
`template_sickvisit`, and `template_ed`.
4. `custom` rows remain visible in settings but are not included in prompt
context.
5. Legacy `correction_*` rows from the removed correction-learning feature are
filtered out rather than deleted.
## Route reference
| File | Mount | Auth | Purpose |
|---|---|---|---|
| `auth.js` | `/api/auth` | Public | Register, login, 2FA, email verify, password reset, backup codes |
| `oidc.js` | `/api/auth` | Public | OIDC SSO (Authorization Code + PKCE) |
| `sessions.js` | `/api/sessions` | Auth | Active sessions list + revoke |
| `hpi.js` | `/api` | Auth | HPI from encounter or dictation |
| `soap.js` | `/api` | Auth | SOAP generation |
| `chartReview.js` | `/api` | Auth | Chart review / pre-charting |
| `hospitalCourse.js` | `/api` | Auth | Hospital course |
| `wellVisit.js` | `/api` | Auth | Well visit + SSHADESS |
| `sickVisit.js` | `/api` | Auth | Sick visit |
| `milestones.js` | `/api` | Auth | Developmental milestone narratives |
| `refine.js` | `/api` | Auth | Refine / shorten / clarify |
| `transcribe.js` | `/api` | Auth | LiteLLM STT |
| `tts.js` | `/api` | Auth | LiteLLM TTS |
| `encounters.js` | `/api` | Auth | Save / load / optimistic-lock encounters |
| `memories.js` | `/api` | Auth | Templates + prompt preferences |
| `audioBackups.js` | `/api` | Auth | Encrypted audio retry store |
| `documents.js` | `/api` | Auth | S3 documents (magic-byte checked) |
| `userPreferences.js` | `/api` | Auth | Per-user STT/TTS choice |
| `nextcloud.js` | `/api` | Auth | Encrypted WebDAV tokens |
| `billing.js` | `/api` | Auth | ICD-10 / CPT code suggestion |
| `logs.js` | `/api` | Auth | Usage + audit dump (admin-only endpoints gated further) |
| `admin.js` | `/api/admin` | Admin | User management |
| `adminConfig.js` | `/api/admin` | Admin | Settings, prompts, models, SMTP, OIDC |
| `adminMilestones.js` | `/api/admin` | Admin | Milestone data management |
| `learningHub.js` | `/api/learning` | Auth | Content delivery + quizzes |
| `learningAdmin.js` | `/api/admin/learning` | Moderator | CMS CRUD |
| `learningAI.js` | `/api/admin/learning` | Moderator | AI content gen, PPTX export |
| `clinicalAssistant.js` | `/api` | Auth | Grounded clinical answers over MCP retrieval |
| `myResources.js` | `/api` | Auth | Personal teaching material: generate, refine, export |
| `notes.js` | `/api` | Auth | Personal notes |
| `edEncounters.js` | `/api` | Auth | ED encounters: staged notes, consolidate, MDM finalize |
| `dontMiss.js` | `/api` | Auth | Don't-miss diagnosis suggestions |
| `patientEducation.js` | `/api` | Auth | Patient education handouts |
| `peGuide.js` | `/api` | Auth | Physical exam guide |
| `diagrams.js` | `/api` | Auth | Diagram rendering |
| `generatedImages.js` | `/api`, `/api/admin/learning/image` | Auth | Image generation jobs and their stored output |
| `extensions.js` | `/api` | Auth | Browser-extension integration |
| `adminDocs.js` | `/api/admin/docs` | Admin | In-app rendering of this `docs/` tree |
## Frontend JS module reference
| File | Purpose |
|---|---|
| `secureStorage.js` | Platform-branched token storage (Keychain / localStorage) |
| `authFetch.js` | Global fetch wrapper (401 → logout + reload) |
| `auth.js` | Login / register / SSO / session management / Turnstile / backup-code modal |
| `app.js` | Tab navigation, model selector, audio recorder, transcription orchestration |
| `liveEncounter.js` | Live recording UI + live preview |
| `soap.js`, `hpi.js` (none — in liveEncounter), `sickVisit.js`, `wellVisit.js`, `hospitalCourse.js`, `chartReview.js` | Clinical tabs |
| `milestones.js` + `milestonesData.js` | Milestones tab |
| `shadess.js` | SSHADESS adolescent assessment |
| `encounters.js` | Save / load / resume with optimistic lock |
| `memories.js` | Physician templates and prompt preferences UI |
| `speechRecognition.js` | Explicit opt-in browser Web Speech support |
| `voicePreferences.js` | Per-user STT/TTS override |
| `audioBackup.js` | Server + IndexedDB backup retries |
| `nextcloud.js` | Connect / export |
| `documents.js` | S3 upload / download |
| `calculators.js` | Pediatric calculators (BP, BMI, growth, bilirubin, vitals, etc.) |
| `learningHub.js` | Content browser + CMS editor |
| `accountBoundary.js` | One verified account owner per JS realm; guards cross-account leakage |
| `ed-encounters.js` | ED encounter workflow: staged notes, consolidate, MDM finalize |
| `voiceDictation.js` | Dictation capture and voice-mode call UI |
| `transcriptionSettings.js` | Transcription provider and model picker |
| `recordingModules.js` | Shared recorder wiring reused by the clinical tabs |
| `calc-math.js` | Pure calculator math, kept separate so it can be tested directly |
| `admin-docs.js` | Documentation viewer inside the Admin panel |
| `drugs-loader.js`, `ui-state.js` | Small shared helpers |
| `e2e-bootstrap.js` | Test-only hook; inert unless the e2e harness sets it up |
| `admin.js` | Admin panel (users, settings, prompts, models) |
Larger features live in their own directory rather than a single file:
`admin/`, `assistant/`, `bedside/`, `calculators/`, `learningHub/`, `notes/`,
and `wellVisit/`.
## Common tasks
### Change default temperature
Edit the default in `callAI()` in `src/utils/ai.js`. Per-call `options.temperature`
overrides.
### Override a prompt without deploying
Admin Panel → Prompts → pick the template → edit → save. Takes effect
immediately; cache is invalidated on write.
### Add a model to the dropdown
Admin Panel → Models → Add Custom Model. Exact provider ID, display name,
cost string, category (`free` / `fast` / `smart` / `premium`). Appears
immediately for all users.
### Trace an AI call
```bash
docker compose logs -f pediatric-scribe | grep '\[AI\]'
```
Or:
```sql
SELECT endpoint, model_used, tokens_input, tokens_output, duration_ms, cost_estimate, error
FROM api_log
ORDER BY timestamp DESC
LIMIT 20;
```
### Force a re-index
```bash
docker exec -w /app pediatric-ai-scribe npm run maint:reindex
```
Runs after any Postgres image upgrade if the startup drift check didn't
catch it.
## Local development
```bash
docker compose up -d postgres # just the DB
npm ci
cp .env.example .env # set JWT_SECRET, DATA_ENCRYPTION_KEY, provider credentials
node server.js
```
App binds `http://localhost:3000`. Without `APP_URL`, production-mode guards
relax (open CORS, non-secure cookies) — never deploy like this.
Or run it the way production does, against the built image:
```bash
cp .env.example .env
./scripts/build-image.sh
docker compose up -d --no-build
curl -fsS http://127.0.0.1:3552/api/health
```
Use Node 24 to match the image. Tests run from the repository root with
`npm ci && npm test`; `node --check <file>` is a fast syntax gate when you are
touching a backend entrypoint.
## Change workflow
1. Read the relevant route, utility, frontend module, and tests before editing.
2. Make the smallest correct change.
3. Add or update a regression test when changing clinical rendering, model routing, auth, settings, or source handling.
4. Run focused tests first if available.
5. Run `npm test` before deploy or commit.
6. Deploy with Docker only after tests pass.
7. Verify `/api/health` after deploy.
## Changing the Clinical Assistant
Clinical Assistant changes should usually include tests because small rendering or prompt changes can affect clinical trust.
High-risk areas:
- citation linking,
- table rendering,
- source title cleanup,
- named-source provenance rules,
- image intent detection,
- MCP result normalization,
- provider/model selection.
When a real answer renders badly, save a de-identified example as a fixture or direct test input. Do not make broad global repairs that convert arbitrary numbers into citation links.
## Frontend rendering rules
Use `textContent` for plain text. Use `innerHTML` only for static templates, sanitized markdown, or HTML built entirely from escaped values.
Safe patterns:
```js
el.textContent = userText;
el.innerHTML = escapeHtml(userText).replace(/\n/g, '<br>');
el.innerHTML = sanitizeHtml(renderMarkdown(modelOutput));
```
Unsafe pattern:
```js
el.innerHTML = modelOutput;
```
If a dynamic value enters an HTML string, escape it at the point of insertion. If it is an attribute value, escape quotes too.
## Deployment checks
After deployment:
```bash
curl -fsS http://127.0.0.1:3552/api/health
docker compose ps pediatric-scribe
```
If the browser still shows old frontend behavior, force-refresh or check the injected `BUILD_ID` asset query string.
## Documentation expectations
Keep docs close to operational truth. If a behavior changes, update the most specific doc in the same change. Prefer short, current docs over long historical explanations.