pediatric-ai-scribe-v3/docs/developer-guide.md

# Developer Guide

This guide explains how the codebase works so any developer can understand, modify, and extend the Pediatric AI Scribe platform.

---

## Project Structure

```
server.js                    -- Express app entry point, middleware stack, route registration
src/
  db/database.js             -- PostgreSQL pool, schema init, query helpers, auto-cleanup
  middleware/
    auth.js                  -- JWT/cookie auth, admin/moderator role checks
    logging.js               -- Request logging middleware
  utils/
    ai.js                    -- Multi-provider AI client (callAI), model discovery
    models.js                -- Built-in model definitions per provider
    prompts.js               -- All AI system prompts (overridable via DB)
    config.js                -- DB-backed settings with 2-minute cache
    logger.js                -- Audit, API, access logging to DB + files
    embeddings.js            -- Vector embedding generation (Vertex/LiteLLM/OpenAI)
    transcribeAWS.js         -- Amazon Transcribe client
    transcribeGoogle.js      -- Google Gemini STT
    transcribeLocal.js       -- Local whisper.cpp / faster-whisper
    ttsGoogle.js             -- Google Cloud TTS
  routes/                    -- 27 route files (see below)
public/
  index.html                 -- Main SPA shell, auth forms, script tags
  sw.js                      -- Service worker (cache shell, network-first API)
  manifest.json              -- PWA manifest
  components/                -- HTML fragments loaded into tabs
  js/                        -- 20+ vanilla JS modules
  css/styles.css             -- All styles in one file
  icons/                     -- PWA icons
  models/                    -- Self-hosted Whisper WASM model files
```

---

## How the Frontend Works

### SPA Architecture

This is a **vanilla JavaScript SPA** -- no React, Vue, or framework. The approach:

1. `index.html` is the only HTML page. It contains two top-level divs:
   - `#auth-screen` -- login/register/forgot forms (hidden when authenticated)
   - `#main-app` -- the actual application (hidden until authenticated)

2. **Tabs** are managed by `app.js`. The sidebar has tab buttons. Clicking a tab calls `activateTab(tabName)` which:
   - Fetches `/components/{tabName}.html` via `loadComponent()`
   - Injects the HTML into `.app-body`
   - Dispatches a `CustomEvent('tabChanged', { detail: { tab: tabName } })`
   - Other modules listen for this event to initialize their UI

3. **Module communication** uses `window` globals and `CustomEvent`:
   - Functions exposed on `window` (e.g., `window.saveAudioBackup`, `window.getAuthHeaders`, `window.showToast`)
   - Events dispatched on `document` (e.g., `recording-started`, `recording-stopped`, `tabChanged`)

4. **Script loading**: all JS files use `defer` attribute, loaded in dependency order defined in `index.html` (lines 302-328). `audioBackup.js` before `app.js` before `auth.js` etc.

### Auth Flow

`auth.js` runs on DOMContentLoaded:
1. Checks for SSO redirect (`?sso=ok` URL param)
2. Tries to restore session from localStorage token or cookie
3. Calls `/api/auth/me` to validate
4. If valid: hides auth screen, shows main app, loads default tab
5. If invalid: shows auth screen

Token is stored in both `localStorage` (for Bearer header) and `ped_auth` cookie (for SSO/cookie-based auth). The auth middleware accepts either.

### Component Lifecycle

When a tab is activated:
1. HTML is fetched and injected into `.app-body`
2. The `tabChanged` event fires
3. Each module has a listener that initializes when its tab is active:
   ```javascript
   document.addEventListener('tabChanged', function(e) {
     if (e.detail.tab === 'settings') {
       loadMemories();
       renderAudioBackups();
     }
   });
   ```
4. Modules query the DOM for elements inside the injected component HTML

### Common Patterns

**API calls**: Always use `getAuthHeaders()` for JSON requests, or manually add Bearer token for FormData uploads. Include `credentials: 'same-origin'` when cookie auth may be needed.

**Toast notifications**: `showToast(message, type)` where type is `success`, `error`, `info`, or `warning`.

**Loading overlay**: `showLoading(message)` and `hideLoading()`.

**Model selection**: `getSelectedModel()` returns the currently selected model ID from the tab's dropdown.

---

## How the Backend Works

### Middleware Stack (server.js)

Requests flow through this chain in order:

```
Request
  -> Helmet (CSP headers)
  -> CORS (restrict to APP_URL)
  -> cookieParser
  -> express.json (10MB limit)
  -> Rate limiters (per-endpoint)
  -> Static file serving (public/)
  -> Route handlers
  -> 404 fallback (serves index.html for SPA routes)
```

### Database Layer (src/db/database.js)

The database module provides:

- `db.get(sql, params)` -- single row (returns object or null)
- `db.all(sql, params)` -- multiple rows (returns array)
- `db.run(sql, params)` -- INSERT/UPDATE/DELETE (returns `{ lastInsertRowid, changes }`)
- `db.query(sql, params)` -- raw pg query
- `db.getSetting(key)` -- read from app_settings
- `db.setSetting(key, value)` -- write to app_settings

SQL uses `?` placeholders which are auto-converted to PostgreSQL `$1, $2, ...` by `convertPlaceholders()`. You can also use `$N` directly.

For INSERT statements, `RETURNING id` is auto-appended if not already present.

**Schema migration**: all tables use `CREATE TABLE IF NOT EXISTS` and `ALTER TABLE ADD COLUMN IF NOT EXISTS`. Migrations run on every startup -- no separate migration tool needed. Just add new columns/tables to `initDatabase()`.

### Authentication Middleware (src/middleware/auth.js)

Three middleware functions:
- `authMiddleware` -- verifies JWT from Bearer header or `ped_auth` cookie. If Bearer header is present but empty, falls through to cookie. Sets `req.user`.
- `adminMiddleware` -- requires `req.user.role === 'admin'` (use after authMiddleware)
- `moderatorMiddleware` -- requires admin or moderator role

### AI Integration (src/utils/ai.js)

The `callAI(messages, options)` function is the single entry point for all AI calls:

```javascript
var result = await callAI([
  { role: 'system', content: systemPrompt },
  { role: 'user',   content: userContent }
], { model: selectedModel });
// result = { text: "...", usage: { input, output } }
```

Internally, `callAI` routes to the active provider:
- **Bedrock**: uses `InvokeModelCommand` with Converse API
- **Azure/OpenRouter/LiteLLM**: uses OpenAI SDK `chat.completions.create()`
- **Vertex**: uses `@google-cloud/vertexai` GenerativeModel

Provider is selected once at startup. The `model` param in options overrides the default.

### Settings System (src/utils/config.js)

Database-backed configuration with in-memory caching:

```javascript
var config = require('../utils/config');
var value = await config.get('feature.read_aloud', 'true'); // key, default
await config.set('registration_enabled', 'false');
```

Cache TTL is 2 minutes. Settings are stored in the `app_settings` table. Environment variables take precedence for provider credentials, but most app settings are DB-backed.

### Prompt System (src/utils/prompts.js)

All AI prompts are defined as a `PROMPTS` object:

```javascript
module.exports = {
  hpiEncounter: "You are a pediatric physician...",
  soapFull: "Generate a complete SOAP note...",
  // ... etc
};
```

On startup, DB overrides are loaded from `app_settings` where `key LIKE 'prompt.%'`. Admin can edit prompts from the Admin Panel without restarting.

### Logging (src/utils/logger.js)

```javascript
var logger = require('../utils/logger');
logger.audit(userId, 'action_name', 'details', req, { category: 'auth' });
logger.apiCall(userId, endpoint, { model, tokens_input, tokens_output, duration_ms });
logger.access(userId, 'login', req, true);
```

All log entries go to both the database and daily log files at `/data/logs/YYYY-MM-DD.log`.

---

## Adding a New Feature

### Adding a New AI Endpoint

1. Create a route file in `src/routes/`:

```javascript
var express = require('express');
var router = express.Router();
var { callAI } = require('../utils/ai');
var { authMiddleware } = require('../middleware/auth');
var PROMPTS = require('../utils/prompts');

router.post('/my-feature', authMiddleware, async function(req, res) {
  try {
    var { transcript, model } = req.body;
    var result = await callAI([
      { role: 'system', content: PROMPTS.myFeature },
      { role: 'user',   content: transcript }
    ], { model });
    res.json({ success: true, text: result.text });
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

module.exports = router;
```

2. Register it in `server.js`:
```javascript
app.use('/api', require('./src/routes/myFeature'));
```

3. Add the prompt to `src/utils/prompts.js`:
```javascript
myFeature: "You are a pediatric physician. Generate..."
```

4. Create a frontend component in `public/components/myfeature.html`

5. Add a tab button in `public/index.html` sidebar

6. Create `public/js/myFeature.js` with a `tabChanged` listener

### Adding a New Database Table

Add the `CREATE TABLE IF NOT EXISTS` statement inside `initDatabase()` in `src/db/database.js`:

```javascript
try { await client.query(`
  CREATE TABLE IF NOT EXISTS my_table (
    id SERIAL PRIMARY KEY,
    user_id INTEGER REFERENCES users(id) ON DELETE CASCADE,
    data TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
  );
  CREATE INDEX IF NOT EXISTS idx_my_table_user ON my_table(user_id);
`); } catch(e) {}
```

No migration files needed. The `IF NOT EXISTS` pattern is idempotent.

### Adding a New Setting

1. Add the default in the `defaults` array in `initDatabase()`:
```javascript
['my_setting.key', 'default_value'],
```

2. Read it in route handlers:
```javascript
var value = await db.getSetting('my_setting.key');
```

3. If it should be admin-editable, ensure the admin config route handles it (the generic `POST /api/admin/config` already saves any key-value pair).

---

## Key Design Decisions

### Why Vanilla JS (No Framework)

The frontend uses plain JavaScript instead of React/Vue because:
- Simpler deployment (no build step, no bundler)
- Components are HTML fragments loaded via fetch
- State is managed via DOM elements and window globals
- Works with aggressive CSP (no eval needed for templates)
- Easy to modify any part without understanding a framework's lifecycle

### Why PostgreSQL Placeholders Are Auto-Converted

The codebase was originally SQLite, then migrated to PostgreSQL. The `convertPlaceholders()` function in `database.js` converts `?` to `$1, $2, ...` so existing queries work unchanged. New code can use either style.

### Why Prompts Are DB-Overridable

Clinicians have specific documentation preferences. Making prompts editable from the admin panel means the team can tune AI output without redeploying. The `prompt.*` keys in `app_settings` override the hardcoded defaults in `prompts.js`.

### Why Audio Backups Use PostgreSQL (Not Filesystem)

Audio is stored as gzip-compressed BYTEA in PostgreSQL because:
- Works in containerized environments without persistent volumes for temp files
- Auto-expires via SQL (`expires_at` column + hourly cleanup)
- Per-user access control is handled by the same auth system
- No orphaned files if the container restarts

### Why Corrections Are Low-Priority Style Hints

The physician memory/correction system injects past edits into AI prompts. Originally these were labeled "APPLY these preferences" which caused smaller models to hallucinate content from the correction examples instead of the current transcript. The injection was changed to `[STYLE HINTS (low priority)]` with truncated 200-char snippets to prevent this.

### Why Auth Middleware Checks Cookie After Bearer

The auth middleware first checks the `Authorization: Bearer` header, then falls back to the `ped_auth` cookie. If a Bearer header is present but the token is empty (which happens with SSO-only users who have no localStorage token), the middleware now correctly treats it as absent and falls through to the cookie. This was a bug fix -- previously, an empty Bearer token would block cookie auth entirely.

---

## Route File Reference

| File | Mount Point | Auth | Purpose |
|------|------------|------|---------|
| `auth.js` | `/api/auth` | Public | Registration, login, 2FA, email verification, password reset |
| `oidc.js` | `/api/auth` | Public | OpenID Connect SSO flow |
| `hpi.js` | `/api` | Auth | HPI generation (encounter + dictation) |
| `soap.js` | `/api` | Auth | SOAP note generation |
| `chartReview.js` | `/api` | Auth | Chart review / precharting |
| `hospitalCourse.js` | `/api` | Auth | Hospital course generation |
| `wellVisit.js` | `/api` | Auth | Well visit + SSHADESS |
| `sickVisit.js` | `/api` | Auth | Sick visit documentation |
| `milestones.js` | `/api` | Auth | Developmental milestone narratives |
| `refine.js` | `/api` | Auth | Refine, shorten, clarify documents |
| `transcribe.js` | `/api` | Auth | Speech-to-text (5 providers) |
| `tts.js` | `/api` | Auth | Text-to-speech (3 providers) |
| `encounters.js` | `/api` | Auth | Save/load/delete encounters |
| `memories.js` | `/api` | Auth | Physician templates + corrections |
| `audioBackups.js` | `/api` | Auth | Audio backup storage |
| `documents.js` | `/api` | Auth | S3 document management |
| `userPreferences.js` | `/api` | Auth | STT/TTS preferences |
| `nextcloud.js` | `/api` | Auth | WebDAV integration |
| `logs.js` | `/api` | Auth | Usage and audit logs |
| `admin.js` | `/api/admin` | Admin | User management |
| `adminConfig.js` | `/api/admin` | Admin | Settings, prompts, models, SMTP, OIDC |
| `adminMilestones.js` | `/api/admin` | Admin | Milestone data management |
| `learningHub.js` | `/api/learning` | Auth | Learning content delivery + quizzes |
| `learningAdmin.js` | `/api/admin/learning` | Moderator | Learning CMS CRUD |
| `learningAI.js` | `/api/admin/learning` | Moderator | AI content generation, PPTX, slides |

---

## Frontend JS File Reference

| File | Loads After | Purpose |
|------|-------------|---------|
| `app.js` | audioBackup, correctionTracker | Tab navigation, model selector, AudioRecorder, transcription |
| `auth.js` | app.js | Login, register, SSO, session management, Turnstile |
| `liveEncounter.js` | auth.js | Recording UI, speech recognition, live transcript |
| `soap.js` | auth.js | SOAP note tab |
| `hospitalCourse.js` | auth.js | Hospital course tab |
| `chartReview.js` | auth.js | Chart review tab |
| `wellVisit.js` | auth.js | Well visit tab |
| `sickVisit.js` | auth.js | Sick visit tab |
| `encounters.js` | auth.js | Save/load/resume encounters |
| `milestones.js` | milestonesData.js | Milestone selection and generation |
| `shadess.js` | auth.js | SSHADESS adolescent assessment |
| `learningHub.js` | auth.js | Learning Hub + CMS (1843 lines) |
| `memories.js` | auth.js | Physician templates + corrections UI |
| `documents.js` | auth.js | S3 document upload/download |
| `admin.js` | auth.js | Admin panel (users, settings, prompts, models) |
| `audioBackup.js` | (early) | Audio backup save/list/retry/delete |
| `correctionTracker.js` | (early) | Track AI output edits for learning |
| `browserWhisper.js` | (early) | In-browser Whisper via WebAssembly |
| `speechRecognition.js` | (early) | Web Speech API wrapper |
| `voicePreferences.js` | auth.js | STT/TTS model/voice selection |
| `nextcloud.js` | auth.js | Nextcloud connection and export |

---

## Testing Locally

```bash
# Start just the database
docker compose up -d postgres

# Install dependencies
npm install

# Copy and configure env
cp .env.example .env
# Edit .env with your provider keys

# Start the app
node server.js
```

The app runs on `http://localhost:3000`. Without `APP_URL` set, CORS allows all origins (development mode).

## Common Tasks

### Change the default AI temperature

Edit the `callAI` function in `src/utils/ai.js`. The default temperature is `0.3` for most providers.

### Add a new AI prompt

1. Add the prompt text to `src/utils/prompts.js`
2. Use it in your route: `var PROMPTS = require('../utils/prompts'); ... PROMPTS.myPrompt`
3. It becomes admin-editable automatically via `prompt.myPrompt` in the DB

### Override a prompt without code changes

In the admin panel, go to Settings > Prompts. Edit any prompt. The override is stored in `app_settings` with key `prompt.{name}` and takes effect immediately (no restart needed).

### Add a model to the dropdown

From the admin panel, go to Models > Add Custom Model. Enter:
- **Model ID**: the exact string the provider expects (e.g., `gemini-2.5-flash` for LiteLLM)
- **Display Name**: what users see
- **Cost**: price string (e.g., `~$0.001`)
- **Category**: determines dropdown group (free/fast/smart/premium)

The model appears immediately for all users.

### Debug an AI call

Check `docker compose logs -f pediatric-scribe` for lines like:
```
[AI] bedrock/anthropic.agent-config-3-haiku... 1247 tokens in 2.3s
```

Or query the `api_log` table for detailed metrics:
```sql
SELECT endpoint, model_used, tokens_input, tokens_output, duration_ms, cost_estimate
FROM api_log ORDER BY timestamp DESC LIMIT 20;
```