Expand DEVELOPER_GUIDE with detailed architecture, DB layer, file descriptions
- Added request flow walkthrough, AI provider selection logic - Documented database layer (raw pg, no ORM) with helper methods - Expanded directory structure with detailed per-file descriptions - Expanded frontend JS descriptions (what each file does) - Rewrote PDF section with current 50k char limit, supported types - Updated version history through v6.0
This commit is contained in:
Daniel Onyejesi
parent
1b1fe535da
commit
805d0fea55
1 changed files with 126 additions and 56 deletions
|
|
@ -1,5 +1,5 @@
|
||||||
# Pediatric AI Scribe — Developer Guide
|
# Pediatric AI Scribe — Developer Guide
|
||||||
**Version:** 3.19 | **Stack:** Node.js / Express / PostgreSQL / Vanilla JS
|
**Version:** 6.0 | **Stack:** Node.js / Express / PostgreSQL / Vanilla JS
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -59,10 +59,39 @@ PostgreSQL (pg driver, no ORM)
|
||||||
|— user_memories, learning_*, access_log, api_log
|
|— user_memories, learning_*, access_log, api_log
|
||||||
```
|
```
|
||||||
|
|
||||||
**AI providers** (configured via environment variables, in priority order):
|
### How Requests Flow
|
||||||
1. AWS Bedrock (if `AWS_BEDROCK_REGION` is set)
|
|
||||||
2. Azure OpenAI (if `AZURE_OPENAI_ENDPOINT` is set)
|
1. **Browser** sends HTTP request with `Authorization: Bearer <jwt>` header
|
||||||
3. OpenRouter (default, requires `OPENROUTER_API_KEY`)
|
2. **Express middleware chain:** Helmet (security headers) → CORS → rate limiter → body parser → logging middleware → route handler
|
||||||
|
3. **Auth middleware** (`src/middleware/auth.js`) decodes JWT, queries `users` table, attaches `req.user` with `{ id, email, name, role }`
|
||||||
|
4. **Route handler** processes the request — for AI routes, calls `callAI()` which routes to the configured provider
|
||||||
|
5. **Database** is accessed via the `pg` driver directly (no ORM). All queries use parameterized placeholders (`$1`, `$2`) to prevent SQL injection
|
||||||
|
6. **Response** is JSON for API calls, or static files served from `/public`
|
||||||
|
|
||||||
|
### AI Providers
|
||||||
|
|
||||||
|
Configured via environment variables. The provider is selected at startup in `src/utils/ai.js` using this priority:
|
||||||
|
|
||||||
|
1. **AWS Bedrock** — if `AWS_BEDROCK_REGION` is set. HIPAA eligible with BAA. Uses `@aws-sdk/client-bedrock-runtime`. Anthropic models use the native Messages API (`InvokeModel`); all others use the Converse API.
|
||||||
|
2. **Azure OpenAI** — if `AZURE_OPENAI_ENDPOINT` is set. HIPAA eligible. Uses the OpenAI SDK pointed at your Azure endpoint.
|
||||||
|
3. **OpenRouter** — default fallback if `OPENROUTER_API_KEY` is set. Routes to 20+ models from various providers. Not HIPAA compliant.
|
||||||
|
|
||||||
|
The provider cannot be changed at runtime — it's determined once at startup. To switch providers, update `.env` and restart the container.
|
||||||
|
|
||||||
|
### Database Layer
|
||||||
|
|
||||||
|
The app uses **raw SQL via the `pg` driver** — no ORM (Sequelize, Prisma, etc.). This is intentional:
|
||||||
|
|
||||||
|
- **Simplicity:** Every query is visible and explicit. No magic, no migrations framework, no model definitions to sync.
|
||||||
|
- **Performance:** No ORM overhead or N+1 query problems.
|
||||||
|
- **Schema management:** `src/db/database.js` runs `CREATE TABLE IF NOT EXISTS` on startup, plus `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` for migrations. This means the schema is always up-to-date when the app starts.
|
||||||
|
- **Future ORM migration:** If needed, the queries are standard PostgreSQL and can be wrapped by any ORM. The main work would be defining models and replacing direct `db.get()`/`db.run()` calls.
|
||||||
|
|
||||||
|
The `database.js` file exports a helper object (`db`) with convenience methods:
|
||||||
|
- `db.get(sql, params)` — returns first row or `null`
|
||||||
|
- `db.all(sql, params)` — returns all rows as array
|
||||||
|
- `db.run(sql, params)` — executes INSERT/UPDATE/DELETE, returns `{ rowCount }`
|
||||||
|
- `db.getSetting(key)` / `db.setSetting(key, value)` — shorthand for `app_settings` table
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -70,19 +99,27 @@ PostgreSQL (pg driver, no ORM)
|
||||||
|
|
||||||
```
|
```
|
||||||
/
|
/
|
||||||
├── server.js # Express app entry point, route registration, CSP
|
├── server.js # Express app entry point (route registration, Helmet CSP,
|
||||||
├── package.json
|
│ # rate limiters, static file serving, error handlers)
|
||||||
├── Dockerfile
|
├── package.json # Dependencies (~25 production deps, no devDeps)
|
||||||
├── docker-compose.yml # Production compose
|
├── Dockerfile # Multi-stage Node.js 20 Alpine build
|
||||||
├── docker-compose.local.yml # Local development (port 3552)
|
├── docker-compose.yml # Production compose (uses Docker Hub image)
|
||||||
|
├── docker-compose.local.yml # Local development (builds from source, port 3552)
|
||||||
|
├── admin-cli.js # CLI tool for admin tasks (create user, reset password)
|
||||||
├── DEVELOPER_GUIDE.md # This file
|
├── DEVELOPER_GUIDE.md # This file
|
||||||
│
|
│
|
||||||
├── src/
|
├── src/
|
||||||
│ ├── db/
|
│ ├── db/
|
||||||
│ │ └── database.js # DB connection pool, schema init, migrations
|
│ │ └── database.js # DB connection pool (pg.Pool), schema init
|
||||||
|
│ │ # (CREATE TABLE IF NOT EXISTS for all tables),
|
||||||
|
│ │ # column migrations (ALTER TABLE ADD COLUMN IF NOT EXISTS),
|
||||||
|
│ │ # helper methods: db.get(), db.all(), db.run(),
|
||||||
|
│ │ # db.getSetting(), db.setSetting()
|
||||||
│ ├── middleware/
|
│ ├── middleware/
|
||||||
│ │ ├── auth.js # authMiddleware, adminMiddleware, moderatorMiddleware
|
│ │ ├── auth.js # authMiddleware (JWT decode → req.user),
|
||||||
│ │ └── logging.js # Access log middleware
|
│ │ # adminMiddleware (role === 'admin'),
|
||||||
|
│ │ # moderatorMiddleware (role === 'admin' || 'moderator')
|
||||||
|
│ │ └── logging.js # Logs every request to api_log table (method, path, user, IP, duration)
|
||||||
│ ├── routes/
|
│ ├── routes/
|
||||||
│ │ ├── auth.js # Login, register, 2FA, password reset, /me
|
│ │ ├── auth.js # Login, register, 2FA, password reset, /me
|
||||||
│ │ ├── admin.js # User management (admin only)
|
│ │ ├── admin.js # User management (admin only)
|
||||||
|
|
@ -105,11 +142,19 @@ PostgreSQL (pg driver, no ORM)
|
||||||
│ │ ├── learningAI.js # AI generation for Learning Hub content
|
│ │ ├── learningAI.js # AI generation for Learning Hub content
|
||||||
│ │ └── logs.js # Usage/audit/API/access logs + client error
|
│ │ └── logs.js # Usage/audit/API/access logs + client error
|
||||||
│ └── utils/
|
│ └── utils/
|
||||||
│ ├── ai.js # callAI() — multi-provider AI client
|
│ ├── ai.js # callAI(messages, options) — routes to OpenRouter/Bedrock/Azure.
|
||||||
│ ├── models.js # Model ID lists, defaults, Bedrock ID mapping
|
│ │ # Handles Anthropic InvokeModel (Messages API) vs Converse API,
|
||||||
│ ├── prompts.js # Prompt templates, DB override loader
|
│ │ # thinking block extraction, fallback model retry, duration tracking.
|
||||||
|
│ ├── models.js # OPENROUTER_MODELS[], BEDROCK_MODELS[], AZURE_MODELS[]
|
||||||
|
│ │ # Each model: { id, name, cost, tag, category, bedrockId, maxOut, regions }
|
||||||
|
│ │ # getBedrockModelId() maps app IDs to Bedrock/inference profile IDs.
|
||||||
|
│ │ # getAvailableModels() filters by region. getAvailableModelsWithOverrides()
|
||||||
|
│ │ # applies admin-disabled/custom models from DB.
|
||||||
|
│ ├── prompts.js # Default prompt templates for every AI route. Loaded on startup,
|
||||||
|
│ │ # then overridden by DB values (app_settings: 'prompt.*' keys).
|
||||||
|
│ │ # PROMPTS.get('key') returns the DB override or default.
|
||||||
│ ├── config.js # App configuration helpers
|
│ ├── config.js # App configuration helpers
|
||||||
│ └── logger.js # Winston logger (file + console)
|
│ └── logger.js # Winston logger (file + console, JSON format)
|
||||||
│
|
│
|
||||||
├── public/
|
├── public/
|
||||||
│ ├── index.html # Single HTML shell, loads all components
|
│ ├── index.html # Single HTML shell, loads all components
|
||||||
|
|
@ -117,24 +162,33 @@ PostgreSQL (pg driver, no ORM)
|
||||||
│ ├── css/
|
│ ├── css/
|
||||||
│ │ └── styles.css # All CSS (single file, ~750 lines)
|
│ │ └── styles.css # All CSS (single file, ~750 lines)
|
||||||
│ ├── js/
|
│ ├── js/
|
||||||
│ │ ├── app.js # Core: tab switching, loadComponent, global helpers
|
│ │ ├── app.js # Core: tab switching via data-tab buttons, loadComponent()
|
||||||
│ │ ├── auth.js # Login/register UI, JWT localStorage, getAuthHeaders()
|
│ │ │ # fetches HTML from /components/, global helpers (showToast,
|
||||||
│ │ ├── admin.js # Admin panel UI
|
│ │ │ # showLoading, getAuthHeaders, getSelectedModel, etc.)
|
||||||
│ │ ├── liveEncounter.js # Live recording tab
|
│ │ ├── auth.js # Login/register/forgot-password forms, JWT storage in
|
||||||
│ │ ├── voiceDictation.js # Dictation tab
|
│ │ │ # localStorage ('ped_scribe_token'), enterApp()/clearSession(),
|
||||||
│ │ ├── hospitalCourse.js # Hospital course tab
|
│ │ │ # resend verification link handler, 2FA code input
|
||||||
│ │ ├── chartReview.js # Chart review tab
|
│ │ ├── admin.js # Admin panel: user management, site settings, SMTP config,
|
||||||
│ │ ├── soap.js # SOAP tab
|
│ │ │ # model enable/disable, prompt editor, announcement banner
|
||||||
│ │ ├── milestones.js # Milestones tab (inside Well Visit)
|
│ │ ├── liveEncounter.js # MediaRecorder → Whisper transcription → AI HPI generation.
|
||||||
│ │ ├── wellVisit.js # Well Visit guide + vaccine schedule
|
│ │ │ # Handles start/stop recording, timer, save/load encounters
|
||||||
│ │ ├── shadess.js # SSHADESS form + Well Visit note generation
|
│ │ ├── voiceDictation.js # Web Speech API (real-time) or Whisper (recorded) dictation
|
||||||
│ │ ├── sickVisit.js # Sick Visit tab
|
│ │ ├── hospitalCourse.js # Paste/dictate hospital course → AI summary
|
||||||
│ │ ├── nextcloud.js # Nextcloud settings UI
|
│ │ ├── chartReview.js # Paste/dictate chart data → AI outpatient review
|
||||||
│ │ ├── encounters.js # Save/load encounter UI (all tabs)
|
│ │ ├── soap.js # Paste/dictate → AI SOAP note
|
||||||
│ │ ├── memories.js # User templates UI (Settings)
|
│ │ ├── milestones.js # Age-based milestone checklist → AI narrative
|
||||||
│ │ ├── learningHub.js # Learning Hub + CMS (entire module, ~1400 lines)
|
│ │ ├── wellVisit.js # Well Visit guide: vaccine schedule display, age calculator
|
||||||
│ │ ├── milestonesData.js # Static milestone data by age group
|
│ │ ├── shadess.js # SSHADESS psychosocial form + ROS/PE checkboxes → AI note
|
||||||
│ │ └── pediatricScheduleData.js # Vaccine schedule data
|
│ │ ├── sickVisit.js # Chief complaint + HPI → AI sick visit SOAP
|
||||||
|
│ │ ├── nextcloud.js # Nextcloud WebDAV connect/disconnect/export settings UI
|
||||||
|
│ │ ├── encounters.js # Save/load/delete encounter drafts (shared across all tabs)
|
||||||
|
│ │ ├── memories.js # User template CRUD (physical exam defaults, ROS, etc.)
|
||||||
|
│ │ ├── learningHub.js # Learning Hub (user feed, content viewer, quiz engine) +
|
||||||
|
│ │ │ # CMS (category CRUD, content editor with Tiptap, question
|
||||||
|
│ │ │ # builder, AI generation panel, Nextcloud file picker,
|
||||||
|
│ │ │ # slide preview modal). Single file, ~1400 lines.
|
||||||
|
│ │ ├── milestonesData.js # Static milestone data by age group (2mo → 6yr)
|
||||||
|
│ │ └── pediatricScheduleData.js # CDC vaccine schedule data + catch-up schedule
|
||||||
│ ├── components/ # Lazy-loaded tab HTML (injected by loadComponent)
|
│ ├── components/ # Lazy-loaded tab HTML (injected by loadComponent)
|
||||||
│ │ ├── encounter.html ├── dictation.html ├── hospital.html
|
│ │ ├── encounter.html ├── dictation.html ├── hospital.html
|
||||||
│ │ ├── chart.html ├── soap.html ├── wellvisit.html
|
│ │ ├── chart.html ├── soap.html ├── wellvisit.html
|
||||||
|
|
@ -793,41 +847,57 @@ pool.end();
|
||||||
| v5.6 | Re-add Qwen3 235B |
|
| v5.6 | Re-add Qwen3 235B |
|
||||||
| v5.7 | Remove Opus 4.6 (JSON issues) |
|
| v5.7 | Remove Opus 4.6 (JSON issues) |
|
||||||
| v5.8 | Fix JSON parse: sanitize literal newlines in strings; re-add Opus 4.6 |
|
| v5.8 | Fix JSON parse: sanitize literal newlines in strings; re-add Opus 4.6 |
|
||||||
|
| v5.9 | Re-add Opus 4.6 with sanitizer; updated DEVELOPER_GUIDE |
|
||||||
|
| v6.0 | Increase PDF/doc context to 50k chars; maxTokens ceiling to 8k |
|
||||||
|
|
||||||
## 16. Current Docker Image
|
## 16. Current Docker Image
|
||||||
|
|
||||||
**Latest stable:** `danielonyejesi/pediatric-ai-scribe-v3:v5.8`
|
**Latest stable:** `danielonyejesi/pediatric-ai-scribe-v3:v6.0`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker pull danielonyejesi/pediatric-ai-scribe-v3:v5.8
|
docker pull danielonyejesi/pediatric-ai-scribe-v3:v6.0
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 17. PDF Uploads — Why No Vector Embeddings
|
## 17. PDF & Document Uploads
|
||||||
|
|
||||||
The current approach for PDF/file uploads in the Learning Hub AI generator:
|
### How It Works
|
||||||
|
|
||||||
1. User uploads a file (or picks from Nextcloud)
|
The Learning Hub AI generator accepts documents via two paths — both produce the same result:
|
||||||
2. `pdf-parse` v1.1.1 extracts the full text from the PDF
|
|
||||||
3. The full text (up to 12,000 characters) is sent as context in the AI prompt
|
|
||||||
4. AI generates structured content (article body + quiz questions) based on that context
|
|
||||||
|
|
||||||
**You do NOT need embeddings or RAG (Retrieval Augmented Generation) for this use case.** Here's why:
|
1. **Direct upload** — user selects a file from their computer (up to 20 MB)
|
||||||
|
2. **Nextcloud WebDAV** — user browses their Nextcloud and picks a file
|
||||||
|
|
||||||
| Scenario | Use embeddings? | Why |
|
The flow:
|
||||||
|----------|----------------|-----|
|
1. `extractText()` in `learningAI.js` detects file type by MIME/extension
|
||||||
| Upload 1 file → generate 1 article | ❌ No | Full text fits in context; AI sees everything |
|
2. **PDF:** `pdf-parse` v1.1.1 extracts all text pages into a single string
|
||||||
| Search across 100+ stored documents | ✅ Yes | Too much text for one context window |
|
3. **PPTX/DOCX/TXT:** extracted via appropriate parser or read as UTF-8
|
||||||
| Long PDF (200+ pages) | ✅ Maybe | Truncation at 12k chars; embeddings enable chunked retrieval |
|
4. Text is truncated to **50,000 characters** (~25-30 pages) and sent as context in the AI prompt
|
||||||
| This app's current use case | ❌ No | Single file, single generation, full context is better |
|
5. AI generates structured content (title, HTML body, quiz questions) from the full context
|
||||||
|
|
||||||
Embeddings (e.g. OpenAI `text-embedding-ada-002`, pgvector in PostgreSQL) would add complexity, cost, and storage requirements with no user-facing benefit for single-document content generation. The current approach is intentionally simple and correct.
|
### Supported File Types
|
||||||
|
| Extension | Handler | Notes |
|
||||||
|
|-----------|---------|-------|
|
||||||
|
| `.pdf` | `pdf-parse` | Extracts text only — images, charts, tables are lost |
|
||||||
|
| `.pptx` | Text extraction from slides | Slide text only |
|
||||||
|
| `.docx` | Text extraction | Body text only |
|
||||||
|
| `.txt`, `.md`, `.csv` | Read as UTF-8 | Full content preserved |
|
||||||
|
|
||||||
If documents exceed 12,000 characters, increase the truncation limit in `learningAI.js`:
|
### Limits
|
||||||
```javascript
|
- **Upload size:** 20 MB (`multer` limit in `learningAI.js`)
|
||||||
docText.substring(0, 12000) // increase if needed (watch token costs)
|
- **Context sent to AI:** 50,000 characters (configurable in `buildGeneratePrompt()`)
|
||||||
```
|
- **AI response tokens:** 8,000 max (ceiling — model stops when done)
|
||||||
|
|
||||||
|
### Why No Vector Embeddings / RAG
|
||||||
|
|
||||||
|
Embeddings and RAG (Retrieval Augmented Generation) are unnecessary for this use case:
|
||||||
|
|
||||||
|
- **Single document → single generation** — the full text fits in the model's context window
|
||||||
|
- Most Bedrock models support 100K-200K token inputs — 50,000 chars is well within that
|
||||||
|
- Embeddings would add complexity (pgvector, chunking, retrieval pipeline) with no benefit
|
||||||
|
|
||||||
|
If you later need to **search across hundreds of stored documents** or handle 200+ page PDFs, then consider pgvector + chunked retrieval. For now, the direct approach is correct.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -902,5 +972,5 @@ httpOnly cookies prevent token *copying* but not token *use* — an XSS attacker
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
*Last updated: March 2026 — v5.8*
|
*Last updated: March 2026 — v6.0*
|
||||||
*Generated for developer handover.*
|
*Generated for developer handover.*
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue