From 689d5e902888aa42ca11d3224c410dc2f13a800e Mon Sep 17 00:00:00 2001 From: Daniel Onyejesi Date: Tue, 24 Mar 2026 19:30:01 -0400 Subject: [PATCH] Expand DEVELOPER_GUIDE with detailed architecture, DB layer, file descriptions - Added request flow walkthrough, AI provider selection logic - Documented database layer (raw pg, no ORM) with helper methods - Expanded directory structure with detailed per-file descriptions - Expanded frontend JS descriptions (what each file does) - Rewrote PDF section with current 50k char limit, supported types - Updated version history through v6.0 --- DEVELOPER_GUIDE.md | 182 +++++++++++++++++++++++++++++++-------------- 1 file changed, 126 insertions(+), 56 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index aa3d9d0..328b80c 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -1,5 +1,5 @@ # Pediatric AI Scribe — Developer Guide -**Version:** 3.19 | **Stack:** Node.js / Express / PostgreSQL / Vanilla JS +**Version:** 6.0 | **Stack:** Node.js / Express / PostgreSQL / Vanilla JS --- @@ -59,10 +59,39 @@ PostgreSQL (pg driver, no ORM) |— user_memories, learning_*, access_log, api_log ``` -**AI providers** (configured via environment variables, in priority order): -1. AWS Bedrock (if `AWS_BEDROCK_REGION` is set) -2. Azure OpenAI (if `AZURE_OPENAI_ENDPOINT` is set) -3. OpenRouter (default, requires `OPENROUTER_API_KEY`) +### How Requests Flow + +1. **Browser** sends HTTP request with `Authorization: Bearer ` header +2. **Express middleware chain:** Helmet (security headers) → CORS → rate limiter → body parser → logging middleware → route handler +3. **Auth middleware** (`src/middleware/auth.js`) decodes JWT, queries `users` table, attaches `req.user` with `{ id, email, name, role }` +4. **Route handler** processes the request — for AI routes, calls `callAI()` which routes to the configured provider +5. **Database** is accessed via the `pg` driver directly (no ORM). All queries use parameterized placeholders (`$1`, `$2`) to prevent SQL injection +6. **Response** is JSON for API calls, or static files served from `/public` + +### AI Providers + +Configured via environment variables. The provider is selected at startup in `src/utils/ai.js` using this priority: + +1. **AWS Bedrock** — if `AWS_BEDROCK_REGION` is set. HIPAA eligible with BAA. Uses `@aws-sdk/client-bedrock-runtime`. Anthropic models use the native Messages API (`InvokeModel`); all others use the Converse API. +2. **Azure OpenAI** — if `AZURE_OPENAI_ENDPOINT` is set. HIPAA eligible. Uses the OpenAI SDK pointed at your Azure endpoint. +3. **OpenRouter** — default fallback if `OPENROUTER_API_KEY` is set. Routes to 20+ models from various providers. Not HIPAA compliant. + +The provider cannot be changed at runtime — it's determined once at startup. To switch providers, update `.env` and restart the container. + +### Database Layer + +The app uses **raw SQL via the `pg` driver** — no ORM (Sequelize, Prisma, etc.). This is intentional: + +- **Simplicity:** Every query is visible and explicit. No magic, no migrations framework, no model definitions to sync. +- **Performance:** No ORM overhead or N+1 query problems. +- **Schema management:** `src/db/database.js` runs `CREATE TABLE IF NOT EXISTS` on startup, plus `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` for migrations. This means the schema is always up-to-date when the app starts. +- **Future ORM migration:** If needed, the queries are standard PostgreSQL and can be wrapped by any ORM. The main work would be defining models and replacing direct `db.get()`/`db.run()` calls. + +The `database.js` file exports a helper object (`db`) with convenience methods: +- `db.get(sql, params)` — returns first row or `null` +- `db.all(sql, params)` — returns all rows as array +- `db.run(sql, params)` — executes INSERT/UPDATE/DELETE, returns `{ rowCount }` +- `db.getSetting(key)` / `db.setSetting(key, value)` — shorthand for `app_settings` table --- @@ -70,19 +99,27 @@ PostgreSQL (pg driver, no ORM) ``` / -├── server.js # Express app entry point, route registration, CSP -├── package.json -├── Dockerfile -├── docker-compose.yml # Production compose -├── docker-compose.local.yml # Local development (port 3552) +├── server.js # Express app entry point (route registration, Helmet CSP, +│ # rate limiters, static file serving, error handlers) +├── package.json # Dependencies (~25 production deps, no devDeps) +├── Dockerfile # Multi-stage Node.js 20 Alpine build +├── docker-compose.yml # Production compose (uses Docker Hub image) +├── docker-compose.local.yml # Local development (builds from source, port 3552) +├── admin-cli.js # CLI tool for admin tasks (create user, reset password) ├── DEVELOPER_GUIDE.md # This file │ ├── src/ │ ├── db/ -│ │ └── database.js # DB connection pool, schema init, migrations +│ │ └── database.js # DB connection pool (pg.Pool), schema init +│ │ # (CREATE TABLE IF NOT EXISTS for all tables), +│ │ # column migrations (ALTER TABLE ADD COLUMN IF NOT EXISTS), +│ │ # helper methods: db.get(), db.all(), db.run(), +│ │ # db.getSetting(), db.setSetting() │ ├── middleware/ -│ │ ├── auth.js # authMiddleware, adminMiddleware, moderatorMiddleware -│ │ └── logging.js # Access log middleware +│ │ ├── auth.js # authMiddleware (JWT decode → req.user), +│ │ # adminMiddleware (role === 'admin'), +│ │ # moderatorMiddleware (role === 'admin' || 'moderator') +│ │ └── logging.js # Logs every request to api_log table (method, path, user, IP, duration) │ ├── routes/ │ │ ├── auth.js # Login, register, 2FA, password reset, /me │ │ ├── admin.js # User management (admin only) @@ -105,11 +142,19 @@ PostgreSQL (pg driver, no ORM) │ │ ├── learningAI.js # AI generation for Learning Hub content │ │ └── logs.js # Usage/audit/API/access logs + client error │ └── utils/ -│ ├── ai.js # callAI() — multi-provider AI client -│ ├── models.js # Model ID lists, defaults, Bedrock ID mapping -│ ├── prompts.js # Prompt templates, DB override loader +│ ├── ai.js # callAI(messages, options) — routes to OpenRouter/Bedrock/Azure. +│ │ # Handles Anthropic InvokeModel (Messages API) vs Converse API, +│ │ # thinking block extraction, fallback model retry, duration tracking. +│ ├── models.js # OPENROUTER_MODELS[], BEDROCK_MODELS[], AZURE_MODELS[] +│ │ # Each model: { id, name, cost, tag, category, bedrockId, maxOut, regions } +│ │ # getBedrockModelId() maps app IDs to Bedrock/inference profile IDs. +│ │ # getAvailableModels() filters by region. getAvailableModelsWithOverrides() +│ │ # applies admin-disabled/custom models from DB. +│ ├── prompts.js # Default prompt templates for every AI route. Loaded on startup, +│ │ # then overridden by DB values (app_settings: 'prompt.*' keys). +│ │ # PROMPTS.get('key') returns the DB override or default. │ ├── config.js # App configuration helpers -│ └── logger.js # Winston logger (file + console) +│ └── logger.js # Winston logger (file + console, JSON format) │ ├── public/ │ ├── index.html # Single HTML shell, loads all components @@ -117,24 +162,33 @@ PostgreSQL (pg driver, no ORM) │ ├── css/ │ │ └── styles.css # All CSS (single file, ~750 lines) │ ├── js/ -│ │ ├── app.js # Core: tab switching, loadComponent, global helpers -│ │ ├── auth.js # Login/register UI, JWT localStorage, getAuthHeaders() -│ │ ├── admin.js # Admin panel UI -│ │ ├── liveEncounter.js # Live recording tab -│ │ ├── voiceDictation.js # Dictation tab -│ │ ├── hospitalCourse.js # Hospital course tab -│ │ ├── chartReview.js # Chart review tab -│ │ ├── soap.js # SOAP tab -│ │ ├── milestones.js # Milestones tab (inside Well Visit) -│ │ ├── wellVisit.js # Well Visit guide + vaccine schedule -│ │ ├── shadess.js # SSHADESS form + Well Visit note generation -│ │ ├── sickVisit.js # Sick Visit tab -│ │ ├── nextcloud.js # Nextcloud settings UI -│ │ ├── encounters.js # Save/load encounter UI (all tabs) -│ │ ├── memories.js # User templates UI (Settings) -│ │ ├── learningHub.js # Learning Hub + CMS (entire module, ~1400 lines) -│ │ ├── milestonesData.js # Static milestone data by age group -│ │ └── pediatricScheduleData.js # Vaccine schedule data +│ │ ├── app.js # Core: tab switching via data-tab buttons, loadComponent() +│ │ │ # fetches HTML from /components/, global helpers (showToast, +│ │ │ # showLoading, getAuthHeaders, getSelectedModel, etc.) +│ │ ├── auth.js # Login/register/forgot-password forms, JWT storage in +│ │ │ # localStorage ('ped_scribe_token'), enterApp()/clearSession(), +│ │ │ # resend verification link handler, 2FA code input +│ │ ├── admin.js # Admin panel: user management, site settings, SMTP config, +│ │ │ # model enable/disable, prompt editor, announcement banner +│ │ ├── liveEncounter.js # MediaRecorder → Whisper transcription → AI HPI generation. +│ │ │ # Handles start/stop recording, timer, save/load encounters +│ │ ├── voiceDictation.js # Web Speech API (real-time) or Whisper (recorded) dictation +│ │ ├── hospitalCourse.js # Paste/dictate hospital course → AI summary +│ │ ├── chartReview.js # Paste/dictate chart data → AI outpatient review +│ │ ├── soap.js # Paste/dictate → AI SOAP note +│ │ ├── milestones.js # Age-based milestone checklist → AI narrative +│ │ ├── wellVisit.js # Well Visit guide: vaccine schedule display, age calculator +│ │ ├── shadess.js # SSHADESS psychosocial form + ROS/PE checkboxes → AI note +│ │ ├── sickVisit.js # Chief complaint + HPI → AI sick visit SOAP +│ │ ├── nextcloud.js # Nextcloud WebDAV connect/disconnect/export settings UI +│ │ ├── encounters.js # Save/load/delete encounter drafts (shared across all tabs) +│ │ ├── memories.js # User template CRUD (physical exam defaults, ROS, etc.) +│ │ ├── learningHub.js # Learning Hub (user feed, content viewer, quiz engine) + +│ │ │ # CMS (category CRUD, content editor with Tiptap, question +│ │ │ # builder, AI generation panel, Nextcloud file picker, +│ │ │ # slide preview modal). Single file, ~1400 lines. +│ │ ├── milestonesData.js # Static milestone data by age group (2mo → 6yr) +│ │ └── pediatricScheduleData.js # CDC vaccine schedule data + catch-up schedule │ ├── components/ # Lazy-loaded tab HTML (injected by loadComponent) │ │ ├── encounter.html ├── dictation.html ├── hospital.html │ │ ├── chart.html ├── soap.html ├── wellvisit.html @@ -793,41 +847,57 @@ pool.end(); | v5.6 | Re-add Qwen3 235B | | v5.7 | Remove Opus 4.6 (JSON issues) | | v5.8 | Fix JSON parse: sanitize literal newlines in strings; re-add Opus 4.6 | +| v5.9 | Re-add Opus 4.6 with sanitizer; updated DEVELOPER_GUIDE | +| v6.0 | Increase PDF/doc context to 50k chars; maxTokens ceiling to 8k | ## 16. Current Docker Image -**Latest stable:** `danielonyejesi/pediatric-ai-scribe-v3:v5.8` +**Latest stable:** `danielonyejesi/pediatric-ai-scribe-v3:v6.0` ```bash -docker pull danielonyejesi/pediatric-ai-scribe-v3:v5.8 +docker pull danielonyejesi/pediatric-ai-scribe-v3:v6.0 ``` --- -## 17. PDF Uploads — Why No Vector Embeddings +## 17. PDF & Document Uploads -The current approach for PDF/file uploads in the Learning Hub AI generator: +### How It Works -1. User uploads a file (or picks from Nextcloud) -2. `pdf-parse` v1.1.1 extracts the full text from the PDF -3. The full text (up to 12,000 characters) is sent as context in the AI prompt -4. AI generates structured content (article body + quiz questions) based on that context +The Learning Hub AI generator accepts documents via two paths — both produce the same result: -**You do NOT need embeddings or RAG (Retrieval Augmented Generation) for this use case.** Here's why: +1. **Direct upload** — user selects a file from their computer (up to 20 MB) +2. **Nextcloud WebDAV** — user browses their Nextcloud and picks a file -| Scenario | Use embeddings? | Why | -|----------|----------------|-----| -| Upload 1 file → generate 1 article | ❌ No | Full text fits in context; AI sees everything | -| Search across 100+ stored documents | ✅ Yes | Too much text for one context window | -| Long PDF (200+ pages) | ✅ Maybe | Truncation at 12k chars; embeddings enable chunked retrieval | -| This app's current use case | ❌ No | Single file, single generation, full context is better | +The flow: +1. `extractText()` in `learningAI.js` detects file type by MIME/extension +2. **PDF:** `pdf-parse` v1.1.1 extracts all text pages into a single string +3. **PPTX/DOCX/TXT:** extracted via appropriate parser or read as UTF-8 +4. Text is truncated to **50,000 characters** (~25-30 pages) and sent as context in the AI prompt +5. AI generates structured content (title, HTML body, quiz questions) from the full context -Embeddings (e.g. OpenAI `text-embedding-ada-002`, pgvector in PostgreSQL) would add complexity, cost, and storage requirements with no user-facing benefit for single-document content generation. The current approach is intentionally simple and correct. +### Supported File Types +| Extension | Handler | Notes | +|-----------|---------|-------| +| `.pdf` | `pdf-parse` | Extracts text only — images, charts, tables are lost | +| `.pptx` | Text extraction from slides | Slide text only | +| `.docx` | Text extraction | Body text only | +| `.txt`, `.md`, `.csv` | Read as UTF-8 | Full content preserved | -If documents exceed 12,000 characters, increase the truncation limit in `learningAI.js`: -```javascript -docText.substring(0, 12000) // increase if needed (watch token costs) -``` +### Limits +- **Upload size:** 20 MB (`multer` limit in `learningAI.js`) +- **Context sent to AI:** 50,000 characters (configurable in `buildGeneratePrompt()`) +- **AI response tokens:** 8,000 max (ceiling — model stops when done) + +### Why No Vector Embeddings / RAG + +Embeddings and RAG (Retrieval Augmented Generation) are unnecessary for this use case: + +- **Single document → single generation** — the full text fits in the model's context window +- Most Bedrock models support 100K-200K token inputs — 50,000 chars is well within that +- Embeddings would add complexity (pgvector, chunking, retrieval pipeline) with no benefit + +If you later need to **search across hundreds of stored documents** or handle 200+ page PDFs, then consider pgvector + chunked retrieval. For now, the direct approach is correct. --- @@ -902,5 +972,5 @@ httpOnly cookies prevent token *copying* but not token *use* — an XSS attacker --- -*Last updated: March 2026 — v5.8* +*Last updated: March 2026 — v6.0* *Generated for developer handover.*