danvics/pediatric-ai-scribe-v3
1
0
Fork 0
Code Issues Pull requests Projects Releases 16 Packages Wiki Activity Actions
pediatric-ai-scribe-v3/docs/logic/ai-and-voice.md
Daniel 67e416c6d9
Some checks failed
Forgejo Android APK / Root app tests (push) Successful in 47s
Details
Forgejo Docker Build / Root app tests (push) Successful in 48s
Details
Forgejo Android APK / Build signed APK (push) Successful in 2m38s
Details
Forgejo Docker Build / Build Docker image (push) Successful in 12s
Details
Forgejo Docker Build / Deploy to the host (push) Failing after 2s
Details
docs: merge the duplicate pairs and correct them against the running app 
Three pairs of docs described the same thing twice, and the copies had drifted
apart. Merged each into one file, keeping the unique content from both:

- ARCHITECTURE.md -> architecture.md (its operational map: ownership, request
  flow, runtime boundaries, source of truth, deployment shape)
- DEVELOPMENT.md -> developer-guide.md (change workflow, Clinical Assistant
  high-risk areas, frontend rendering rules, deployment checks)
- transcription-options.md -> speech.md (the clinic setup table, and the list
  of browser-Whisper paths that must stay removed)

Then audited what remained against the code and the live database rather than
against the previous docs. Corrected:

- Google Vertex was still documented as a provider across nine files. The SDK
  is gone; AI_PROVIDER=vertex now logs an advisory and falls back to
  OpenRouter, and Gemini is reached through LiteLLM. Fixed the provider
  selection order to match src/utils/ai.js, which starts from LITELLM_API_BASE.
- promptSafe was documented on 8 routes; it is on 13.
- Node 20 -> 24, "24 vanilla JS modules" -> no fixed count, and
  transcribe.js/tts.js -> sttProvider.js/ttsProvider.js, which is what exists.
- STT/TTS are LiteLLM-only; README listed direct Google, AWS Transcribe and
  ElevenLabs paths that are not in the runtime.
- Learning Hub PPTX export was documented as pptxgenjs, which is not a
  dependency. It is pandoc against a reference deck.
- POST /api/admin/milestones/seed does not exist; it is /bulk-import.
- NEXTCLOUD_URL and NTFY_TOPIC are not read anywhere. Nextcloud is per-user in
  the users table, and the ntfy topic is derived as pedscribe-{userId}.
- A prose paragraph sat inside the Clinical Assistant settings table, so half
  the rows rendered as text.

Filled the gaps the audit exposed:

- database.md was missing 12 of 29 tables, including user_resources,
  personal_notes, login_codes, registration_invites and generated_image_jobs.
- developer-guide.md was missing 11 routers and 10 frontend modules.
- api-reference.md detailed 121 of 244 endpoints and said so, but whole
  features were absent. Added an endpoint index covering Clinical Assistant,
  My Resources, Notes, Diagrams, ED Encounters, invites and sign-in codes.
- configuration.md was missing METRICS_TOKEN, REDIS_URL, API_RATE_LIMIT_MAX,
  the LITELLM_* model variables, the DB_* ones maintenance.js reads, and the
  per-purpose S3 resolution scheme.
- clinical-assistant.md documented 2 of its 17 environment variables.
- features-explained.md had no entry for My Resources or Clinical Assistant.

Renamed the three remaining SHOUTING filenames to kebab-case, which is what the
docs viewer's prettyName() was working around, and rewrote README's index,
which listed architecture.md twice and omitted nine files.

Noted but not changed: the Turnstile site key is hardcoded in index.html rather
than read from TURNSTILE_SITE_KEY, and /api/health/detailed can report
tts: 'elevenlabs' though no ElevenLabs path exists.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv6sqaY6Vq3ChZHMem3cnU
2026-09-12 04:57:35 +02:00

3.8 KiB
Raw Blame History

AI, Speech, And Post-Note Helpers

This doc summarizes the current AI/STT/TTS pipeline without line-number citations. For exact behavior, read src/utils/ai.js, src/routes/transcribe.js, src/routes/tts.js, and the relevant frontend scripts.

Text Generation

All text-generation routes call callAI(messages, options) from src/utils/ai.js.

Supported providers:

  • LiteLLM or another OpenAI-compatible gateway.
  • OpenRouter.
  • AWS Bedrock.
  • Azure OpenAI.

Google Vertex is no longer a provider of its own; the Google SDK was removed and Gemini is reached as a LiteLLM-configured model.

AI_PROVIDER can explicitly choose the provider. If unset, the startup loader initializes configured clients and the final active provider follows the current load order described in ../ai-providers.md. Route handlers do not call provider SDKs directly.

Model Allowlist

callAI() rejects model IDs outside the active server-side allowlist unless a specific admin test path opts out. The allowlist is assembled from built-in provider models, models.disabled, and models.custom in app_settings.

The default model comes from the configured provider/model settings. Admins can set defaults and custom models from the Admin Panel.

Prompt Safety

Clinical routes should build prompts with:

  • canonical templates from src/utils/prompts.js
  • optional DB prompt overrides through app_settings keys prompt.*
  • INJECTION_GUARD
  • wrapUserText(label, text) around user-derived text

User-derived text includes transcripts, dictated notes, pasted chart data, refine instructions, template preferences, and patient education source notes.

User Templates

getUserMemoryContext() fetches /api/memories/context and passes the returned template/preference context as physicianMemories. Server routes wrap that block as low-priority style/template context. custom memories and legacy correction_* rows are not prompt context.

Speech-To-Text

POST /api/transcribe accepts one audio file up to 25 MB. Provider selection:

  • explicit TRANSCRIBE_PROVIDER=litellm, or
  • auto mode when LITELLM_API_BASE is configured.

Direct Google, AWS, local Whisper, and OpenAI Whisper branches are not part of the runtime. Browser Whisper/browser-local model downloads are also absent.

Browser Web Speech

Browser-native Web Speech is an explicit opt-in preview. It may rely on browser vendor cloud services and must not be treated as the final clinical transcript. Final transcription should come from the configured server-side STT provider.

Audio Backup

Failed transcription attempts can create encrypted 24-hour audio backups through src/routes/audioBackups.js. The user can retry or delete backups from Settings. Browser fallback storage is only for cases where the server cannot store the failed recording.

Text-To-Speech

POST /api/text-to-speech returns audio from LiteLLM and marks the LiteLLM model in X-TTS-Provider. Voices are LiteLLM-compatible strings configured by LITELLM_TTS_VOICES.

Post-Note Helpers

Generated note outputs can expose helper panels:

  • refineDocument for editing/refining/shortening generated text.
  • suggestBillingCodes for clinician-facing ICD/CPT suggestions.
  • suggestDontMiss for clinician-facing safety review.
  • attachPatientEducation for parent-facing handout drafts.

These helpers are authenticated API-backed actions. They should treat the edited note as the source of truth and keep the clinician in the review loop.

Change Checklist

When changing this area:

  1. Keep provider-specific code inside utility/provider modules.
  2. Wrap all user-derived text with wrapUserText before AI calls.
  3. Do not add browser-local Whisper back without a new design review.
  4. Do not add clinical answer response caching.
  5. Run touched-file node --check commands and npm test.