danvics/pediatric-ai-scribe-v3
1
0
Fork 0
Code Issues Pull requests Projects Releases 14 Packages Wiki Activity Actions
pediatric-ai-scribe-v3/docs/logic/ai-and-voice.md

3.7 KiB
Raw Blame History

AI, Speech, And Post-Note Helpers

This doc summarizes the current AI/STT/TTS pipeline without line-number citations. For exact behavior, read src/utils/ai.js, src/routes/transcribe.js, src/routes/tts.js, and the relevant frontend scripts.

Text Generation

All text-generation routes call callAI(messages, options) from src/utils/ai.js.

Supported providers:

  • OpenRouter.
  • AWS Bedrock.
  • Azure OpenAI.
  • Google Vertex AI.
  • LiteLLM or another OpenAI-compatible gateway.

AI_PROVIDER can explicitly choose the provider. If unset, the startup loader initializes configured clients and the final active provider follows the current load order described in ../ai-providers.md. Route handlers do not call provider SDKs directly.

Model Allowlist

callAI() rejects model IDs outside the active server-side allowlist unless a specific admin test path opts out. The allowlist is assembled from built-in provider models, models.disabled, and models.custom in app_settings.

The default model comes from the configured provider/model settings. Admins can set defaults and custom models from the Admin Panel.

Prompt Safety

Clinical routes should build prompts with:

  • canonical templates from src/utils/prompts.js
  • optional DB prompt overrides through app_settings keys prompt.*
  • INJECTION_GUARD
  • wrapUserText(label, text) around user-derived text

User-derived text includes transcripts, dictated notes, pasted chart data, refine instructions, template preferences, and patient education source notes.

User Templates

getUserMemoryContext() fetches /api/memories/context and passes the returned template/preference context as physicianMemories. Server routes wrap that block as low-priority style/template context. custom memories and legacy correction_* rows are not prompt context.

Speech-To-Text

POST /api/transcribe accepts one audio file up to 25 MB. Provider selection:

  • explicit TRANSCRIBE_PROVIDER=litellm, or
  • auto mode when LITELLM_API_BASE is configured.

Direct Google, AWS, local Whisper, and OpenAI Whisper branches are not part of the runtime. Browser Whisper/browser-local model downloads are also absent.

Browser Web Speech

Browser-native Web Speech is an explicit opt-in preview. It may rely on browser vendor cloud services and must not be treated as the final clinical transcript. Final transcription should come from the configured server-side STT provider.

Audio Backup

Failed transcription attempts can create encrypted 24-hour audio backups through src/routes/audioBackups.js. The user can retry or delete backups from Settings. Browser fallback storage is only for cases where the server cannot store the failed recording.

Text-To-Speech

POST /api/text-to-speech returns audio from LiteLLM and marks the LiteLLM model in X-TTS-Provider. Voices are LiteLLM-compatible strings configured by LITELLM_TTS_VOICES.

Post-Note Helpers

Generated note outputs can expose helper panels:

  • refineDocument for editing/refining/shortening generated text.
  • suggestBillingCodes for clinician-facing ICD/CPT suggestions.
  • suggestDontMiss for clinician-facing safety review.
  • attachPatientEducation for parent-facing handout drafts.

These helpers are authenticated API-backed actions. They should treat the edited note as the source of truth and keep the clinician in the review loop.

Change Checklist

When changing this area:

  1. Keep provider-specific code inside utility/provider modules.
  2. Wrap all user-derived text with wrapUserText before AI calls.
  3. Do not add browser-local Whisper back without a new design review.
  4. Do not add clinical answer response caching.
  5. Run touched-file node --check commands and npm test.