101 lines
3.7 KiB
Markdown
101 lines
3.7 KiB
Markdown
# AI, Speech, And Post-Note Helpers
|
|
|
|
This doc summarizes the current AI/STT/TTS pipeline without line-number
|
|
citations. For exact behavior, read `src/utils/ai.js`, `src/routes/transcribe.js`,
|
|
`src/routes/tts.js`, and the relevant frontend scripts.
|
|
|
|
## Text Generation
|
|
|
|
All text-generation routes call `callAI(messages, options)` from
|
|
`src/utils/ai.js`.
|
|
|
|
Supported providers:
|
|
|
|
- OpenRouter.
|
|
- AWS Bedrock.
|
|
- Azure OpenAI.
|
|
- Google Vertex AI.
|
|
- LiteLLM or another OpenAI-compatible gateway.
|
|
|
|
`AI_PROVIDER` can explicitly choose the provider. If unset, the startup loader
|
|
initializes configured clients and the final active provider follows the current
|
|
load order described in [`../ai-providers.md`](../ai-providers.md). Route
|
|
handlers do not call provider SDKs directly.
|
|
|
|
## Model Allowlist
|
|
|
|
`callAI()` rejects model IDs outside the active server-side allowlist unless a
|
|
specific admin test path opts out. The allowlist is assembled from built-in
|
|
provider models, `models.disabled`, and `models.custom` in `app_settings`.
|
|
|
|
The default model comes from the configured provider/model settings. Admins can
|
|
set defaults and custom models from the Admin Panel.
|
|
|
|
## Prompt Safety
|
|
|
|
Clinical routes should build prompts with:
|
|
|
|
- canonical templates from `src/utils/prompts.js`
|
|
- optional DB prompt overrides through `app_settings` keys `prompt.*`
|
|
- `INJECTION_GUARD`
|
|
- `wrapUserText(label, text)` around user-derived text
|
|
|
|
User-derived text includes transcripts, dictated notes, pasted chart data,
|
|
refine instructions, template preferences, and patient education source notes.
|
|
|
|
## User Templates
|
|
|
|
`getUserMemoryContext()` fetches `/api/memories/context` and passes the returned
|
|
template/preference context as `physicianMemories`. Server routes wrap that block
|
|
as low-priority style/template context. `custom` memories and legacy
|
|
`correction_*` rows are not prompt context.
|
|
|
|
## Speech-To-Text
|
|
|
|
`POST /api/transcribe` accepts one audio file up to 25 MB. Provider selection:
|
|
|
|
- explicit `TRANSCRIBE_PROVIDER=litellm`, or
|
|
- auto mode when `LITELLM_API_BASE` is configured.
|
|
|
|
Direct Google, AWS, local Whisper, and OpenAI Whisper branches are not part of the runtime. Browser Whisper/browser-local model downloads are also absent.
|
|
|
|
## Browser Web Speech
|
|
|
|
Browser-native Web Speech is an explicit opt-in preview. It may rely on browser
|
|
vendor cloud services and must not be treated as the final clinical transcript.
|
|
Final transcription should come from the configured server-side STT provider.
|
|
|
|
## Audio Backup
|
|
|
|
Failed transcription attempts can create encrypted 24-hour audio backups through
|
|
`src/routes/audioBackups.js`. The user can retry or delete backups from
|
|
Settings. Browser fallback storage is only for cases where the server cannot
|
|
store the failed recording.
|
|
|
|
## Text-To-Speech
|
|
|
|
`POST /api/text-to-speech` returns audio from LiteLLM and marks the LiteLLM
|
|
model in `X-TTS-Provider`. Voices are LiteLLM-compatible strings configured by
|
|
`LITELLM_TTS_VOICES`.
|
|
|
|
## Post-Note Helpers
|
|
|
|
Generated note outputs can expose helper panels:
|
|
|
|
- `refineDocument` for editing/refining/shortening generated text.
|
|
- `suggestBillingCodes` for clinician-facing ICD/CPT suggestions.
|
|
- `suggestDontMiss` for clinician-facing safety review.
|
|
- `attachPatientEducation` for parent-facing handout drafts.
|
|
|
|
These helpers are authenticated API-backed actions. They should treat the edited
|
|
note as the source of truth and keep the clinician in the review loop.
|
|
|
|
## Change Checklist
|
|
|
|
When changing this area:
|
|
|
|
1. Keep provider-specific code inside utility/provider modules.
|
|
2. Wrap all user-derived text with `wrapUserText` before AI calls.
|
|
3. Do not add browser-local Whisper back without a new design review.
|
|
4. Do not add clinical answer response caching.
|
|
5. Run touched-file `node --check` commands and `npm test`.
|