857ed341f5
- Drop first/second-person voice; reference-style prose throughout - Remove stale information; align with current code (argon2id primary, hybrid cookie/Bearer auth, sliding 24h idle, AES-256-GCM PHI at rest, backup codes, node-pg-migrate, collation-drift guard, multi-arch Docker, auto-version pipeline) - Preserve all technical accuracy and code examples - Remove any remaining references to separate PedsHub Quiz app - Keep consistent tone across files (tables + code blocks, imperatives where needed) - api-reference.md and developer-guide.md route tables expanded to reflect current routes (billing, sessions)
83 lines
3.4 KiB
Markdown
83 lines
3.4 KiB
Markdown
# Speech: STT, TTS, audio backup
|
|
|
|
## Transcription (speech-to-text)
|
|
|
|
### Overview
|
|
|
|
`POST /api/transcribe` accepts `multipart/form-data` with a single audio
|
|
file (≤ 25 MB). Provider is `TRANSCRIBE_PROVIDER` env var, or auto-detected
|
|
(`google > aws > openai`) from available credentials. Each user may override
|
|
via `users.stt_model`; admin-wide default via `stt.model` in `app_settings`.
|
|
|
|
### Providers
|
|
|
|
| Provider | Transport | HIPAA (with BAA) |
|
|
|---|---|---|
|
|
| **Google Gemini** | Inline audio in `generateContent` call. Default model `gemini-2.0-flash`. | Yes |
|
|
| **Amazon Transcribe** | Streaming. `AWS_TRANSCRIBE_MEDICAL=true` + `AWS_TRANSCRIBE_SPECIALTY` switches to Transcribe Medical. Specialties: `PRIMARYCARE`, `CARDIOLOGY`, `NEUROLOGY`, `ONCOLOGY`, `RADIOLOGY`, `UROLOGY`. | Yes |
|
|
| **Local Whisper** | Spawns `whisper.cpp` or `faster-whisper` via `WHISPER_BINARY`. Fully offline. Model sizes `tiny`/`base`/`small`/`medium`/`large`. | N/A (nothing leaves host) |
|
|
| **OpenAI Whisper** | `whisper-1` via `/v1/audio/transcriptions`. Medical-context prompt prepended: `"Medical patient encounter. Pediatric."` | No |
|
|
| **LiteLLM** | Inline audio via LiteLLM's `chat.completions` endpoint (not the `/audio/transcriptions` path). Model from `LITELLM_STT_MODEL`. | Depends on LiteLLM backend |
|
|
|
|
## Browser Whisper (fully offline)
|
|
|
|
Runs entirely in the browser via WebAssembly. Zero network. Suitable when
|
|
no external transcription is acceptable.
|
|
|
|
- Runtime: `@xenova/transformers` (WASM).
|
|
- Models (bundled in the Docker image, no CDN fetch):
|
|
- `whisper-tiny.en` — 39 MB
|
|
- `whisper-base.en` — 74 MB
|
|
- `whisper-small.en` — 244 MB
|
|
- Executes in a dedicated Web Worker; UI thread is never blocked.
|
|
- Models cached in IndexedDB after first load.
|
|
- Per-user toggle. On browser transcription failure, the client falls back to
|
|
server-side transcription without user intervention.
|
|
|
|
## Live speech preview
|
|
|
|
Chrome / Edge `webkitSpeechRecognition` streams interim text to the UI during
|
|
recording. Used for real-time preview only — **not** for final transcription.
|
|
The actual transcript comes from the configured STT provider after recording
|
|
ends.
|
|
|
|
## Text-to-speech
|
|
|
|
### Overview
|
|
|
|
`POST /api/text-to-speech`. Returns `audio/mpeg`. `X-TTS-Provider` response
|
|
header identifies the provider used. 5000-character limit per request. Each
|
|
user may override via `users.tts_voice`; admin-wide default via `tts.voice`.
|
|
|
|
### Providers
|
|
|
|
| Provider | Notes |
|
|
|---|---|
|
|
| **Google Cloud TTS** | `@google-cloud/text-to-speech`. Voice families: Journey, Studio, Neural2. |
|
|
| **LiteLLM** | Configured via `LITELLM_TTS_MODEL` + `LITELLM_TTS_VOICE`. Backend-agnostic. |
|
|
| **ElevenLabs** | `eleven_turbo_v2_5`. **Not HIPAA-compliant**. |
|
|
|
|
## Audio backup
|
|
|
|
Raw audio is saved to Postgres **only when transcription fails**, providing a
|
|
retry window without persisting every recording.
|
|
|
|
### Storage
|
|
|
|
- Gzip-compressed, then AES-256-GCM encrypted (0x01 version byte prefix).
|
|
- `BYTEA` column in `audio_backups`.
|
|
- 24-hour `expires_at`, swept hourly.
|
|
- Legacy rows (gzip magic `0x1F` as first byte, no encryption envelope)
|
|
decompress as-is — detection is deterministic because `0x1F ≠ 0x01`.
|
|
|
|
### Retry UI
|
|
|
|
Settings → Audio Backups:
|
|
- List: module, size, created, expiry.
|
|
- **Retry** — resubmits to `POST /api/transcribe`.
|
|
- **Delete** — purge now.
|
|
|
|
### Browser fallback
|
|
|
|
If the server-side save fails (network, 500, etc.), the client stores the audio
|
|
in IndexedDB so it can retry later. Cleared after successful submission.
|