b97c5f146b
- Add Cloudflare Turnstile to login, register, and password reset forms - Switch AI provider to LiteLLM, transcription to OpenAI Whisper - Change domain to scribe.pedshub.com - Fix PPTX export: add tables, bold/italic, numbered lists, code blocks, blockquotes - Fix announcement banner close button (CSP was blocking inline onclick) - Fix auth middleware: empty Bearer token now falls through to cookie auth - Fix audio backups: only save on transcription failure, stop auto-deleting on success - Soften AI correction injection to prevent model hallucination from correction history - Fix LiteLLM TTS model name handling (no incorrect openai/ prefix) - Expand AI instructions textarea in Learning Hub CMS - Update README for v6 with all features and providers - Add comprehensive docs/: architecture, API reference, database schema, authentication, AI providers, speech, learning hub, configuration, deployment
129 lines
4.9 KiB
Markdown
129 lines
4.9 KiB
Markdown
# Speech-to-Text and Text-to-Speech Systems
|
|
|
|
This document covers all audio processing capabilities in the Pediatric AI Scribe, including server-side transcription, client-side transcription, live speech preview, text-to-speech, and audio backup.
|
|
|
|
---
|
|
|
|
## Speech-to-Text
|
|
|
|
### Overview
|
|
|
|
The transcription system supports multiple providers with automatic fallback. The active provider is selected via the `TRANSCRIBE_PROVIDER` environment variable, or auto-detected in priority order: Google > AWS > OpenAI.
|
|
|
|
- **Endpoint:** `POST /api/transcribe`
|
|
- **Max upload size:** 25 MB (multipart form data via multer)
|
|
- **User override:** Each user can select a preferred STT model in their settings, stored in the `stt_model` column of the `users` table.
|
|
- **Admin default:** Administrators can set the system-wide default STT model via the admin settings panel.
|
|
|
|
### Providers
|
|
|
|
#### 1. Google Gemini
|
|
|
|
- Sends inline audio data within chat completion requests (not a separate transcription API).
|
|
- Model is configurable; default is `gemini-2.0-flash`.
|
|
- HIPAA eligible.
|
|
|
|
#### 2. Amazon Transcribe
|
|
|
|
- Uses streaming audio for real-time transcription.
|
|
- Supports **Medical mode** with specialty selection:
|
|
- `PRIMARYCARE`, `CARDIOLOGY`, `NEUROLOGY`, `ONCOLOGY`, `RADIOLOGY`, `UROLOGY`
|
|
- Configured via `AWS_TRANSCRIBE_MEDICAL` and `AWS_TRANSCRIBE_SPECIALTY` environment variables.
|
|
- HIPAA eligible.
|
|
|
|
#### 3. Local Whisper
|
|
|
|
- Runs `whisper.cpp` or `faster-whisper` as a local binary process.
|
|
- Supported model sizes: `tiny`, `base`, `small`, `medium`, `large`.
|
|
- Configurable threads and language via environment variables (`WHISPER_THREADS`, `WHISPER_LANGUAGE`).
|
|
- No external API calls -- fully offline.
|
|
|
|
#### 4. OpenAI Whisper
|
|
|
|
- Uses the `whisper-1` model via the OpenAI API.
|
|
- Sends a medical context prompt: `"Medical patient encounter. Pediatric."`
|
|
|
|
#### 5. LiteLLM
|
|
|
|
- Routes transcription through LiteLLM's `chat/completions` endpoint using Gemini-style inline audio.
|
|
- Does **not** use the `/audio/transcriptions` endpoint.
|
|
- Model name configured via `LITELLM_STT_MODEL`.
|
|
|
|
---
|
|
|
|
## Browser Whisper (Client-Side Transcription)
|
|
|
|
Client-side transcription runs entirely in the browser with zero network traffic, providing maximum privacy.
|
|
|
|
- **Runtime:** WebAssembly via `@xenova/transformers`
|
|
- **Available models:**
|
|
- `whisper-tiny.en` -- 39 MB
|
|
- `whisper-base.en` -- 74 MB
|
|
- `whisper-small.en` -- 244 MB
|
|
- **Self-hosted:** Model files are bundled in the Docker image. There is no CDN dependency.
|
|
- **Web Worker:** Transcription runs in a dedicated Web Worker to avoid blocking the UI thread.
|
|
- **Caching:** Downloaded models are cached in IndexedDB so subsequent loads are instant.
|
|
- **User toggle:** Enabled or disabled per user in settings. If browser transcription fails, it falls back to server-side transcription automatically.
|
|
|
|
---
|
|
|
|
## Web Speech Recognition (Live Preview)
|
|
|
|
- Uses the Chrome/Edge **Web Speech API** (`webkitSpeechRecognition`) for live preview during recording.
|
|
- Streams interim (partial) results to the UI while the user is still speaking.
|
|
- This is **not** used for final transcription. It serves only as a real-time visual preview. The actual transcription is performed by the configured STT provider (server-side or browser Whisper) after recording completes.
|
|
|
|
---
|
|
|
|
## Text-to-Speech
|
|
|
|
### Overview
|
|
|
|
- **Endpoint:** `POST /api/text-to-speech`
|
|
- **Character limit:** 5000 characters per request.
|
|
- **Response format:** `audio/mpeg`
|
|
- **Provider header:** The response includes an `X-TTS-Provider` header indicating which provider was used.
|
|
- **User override:** Each user can select a preferred voice in their settings, stored in the `tts_voice` column of the `users` table.
|
|
|
|
### Providers
|
|
|
|
#### 1. Google Cloud TTS
|
|
|
|
- Uses the `@google-cloud/text-to-speech` client library.
|
|
- Supported voice families:
|
|
- **Journey** voices: `Journey-F`, `Journey-D`
|
|
- **Studio** voices
|
|
- **Neural2** voices
|
|
|
|
#### 2. LiteLLM
|
|
|
|
- Routes TTS requests to downstream providers (OpenAI, ElevenLabs, Gemini) via the configured LiteLLM model name.
|
|
- Configured via `LITELLM_TTS_MODEL` and `LITELLM_TTS_VOICE`.
|
|
|
|
#### 3. ElevenLabs
|
|
|
|
- Uses the `eleven_turbo_v2_5` model.
|
|
- **Not HIPAA compliant.** Do not use in production environments handling protected health information.
|
|
|
|
---
|
|
|
|
## Audio Backup System
|
|
|
|
The audio backup system preserves original audio recordings when transcription fails, allowing later retry.
|
|
|
|
### Storage
|
|
|
|
- Audio is saved to **PostgreSQL** only when transcription fails (not on every recording).
|
|
- Stored as gzip-compressed binary data in a `BYTEA` column.
|
|
- Backups auto-expire after **24 hours**.
|
|
|
|
### User Interface
|
|
|
|
- The Settings page displays a list of saved audio backups.
|
|
- Each backup has two actions:
|
|
- **Retry** -- re-submits the audio to the transcription provider.
|
|
- **Delete** -- permanently removes the backup.
|
|
|
|
### Browser Fallback
|
|
|
|
- If the server-side backup save fails (e.g., network error), the audio is saved to **IndexedDB** in the browser as a secondary fallback.
|