# Speech-to-Text and Text-to-Speech Systems

This document covers all audio processing capabilities in the Pediatric AI Scribe, including server-side transcription, client-side transcription, live speech preview, text-to-speech, and audio backup.

---

## Speech-to-Text

### Overview

The transcription system supports multiple providers with automatic fallback. The active provider is selected via the `TRANSCRIBE_PROVIDER` environment variable, or auto-detected in priority order: Google > AWS > OpenAI.

- **Endpoint:** `POST /api/transcribe`
- **Max upload size:** 25 MB (multipart form data via multer)
- **User override:** Each user can select a preferred STT model in their settings, stored in the `stt_model` column of the `users` table.
- **Admin default:** Administrators can set the system-wide default STT model via the admin settings panel.

### Providers

#### 1. Google Gemini

- Sends inline audio data within chat completion requests (not a separate transcription API).
- Model is configurable; default is `gemini-2.0-flash`.
- HIPAA eligible.

#### 2. Amazon Transcribe

- Uses streaming audio for real-time transcription.
- Supports **Medical mode** with specialty selection:
  - `PRIMARYCARE`, `CARDIOLOGY`, `NEUROLOGY`, `ONCOLOGY`, `RADIOLOGY`, `UROLOGY`
- Configured via `AWS_TRANSCRIBE_MEDICAL` and `AWS_TRANSCRIBE_SPECIALTY` environment variables.
- HIPAA eligible.

#### 3. Local Whisper

- Runs `whisper.cpp` or `faster-whisper` as a local binary process.
- Supported model sizes: `tiny`, `base`, `small`, `medium`, `large`.
- Configurable threads and language via environment variables (`WHISPER_THREADS`, `WHISPER_LANGUAGE`).
- No external API calls -- fully offline.

#### 4. OpenAI Whisper

- Uses the `whisper-1` model via the OpenAI API.
- Sends a medical context prompt: `"Medical patient encounter. Pediatric."`

#### 5. LiteLLM

- Routes transcription through LiteLLM's `chat/completions` endpoint using Gemini-style inline audio.
- Does **not** use the `/audio/transcriptions` endpoint.
- Model name configured via `LITELLM_STT_MODEL`.

---

## Browser Whisper (Client-Side Transcription)

Client-side transcription runs entirely in the browser with zero network traffic, providing maximum privacy.

- **Runtime:** WebAssembly via `@xenova/transformers`
- **Available models:**
  - `whisper-tiny.en` -- 39 MB
  - `whisper-base.en` -- 74 MB
  - `whisper-small.en` -- 244 MB
- **Self-hosted:** Model files are bundled in the Docker image. There is no CDN dependency.
- **Web Worker:** Transcription runs in a dedicated Web Worker to avoid blocking the UI thread.
- **Caching:** Downloaded models are cached in IndexedDB so subsequent loads are instant.
- **User toggle:** Enabled or disabled per user in settings. If browser transcription fails, it falls back to server-side transcription automatically.

---

## Web Speech Recognition (Live Preview)

- Uses the Chrome/Edge **Web Speech API** (`webkitSpeechRecognition`) for live preview during recording.
- Streams interim (partial) results to the UI while the user is still speaking.
- This is **not** used for final transcription. It serves only as a real-time visual preview. The actual transcription is performed by the configured STT provider (server-side or browser Whisper) after recording completes.

---

## Text-to-Speech

### Overview

- **Endpoint:** `POST /api/text-to-speech`
- **Character limit:** 5000 characters per request.
- **Response format:** `audio/mpeg`
- **Provider header:** The response includes an `X-TTS-Provider` header indicating which provider was used.
- **User override:** Each user can select a preferred voice in their settings, stored in the `tts_voice` column of the `users` table.

### Providers

#### 1. Google Cloud TTS

- Uses the `@google-cloud/text-to-speech` client library.
- Supported voice families:
  - **Journey** voices: `Journey-F`, `Journey-D`
  - **Studio** voices
  - **Neural2** voices

#### 2. LiteLLM

- Routes TTS requests to downstream providers (OpenAI, ElevenLabs, Gemini) via the configured LiteLLM model name.
- Configured via `LITELLM_TTS_MODEL` and `LITELLM_TTS_VOICE`.

#### 3. ElevenLabs

- Uses the `eleven_turbo_v2_5` model.
- **Not HIPAA compliant.** Do not use in production environments handling protected health information.

---

## Audio Backup System

The audio backup system preserves original audio recordings when transcription fails, allowing later retry.

### Storage

- Audio is saved to **PostgreSQL** only when transcription fails (not on every recording).
- Stored as gzip-compressed binary data in a `BYTEA` column.
- Backups auto-expire after **24 hours**.

### User Interface

- The Settings page displays a list of saved audio backups.
- Each backup has two actions:
  - **Retry** -- re-submits the audio to the transcription provider.
  - **Delete** -- permanently removes the backup.

### Browser Fallback

- If the server-side backup save fails (e.g., network error), the audio is saved to **IndexedDB** in the browser as a secondary fallback.