pediatric-ai-scribe-v3/docs/speech.md

# Speech: STT, TTS, Audio Backup

## Transcription

`POST /api/transcribe` accepts `multipart/form-data` with one audio file up to 25 MB. Server STT is routed through LiteLLM.

Set `TRANSCRIBE_PROVIDER=litellm`, `LITELLM_API_BASE`, and `LITELLM_STT_MODEL`. Auto mode also uses LiteLLM when the gateway is configured.

The model for a request is `users.stt_model` (the person's own choice), then
the `stt.model` setting, then `LITELLM_STT_MODEL`. A user's choice wins, so the
Settings picker must never offer a model the gateway does not have: it lists
what `/model/info` advertises as `audio_transcription`
(`discoverSTTModels()` in `src/utils/sttProvider.js`, cached five minutes).
`LITELLM_STT_MODELS` is a fallback for when discovery fails and is not a list
of models known to work — its ids do not resolve on every gateway.

| Provider | Notes | HIPAA posture |
|---|---|---|
| LiteLLM | Sends audio through the configured LiteLLM `/audio/transcriptions` backend. | Depends on the selected upstream. |

Browser Whisper and browser-local Whisper workers are not part of the runtime. Do not add browser model downloads or Transformers.js STT back into the public app.

### What to run in a clinic

| Need | Use |
|---|---|
| Server STT | LiteLLM, with a compliant upstream configured in the gateway. |
| Real-time draft preview | Browser Web Speech, only on explicit opt-in and with the privacy warning. |

```env
TRANSCRIBE_PROVIDER=litellm
LITELLM_API_BASE=https://your-litellm.example/v1
LITELLM_API_KEY=<key>
LITELLM_STT_MODEL=local-parakeet-v3
```

Direct Google, AWS Transcribe, OpenAI Whisper and local Whisper branches are
not part of the runtime. Choosing among those upstreams is a LiteLLM
configuration question, not an app one.

### Paths that must stay removed

Browser-local STT was removed deliberately. These should stay absent unless the
project reintroduces it with a fresh design review — and they should not be
offered in settings, docs, public workers, or model download scripts either:

- `public/js/browserWhisper.js`
- `public/js/whisperWorker.js`
- `public/js/whisperWorkerV2.js`
- `public/models/Xenova/*`
- Browser Whisper setup and troubleshooting docs
- Whisper model download scripts for public browser models

## Web Speech Preview

Browser-native Web Speech can show interim text when the user explicitly enables it. It is browser/vendor dependent, may send audio to browser-provider cloud services, and should not be treated as the final clinical transcript.

## Text To Speech

`POST /api/text-to-speech` returns audio from LiteLLM `/audio/speech`. The `X-TTS-Provider` response header identifies the LiteLLM model used. Requests are limited to 5000 characters.

Speech models sit on a roster (`tts.roster`, Admin → Models → Speech), and the
default is a pair — `tts.model` and `tts.voice`, set together through
`PUT /api/admin/config/tts/default`, which refuses a voice the model does not
accept. `LITELLM_TTS_MODEL` / `LITELLM_TTS_VOICE` are the fallback when no
default has been chosen.

Which voices a model accepts is a property of the model, not the gateway
(`/model/info` carries no voice field), so the table lives in
`src/utils/ttsProvider.js` (`MODEL_VOICES`, keyed by family: Kokoro, Orpheus
English/Arabic, Fish). `LITELLM_TTS_VOICES` names the voices of
`LITELLM_TTS_MODEL` only. A user's choice is stored as `model|voice`, so Read
Aloud sends each voice to the model that accepts it; `chooseTTS` is the one
place that decision is made.

## Recording

A recording holds a screen wake lock for as long as it runs, so the screen
going to sleep cannot suspend it. Browsers release the lock whenever the page
is hidden, so it is requested again when the page becomes visible; a lock is
never requested while hidden, because that request is rejected. The count is
shared, so two recorders cannot release each other's lock. A browser that
denies or lacks the API keeps recording without one.

Recording continues while the user moves around the workspace — the app is a
single page and switching tabs does not touch the recorder. Signing out stops
it and releases the lock; nothing is sent, because the session that owned the
audio is gone.

`AudioRecorder.start()` on a running recorder is a no-op. Calling it again
would replace the `MediaRecorder` and drop everything captured so far.

A recording can stop without anyone pressing Stop: the recorder can error, and
the microphone can be claimed by another app, unplugged, or revoked. Both are
reported once and dispatch `audio-recorder-failed`, which runs the same path as
Stop — so the audio is transcribed and stored rather than left in a tab that
still claims to be recording. Whatever was captured before the failure is kept.

## Audio Backup

Every recording is kept for 24 hours, whether its transcription succeeded or
not. `POST /api/transcribe` already holds the audio, so keeping it costs no
second upload; a storage failure there is logged and the transcription still
returns, because losing the transcript someone is waiting for would be worse
than losing the copy.

`src/utils/audioBackupStore.js` is the only place that knows how a recording is
kept, shared by `/api/transcribe` and `/api/audio-backups` so the two cannot
drift apart.

- Audio is gzipped, then AES-256-GCM encrypted (`DATA_ENCRYPTION_KEY`), before
  it is stored — in either backend.
- **Object storage** is used when `AUDIO_BACKUPS_S3_*` is set; otherwise the
  payload goes in the `audio_backups.audio_data` column. Metadata (owner,
  module, sizes, expiry) is always in Postgres, so listing, ownership and
  expiry behave the same either way.
- Object keys are `recordings/<user id>/<timestamp>-<random>`, scoped to their
  owner, so a leaked id cannot address someone else's audio.
- Reads carry `user_id` and `expires_at > NOW()` in the query, so an expired or
  borrowed id reads as missing rather than as another person's recording.
- The expiry sweep (`cleanupExpired` in `src/db/database.js`) deletes each
  object with its row, so audio cannot outlive its 24 hours in the bucket.
- Rows written before encryption was added are passed through unencrypted on
  read, so old backups still play.
- The Settings list can retry, **download** or delete a recording. Download is
  how a copy leaves the app — onto a phone's Files, a shared drive, an external
  recorder. It works for both server-side and browser-fallback copies, and a
  browser copy is only handed to the account that owns it.
- Browser fallback storage (IndexedDB) is used only when the server cannot be
  reached at all.

Treat audio backups as sensitive clinical data even when encrypted.

### Where object storage settings come from

`src/utils/objectStorage.js` resolves them for every purpose the same way, so
moving the app to a different MinIO — or to a real S3 — is one set of variables
rather than three schemes. For a purpose (`documents`, `generated-images`,
`audio-backups`) it reads, in order:

1. that purpose's own variables — `AUDIO_BACKUPS_S3_ENDPOINT`, `..._BUCKET`,
   `..._REGION`, `..._ACCESS_KEY[_FILE]`, `..._SECRET_KEY[_FILE]`;
2. the shared ones — `S3_ENDPOINT`, `S3_REGION`, `S3_ACCESS_KEY[_FILE]`,
   `S3_SECRET_KEY[_FILE]`;
3. a per-purpose bucket — `S3_BUCKET_AUDIO_BACKUPS`, `S3_BUCKET_GENERATED_IMAGES`.

So one endpoint plus three bucket names covers everything, while a purpose that
needs its own account overrides all of it. A `_FILE` variant always beats an
inline value, because a mounted secret should not be shadowed by an inherited
environment variable. No bucket means "not configured", which is never an error
— all three are optional.

Every name previously accepted still works, including `S3_ACCESS_KEY_ID`,
`S3_SECRET_ACCESS_KEY` and the `AWS_*` fallbacks for documents, and
`GENERATED_IMAGES_S3_*`. Path-style addressing keeps each purpose's old default
(off for documents, so Backblaze keeps working) unless `S3_FORCE_PATH_STYLE`
says otherwise; a custom endpoint turns it on where there was no older default,
because that is nearly always MinIO.

### Switching audio backups to MinIO

Storing every recording, rather than only the failures, makes object storage
the better home. It is off by default because it needs a bucket and its own
credentials; the existing `generated-images` key is scoped to that bucket and
cannot reach another.

1. Create an `audio-backups` bucket and a user with read/write/delete on it.
2. Give the bucket a 24-hour expiry lifecycle rule, as a backstop for objects
   the sweep could not delete.
3. Set `AUDIO_BACKUPS_S3_ENDPOINT`, `AUDIO_BACKUPS_S3_BUCKET`,
   `AUDIO_BACKUPS_S3_REGION`, and either `AUDIO_BACKUPS_S3_ACCESS_KEY_FILE` /
   `AUDIO_BACKUPS_S3_SECRET_KEY_FILE` (preferred — credentials stay out of the
   process environment) or `AUDIO_BACKUPS_S3_ACCESS_KEY` /
   `AUDIO_BACKUPS_S3_SECRET_KEY`.

New recordings then go to the bucket. Rows already in Postgres keep working:
a row without `storage_key` is read from the column.