danvics/pediatric-ai-scribe-v3
1
0
Fork 0
Code Issues Pull requests Projects Releases 14 Packages Wiki Activity Actions
pediatric-ai-scribe-v3/docs/speech.md
Daniel b97c5f146b v6.1: Turnstile bot protection, LiteLLM provider, PPTX tables, audio backup fixes, docs 
- Add Cloudflare Turnstile to login, register, and password reset forms
- Switch AI provider to LiteLLM, transcription to OpenAI Whisper
- Change domain to scribe.pedshub.com
- Fix PPTX export: add tables, bold/italic, numbered lists, code blocks, blockquotes
- Fix announcement banner close button (CSP was blocking inline onclick)
- Fix auth middleware: empty Bearer token now falls through to cookie auth
- Fix audio backups: only save on transcription failure, stop auto-deleting on success
- Soften AI correction injection to prevent model hallucination from correction history
- Fix LiteLLM TTS model name handling (no incorrect openai/ prefix)
- Expand AI instructions textarea in Learning Hub CMS
- Update README for v6 with all features and providers
- Add comprehensive docs/: architecture, API reference, database schema,
  authentication, AI providers, speech, learning hub, configuration, deployment
2026-04-04 22:56:24 +02:00

4.9 KiB
Raw Blame History

Speech-to-Text and Text-to-Speech Systems

This document covers all audio processing capabilities in the Pediatric AI Scribe, including server-side transcription, client-side transcription, live speech preview, text-to-speech, and audio backup.

Speech-to-Text

Overview

The transcription system supports multiple providers with automatic fallback. The active provider is selected via the TRANSCRIBE_PROVIDER environment variable, or auto-detected in priority order: Google > AWS > OpenAI.

  • Endpoint: POST /api/transcribe
  • Max upload size: 25 MB (multipart form data via multer)
  • User override: Each user can select a preferred STT model in their settings, stored in the stt_model column of the users table.
  • Admin default: Administrators can set the system-wide default STT model via the admin settings panel.

Providers

1. Google Gemini

  • Sends inline audio data within chat completion requests (not a separate transcription API).
  • Model is configurable; default is gemini-2.0-flash.
  • HIPAA eligible.

2. Amazon Transcribe

  • Uses streaming audio for real-time transcription.
  • Supports Medical mode with specialty selection:
    • PRIMARYCARE, CARDIOLOGY, NEUROLOGY, ONCOLOGY, RADIOLOGY, UROLOGY
  • Configured via AWS_TRANSCRIBE_MEDICAL and AWS_TRANSCRIBE_SPECIALTY environment variables.
  • HIPAA eligible.

3. Local Whisper

  • Runs whisper.cpp or faster-whisper as a local binary process.
  • Supported model sizes: tiny, base, small, medium, large.
  • Configurable threads and language via environment variables (WHISPER_THREADS, WHISPER_LANGUAGE).
  • No external API calls -- fully offline.

4. OpenAI Whisper

  • Uses the whisper-1 model via the OpenAI API.
  • Sends a medical context prompt: "Medical patient encounter. Pediatric."

5. LiteLLM

  • Routes transcription through LiteLLM's chat/completions endpoint using Gemini-style inline audio.
  • Does not use the /audio/transcriptions endpoint.
  • Model name configured via LITELLM_STT_MODEL.

Browser Whisper (Client-Side Transcription)

Client-side transcription runs entirely in the browser with zero network traffic, providing maximum privacy.

  • Runtime: WebAssembly via @xenova/transformers
  • Available models:
    • whisper-tiny.en -- 39 MB
    • whisper-base.en -- 74 MB
    • whisper-small.en -- 244 MB
  • Self-hosted: Model files are bundled in the Docker image. There is no CDN dependency.
  • Web Worker: Transcription runs in a dedicated Web Worker to avoid blocking the UI thread.
  • Caching: Downloaded models are cached in IndexedDB so subsequent loads are instant.
  • User toggle: Enabled or disabled per user in settings. If browser transcription fails, it falls back to server-side transcription automatically.

Web Speech Recognition (Live Preview)

  • Uses the Chrome/Edge Web Speech API (webkitSpeechRecognition) for live preview during recording.
  • Streams interim (partial) results to the UI while the user is still speaking.
  • This is not used for final transcription. It serves only as a real-time visual preview. The actual transcription is performed by the configured STT provider (server-side or browser Whisper) after recording completes.

Text-to-Speech

Overview

  • Endpoint: POST /api/text-to-speech
  • Character limit: 5000 characters per request.
  • Response format: audio/mpeg
  • Provider header: The response includes an X-TTS-Provider header indicating which provider was used.
  • User override: Each user can select a preferred voice in their settings, stored in the tts_voice column of the users table.

Providers

1. Google Cloud TTS

  • Uses the @google-cloud/text-to-speech client library.
  • Supported voice families:
    • Journey voices: Journey-F, Journey-D
    • Studio voices
    • Neural2 voices

2. LiteLLM

  • Routes TTS requests to downstream providers (OpenAI, ElevenLabs, Gemini) via the configured LiteLLM model name.
  • Configured via LITELLM_TTS_MODEL and LITELLM_TTS_VOICE.

3. ElevenLabs

  • Uses the eleven_turbo_v2_5 model.
  • Not HIPAA compliant. Do not use in production environments handling protected health information.

Audio Backup System

The audio backup system preserves original audio recordings when transcription fails, allowing later retry.

Storage

  • Audio is saved to PostgreSQL only when transcription fails (not on every recording).
  • Stored as gzip-compressed binary data in a BYTEA column.
  • Backups auto-expire after 24 hours.

User Interface

  • The Settings page displays a list of saved audio backups.
  • Each backup has two actions:
    • Retry -- re-submits the audio to the transcription provider.
    • Delete -- permanently removes the backup.

Browser Fallback

  • If the server-side backup save fails (e.g., network error), the audio is saved to IndexedDB in the browser as a secondary fallback.