pediatric-ai-scribe-v3/BROWSER_WHISPER_SETUP.md

Browser Whisper Self-Hosted Setup

Overview

As of v3, Browser Whisper is fully self-hosted with zero CDN dependencies. All models and libraries are bundled with the application and served from your own server.

What Changed

Before (v2 and earlier):

• Loaded transformers.js from cdn.jsdelivr.net
• Downloaded models from cdn-lfs.huggingface.co
• Failed in corporate/clinical networks with firewall restrictions

Now (v3+):

• Transformers.js library (v2.6.2) bundled at /models/transformers.min.js (760KB)
• Whisper model bundled at /models/Xenova/whisper-tiny.en/ (42MB)
• Everything served from your own server
• Works in any network environment (firewalled, air-gapped, offline)

Files Included

public/models/
├── transformers.min.js                    (760KB) - Transformers.js v2.6.2 (worker-compatible)
└── Xenova/
    └── whisper-tiny.en/                   (42MB total)
        ├── config.json
        ├── tokenizer.json
        ├── preprocessor_config.json
        ├── generation_config.json
        └── onnx/
            ├── encoder_model_quantized.onnx
            └── decoder_model_merged_quantized.onnx

How It Works

1. Worker loads transformers.js locally:

   importScripts('/models/transformers.min.js');
   

2. Transformers.js configured for local models:

   T.env.localModelPath = '/models/';
   T.env.allowRemoteModels = false;
   

3. Models load from your server:

   • Browser requests: GET /models/Xenova/whisper-tiny.en/config.json
   • Served by Express static middleware
   • No external network calls

Docker Build

Models are downloaded during Docker build (not runtime):

RUN curl -sL -o onnx/encoder_model_quantized.onnx \
    https://huggingface.co/Xenova/whisper-tiny.en/resolve/main/onnx/encoder_model_quantized.onnx

This means:

• Docker image is ~200MB larger (one-time cost)
• Runtime has zero dependencies
• Works in air-gapped environments (after image is pulled)

Development Setup

If you're running locally (not Docker), download models:

cd public/models
mkdir -p Xenova/whisper-tiny.en/onnx

# Download transformers.js
curl -L -o transformers.min.js \
  https://cdn.jsdelivr.net/npm/@xenova/transformers@2.17.2/dist/transformers.min.js

# Download model files
cd Xenova/whisper-tiny.en
curl -L -o config.json \
  https://huggingface.co/Xenova/whisper-tiny.en/resolve/main/config.json
curl -L -o tokenizer.json \
  https://huggingface.co/Xenova/whisper-tiny.en/resolve/main/tokenizer.json
curl -L -o preprocessor_config.json \
  https://huggingface.co/Xenova/whisper-tiny.en/resolve/main/preprocessor_config.json
curl -L -o generation_config.json \
  https://huggingface.co/Xenova/whisper-tiny.en/resolve/main/generation_config.json
curl -L -o onnx/encoder_model_quantized.onnx \
  https://huggingface.co/Xenova/whisper-tiny.en/resolve/main/onnx/encoder_model_quantized.onnx
curl -L -o onnx/decoder_model_merged_quantized.onnx \
  https://huggingface.co/Xenova/whisper-tiny.en/resolve/main/onnx/decoder_model_merged_quantized.onnx

Or use the helper script:

./scripts/download-whisper-models.sh

Adding More Models

To add base or small models:

1. Create directory:

   mkdir -p public/models/Xenova/whisper-base.en/onnx
   

2. Download from HuggingFace:

   • https://huggingface.co/Xenova/whisper-base.en
   • https://huggingface.co/Xenova/whisper-small.en

3. Update UI in settings.html:

   <option value="Xenova/whisper-base.en">Base (~74MB, better quality)</option>
   

4. Update Dockerfile to download during build

Benefits

✅ Works everywhere - No firewall/CDN issues ✅ Privacy-first - Audio never leaves browser ✅ Offline capable - After initial page load ✅ No API costs - Zero transcription expenses ✅ Predictable - Same model, same results ✅ Fast - Local processing, no network latency

Limitations

• Docker image is larger (~200MB vs ~150MB)
• Only tiny model included by default (base/small optional)
• Slower than cloud APIs for long recordings
• Requires modern browser with WebAssembly support

Testing

# 1. Start server
docker-compose up -d

# 2. Open browser DevTools → Network tab
# 3. Go to Settings → Browser Transcription
# 4. Click "Pre-download model"
# 5. Watch for requests to /models/* (should all be 200 OK from your server)
# 6. NO requests to cdn.jsdelivr.net or huggingface.co

Troubleshooting

Issue: "Failed to load transformers library"

• Check: GET /models/transformers.min.js returns 200 OK
• Verify file exists: ls public/models/transformers.min.js

Issue: "Model load failed"

• Check: GET /models/Xenova/whisper-tiny.en/config.json returns 200 OK
• Verify files exist: ls public/models/Xenova/whisper-tiny.en/

Issue: Still seeing CDN requests

• Clear browser cache (Ctrl+Shift+R)
• Check you're running v18+ (/api/health should show version)

Migration from v17

If upgrading from v17:

1. Pull new Docker image: docker-compose pull
2. Restart: docker-compose up -d
3. Clear browser cache
4. Test: Settings → Browser Transcription → Pre-download

No configuration changes needed - it just works!