pediatric-ai-scribe-v3/docs/ai-providers.md

AI Providers

This document covers the AI provider system, model management, prompt configuration, and usage logging for the Pediatric AI Scribe application.

---

Provider Selection

The active AI provider is determined in one of two ways:

1. Explicit: Set the AI_PROVIDER environment variable to one of the supported provider names.

2. Auto-detect: If AI_PROVIDER is not set, the system checks for provider-specific credentials in the environment and selects the first match using this priority order:

   bedrock > azure > vertex > litellm > openrouter

All providers expose a unified callAI() interface defined in src/utils/ai.js. Calling code does not need to know which backend is active.

---

Provider Details

1. AWS Bedrock (HIPAA-eligible with BAA)

• SDK: @aws-sdk/client-bedrock-runtime
• Uses inference profiles for newer models, enabling cross-region routing.
• Available model families: vendor model (Anthropic), Amazon Nova, Llama (Meta), Mistral, DeepSeek, Cohere.
• Default temperature: 0.3

2. Azure OpenAI (HIPAA-eligible with BAA)

• SDK: OpenAI SDK configured to point at an Azure endpoint.
• Requires a deployment name that maps to the desired model.
• Available model families: GPT-4o family, GPT-4.1 family.

3. Google Vertex AI (HIPAA-eligible with BAA)

• SDK: @google-cloud/vertexai
• In addition to text generation, Vertex AI handles:
  • Speech-to-Text (STT): via Gemini inline audio capabilities.
  • Text-to-Speech (TTS): via Vertex AI TTS endpoint.
• Available model families: Gemini 2.5/2.0, vendor model on Vertex (Anthropic models hosted on Google Cloud), Llama.

4. LiteLLM Proxy (self-hosted)

• SDK: OpenAI SDK pointed at the LITELLM_API_BASE URL.
• Acts as a proxy that routes requests to any backend configured within LiteLLM.
• Model discovery: queries /v1/models on the LiteLLM instance to populate the available model list.
• Also supports TTS and STT passthrough.
• Model names are used as-configured in LiteLLM -- the application does not auto-prefix or transform them.

5. OpenRouter (NOT HIPAA-compliant)

• SDK: OpenAI SDK pointed at https://openrouter.ai.
• Cost discovery: queries the OpenRouter API to retrieve per-model pricing.
• Offers the cheapest option and the widest model selection across many providers.
• Not suitable for environments that require HIPAA compliance.

---

Model Management

Model Definitions

Models are defined in src/utils/models.js and organized into four categories:

Category	Description
free	No-cost models (typically smaller or rate-limited)
fast	Low-latency models optimized for speed
smart	Balanced models with strong reasoning ability
premium	Top-tier models with the highest capability

Admin Controls

Administrators manage models from Admin Panel > Models:

• Enable/Disable -- toggle visibility of any model for users (PUT /api/admin/config/models/toggle). Disabled models are stored in the models.disabled setting as a JSON array of model IDs.
• Set Default -- choose which model is pre-selected for new users (PUT /api/admin/config/models/default). Stored in models.default setting.
• Add Custom Models -- manually add any model not in the built-in list (POST /api/admin/config/models/custom). Each custom model has:
  • id -- the model identifier as the provider expects it (e.g., openai-gpt-4.1-mini for LiteLLM, anthropic.agent-config-3-haiku for Bedrock)
  • name -- display name shown to users
  • cost -- cost string shown in the UI (e.g., ~$0.002, FREE)
  • category -- one of free, fast, smart, premium (determines grouping in dropdown)
• Delete Custom Models -- remove a manually added model (DELETE /api/admin/config/models/custom/:modelId)
• Clear All -- remove all custom models and reset the disabled list (POST /api/admin/config/models/clear)

Custom models are stored in the models.custom setting as a JSON array in the app_settings table.

Model Discovery

The Discover button (GET /api/admin/config/models/discover) queries the active provider's API:

• LiteLLM: calls /v1/models on the LiteLLM proxy
• OpenRouter: calls https://openrouter.ai/api/v1/models (includes pricing data)
• Bedrock: uses ListFoundationModelsCommand

Discovered models can be added individually via POST /api/admin/config/models/add-discovered, which merges them into the custom models list.

For LiteLLM specifically, since models are manually configured in the LiteLLM proxy, the model IDs returned by discovery are the exact names to use -- no provider prefix is added.

Frontend Display

The model selector dropdown (present on every tab) groups models by category:

• Free -- no-cost models
• Fast & Cheap -- low-latency, low-cost
• Smart -- balanced capability
• Premium -- highest quality

Each model shows its display name and cost string. The dropdown is populated from GET /api/models which merges built-in models, custom models, and respects the enabled/disabled list.

---

AI Prompts

Storage and Override

• All default prompts are defined in src/utils/prompts.js.
• Prompts can be overridden via the database using the app_settings table with keys following the pattern prompt.{name}.
• Administrators can view and edit all prompts directly from the Admin Panel.

Physician Memories

When a physician saves corrections or preferences (referred to as "memories"), these are injected into the prompt as low-priority style hints. This allows the AI to adapt its output to the physician's preferred documentation style without overriding the core clinical prompt.

---

Logging

Every AI call is recorded in the api_log database table with the following fields:

Field	Description
model	The model identifier used for the request
tokens	Input and output token counts
cost	Estimated cost of the call
duration	Wall-clock time for the request in milliseconds

Cost estimates are calculated from hardcoded per-model rates defined in the codebase. For OpenRouter, rates may also be fetched from the OpenRouter pricing API.