33b47d4cb1
- New docs/developer-guide.md: full walkthrough of frontend SPA architecture, backend middleware stack, database layer, AI integration, settings system, how to add features/routes/tables, key design decisions, file references - Expand ai-providers.md: detailed admin model management (add custom models with ID/name/cost/category, discover from provider, enable/disable, set default) - Update README docs index
139 lines
6.4 KiB
Markdown
139 lines
6.4 KiB
Markdown
# AI Providers
|
|
|
|
This document covers the AI provider system, model management, prompt configuration, and usage logging for the Pediatric AI Scribe application.
|
|
|
|
---
|
|
|
|
## Provider Selection
|
|
|
|
The active AI provider is determined in one of two ways:
|
|
|
|
1. **Explicit**: Set the `AI_PROVIDER` environment variable to one of the supported provider names.
|
|
2. **Auto-detect**: If `AI_PROVIDER` is not set, the system checks for provider-specific credentials in the environment and selects the first match using this priority order:
|
|
|
|
`bedrock` > `azure` > `vertex` > `litellm` > `openrouter`
|
|
|
|
All providers expose a unified `callAI()` interface defined in `src/utils/ai.js`. Calling code does not need to know which backend is active.
|
|
|
|
---
|
|
|
|
## Provider Details
|
|
|
|
### 1. AWS Bedrock (HIPAA-eligible with BAA)
|
|
|
|
- **SDK**: `@aws-sdk/client-bedrock-runtime`
|
|
- Uses **inference profiles** for newer models, enabling cross-region routing.
|
|
- **Available model families**: vendor model (Anthropic), Amazon Nova, Llama (Meta), Mistral, DeepSeek, Cohere.
|
|
- **Default temperature**: 0.3
|
|
|
|
### 2. Azure OpenAI (HIPAA-eligible with BAA)
|
|
|
|
- **SDK**: OpenAI SDK configured to point at an Azure endpoint.
|
|
- Requires a **deployment name** that maps to the desired model.
|
|
- **Available model families**: GPT-4o family, GPT-4.1 family.
|
|
|
|
### 3. Google Vertex AI (HIPAA-eligible with BAA)
|
|
|
|
- **SDK**: `@google-cloud/vertexai`
|
|
- In addition to text generation, Vertex AI handles:
|
|
- **Speech-to-Text (STT)**: via Gemini inline audio capabilities.
|
|
- **Text-to-Speech (TTS)**: via Vertex AI TTS endpoint.
|
|
- **Available model families**: Gemini 2.5/2.0, vendor model on Vertex (Anthropic models hosted on Google Cloud), Llama.
|
|
|
|
### 4. LiteLLM Proxy (self-hosted)
|
|
|
|
- **SDK**: OpenAI SDK pointed at the `LITELLM_API_BASE` URL.
|
|
- Acts as a proxy that routes requests to any backend configured within LiteLLM.
|
|
- **Model discovery**: queries `/v1/models` on the LiteLLM instance to populate the available model list.
|
|
- Also supports **TTS and STT** passthrough.
|
|
- Model names are used **as-configured in LiteLLM** -- the application does not auto-prefix or transform them.
|
|
|
|
### 5. OpenRouter (NOT HIPAA-compliant)
|
|
|
|
- **SDK**: OpenAI SDK pointed at `https://openrouter.ai`.
|
|
- **Cost discovery**: queries the OpenRouter API to retrieve per-model pricing.
|
|
- Offers the **cheapest option** and the **widest model selection** across many providers.
|
|
- Not suitable for environments that require HIPAA compliance.
|
|
|
|
---
|
|
|
|
## Model Management
|
|
|
|
### Model Definitions
|
|
|
|
Models are defined in `src/utils/models.js` and organized into four categories:
|
|
|
|
| Category | Description |
|
|
|-----------|------------------------------------------------|
|
|
| `free` | No-cost models (typically smaller or rate-limited) |
|
|
| `fast` | Low-latency models optimized for speed |
|
|
| `smart` | Balanced models with strong reasoning ability |
|
|
| `premium` | Top-tier models with the highest capability |
|
|
|
|
### Admin Controls
|
|
|
|
Administrators manage models from **Admin Panel > Models**:
|
|
|
|
- **Enable/Disable** -- toggle visibility of any model for users (`PUT /api/admin/config/models/toggle`). Disabled models are stored in the `models.disabled` setting as a JSON array of model IDs.
|
|
- **Set Default** -- choose which model is pre-selected for new users (`PUT /api/admin/config/models/default`). Stored in `models.default` setting.
|
|
- **Add Custom Models** -- manually add any model not in the built-in list (`POST /api/admin/config/models/custom`). Each custom model has:
|
|
- `id` -- the model identifier as the provider expects it (e.g., `openai-gpt-4.1-mini` for LiteLLM, `anthropic.agent-config-3-haiku` for Bedrock)
|
|
- `name` -- display name shown to users
|
|
- `cost` -- cost string shown in the UI (e.g., `~$0.002`, `FREE`)
|
|
- `category` -- one of `free`, `fast`, `smart`, `premium` (determines grouping in dropdown)
|
|
- **Delete Custom Models** -- remove a manually added model (`DELETE /api/admin/config/models/custom/:modelId`)
|
|
- **Clear All** -- remove all custom models and reset the disabled list (`POST /api/admin/config/models/clear`)
|
|
|
|
Custom models are stored in the `models.custom` setting as a JSON array in the `app_settings` table.
|
|
|
|
### Model Discovery
|
|
|
|
The **Discover** button (`GET /api/admin/config/models/discover`) queries the active provider's API:
|
|
|
|
- **LiteLLM**: calls `/v1/models` on the LiteLLM proxy
|
|
- **OpenRouter**: calls `https://openrouter.ai/api/v1/models` (includes pricing data)
|
|
- **Bedrock**: uses `ListFoundationModelsCommand`
|
|
|
|
Discovered models can be added individually via `POST /api/admin/config/models/add-discovered`, which merges them into the custom models list.
|
|
|
|
For **LiteLLM** specifically, since models are manually configured in the LiteLLM proxy, the model IDs returned by discovery are the exact names to use -- no provider prefix is added.
|
|
|
|
### Frontend Display
|
|
|
|
The model selector dropdown (present on every tab) groups models by category:
|
|
|
|
- Free -- no-cost models
|
|
- Fast & Cheap -- low-latency, low-cost
|
|
- Smart -- balanced capability
|
|
- Premium -- highest quality
|
|
|
|
Each model shows its display name and cost string. The dropdown is populated from `GET /api/models` which merges built-in models, custom models, and respects the enabled/disabled list.
|
|
|
|
---
|
|
|
|
## AI Prompts
|
|
|
|
### Storage and Override
|
|
|
|
- All default prompts are defined in `src/utils/prompts.js`.
|
|
- Prompts can be **overridden via the database** using the `app_settings` table with keys following the pattern `prompt.{name}`.
|
|
- Administrators can view and edit all prompts directly from the Admin Panel.
|
|
|
|
### Physician Memories
|
|
|
|
When a physician saves corrections or preferences (referred to as "memories"), these are injected into the prompt as **low-priority style hints**. This allows the AI to adapt its output to the physician's preferred documentation style without overriding the core clinical prompt.
|
|
|
|
---
|
|
|
|
## Logging
|
|
|
|
Every AI call is recorded in the `api_log` database table with the following fields:
|
|
|
|
| Field | Description |
|
|
|------------|-----------------------------------------------------|
|
|
| `model` | The model identifier used for the request |
|
|
| `tokens` | Input and output token counts |
|
|
| `cost` | Estimated cost of the call |
|
|
| `duration` | Wall-clock time for the request in milliseconds |
|
|
|
|
Cost estimates are calculated from **hardcoded per-model rates** defined in the codebase. For OpenRouter, rates may also be fetched from the OpenRouter pricing API.
|