scrub vendor model docs

This commit is contained in:
Daniel 2026-05-08 19:15:54 +02:00
parent 210ec06fe5
commit aeb31a2d15
4 changed files with 15 additions and 24 deletions

View file

@ -15,7 +15,7 @@ Provider is selected once at startup and is transparent to callers.
- SDK: `@aws-sdk/client-bedrock-runtime`.
- Uses Bedrock **inference profiles** for newer models (cross-region routing).
- Model families: vendor model (Anthropic), Amazon Nova, Llama (Meta), Mistral, DeepSeek, Cohere.
- Model families: Amazon Nova, Llama (Meta), Mistral, DeepSeek, Cohere, and other Bedrock-hosted families.
### Azure OpenAI (BAA-eligible)
@ -27,7 +27,7 @@ Provider is selected once at startup and is transparent to callers.
- SDK: `@google-cloud/vertexai`.
- Also serves STT (Gemini inline audio) and TTS (Vertex TTS endpoint).
- Families: Gemini 2.5 / 2.0, vendor model on Vertex (Anthropic via GCP), Llama.
- Families: Gemini 2.5 / 2.0 and Llama.
### LiteLLM proxy (self-hosted)

View file

@ -178,8 +178,8 @@ OpenAI-compatible gateway — LiteLLM, Bifrost, or other proxies.
3. **Update model names** — Different gateways use different naming
conventions. Bifrost requires `provider/model` format
(e.g., `openrouter/vendor-model-sonnet-4.6`), while LiteLLM uses aliases
(e.g., `openrouter-vendor-model-sonnet-4.6`). Update model names in:
(e.g., `openrouter/gpt-4.1`), while LiteLLM can use deployment aliases
(e.g., `openrouter-gpt-4.1`). Update model names in:
- Admin Panel → Models (chat models)
- Admin Panel → Settings → `stt.model` (speech-to-text)
- Admin Panel → Settings → `tts.model` (text-to-speech)

View file

@ -542,11 +542,11 @@ const result = await callAI(
### Bedrock Model Notes
**Inference Profiles:** Most newer models (Anthropic vendor model 4.x, Meta Llama 4, DeepSeek R1, Amazon Nova, Writer) require cross-region inference profiles. These use a `us.` prefix on the model ID (e.g. `us.anthropic.agent-config-sonnet-4-6`). Direct model IDs will return "on-demand throughput not supported" errors.
**Inference Profiles:** Most newer Bedrock models require cross-region inference profiles. These use a `us.` prefix on the model ID (for example, `us.amazon.nova-pro-v1:0`). Direct model IDs may return "on-demand throughput not supported" errors.
**Max Output Tokens:** Some models have low output limits (Cohere Command R/R+: 4096, AI21 Jamba: 4096). The `maxOut` field in `models.js` auto-clamps `maxTokens` in `callBedrock()`.
**JSON Sanitization:** Some models (notably vendor model Sonnet 4.6, Opus 4.6) output literal newline characters inside JSON string values. `learningAI.js` includes a `sanitizeJsonString()` function that escapes these before parsing.
**JSON Sanitization:** Some models output literal newline characters inside JSON string values. `learningAI.js` includes a `sanitizeJsonString()` function that escapes these before parsing.
### Prompt System
Prompts are defined in `src/utils/prompts.js`. Admins can override any prompt via the Admin panel (`/admin/config/prompts`). Overrides are stored in `app_settings` table and loaded into memory on startup (with 3s grace period for DB readiness).

View file

@ -175,7 +175,7 @@ Lines 478514. On primary-provider error:
("Pediatric AI Scribe") — this is what shows up in the OpenRouter
dashboard's app attribution.
- Model IDs are passed straight through (`google/gemini-2.5-flash`,
`anthropic/vendor-model-sonnet-4-6`, etc.).
`openai/gpt-4.1`, etc.).
- **Not BAA-eligible.** The system enforces nothing about that — the operator
must just not select it for PHI.
@ -183,27 +183,18 @@ Lines 478514. On primary-provider error:
- Auth: standard AWS SDK credentials (env or instance profile). Region
comes from `AWS_BEDROCK_REGION`.
- Two API paths:
- **Anthropic models** (line 223 — detected by `modelId.indexOf('anthropic.')`)
use the native `InvokeModel` Messages API (lines 225273). System
message is hoisted out of the messages array into the top-level
`system` field — Anthropic's Messages API takes it separately, not as
a `role: system` entry.
- **All other models** (Nova, Llama, Mistral, DeepSeek, Cohere, etc.) use
the unified Bedrock **Converse API** (lines 276313). System message
becomes `system: [{ text: ... }]`; chat content is wrapped as
`content: [{ text: ... }]`.
- Bedrock models use either native provider payloads or the unified Bedrock
**Converse API**. For Converse requests, the system message becomes
`system: [{ text: ... }]`; chat content is wrapped as `content: [{ text: ... }]`.
- **Inference profile mapping.** `getBedrockModelId(model)` in
`models.js` translates the friendly id (`anthropic/vendor-model-sonnet-4-6`)
into the AWS-required cross-region inference profile id
(`us.anthropic.agent-config-sonnet-4-6`). Most newer models *require* the
inference profile; using the raw foundation model id will fail.
`models.js` translates the friendly id into the AWS-required cross-region
inference profile id. Most newer models require the inference profile;
using the raw foundation model id can fail.
- **Output token clamping** (lines 209210). `getBedrockMaxOut` returns the
per-model output limit (e.g. Cohere is 4096) so we don't ask for 4000
output and get a 400 from the SDK.
- **Multi-block content extraction** (lines 245263). vendor model Opus 4.6+ may
return multiple content blocks (e.g. a thinking block + a text block).
We iterate, keep only `type === 'text'`, and concatenate.
- **Multi-block content extraction.** Some provider-native responses may return
multiple content blocks. We iterate, keep only `type === 'text'`, and concatenate.
- Usage is normalised: AWS's `input_tokens` / `output_tokens`
`prompt_tokens` / `completion_tokens` to match the OpenAI shape.