scrub vendor model docs
This commit is contained in:
parent
210ec06fe5
commit
aeb31a2d15
4 changed files with 15 additions and 24 deletions
|
|
@ -15,7 +15,7 @@ Provider is selected once at startup and is transparent to callers.
|
||||||
|
|
||||||
- SDK: `@aws-sdk/client-bedrock-runtime`.
|
- SDK: `@aws-sdk/client-bedrock-runtime`.
|
||||||
- Uses Bedrock **inference profiles** for newer models (cross-region routing).
|
- Uses Bedrock **inference profiles** for newer models (cross-region routing).
|
||||||
- Model families: vendor model (Anthropic), Amazon Nova, Llama (Meta), Mistral, DeepSeek, Cohere.
|
- Model families: Amazon Nova, Llama (Meta), Mistral, DeepSeek, Cohere, and other Bedrock-hosted families.
|
||||||
|
|
||||||
### Azure OpenAI (BAA-eligible)
|
### Azure OpenAI (BAA-eligible)
|
||||||
|
|
||||||
|
|
@ -27,7 +27,7 @@ Provider is selected once at startup and is transparent to callers.
|
||||||
|
|
||||||
- SDK: `@google-cloud/vertexai`.
|
- SDK: `@google-cloud/vertexai`.
|
||||||
- Also serves STT (Gemini inline audio) and TTS (Vertex TTS endpoint).
|
- Also serves STT (Gemini inline audio) and TTS (Vertex TTS endpoint).
|
||||||
- Families: Gemini 2.5 / 2.0, vendor model on Vertex (Anthropic via GCP), Llama.
|
- Families: Gemini 2.5 / 2.0 and Llama.
|
||||||
|
|
||||||
### LiteLLM proxy (self-hosted)
|
### LiteLLM proxy (self-hosted)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -178,8 +178,8 @@ OpenAI-compatible gateway — LiteLLM, Bifrost, or other proxies.
|
||||||
|
|
||||||
3. **Update model names** — Different gateways use different naming
|
3. **Update model names** — Different gateways use different naming
|
||||||
conventions. Bifrost requires `provider/model` format
|
conventions. Bifrost requires `provider/model` format
|
||||||
(e.g., `openrouter/vendor-model-sonnet-4.6`), while LiteLLM uses aliases
|
(e.g., `openrouter/gpt-4.1`), while LiteLLM can use deployment aliases
|
||||||
(e.g., `openrouter-vendor-model-sonnet-4.6`). Update model names in:
|
(e.g., `openrouter-gpt-4.1`). Update model names in:
|
||||||
- Admin Panel → Models (chat models)
|
- Admin Panel → Models (chat models)
|
||||||
- Admin Panel → Settings → `stt.model` (speech-to-text)
|
- Admin Panel → Settings → `stt.model` (speech-to-text)
|
||||||
- Admin Panel → Settings → `tts.model` (text-to-speech)
|
- Admin Panel → Settings → `tts.model` (text-to-speech)
|
||||||
|
|
|
||||||
|
|
@ -542,11 +542,11 @@ const result = await callAI(
|
||||||
|
|
||||||
### Bedrock Model Notes
|
### Bedrock Model Notes
|
||||||
|
|
||||||
**Inference Profiles:** Most newer models (Anthropic vendor model 4.x, Meta Llama 4, DeepSeek R1, Amazon Nova, Writer) require cross-region inference profiles. These use a `us.` prefix on the model ID (e.g. `us.anthropic.agent-config-sonnet-4-6`). Direct model IDs will return "on-demand throughput not supported" errors.
|
**Inference Profiles:** Most newer Bedrock models require cross-region inference profiles. These use a `us.` prefix on the model ID (for example, `us.amazon.nova-pro-v1:0`). Direct model IDs may return "on-demand throughput not supported" errors.
|
||||||
|
|
||||||
**Max Output Tokens:** Some models have low output limits (Cohere Command R/R+: 4096, AI21 Jamba: 4096). The `maxOut` field in `models.js` auto-clamps `maxTokens` in `callBedrock()`.
|
**Max Output Tokens:** Some models have low output limits (Cohere Command R/R+: 4096, AI21 Jamba: 4096). The `maxOut` field in `models.js` auto-clamps `maxTokens` in `callBedrock()`.
|
||||||
|
|
||||||
**JSON Sanitization:** Some models (notably vendor model Sonnet 4.6, Opus 4.6) output literal newline characters inside JSON string values. `learningAI.js` includes a `sanitizeJsonString()` function that escapes these before parsing.
|
**JSON Sanitization:** Some models output literal newline characters inside JSON string values. `learningAI.js` includes a `sanitizeJsonString()` function that escapes these before parsing.
|
||||||
|
|
||||||
### Prompt System
|
### Prompt System
|
||||||
Prompts are defined in `src/utils/prompts.js`. Admins can override any prompt via the Admin panel (`/admin/config/prompts`). Overrides are stored in `app_settings` table and loaded into memory on startup (with 3s grace period for DB readiness).
|
Prompts are defined in `src/utils/prompts.js`. Admins can override any prompt via the Admin panel (`/admin/config/prompts`). Overrides are stored in `app_settings` table and loaded into memory on startup (with 3s grace period for DB readiness).
|
||||||
|
|
|
||||||
|
|
@ -175,7 +175,7 @@ Lines 478–514. On primary-provider error:
|
||||||
("Pediatric AI Scribe") — this is what shows up in the OpenRouter
|
("Pediatric AI Scribe") — this is what shows up in the OpenRouter
|
||||||
dashboard's app attribution.
|
dashboard's app attribution.
|
||||||
- Model IDs are passed straight through (`google/gemini-2.5-flash`,
|
- Model IDs are passed straight through (`google/gemini-2.5-flash`,
|
||||||
`anthropic/vendor-model-sonnet-4-6`, etc.).
|
`openai/gpt-4.1`, etc.).
|
||||||
- **Not BAA-eligible.** The system enforces nothing about that — the operator
|
- **Not BAA-eligible.** The system enforces nothing about that — the operator
|
||||||
must just not select it for PHI.
|
must just not select it for PHI.
|
||||||
|
|
||||||
|
|
@ -183,27 +183,18 @@ Lines 478–514. On primary-provider error:
|
||||||
|
|
||||||
- Auth: standard AWS SDK credentials (env or instance profile). Region
|
- Auth: standard AWS SDK credentials (env or instance profile). Region
|
||||||
comes from `AWS_BEDROCK_REGION`.
|
comes from `AWS_BEDROCK_REGION`.
|
||||||
- Two API paths:
|
- Bedrock models use either native provider payloads or the unified Bedrock
|
||||||
- **Anthropic models** (line 223 — detected by `modelId.indexOf('anthropic.')`)
|
**Converse API**. For Converse requests, the system message becomes
|
||||||
use the native `InvokeModel` Messages API (lines 225–273). System
|
`system: [{ text: ... }]`; chat content is wrapped as `content: [{ text: ... }]`.
|
||||||
message is hoisted out of the messages array into the top-level
|
|
||||||
`system` field — Anthropic's Messages API takes it separately, not as
|
|
||||||
a `role: system` entry.
|
|
||||||
- **All other models** (Nova, Llama, Mistral, DeepSeek, Cohere, etc.) use
|
|
||||||
the unified Bedrock **Converse API** (lines 276–313). System message
|
|
||||||
becomes `system: [{ text: ... }]`; chat content is wrapped as
|
|
||||||
`content: [{ text: ... }]`.
|
|
||||||
- **Inference profile mapping.** `getBedrockModelId(model)` in
|
- **Inference profile mapping.** `getBedrockModelId(model)` in
|
||||||
`models.js` translates the friendly id (`anthropic/vendor-model-sonnet-4-6`)
|
`models.js` translates the friendly id into the AWS-required cross-region
|
||||||
into the AWS-required cross-region inference profile id
|
inference profile id. Most newer models require the inference profile;
|
||||||
(`us.anthropic.agent-config-sonnet-4-6`). Most newer models *require* the
|
using the raw foundation model id can fail.
|
||||||
inference profile; using the raw foundation model id will fail.
|
|
||||||
- **Output token clamping** (lines 209–210). `getBedrockMaxOut` returns the
|
- **Output token clamping** (lines 209–210). `getBedrockMaxOut` returns the
|
||||||
per-model output limit (e.g. Cohere is 4096) so we don't ask for 4000
|
per-model output limit (e.g. Cohere is 4096) so we don't ask for 4000
|
||||||
output and get a 400 from the SDK.
|
output and get a 400 from the SDK.
|
||||||
- **Multi-block content extraction** (lines 245–263). vendor model Opus 4.6+ may
|
- **Multi-block content extraction.** Some provider-native responses may return
|
||||||
return multiple content blocks (e.g. a thinking block + a text block).
|
multiple content blocks. We iterate, keep only `type === 'text'`, and concatenate.
|
||||||
We iterate, keep only `type === 'text'`, and concatenate.
|
|
||||||
- Usage is normalised: AWS's `input_tokens` / `output_tokens` →
|
- Usage is normalised: AWS's `input_tokens` / `output_tokens` →
|
||||||
`prompt_tokens` / `completion_tokens` to match the OpenAI shape.
|
`prompt_tokens` / `completion_tokens` to match the OpenAI shape.
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue