scrub vendor model docs
This commit is contained in:
parent
210ec06fe5
commit
aeb31a2d15
4 changed files with 15 additions and 24 deletions
|
|
@ -15,7 +15,7 @@ Provider is selected once at startup and is transparent to callers.
|
|||
|
||||
- SDK: `@aws-sdk/client-bedrock-runtime`.
|
||||
- Uses Bedrock **inference profiles** for newer models (cross-region routing).
|
||||
- Model families: vendor model (Anthropic), Amazon Nova, Llama (Meta), Mistral, DeepSeek, Cohere.
|
||||
- Model families: Amazon Nova, Llama (Meta), Mistral, DeepSeek, Cohere, and other Bedrock-hosted families.
|
||||
|
||||
### Azure OpenAI (BAA-eligible)
|
||||
|
||||
|
|
@ -27,7 +27,7 @@ Provider is selected once at startup and is transparent to callers.
|
|||
|
||||
- SDK: `@google-cloud/vertexai`.
|
||||
- Also serves STT (Gemini inline audio) and TTS (Vertex TTS endpoint).
|
||||
- Families: Gemini 2.5 / 2.0, vendor model on Vertex (Anthropic via GCP), Llama.
|
||||
- Families: Gemini 2.5 / 2.0 and Llama.
|
||||
|
||||
### LiteLLM proxy (self-hosted)
|
||||
|
||||
|
|
|
|||
|
|
@ -178,8 +178,8 @@ OpenAI-compatible gateway — LiteLLM, Bifrost, or other proxies.
|
|||
|
||||
3. **Update model names** — Different gateways use different naming
|
||||
conventions. Bifrost requires `provider/model` format
|
||||
(e.g., `openrouter/vendor-model-sonnet-4.6`), while LiteLLM uses aliases
|
||||
(e.g., `openrouter-vendor-model-sonnet-4.6`). Update model names in:
|
||||
(e.g., `openrouter/gpt-4.1`), while LiteLLM can use deployment aliases
|
||||
(e.g., `openrouter-gpt-4.1`). Update model names in:
|
||||
- Admin Panel → Models (chat models)
|
||||
- Admin Panel → Settings → `stt.model` (speech-to-text)
|
||||
- Admin Panel → Settings → `tts.model` (text-to-speech)
|
||||
|
|
|
|||
|
|
@ -542,11 +542,11 @@ const result = await callAI(
|
|||
|
||||
### Bedrock Model Notes
|
||||
|
||||
**Inference Profiles:** Most newer models (Anthropic vendor model 4.x, Meta Llama 4, DeepSeek R1, Amazon Nova, Writer) require cross-region inference profiles. These use a `us.` prefix on the model ID (e.g. `us.anthropic.agent-config-sonnet-4-6`). Direct model IDs will return "on-demand throughput not supported" errors.
|
||||
**Inference Profiles:** Most newer Bedrock models require cross-region inference profiles. These use a `us.` prefix on the model ID (for example, `us.amazon.nova-pro-v1:0`). Direct model IDs may return "on-demand throughput not supported" errors.
|
||||
|
||||
**Max Output Tokens:** Some models have low output limits (Cohere Command R/R+: 4096, AI21 Jamba: 4096). The `maxOut` field in `models.js` auto-clamps `maxTokens` in `callBedrock()`.
|
||||
|
||||
**JSON Sanitization:** Some models (notably vendor model Sonnet 4.6, Opus 4.6) output literal newline characters inside JSON string values. `learningAI.js` includes a `sanitizeJsonString()` function that escapes these before parsing.
|
||||
**JSON Sanitization:** Some models output literal newline characters inside JSON string values. `learningAI.js` includes a `sanitizeJsonString()` function that escapes these before parsing.
|
||||
|
||||
### Prompt System
|
||||
Prompts are defined in `src/utils/prompts.js`. Admins can override any prompt via the Admin panel (`/admin/config/prompts`). Overrides are stored in `app_settings` table and loaded into memory on startup (with 3s grace period for DB readiness).
|
||||
|
|
|
|||
|
|
@ -175,7 +175,7 @@ Lines 478–514. On primary-provider error:
|
|||
("Pediatric AI Scribe") — this is what shows up in the OpenRouter
|
||||
dashboard's app attribution.
|
||||
- Model IDs are passed straight through (`google/gemini-2.5-flash`,
|
||||
`anthropic/vendor-model-sonnet-4-6`, etc.).
|
||||
`openai/gpt-4.1`, etc.).
|
||||
- **Not BAA-eligible.** The system enforces nothing about that — the operator
|
||||
must just not select it for PHI.
|
||||
|
||||
|
|
@ -183,27 +183,18 @@ Lines 478–514. On primary-provider error:
|
|||
|
||||
- Auth: standard AWS SDK credentials (env or instance profile). Region
|
||||
comes from `AWS_BEDROCK_REGION`.
|
||||
- Two API paths:
|
||||
- **Anthropic models** (line 223 — detected by `modelId.indexOf('anthropic.')`)
|
||||
use the native `InvokeModel` Messages API (lines 225–273). System
|
||||
message is hoisted out of the messages array into the top-level
|
||||
`system` field — Anthropic's Messages API takes it separately, not as
|
||||
a `role: system` entry.
|
||||
- **All other models** (Nova, Llama, Mistral, DeepSeek, Cohere, etc.) use
|
||||
the unified Bedrock **Converse API** (lines 276–313). System message
|
||||
becomes `system: [{ text: ... }]`; chat content is wrapped as
|
||||
`content: [{ text: ... }]`.
|
||||
- Bedrock models use either native provider payloads or the unified Bedrock
|
||||
**Converse API**. For Converse requests, the system message becomes
|
||||
`system: [{ text: ... }]`; chat content is wrapped as `content: [{ text: ... }]`.
|
||||
- **Inference profile mapping.** `getBedrockModelId(model)` in
|
||||
`models.js` translates the friendly id (`anthropic/vendor-model-sonnet-4-6`)
|
||||
into the AWS-required cross-region inference profile id
|
||||
(`us.anthropic.agent-config-sonnet-4-6`). Most newer models *require* the
|
||||
inference profile; using the raw foundation model id will fail.
|
||||
`models.js` translates the friendly id into the AWS-required cross-region
|
||||
inference profile id. Most newer models require the inference profile;
|
||||
using the raw foundation model id can fail.
|
||||
- **Output token clamping** (lines 209–210). `getBedrockMaxOut` returns the
|
||||
per-model output limit (e.g. Cohere is 4096) so we don't ask for 4000
|
||||
output and get a 400 from the SDK.
|
||||
- **Multi-block content extraction** (lines 245–263). vendor model Opus 4.6+ may
|
||||
return multiple content blocks (e.g. a thinking block + a text block).
|
||||
We iterate, keep only `type === 'text'`, and concatenate.
|
||||
- **Multi-block content extraction.** Some provider-native responses may return
|
||||
multiple content blocks. We iterate, keep only `type === 'text'`, and concatenate.
|
||||
- Usage is normalised: AWS's `input_tokens` / `output_tokens` →
|
||||
`prompt_tokens` / `completion_tokens` to match the OpenAI shape.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue