Merge pull request #118 from btpod/btpod/speech-sdk-provider
feat(tts): add Speech SDK provider for ElevenLabs, Cartesia, Deepgram, Google, Inworld, and more
This commit is contained in:
Richard R
GitHub
commit
d216ecb713
15 changed files with 328 additions and 4 deletions
63
docs-site/docs/configure/tts-provider-guides/speech-sdk.md
Normal file
63
docs-site/docs/configure/tts-provider-guides/speech-sdk.md
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
---
|
||||
title: Speech SDK
|
||||
---
|
||||
|
||||
Use [speech-sdk](https://github.com/Jellypod-Inc/speech-sdk) (Apache 2.0) to reach additional cloud TTS providers (ElevenLabs, Cartesia, Hume, Deepgram, Google Gemini TTS, Inworld, and more) with your own provider API keys. Requests go from the OpenReader server directly to the provider's API; no extra account or proxy is involved.
|
||||
|
||||
Models use the `provider/model` format. The API key you enter belongs to the provider named by the model prefix: for `elevenlabs/eleven_multilingual_v2` enter an ElevenLabs key, for `cartesia/sonic-3.5` a Cartesia key, and so on.
|
||||
|
||||
## Setup
|
||||
|
||||
**Recommended (auth + admin): Settings → Admin → Shared providers**
|
||||
|
||||
1. Add a shared provider with type `speech-sdk`.
|
||||
2. Enter the API key for the provider you want to use.
|
||||
3. Set default model to a matching `provider/model` (for example `elevenlabs/eleven_multilingual_v2`).
|
||||
|
||||
**Per-user Settings → TTS Provider (only when `restrictUserApiKeys=false`):**
|
||||
|
||||
1. Set provider to `Speech SDK`.
|
||||
2. Choose a model; enter the API key for that model's provider.
|
||||
3. Choose a voice.
|
||||
|
||||
See [TTS Providers](../tts-providers) for admin-shared vs per-user behavior.
|
||||
|
||||
## Built-in models
|
||||
|
||||
- `openai/gpt-4o-mini-tts` (works with your existing OpenAI API key)
|
||||
- `elevenlabs/eleven_multilingual_v2`
|
||||
- `cartesia/sonic-3.5`
|
||||
- `deepgram/aura-2`
|
||||
- `google/gemini-2.5-flash-preview-tts`
|
||||
- `inworld/inworld-tts-1.5-max`
|
||||
|
||||
You can also choose `Other` and enter any `provider/model` the SDK supports. Recognized prefixes: `openai`, `elevenlabs`, `cartesia`, `hume`, `deepgram`, `google`, `inworld`, `minimax`, `fish-audio`, `murf`, `resemble`, `fal-ai`, `mistral`, `xai`, `smallest-ai`.
|
||||
|
||||
## Voice IDs
|
||||
|
||||
ElevenLabs and Cartesia identify voices by opaque IDs. The built-in lists map to these shared library voices:
|
||||
|
||||
| ElevenLabs ID | Name | | Cartesia ID | Name |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `JBFqnCBsd6RMkjVDRZzb` | George | | `a0e99841-438c-4a64-b679-ae501e7d6091` | Barbershop Man |
|
||||
| `IKne3meq5aSn9XLyUdCD` | Charlie | | `156fb8d2-335b-4950-9cb3-a2d33f0c0c2a` | British Lady |
|
||||
| `XB0fDUnXU5powFXDhCwa` | Charlotte | | `694f9389-aac1-45b6-b726-9d9369183238` | California Girl |
|
||||
| `Xb7hH8MSUJpSbSDYk0k2` | Alice | | `87748186-23bb-4571-8b8b-a73da9bf9c4f` | Commercial Lady |
|
||||
| `iP95p4xoKVk53GoZ742B` | Chris | | `ee7ea9f8-c0c1-498c-9f62-dc2da49a6f98` | Friendly Reading Man |
|
||||
| `nPczCjzI2devNBz1zQrb` | Brian | | `248be419-c632-4f23-adf1-5324ed7dbf1d` | Hannah |
|
||||
| `onwK4e9ZLuTAKqWW03F9` | Daniel | | | |
|
||||
| `pFZP5JQG7iQjIQuC4Bku` | Lily | | | |
|
||||
| `pqHfZKP75CvOlQylNhV4` | Bill | | | |
|
||||
|
||||
## Notes
|
||||
|
||||
- One voice per request; Kokoro-style multi-voice mixing does not apply to this provider.
|
||||
- Playback speed is applied client-side, so cached audio segments stay valid when you change speed.
|
||||
- Providers without a built-in voice list fall back to a `default` entry, which lets the provider pick its default voice.
|
||||
- Word-by-word highlighting works the same as with every other provider (alignment runs in OpenReader, not the provider).
|
||||
- TTS requests are sent from the server, not the browser. The API key is never exposed to clients.
|
||||
|
||||
## References
|
||||
|
||||
- [speech-sdk on GitHub](https://github.com/Jellypod-Inc/speech-sdk)
|
||||
- [TTS Providers](../tts-providers)
|
||||
|
|
@ -19,6 +19,7 @@ If you're running a private/self-hosted instance and want per-user BYOK behavior
|
|||
- **OpenAI**: Cloud. Base URL pre-filled (`https://api.openai.com/v1`). API key required.
|
||||
- **Replicate**: Cloud. Base URL managed internally by OpenReader. API key required.
|
||||
- **DeepInfra**: Cloud. Base URL pre-filled (`https://api.deepinfra.com/v1/openai`). API key required.
|
||||
- **Speech SDK**: Cloud. Reaches additional providers (ElevenLabs, Cartesia, Hume, Deepgram, Google, Inworld, and more) directly with your own provider API keys via [speech-sdk](./tts-provider-guides/speech-sdk). No base URL. API key required (the key for the model's provider).
|
||||
- **Custom OpenAI-Like**: Self-hosted or any custom endpoint. `API_BASE` must be set manually (typically ending in `/v1`). API key optional.
|
||||
|
||||
For `OpenAI`, `DeepInfra`, and `Replicate` you only need to supply an API key. For `Custom OpenAI-Like` you must also set `API_BASE`.
|
||||
|
|
@ -28,6 +29,7 @@ For `OpenAI`, `DeepInfra`, and `Replicate` you only need to supply an API key. F
|
|||
- **Replicate** models: `alphanumericuser/kokoro-82m`, `google/gemini-3.1-flash-tts`, `minimax/speech-2.8-turbo`, `qwen/qwen3-tts`, `inworld/tts-1.5-mini` (or choose `Other` and enter any Replicate model ID, such as `owner/model` or `owner/model:version`)
|
||||
- **OpenAI** models: `tts-1`, `tts-1-hd`, `gpt-4o-mini-tts`
|
||||
- **DeepInfra** models: includes `hexgrad/Kokoro-82M` and additional hosted models (depending on API key / feature flags)
|
||||
- **Speech SDK** models: `openai/gpt-4o-mini-tts`, `elevenlabs/eleven_multilingual_v2`, `cartesia/sonic-3.5`, `deepgram/aura-2`, `google/gemini-2.5-flash-preview-tts`, `inworld/inworld-tts-1.5-max` (or choose `Other` and enter any `provider/model` the SDK supports)
|
||||
|
||||
## Custom provider requirements
|
||||
|
||||
|
|
@ -51,6 +53,7 @@ TTS requests originate from the **Next.js server**, not the browser. `API_BASE`
|
|||
- [Replicate](./tts-provider-guides/replicate)
|
||||
- [DeepInfra](./tts-provider-guides/deepinfra)
|
||||
- [OpenAI](./tts-provider-guides/openai)
|
||||
- [Speech SDK](./tts-provider-guides/speech-sdk)
|
||||
- [Other](./tts-provider-guides/other)
|
||||
|
||||
## Related
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ const sidebars: SidebarsConfig = {
|
|||
'configure/tts-provider-guides/replicate',
|
||||
'configure/tts-provider-guides/deepinfra',
|
||||
'configure/tts-provider-guides/openai',
|
||||
'configure/tts-provider-guides/speech-sdk',
|
||||
'configure/tts-provider-guides/other',
|
||||
],
|
||||
},
|
||||
|
|
|
|||
|
|
@ -38,6 +38,7 @@
|
|||
"@microsoft/antissrf": "^1.0.0",
|
||||
"@mozilla/readability": "^0.6.0",
|
||||
"@napi-rs/canvas": "^0.1.100",
|
||||
"@speech-sdk/core": "^0.15.0",
|
||||
"@tanstack/react-query": "^5.101.0",
|
||||
"@types/archiver": "^7.0.0",
|
||||
"@vercel/analytics": "^1.6.1",
|
||||
|
|
|
|||
|
|
@ -32,6 +32,9 @@ importers:
|
|||
'@napi-rs/canvas':
|
||||
specifier: ^0.1.100
|
||||
version: 0.1.100
|
||||
'@speech-sdk/core':
|
||||
specifier: ^0.15.0
|
||||
version: 0.15.0
|
||||
'@tanstack/react-query':
|
||||
specifier: ^5.101.0
|
||||
version: 5.101.0(react@19.2.7)
|
||||
|
|
@ -1258,6 +1261,11 @@ packages:
|
|||
'@jridgewell/trace-mapping@0.3.31':
|
||||
resolution: {integrity: sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==}
|
||||
|
||||
'@mediabunny/mp3-encoder@1.46.0':
|
||||
resolution: {integrity: sha512-SG69nm0ntRR4yc5TP9LG1/oxrh5PcoIEZwFm8BXecj3xNF0V+PlnpyJ1IQVS/JEXRIOvBJQ09YFknVGnJNAJrQ==}
|
||||
peerDependencies:
|
||||
mediabunny: ^1.0.0
|
||||
|
||||
'@microsoft/antissrf@1.0.0':
|
||||
resolution: {integrity: sha512-AmO1ykSha52Ef/7UXvHgu8ElsGdgcLkF5nTl7YZGyR1AkbmlQYtiVeWp+CsjkFaJzKAH5ofXLZveMmOHwX/7Jw==}
|
||||
|
||||
|
|
@ -1640,6 +1648,9 @@ packages:
|
|||
resolution: {integrity: sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A==}
|
||||
engines: {node: '>=14.0.0'}
|
||||
|
||||
'@speech-sdk/core@0.15.0':
|
||||
resolution: {integrity: sha512-GHM6Cno7PuU7nEtl6rRZVb+xEFJKZ1AB8WvdeCTIImEtzuWWaztLK3CG4h9KLXd8EQJHEbwVfcZh+CeAJBQrpw==}
|
||||
|
||||
'@standard-schema/spec@1.1.0':
|
||||
resolution: {integrity: sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==}
|
||||
|
||||
|
|
@ -1689,6 +1700,12 @@ packages:
|
|||
'@types/deep-eql@4.0.2':
|
||||
resolution: {integrity: sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==}
|
||||
|
||||
'@types/dom-mediacapture-transform@0.1.11':
|
||||
resolution: {integrity: sha512-Y2p+nGf1bF2XMttBnsVPHUWzRRZzqUoJAKmiP10b5umnO6DDrWI0BrGDJy1pOHoOULVmGSfFNkQrAlC5dcj6nQ==}
|
||||
|
||||
'@types/dom-webcodecs@0.1.13':
|
||||
resolution: {integrity: sha512-O5hkiFIcjjszPIYyUSyvScyvrBoV3NOEEZx/pMlsu44TKzWNkLVBBxnxJz42in5n3QIolYOcBYFCPZZ0h8SkwQ==}
|
||||
|
||||
'@types/estree-jsx@1.0.5':
|
||||
resolution: {integrity: sha512-52CcUVNFyfb1A2ALocQw/Dd1BQFNmSdkuC3BkZ6iqhdMfQz7JWOFRuJFloOzjk+6WijU56m9oKXFAXc7o3Towg==}
|
||||
|
||||
|
|
@ -3313,6 +3330,10 @@ packages:
|
|||
resolution: {integrity: sha512-5KoIu2Ngpyek75jXodFvnafB6DJgr3u8uuK0LEZJjrU19DrMD3EVERaR8sjz8CCGgpZvxPl9SuE1GMVPFHx1mw==}
|
||||
engines: {node: '>= 0.4'}
|
||||
|
||||
is-network-error@1.3.2:
|
||||
resolution: {integrity: sha512-PhBY86zaxNZUuWP6h13Vu5oFe0XY6/UlKzQnYFELzGVHygP3MxmvTfYSG7GN3aIab/iWudSMgjSnG9Dq+nHrgA==}
|
||||
engines: {node: '>=16'}
|
||||
|
||||
is-number-object@1.1.1:
|
||||
resolution: {integrity: sha512-lZhclumE1G6VYD8VHe35wFaIif+CTy5SJIi5+3y4psDgWu4wPDoBhF8NxUOinEc7pHgiTsT6MaBb92rKhhD+Xw==}
|
||||
engines: {node: '>= 0.4'}
|
||||
|
|
@ -3641,6 +3662,9 @@ packages:
|
|||
mdast-util-to-string@4.0.0:
|
||||
resolution: {integrity: sha512-0H44vDimn51F0YwvxSJSm0eCDOJTRlmN0R1yBh4HLj9wiV1Dn0QoXGbvFAWj2hSItVTlCmBF1hqKlIyUBVFLPg==}
|
||||
|
||||
mediabunny@1.46.0:
|
||||
resolution: {integrity: sha512-HIaMQ6PVMndrFb9sNCh9/uAsLFBfLzyAgbk3kN7ZqlRRq//9RPYqIOQtXYpkbWBZU8ZJkaAUNLDtwTcdKo7yQw==}
|
||||
|
||||
merge-refs@1.3.0:
|
||||
resolution: {integrity: sha512-nqXPXbso+1dcKDpPCXvwZyJILz+vSLqGGOnDrYHQYE+B8n9JTCekVLC65AfCpR4ggVyA/45Y0iR9LDyS2iI+zA==}
|
||||
peerDependencies:
|
||||
|
|
@ -3919,6 +3943,10 @@ packages:
|
|||
resolution: {integrity: sha512-LaNjtRWUBY++zB5nE/NwcaoMylSPk+S+ZHNB1TzdbMJMny6dynpAGt7X/tl/QYq3TIeE6nxHppbo2LGymrG5Pw==}
|
||||
engines: {node: '>=10'}
|
||||
|
||||
p-retry@8.0.0:
|
||||
resolution: {integrity: sha512-kFVqH1HxOHp8LupNsOys7bSV09VYTRLxarH/mokO4Rqhk6wGi70E0jh4VzvVGXfEVNggHoHLAMWsQqHyU1Ey9A==}
|
||||
engines: {node: '>=22'}
|
||||
|
||||
package-json-from-dist@1.0.1:
|
||||
resolution: {integrity: sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw==}
|
||||
|
||||
|
|
@ -5762,6 +5790,10 @@ snapshots:
|
|||
'@jridgewell/resolve-uri': 3.1.2
|
||||
'@jridgewell/sourcemap-codec': 1.5.5
|
||||
|
||||
'@mediabunny/mp3-encoder@1.46.0(mediabunny@1.46.0)':
|
||||
dependencies:
|
||||
mediabunny: 1.46.0
|
||||
|
||||
'@microsoft/antissrf@1.0.0': {}
|
||||
|
||||
'@mixmark-io/domino@2.2.0': {}
|
||||
|
|
@ -6039,6 +6071,13 @@ snapshots:
|
|||
'@smithy/util-buffer-from': 2.2.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@speech-sdk/core@0.15.0':
|
||||
dependencies:
|
||||
'@mediabunny/mp3-encoder': 1.46.0(mediabunny@1.46.0)
|
||||
mediabunny: 1.46.0
|
||||
p-retry: 8.0.0
|
||||
zod: 4.4.3
|
||||
|
||||
'@standard-schema/spec@1.1.0': {}
|
||||
|
||||
'@swc/helpers@0.5.15':
|
||||
|
|
@ -6093,6 +6132,12 @@ snapshots:
|
|||
|
||||
'@types/deep-eql@4.0.2': {}
|
||||
|
||||
'@types/dom-mediacapture-transform@0.1.11':
|
||||
dependencies:
|
||||
'@types/dom-webcodecs': 0.1.13
|
||||
|
||||
'@types/dom-webcodecs@0.1.13': {}
|
||||
|
||||
'@types/estree-jsx@1.0.5':
|
||||
dependencies:
|
||||
'@types/estree': 1.0.9
|
||||
|
|
@ -7848,6 +7893,8 @@ snapshots:
|
|||
|
||||
is-negative-zero@2.0.3: {}
|
||||
|
||||
is-network-error@1.3.2: {}
|
||||
|
||||
is-number-object@1.1.1:
|
||||
dependencies:
|
||||
call-bound: 1.0.4
|
||||
|
|
@ -8252,6 +8299,11 @@ snapshots:
|
|||
dependencies:
|
||||
'@types/mdast': 4.0.4
|
||||
|
||||
mediabunny@1.46.0:
|
||||
dependencies:
|
||||
'@types/dom-mediacapture-transform': 0.1.11
|
||||
'@types/dom-webcodecs': 0.1.13
|
||||
|
||||
merge-refs@1.3.0(@types/react@19.2.16):
|
||||
optionalDependencies:
|
||||
'@types/react': 19.2.16
|
||||
|
|
@ -8629,6 +8681,10 @@ snapshots:
|
|||
dependencies:
|
||||
p-limit: 3.1.0
|
||||
|
||||
p-retry@8.0.0:
|
||||
dependencies:
|
||||
is-network-error: 1.3.2
|
||||
|
||||
package-json-from-dist@1.0.1: {}
|
||||
|
||||
pako@1.0.11: {}
|
||||
|
|
|
|||
|
|
@ -550,6 +550,7 @@ export function SettingsModal({
|
|||
&& !isSharedSelected
|
||||
&& providerModelPolicy.isResolvedProviderType
|
||||
&& providerModelPolicy.providerType !== 'replicate'
|
||||
&& providerModelPolicy.providerType !== 'speech-sdk'
|
||||
&& (providerModelPolicy.providerType === 'custom-openai' || !localBaseUrl || localBaseUrl === '');
|
||||
const shouldShowApiKey = !restrictUserApiKeys && !isSharedSelected;
|
||||
const selectedModel = ttsModels.find(m => m.id === selectedModelId) || ttsModels[0];
|
||||
|
|
|
|||
|
|
@ -44,6 +44,7 @@ const PROVIDER_TYPE_OPTIONS: { value: ProviderType; label: string }[] = [
|
|||
{ value: 'openai', label: 'OpenAI' },
|
||||
{ value: 'deepinfra', label: 'Deepinfra' },
|
||||
{ value: 'replicate', label: 'Replicate' },
|
||||
{ value: 'speech-sdk', label: 'Speech SDK' },
|
||||
];
|
||||
|
||||
interface FormState {
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@ export const BUILT_IN_PROVIDER_IDS: readonly TtsProviderId[] = [
|
|||
'replicate',
|
||||
'deepinfra',
|
||||
'openai',
|
||||
'speech-sdk',
|
||||
];
|
||||
|
||||
const SLUG_PATTERN = /^[a-z0-9][a-z0-9-]{0,62}[a-z0-9]$|^[a-z0-9]$/;
|
||||
|
|
|
|||
|
|
@ -89,7 +89,7 @@ export async function resolveTtsCredentials(opts: {
|
|||
};
|
||||
}
|
||||
|
||||
const BUILT_IN_IDS = new Set(['custom-openai', 'replicate', 'deepinfra', 'openai']);
|
||||
const BUILT_IN_IDS = new Set(['custom-openai', 'replicate', 'deepinfra', 'openai', 'speech-sdk']);
|
||||
|
||||
function isBuiltInProviderId(value: string): boolean {
|
||||
return BUILT_IN_IDS.has(value);
|
||||
|
|
|
|||
|
|
@ -1,9 +1,29 @@
|
|||
import OpenAI from 'openai';
|
||||
import Replicate from 'replicate';
|
||||
import { SpeechCreateParams } from 'openai/resources/audio/speech.mjs';
|
||||
import { generateSpeech } from '@speech-sdk/core';
|
||||
import type { ResolvedModel } from '@speech-sdk/core/types';
|
||||
import {
|
||||
createCartesia,
|
||||
createDeepgram,
|
||||
createElevenLabs,
|
||||
createFal,
|
||||
createFishAudio,
|
||||
createGoogle,
|
||||
createHume,
|
||||
createInworld,
|
||||
createMiniMax,
|
||||
createMistral,
|
||||
createMurf,
|
||||
createOpenAI,
|
||||
createResemble,
|
||||
createSmallestAI,
|
||||
createXai,
|
||||
} from '@speech-sdk/core/providers';
|
||||
import {
|
||||
isBuiltInTtsProviderId,
|
||||
REPLICATE_KOKORO_82M_VERSIONED_MODEL,
|
||||
speechSdkProviderPrefix,
|
||||
} from '@/lib/shared/tts-provider-catalog';
|
||||
import { resolveTtsProviderModelPolicy } from '@/lib/shared/tts-provider-policy';
|
||||
import {
|
||||
|
|
@ -333,6 +353,7 @@ function resolveTTSRequest(input: ServerTTSRequest): ResolvedServerTTSRequest {
|
|||
const providerType = isBuiltInTtsProviderId(provider) ? provider : 'openai';
|
||||
const rawModel = provider === 'deepinfra' && !input.model ? 'hexgrad/Kokoro-82M'
|
||||
: provider === 'replicate' && !input.model ? REPLICATE_KOKORO_82M_VERSIONED_MODEL
|
||||
: provider === 'speech-sdk' && !input.model ? 'openai/gpt-4o-mini-tts'
|
||||
: input.model;
|
||||
const model = (rawModel ?? 'gpt-4o-mini-tts') as SpeechCreateParams['model'];
|
||||
const providerModelPolicy = resolveTtsProviderModelPolicy({
|
||||
|
|
@ -342,7 +363,7 @@ function resolveTTSRequest(input: ServerTTSRequest): ResolvedServerTTSRequest {
|
|||
});
|
||||
|
||||
const normalizedVoice = (
|
||||
(providerType === 'replicate' || !providerModelPolicy.isKokoroModel) && input.voice.includes('+')
|
||||
(providerType === 'replicate' || providerType === 'speech-sdk' || !providerModelPolicy.isKokoroModel) && input.voice.includes('+')
|
||||
? input.voice.split('+')[0].trim()
|
||||
: input.voice
|
||||
) as string;
|
||||
|
|
@ -652,6 +673,71 @@ async function runReplicateRequest(
|
|||
});
|
||||
}
|
||||
|
||||
type SpeechSdkModelFactory = (config: { apiKey?: string }) => (modelId?: string) => ResolvedModel;
|
||||
|
||||
const SPEECH_SDK_PROVIDER_FACTORIES: Record<string, SpeechSdkModelFactory> = {
|
||||
openai: createOpenAI,
|
||||
elevenlabs: createElevenLabs,
|
||||
cartesia: createCartesia,
|
||||
hume: createHume,
|
||||
deepgram: createDeepgram,
|
||||
google: createGoogle,
|
||||
inworld: createInworld,
|
||||
minimax: createMiniMax,
|
||||
'fish-audio': createFishAudio,
|
||||
murf: createMurf,
|
||||
resemble: createResemble,
|
||||
'fal-ai': createFal,
|
||||
mistral: createMistral,
|
||||
xai: createXai,
|
||||
'smallest-ai': createSmallestAI,
|
||||
};
|
||||
|
||||
async function runSpeechSdkRequest(
|
||||
request: ResolvedServerTTSRequest,
|
||||
signal: AbortSignal,
|
||||
upstreamSettings: ResolvedTtsUpstreamRuntimeSettings,
|
||||
): Promise<Buffer> {
|
||||
const model = request.model as string;
|
||||
const prefix = speechSdkProviderPrefix(model);
|
||||
const modelId = model.slice(prefix.length + 1);
|
||||
if (!prefix || !modelId) {
|
||||
throw new Error(`Invalid Speech SDK model "${model}". Expected "provider/model".`);
|
||||
}
|
||||
|
||||
const factory = SPEECH_SDK_PROVIDER_FACTORIES[prefix];
|
||||
if (!factory) {
|
||||
throw new Error(
|
||||
`Unknown Speech SDK provider prefix "${prefix}". Use "provider/model" with one of: ${Object.keys(SPEECH_SDK_PROVIDER_FACTORIES).join(', ')}.`
|
||||
);
|
||||
}
|
||||
|
||||
// 'default' is the placeholder voice for providers without a static voice
|
||||
// list; omit it so the provider's own default applies.
|
||||
const voice = request.voice === 'default' ? undefined : request.voice;
|
||||
|
||||
// Overall budget across the SDK's internal retries, mirroring the timeout
|
||||
// the OpenAI-compatible path configures on its client.
|
||||
const timeoutController = new AbortController();
|
||||
const timeoutId = setTimeout(() => timeoutController.abort(), upstreamSettings.ttsUpstreamTimeoutMs);
|
||||
const onAbort = () => timeoutController.abort();
|
||||
signal.addEventListener('abort', onAbort, { once: true });
|
||||
try {
|
||||
const result = await generateSpeech({
|
||||
model: factory({ apiKey: request.apiKey || undefined })(modelId),
|
||||
text: request.text,
|
||||
voice: voice as string,
|
||||
output: { format: 'mp3' },
|
||||
maxRetries: upstreamSettings.ttsUpstreamMaxRetries,
|
||||
abortSignal: timeoutController.signal,
|
||||
});
|
||||
return Buffer.from(result.audio.uint8Array);
|
||||
} finally {
|
||||
clearTimeout(timeoutId);
|
||||
signal.removeEventListener('abort', onAbort);
|
||||
}
|
||||
}
|
||||
|
||||
async function runProviderRequest(
|
||||
request: ResolvedServerTTSRequest,
|
||||
signal: AbortSignal,
|
||||
|
|
@ -662,7 +748,9 @@ async function runProviderRequest(
|
|||
|
||||
const raw = request.provider === 'replicate'
|
||||
? await runReplicateRequest(request, signal, upstreamSettings.ttsUpstreamMaxRetries)
|
||||
: await runOpenAiCompatibleRequest(request, signal, upstreamSettings);
|
||||
: request.provider === 'speech-sdk'
|
||||
? await runSpeechSdkRequest(request, signal, upstreamSettings)
|
||||
: await runOpenAiCompatibleRequest(request, signal, upstreamSettings);
|
||||
|
||||
// OpenAI-compatible servers (and some Replicate models) may emit wav/ogg/etc.;
|
||||
// normalize to mp3 so the cache, storage, and audiobook pipeline stay mp3-only.
|
||||
|
|
|
|||
|
|
@ -59,6 +59,7 @@ export const getMaxVoicesForProvider = (provider: TtsProviderId, model: string):
|
|||
// Replicate must always stay single-voice, even for Kokoro models.
|
||||
if (provider === 'replicate') return 1;
|
||||
if (provider === 'deepinfra') return 1;
|
||||
if (provider === 'speech-sdk') return 1;
|
||||
|
||||
// Other providers with Kokoro support unlimited voices
|
||||
return Infinity;
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import { isKokoroModel } from '@/lib/shared/kokoro';
|
||||
|
||||
export type TtsProviderId = 'custom-openai' | 'replicate' | 'deepinfra' | 'openai';
|
||||
export type TtsProviderId = 'custom-openai' | 'replicate' | 'deepinfra' | 'openai' | 'speech-sdk';
|
||||
export type TtsProviderType = TtsProviderId | 'unknown';
|
||||
export type TtsVoiceSource = 'static' | 'deepinfra-api' | 'custom-openai-api' | 'replicate-api';
|
||||
export type ReplicateVoiceInputKey = 'voice' | 'voice_id' | 'speaker';
|
||||
|
|
@ -72,6 +72,16 @@ const REPLICATE_MODELS: TtsModelDefinition[] = [
|
|||
{ id: 'inworld/tts-1.5-mini', name: 'inworld/tts-1.5-mini' },
|
||||
{ id: 'custom', name: 'Other' },
|
||||
];
|
||||
|
||||
const SPEECH_SDK_MODELS: TtsModelDefinition[] = [
|
||||
{ id: 'openai/gpt-4o-mini-tts', name: 'openai/gpt-4o-mini-tts' },
|
||||
{ id: 'elevenlabs/eleven_multilingual_v2', name: 'elevenlabs/eleven_multilingual_v2' },
|
||||
{ id: 'cartesia/sonic-3.5', name: 'cartesia/sonic-3.5' },
|
||||
{ id: 'deepgram/aura-2', name: 'deepgram/aura-2' },
|
||||
{ id: 'google/gemini-2.5-flash-preview-tts', name: 'google/gemini-2.5-flash-preview-tts' },
|
||||
{ id: 'inworld/inworld-tts-1.5-max', name: 'inworld/inworld-tts-1.5-max' },
|
||||
{ id: 'custom', name: 'Other' },
|
||||
];
|
||||
const DEEPINFRA_API_VOICE_MODELS = new Set([
|
||||
'ResembleAI/chatterbox',
|
||||
'Zyphra/Zonos-v0.1-hybrid',
|
||||
|
|
@ -110,6 +120,40 @@ export const MINIMAX_SPEECH_VOICES = [
|
|||
] as const;
|
||||
export const QWEN3_TTS_VOICES = ['Aiden', 'Dylan'] as const;
|
||||
export const INWORLD_TTS_VOICES = ['Ashley', 'Dennis', 'Alex', 'Darlene'] as const;
|
||||
|
||||
// Speech SDK direct-provider voices. ElevenLabs and Cartesia identify voices by
|
||||
// opaque IDs; the listed ones are stable, publicly shared library voices.
|
||||
export const ELEVENLABS_DEFAULT_VOICES = [
|
||||
'JBFqnCBsd6RMkjVDRZzb', 'IKne3meq5aSn9XLyUdCD', 'XB0fDUnXU5powFXDhCwa',
|
||||
'Xb7hH8MSUJpSbSDYk0k2', 'iP95p4xoKVk53GoZ742B', 'nPczCjzI2devNBz1zQrb',
|
||||
'onwK4e9ZLuTAKqWW03F9', 'pFZP5JQG7iQjIQuC4Bku', 'pqHfZKP75CvOlQylNhV4',
|
||||
] as const;
|
||||
export const CARTESIA_DEFAULT_VOICES = [
|
||||
'a0e99841-438c-4a64-b679-ae501e7d6091', '156fb8d2-335b-4950-9cb3-a2d33f0c0c2a',
|
||||
'694f9389-aac1-45b6-b726-9d9369183238', '87748186-23bb-4571-8b8b-a73da9bf9c4f',
|
||||
'ee7ea9f8-c0c1-498c-9f62-dc2da49a6f98', '248be419-c632-4f23-adf1-5324ed7dbf1d',
|
||||
] as const;
|
||||
export const HUME_DEFAULT_VOICES = ['KORA', 'STELLA', 'DACHER'] as const;
|
||||
export const DEEPGRAM_AURA2_DEFAULT_VOICES = [
|
||||
'thalia-en', 'andromeda-en', 'helena-en', 'apollo-en', 'arcas-en',
|
||||
'aries-en', 'asteria-en', 'athena-en', 'atlas-en', 'aurora-en',
|
||||
] as const;
|
||||
export const INWORLD_DIRECT_DEFAULT_VOICES = ['Ashley', 'Dominus', 'Edward', 'Hades', 'Priya'] as const;
|
||||
const SPEECH_SDK_DEFAULT_VOICES_BY_PREFIX: Record<string, readonly string[]> = {
|
||||
openai: GPT4O_MINI_DEFAULT_VOICES,
|
||||
elevenlabs: ELEVENLABS_DEFAULT_VOICES,
|
||||
cartesia: CARTESIA_DEFAULT_VOICES,
|
||||
hume: HUME_DEFAULT_VOICES,
|
||||
deepgram: DEEPGRAM_AURA2_DEFAULT_VOICES,
|
||||
google: GEMINI_FLASH_TTS_VOICES,
|
||||
inworld: INWORLD_DIRECT_DEFAULT_VOICES,
|
||||
minimax: MINIMAX_SPEECH_VOICES,
|
||||
};
|
||||
|
||||
export function speechSdkProviderPrefix(model: string): string {
|
||||
const slashIndex = model.indexOf('/');
|
||||
return slashIndex === -1 ? model : model.slice(0, slashIndex);
|
||||
}
|
||||
const REPLICATE_DEFAULT_VOICES_BY_MODEL: Record<string, readonly string[]> = {
|
||||
[REPLICATE_KOKORO_82M_VERSIONED_MODEL]: KOKORO_DEFAULT_VOICES,
|
||||
'google/gemini-3.1-flash-tts': GEMINI_FLASH_TTS_VOICES,
|
||||
|
|
@ -151,6 +195,12 @@ export const TTS_PROVIDER_DEFINITIONS: TtsProviderDefinition[] = [
|
|||
supportsCustomModel: false,
|
||||
models: () => OPENAI_MODELS,
|
||||
},
|
||||
{
|
||||
id: 'speech-sdk',
|
||||
name: 'Speech SDK',
|
||||
supportsCustomModel: true,
|
||||
models: () => SPEECH_SDK_MODELS,
|
||||
},
|
||||
];
|
||||
|
||||
const BUILT_IN_PROVIDER_ID_SET: ReadonlySet<TtsProviderId> = new Set(
|
||||
|
|
@ -209,6 +259,12 @@ export function supportsNativeModelSpeed(provider: TtsProviderId, model: string
|
|||
return !REPLICATE_MODELS_WITHOUT_NATIVE_SPEED.has(model);
|
||||
}
|
||||
|
||||
// Speed support varies per underlying provider; play back at the requested
|
||||
// rate client-side so cached audio stays reusable across speed changes.
|
||||
if (provider === 'speech-sdk') {
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
|
|
@ -247,6 +303,11 @@ export function getDefaultVoices(provider: TtsProviderId, model: string): string
|
|||
: [...CUSTOM_OPENAI_DEFAULT_VOICES];
|
||||
}
|
||||
|
||||
if (provider === 'speech-sdk') {
|
||||
const prefixVoices = SPEECH_SDK_DEFAULT_VOICES_BY_PREFIX[speechSdkProviderPrefix(model)];
|
||||
return prefixVoices ? [...prefixVoices] : ['default'];
|
||||
}
|
||||
|
||||
return [...OPENAI_DEFAULT_VOICES];
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -61,6 +61,7 @@ export function defaultModelForProviderType(providerType: TtsProviderId): string
|
|||
if (providerType === 'openai') return 'tts-1';
|
||||
if (providerType === 'deepinfra') return 'hexgrad/Kokoro-82M';
|
||||
if (providerType === 'replicate') return REPLICATE_KOKORO_82M_VERSIONED_MODEL;
|
||||
if (providerType === 'speech-sdk') return 'openai/gpt-4o-mini-tts';
|
||||
return 'kokoro';
|
||||
}
|
||||
|
||||
|
|
@ -73,6 +74,7 @@ export function defaultBaseUrlForProviderType(providerType: TtsProviderId): stri
|
|||
export function defaultVoiceForProviderType(providerType: TtsProviderId): string {
|
||||
if (providerType === 'openai') return 'alloy';
|
||||
if (providerType === 'deepinfra') return 'af_bella';
|
||||
if (providerType === 'speech-sdk') return 'alloy';
|
||||
return 'af_sarah';
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import {
|
|||
buildReplicateInput,
|
||||
buildTTSCacheKey,
|
||||
extractReplicateAudioUrl,
|
||||
generateTTSBuffer,
|
||||
resolveReplicateLanguageValue,
|
||||
} from '../../src/lib/server/tts/generate';
|
||||
import { REPLICATE_KOKORO_82M_VERSIONED_MODEL } from '../../src/lib/shared/tts-provider-catalog';
|
||||
|
|
@ -136,3 +137,29 @@ describe('Replicate language schema values', () => {
|
|||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('speech-sdk request resolution', () => {
|
||||
test('rejects unknown provider prefixes before any network call', async () => {
|
||||
await expect(generateTTSBuffer({
|
||||
text: 'hello',
|
||||
voice: 'alloy',
|
||||
speed: 1,
|
||||
model: 'doesnotexist/some-model',
|
||||
provider: 'speech-sdk',
|
||||
apiKey: 'unused',
|
||||
})).rejects.toThrow('Unknown Speech SDK provider prefix');
|
||||
});
|
||||
|
||||
test('rejects models missing the provider/model form', async () => {
|
||||
for (const model of ['openai', 'openai/', '/gpt-4o-mini-tts']) {
|
||||
await expect(generateTTSBuffer({
|
||||
text: 'hello',
|
||||
voice: 'alloy',
|
||||
speed: 1,
|
||||
model,
|
||||
provider: 'speech-sdk',
|
||||
apiKey: 'unused',
|
||||
})).rejects.toThrow('Expected "provider/model"');
|
||||
}
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import {
|
|||
getDefaultVoices,
|
||||
providerSupportsCustomModel,
|
||||
resolveProviderModels,
|
||||
speechSdkProviderPrefix,
|
||||
supportsNativeModelSpeed,
|
||||
supportsTtsInstructions,
|
||||
} from '../../src/lib/shared/tts-provider-catalog';
|
||||
|
|
@ -51,6 +52,23 @@ describe('tts provider catalog', () => {
|
|||
expect(providerSupportsCustomModel('deepinfra')).toBe(true);
|
||||
});
|
||||
|
||||
test('resolves speech-sdk models, voices, and prefix parsing', () => {
|
||||
expect(resolveProviderModels('speech-sdk').map((model) => model.id)).toContain('openai/gpt-4o-mini-tts');
|
||||
expect(providerSupportsCustomModel('speech-sdk')).toBe(true);
|
||||
|
||||
expect(speechSdkProviderPrefix('elevenlabs/eleven_multilingual_v2')).toBe('elevenlabs');
|
||||
// fal model ids are path-style, so only the first slash delimits the provider.
|
||||
expect(speechSdkProviderPrefix('fal-ai/kokoro/american-english')).toBe('fal-ai');
|
||||
|
||||
expect(getDefaultVoices('speech-sdk', 'openai/gpt-4o-mini-tts')).toContain('sage');
|
||||
expect(getDefaultVoices('speech-sdk', 'deepgram/aura-2')).toContain('thalia-en');
|
||||
// Providers without a static voice list fall back to the provider default.
|
||||
expect(getDefaultVoices('speech-sdk', 'mistral/voxtral-mini-tts-2603')).toEqual(['default']);
|
||||
|
||||
// Speed is applied client-side for speech-sdk so cached audio is speed-independent.
|
||||
expect(supportsNativeModelSpeed('speech-sdk', 'openai/gpt-4o-mini-tts')).toBe(false);
|
||||
});
|
||||
|
||||
test('uses blocklist semantics for Replicate native speed support', () => {
|
||||
expect(supportsNativeModelSpeed('replicate', REPLICATE_KOKORO_82M_VERSIONED_MODEL)).toBe(true);
|
||||
expect(supportsNativeModelSpeed('replicate', 'acme/runtime-speed-model')).toBe(true);
|
||||
|
|
|
|||
Loading…
Reference in a new issue