diff --git a/docs-site/docs/configure/tts-provider-guides/speech-sdk.md b/docs-site/docs/configure/tts-provider-guides/speech-sdk.md new file mode 100644 index 0000000..11ceb2d --- /dev/null +++ b/docs-site/docs/configure/tts-provider-guides/speech-sdk.md @@ -0,0 +1,63 @@ +--- +title: Speech SDK +--- + +Use [speech-sdk](https://github.com/Jellypod-Inc/speech-sdk) (Apache 2.0) to reach additional cloud TTS providers (ElevenLabs, Cartesia, Hume, Deepgram, Google Gemini TTS, Inworld, and more) with your own provider API keys. Requests go from the OpenReader server directly to the provider's API; no extra account or proxy is involved. + +Models use the `provider/model` format. The API key you enter belongs to the provider named by the model prefix: for `elevenlabs/eleven_multilingual_v2` enter an ElevenLabs key, for `cartesia/sonic-3.5` a Cartesia key, and so on. + +## Setup + +**Recommended (auth + admin): Settings → Admin → Shared providers** + +1. Add a shared provider with type `speech-sdk`. +2. Enter the API key for the provider you want to use. +3. Set default model to a matching `provider/model` (for example `elevenlabs/eleven_multilingual_v2`). + +**Per-user Settings → TTS Provider (only when `restrictUserApiKeys=false`):** + +1. Set provider to `Speech SDK`. +2. Choose a model; enter the API key for that model's provider. +3. Choose a voice. + +See [TTS Providers](../tts-providers) for admin-shared vs per-user behavior. + +## Built-in models + +- `openai/gpt-4o-mini-tts` +- `elevenlabs/eleven_multilingual_v2` +- `cartesia/sonic-3.5` +- `deepgram/aura-2` +- `google/gemini-2.5-flash-preview-tts` +- `inworld/inworld-tts-1.5-max` + +You can also choose `Other` and enter any `provider/model` the SDK supports. Recognized prefixes: `openai`, `elevenlabs`, `cartesia`, `hume`, `deepgram`, `google`, `inworld`, `minimax`, `fish-audio`, `murf`, `resemble`, `fal-ai`, `mistral`, `xai`, `smallest-ai`. + +## Voice IDs + +ElevenLabs and Cartesia identify voices by opaque IDs. The built-in lists map to these shared library voices: + +| ElevenLabs ID | Name | | Cartesia ID | Name | +| --- | --- | --- | --- | --- | +| `JBFqnCBsd6RMkjVDRZzb` | George | | `a0e99841-438c-4a64-b679-ae501e7d6091` | Barbershop Man | +| `IKne3meq5aSn9XLyUdCD` | Charlie | | `156fb8d2-335b-4950-9cb3-a2d33f0c0c2a` | British Lady | +| `XB0fDUnXU5powFXDhCwa` | Charlotte | | `694f9389-aac1-45b6-b726-9d9369183238` | California Girl | +| `Xb7hH8MSUJpSbSDYk0k2` | Alice | | `87748186-23bb-4571-8b8b-a73da9bf9c4f` | Commercial Lady | +| `iP95p4xoKVk53GoZ742B` | Chris | | `ee7ea9f8-c0c1-498c-9f62-dc2da49a6f98` | Friendly Reading Man | +| `nPczCjzI2devNBz1zQrb` | Brian | | `248be419-c632-4f23-adf1-5324ed7dbf1d` | Hannah | +| `onwK4e9ZLuTAKqWW03F9` | Daniel | | | | +| `pFZP5JQG7iQjIQuC4Bku` | Lily | | | | +| `pqHfZKP75CvOlQylNhV4` | Bill | | | | + +## Notes + +- One voice per request; Kokoro-style multi-voice mixing does not apply to this provider. +- Playback speed is applied client-side, so cached audio segments stay valid when you change speed. +- Providers without a built-in voice list fall back to a `default` entry, which lets the provider pick its default voice. +- Word-by-word highlighting works the same as with every other provider (alignment runs in OpenReader, not the provider). +- TTS requests are sent from the server, not the browser. The API key is never exposed to clients. + +## References + +- [speech-sdk on GitHub](https://github.com/Jellypod-Inc/speech-sdk) +- [TTS Providers](../tts-providers) diff --git a/docs-site/docs/configure/tts-providers.md b/docs-site/docs/configure/tts-providers.md index 9f5ba7c..4aee629 100644 --- a/docs-site/docs/configure/tts-providers.md +++ b/docs-site/docs/configure/tts-providers.md @@ -19,6 +19,7 @@ If you're running a private/self-hosted instance and want per-user BYOK behavior - **OpenAI**: Cloud. Base URL pre-filled (`https://api.openai.com/v1`). API key required. - **Replicate**: Cloud. Base URL managed internally by OpenReader. API key required. - **DeepInfra**: Cloud. Base URL pre-filled (`https://api.deepinfra.com/v1/openai`). API key required. +- **Speech SDK**: Cloud. Reaches additional providers (ElevenLabs, Cartesia, Hume, Deepgram, Google, Inworld, and more) directly with your own provider API keys via [speech-sdk](./tts-provider-guides/speech-sdk). No base URL. API key required (the key for the model's provider). - **Custom OpenAI-Like**: Self-hosted or any custom endpoint. `API_BASE` must be set manually (typically ending in `/v1`). API key optional. For `OpenAI`, `DeepInfra`, and `Replicate` you only need to supply an API key. For `Custom OpenAI-Like` you must also set `API_BASE`. @@ -28,6 +29,7 @@ For `OpenAI`, `DeepInfra`, and `Replicate` you only need to supply an API key. F - **Replicate** models: `alphanumericuser/kokoro-82m`, `google/gemini-3.1-flash-tts`, `minimax/speech-2.8-turbo`, `qwen/qwen3-tts`, `inworld/tts-1.5-mini` (or choose `Other` and enter any Replicate model ID, such as `owner/model` or `owner/model:version`) - **OpenAI** models: `tts-1`, `tts-1-hd`, `gpt-4o-mini-tts` - **DeepInfra** models: includes `hexgrad/Kokoro-82M` and additional hosted models (depending on API key / feature flags) +- **Speech SDK** models: `openai/gpt-4o-mini-tts`, `elevenlabs/eleven_multilingual_v2`, `cartesia/sonic-3.5`, `deepgram/aura-2`, `google/gemini-2.5-flash-preview-tts`, `inworld/inworld-tts-1.5-max` (or choose `Other` and enter any `provider/model` the SDK supports) ## Custom provider requirements @@ -51,6 +53,7 @@ TTS requests originate from the **Next.js server**, not the browser. `API_BASE` - [Replicate](./tts-provider-guides/replicate) - [DeepInfra](./tts-provider-guides/deepinfra) - [OpenAI](./tts-provider-guides/openai) +- [Speech SDK](./tts-provider-guides/speech-sdk) - [Other](./tts-provider-guides/other) ## Related diff --git a/docs-site/sidebars.ts b/docs-site/sidebars.ts index 30033c2..63041f9 100644 --- a/docs-site/sidebars.ts +++ b/docs-site/sidebars.ts @@ -27,6 +27,7 @@ const sidebars: SidebarsConfig = { 'configure/tts-provider-guides/replicate', 'configure/tts-provider-guides/deepinfra', 'configure/tts-provider-guides/openai', + 'configure/tts-provider-guides/speech-sdk', 'configure/tts-provider-guides/other', ], }, diff --git a/package.json b/package.json index 1ab8608..de098e0 100644 --- a/package.json +++ b/package.json @@ -38,6 +38,7 @@ "@microsoft/antissrf": "^1.0.0", "@mozilla/readability": "^0.6.0", "@napi-rs/canvas": "^0.1.100", + "@speech-sdk/core": "^0.15.0", "@tanstack/react-query": "^5.101.0", "@types/archiver": "^7.0.0", "@vercel/analytics": "^1.6.1", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 1749ccb..e6e0608 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -32,6 +32,9 @@ importers: '@napi-rs/canvas': specifier: ^0.1.100 version: 0.1.100 + '@speech-sdk/core': + specifier: ^0.15.0 + version: 0.15.0 '@tanstack/react-query': specifier: ^5.101.0 version: 5.101.0(react@19.2.7) @@ -1258,6 +1261,11 @@ packages: '@jridgewell/trace-mapping@0.3.31': resolution: {integrity: sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==} + '@mediabunny/mp3-encoder@1.46.0': + resolution: {integrity: sha512-SG69nm0ntRR4yc5TP9LG1/oxrh5PcoIEZwFm8BXecj3xNF0V+PlnpyJ1IQVS/JEXRIOvBJQ09YFknVGnJNAJrQ==} + peerDependencies: + mediabunny: ^1.0.0 + '@microsoft/antissrf@1.0.0': resolution: {integrity: sha512-AmO1ykSha52Ef/7UXvHgu8ElsGdgcLkF5nTl7YZGyR1AkbmlQYtiVeWp+CsjkFaJzKAH5ofXLZveMmOHwX/7Jw==} @@ -1640,6 +1648,9 @@ packages: resolution: {integrity: sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A==} engines: {node: '>=14.0.0'} + '@speech-sdk/core@0.15.0': + resolution: {integrity: sha512-GHM6Cno7PuU7nEtl6rRZVb+xEFJKZ1AB8WvdeCTIImEtzuWWaztLK3CG4h9KLXd8EQJHEbwVfcZh+CeAJBQrpw==} + '@standard-schema/spec@1.1.0': resolution: {integrity: sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==} @@ -1689,6 +1700,12 @@ packages: '@types/deep-eql@4.0.2': resolution: {integrity: sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==} + '@types/dom-mediacapture-transform@0.1.11': + resolution: {integrity: sha512-Y2p+nGf1bF2XMttBnsVPHUWzRRZzqUoJAKmiP10b5umnO6DDrWI0BrGDJy1pOHoOULVmGSfFNkQrAlC5dcj6nQ==} + + '@types/dom-webcodecs@0.1.13': + resolution: {integrity: sha512-O5hkiFIcjjszPIYyUSyvScyvrBoV3NOEEZx/pMlsu44TKzWNkLVBBxnxJz42in5n3QIolYOcBYFCPZZ0h8SkwQ==} + '@types/estree-jsx@1.0.5': resolution: {integrity: sha512-52CcUVNFyfb1A2ALocQw/Dd1BQFNmSdkuC3BkZ6iqhdMfQz7JWOFRuJFloOzjk+6WijU56m9oKXFAXc7o3Towg==} @@ -3313,6 +3330,10 @@ packages: resolution: {integrity: sha512-5KoIu2Ngpyek75jXodFvnafB6DJgr3u8uuK0LEZJjrU19DrMD3EVERaR8sjz8CCGgpZvxPl9SuE1GMVPFHx1mw==} engines: {node: '>= 0.4'} + is-network-error@1.3.2: + resolution: {integrity: sha512-PhBY86zaxNZUuWP6h13Vu5oFe0XY6/UlKzQnYFELzGVHygP3MxmvTfYSG7GN3aIab/iWudSMgjSnG9Dq+nHrgA==} + engines: {node: '>=16'} + is-number-object@1.1.1: resolution: {integrity: sha512-lZhclumE1G6VYD8VHe35wFaIif+CTy5SJIi5+3y4psDgWu4wPDoBhF8NxUOinEc7pHgiTsT6MaBb92rKhhD+Xw==} engines: {node: '>= 0.4'} @@ -3641,6 +3662,9 @@ packages: mdast-util-to-string@4.0.0: resolution: {integrity: sha512-0H44vDimn51F0YwvxSJSm0eCDOJTRlmN0R1yBh4HLj9wiV1Dn0QoXGbvFAWj2hSItVTlCmBF1hqKlIyUBVFLPg==} + mediabunny@1.46.0: + resolution: {integrity: sha512-HIaMQ6PVMndrFb9sNCh9/uAsLFBfLzyAgbk3kN7ZqlRRq//9RPYqIOQtXYpkbWBZU8ZJkaAUNLDtwTcdKo7yQw==} + merge-refs@1.3.0: resolution: {integrity: sha512-nqXPXbso+1dcKDpPCXvwZyJILz+vSLqGGOnDrYHQYE+B8n9JTCekVLC65AfCpR4ggVyA/45Y0iR9LDyS2iI+zA==} peerDependencies: @@ -3919,6 +3943,10 @@ packages: resolution: {integrity: sha512-LaNjtRWUBY++zB5nE/NwcaoMylSPk+S+ZHNB1TzdbMJMny6dynpAGt7X/tl/QYq3TIeE6nxHppbo2LGymrG5Pw==} engines: {node: '>=10'} + p-retry@8.0.0: + resolution: {integrity: sha512-kFVqH1HxOHp8LupNsOys7bSV09VYTRLxarH/mokO4Rqhk6wGi70E0jh4VzvVGXfEVNggHoHLAMWsQqHyU1Ey9A==} + engines: {node: '>=22'} + package-json-from-dist@1.0.1: resolution: {integrity: sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw==} @@ -5762,6 +5790,10 @@ snapshots: '@jridgewell/resolve-uri': 3.1.2 '@jridgewell/sourcemap-codec': 1.5.5 + '@mediabunny/mp3-encoder@1.46.0(mediabunny@1.46.0)': + dependencies: + mediabunny: 1.46.0 + '@microsoft/antissrf@1.0.0': {} '@mixmark-io/domino@2.2.0': {} @@ -6039,6 +6071,13 @@ snapshots: '@smithy/util-buffer-from': 2.2.0 tslib: 2.8.1 + '@speech-sdk/core@0.15.0': + dependencies: + '@mediabunny/mp3-encoder': 1.46.0(mediabunny@1.46.0) + mediabunny: 1.46.0 + p-retry: 8.0.0 + zod: 4.4.3 + '@standard-schema/spec@1.1.0': {} '@swc/helpers@0.5.15': @@ -6093,6 +6132,12 @@ snapshots: '@types/deep-eql@4.0.2': {} + '@types/dom-mediacapture-transform@0.1.11': + dependencies: + '@types/dom-webcodecs': 0.1.13 + + '@types/dom-webcodecs@0.1.13': {} + '@types/estree-jsx@1.0.5': dependencies: '@types/estree': 1.0.9 @@ -7848,6 +7893,8 @@ snapshots: is-negative-zero@2.0.3: {} + is-network-error@1.3.2: {} + is-number-object@1.1.1: dependencies: call-bound: 1.0.4 @@ -8252,6 +8299,11 @@ snapshots: dependencies: '@types/mdast': 4.0.4 + mediabunny@1.46.0: + dependencies: + '@types/dom-mediacapture-transform': 0.1.11 + '@types/dom-webcodecs': 0.1.13 + merge-refs@1.3.0(@types/react@19.2.16): optionalDependencies: '@types/react': 19.2.16 @@ -8629,6 +8681,10 @@ snapshots: dependencies: p-limit: 3.1.0 + p-retry@8.0.0: + dependencies: + is-network-error: 1.3.2 + package-json-from-dist@1.0.1: {} pako@1.0.11: {} diff --git a/src/components/SettingsModal.tsx b/src/components/SettingsModal.tsx index f4a0b89..fb80ca3 100644 --- a/src/components/SettingsModal.tsx +++ b/src/components/SettingsModal.tsx @@ -550,6 +550,7 @@ export function SettingsModal({ && !isSharedSelected && providerModelPolicy.isResolvedProviderType && providerModelPolicy.providerType !== 'replicate' + && providerModelPolicy.providerType !== 'speech-sdk' && (providerModelPolicy.providerType === 'custom-openai' || !localBaseUrl || localBaseUrl === ''); const shouldShowApiKey = !restrictUserApiKeys && !isSharedSelected; const selectedModel = ttsModels.find(m => m.id === selectedModelId) || ttsModels[0]; diff --git a/src/components/admin/AdminProvidersPanel.tsx b/src/components/admin/AdminProvidersPanel.tsx index 4a2f202..312f279 100644 --- a/src/components/admin/AdminProvidersPanel.tsx +++ b/src/components/admin/AdminProvidersPanel.tsx @@ -44,6 +44,7 @@ const PROVIDER_TYPE_OPTIONS: { value: ProviderType; label: string }[] = [ { value: 'openai', label: 'OpenAI' }, { value: 'deepinfra', label: 'Deepinfra' }, { value: 'replicate', label: 'Replicate' }, + { value: 'speech-sdk', label: 'Speech SDK' }, ]; interface FormState { diff --git a/src/lib/server/admin/providers.ts b/src/lib/server/admin/providers.ts index 0b99533..a562c9c 100644 --- a/src/lib/server/admin/providers.ts +++ b/src/lib/server/admin/providers.ts @@ -13,6 +13,7 @@ export const BUILT_IN_PROVIDER_IDS: readonly TtsProviderId[] = [ 'replicate', 'deepinfra', 'openai', + 'speech-sdk', ]; const SLUG_PATTERN = /^[a-z0-9][a-z0-9-]{0,62}[a-z0-9]$|^[a-z0-9]$/; diff --git a/src/lib/server/admin/resolve-credentials.ts b/src/lib/server/admin/resolve-credentials.ts index 2193b39..39c039f 100644 --- a/src/lib/server/admin/resolve-credentials.ts +++ b/src/lib/server/admin/resolve-credentials.ts @@ -89,7 +89,7 @@ export async function resolveTtsCredentials(opts: { }; } -const BUILT_IN_IDS = new Set(['custom-openai', 'replicate', 'deepinfra', 'openai']); +const BUILT_IN_IDS = new Set(['custom-openai', 'replicate', 'deepinfra', 'openai', 'speech-sdk']); function isBuiltInProviderId(value: string): boolean { return BUILT_IN_IDS.has(value); diff --git a/src/lib/server/tts/generate.ts b/src/lib/server/tts/generate.ts index 7f359bd..7c98c6d 100644 --- a/src/lib/server/tts/generate.ts +++ b/src/lib/server/tts/generate.ts @@ -1,9 +1,29 @@ import OpenAI from 'openai'; import Replicate from 'replicate'; import { SpeechCreateParams } from 'openai/resources/audio/speech.mjs'; +import { generateSpeech } from '@speech-sdk/core'; +import type { ResolvedModel } from '@speech-sdk/core/types'; +import { + createCartesia, + createDeepgram, + createElevenLabs, + createFal, + createFishAudio, + createGoogle, + createHume, + createInworld, + createMiniMax, + createMistral, + createMurf, + createOpenAI, + createResemble, + createSmallestAI, + createXai, +} from '@speech-sdk/core/providers'; import { isBuiltInTtsProviderId, REPLICATE_KOKORO_82M_VERSIONED_MODEL, + speechSdkProviderPrefix, } from '@/lib/shared/tts-provider-catalog'; import { resolveTtsProviderModelPolicy } from '@/lib/shared/tts-provider-policy'; import { @@ -333,6 +353,7 @@ function resolveTTSRequest(input: ServerTTSRequest): ResolvedServerTTSRequest { const providerType = isBuiltInTtsProviderId(provider) ? provider : 'openai'; const rawModel = provider === 'deepinfra' && !input.model ? 'hexgrad/Kokoro-82M' : provider === 'replicate' && !input.model ? REPLICATE_KOKORO_82M_VERSIONED_MODEL + : provider === 'speech-sdk' && !input.model ? 'openai/gpt-4o-mini-tts' : input.model; const model = (rawModel ?? 'gpt-4o-mini-tts') as SpeechCreateParams['model']; const providerModelPolicy = resolveTtsProviderModelPolicy({ @@ -342,7 +363,7 @@ function resolveTTSRequest(input: ServerTTSRequest): ResolvedServerTTSRequest { }); const normalizedVoice = ( - (providerType === 'replicate' || !providerModelPolicy.isKokoroModel) && input.voice.includes('+') + (providerType === 'replicate' || providerType === 'speech-sdk' || !providerModelPolicy.isKokoroModel) && input.voice.includes('+') ? input.voice.split('+')[0].trim() : input.voice ) as string; @@ -652,6 +673,55 @@ async function runReplicateRequest( }); } +type SpeechSdkModelFactory = (config: { apiKey?: string }) => (modelId?: string) => ResolvedModel; + +const SPEECH_SDK_PROVIDER_FACTORIES: Record = { + openai: createOpenAI, + elevenlabs: createElevenLabs, + cartesia: createCartesia, + hume: createHume, + deepgram: createDeepgram, + google: createGoogle, + inworld: createInworld, + minimax: createMiniMax, + 'fish-audio': createFishAudio, + murf: createMurf, + resemble: createResemble, + 'fal-ai': createFal, + mistral: createMistral, + xai: createXai, + 'smallest-ai': createSmallestAI, +}; + +async function runSpeechSdkRequest( + request: ResolvedServerTTSRequest, + signal: AbortSignal, + maxRetries: number, +): Promise { + const model = request.model as string; + const prefix = speechSdkProviderPrefix(model); + const factory = SPEECH_SDK_PROVIDER_FACTORIES[prefix]; + if (!factory) { + throw new Error( + `Unknown Speech SDK provider prefix "${prefix}". Use "provider/model" with one of: ${Object.keys(SPEECH_SDK_PROVIDER_FACTORIES).join(', ')}.` + ); + } + + const modelId = model.slice(prefix.length + 1); + // 'default' is the placeholder voice for providers without a static voice + // list; omit it so the provider's own default applies. + const voice = request.voice === 'default' ? undefined : request.voice; + const result = await generateSpeech({ + model: factory({ apiKey: request.apiKey || undefined })(modelId || undefined), + text: request.text, + voice: voice as string, + output: { format: 'mp3' }, + maxRetries, + abortSignal: signal, + }); + return Buffer.from(result.audio.uint8Array); +} + async function runProviderRequest( request: ResolvedServerTTSRequest, signal: AbortSignal, @@ -662,7 +732,9 @@ async function runProviderRequest( const raw = request.provider === 'replicate' ? await runReplicateRequest(request, signal, upstreamSettings.ttsUpstreamMaxRetries) - : await runOpenAiCompatibleRequest(request, signal, upstreamSettings); + : request.provider === 'speech-sdk' + ? await runSpeechSdkRequest(request, signal, upstreamSettings.ttsUpstreamMaxRetries) + : await runOpenAiCompatibleRequest(request, signal, upstreamSettings); // OpenAI-compatible servers (and some Replicate models) may emit wav/ogg/etc.; // normalize to mp3 so the cache, storage, and audiobook pipeline stay mp3-only. diff --git a/src/lib/shared/kokoro.ts b/src/lib/shared/kokoro.ts index 8698880..0d4295a 100644 --- a/src/lib/shared/kokoro.ts +++ b/src/lib/shared/kokoro.ts @@ -59,6 +59,7 @@ export const getMaxVoicesForProvider = (provider: TtsProviderId, model: string): // Replicate must always stay single-voice, even for Kokoro models. if (provider === 'replicate') return 1; if (provider === 'deepinfra') return 1; + if (provider === 'speech-sdk') return 1; // Other providers with Kokoro support unlimited voices return Infinity; diff --git a/src/lib/shared/tts-provider-catalog.ts b/src/lib/shared/tts-provider-catalog.ts index cb30fc2..423b1a5 100644 --- a/src/lib/shared/tts-provider-catalog.ts +++ b/src/lib/shared/tts-provider-catalog.ts @@ -1,6 +1,6 @@ import { isKokoroModel } from '@/lib/shared/kokoro'; -export type TtsProviderId = 'custom-openai' | 'replicate' | 'deepinfra' | 'openai'; +export type TtsProviderId = 'custom-openai' | 'replicate' | 'deepinfra' | 'openai' | 'speech-sdk'; export type TtsProviderType = TtsProviderId | 'unknown'; export type TtsVoiceSource = 'static' | 'deepinfra-api' | 'custom-openai-api' | 'replicate-api'; export type ReplicateVoiceInputKey = 'voice' | 'voice_id' | 'speaker'; @@ -72,6 +72,16 @@ const REPLICATE_MODELS: TtsModelDefinition[] = [ { id: 'inworld/tts-1.5-mini', name: 'inworld/tts-1.5-mini' }, { id: 'custom', name: 'Other' }, ]; + +const SPEECH_SDK_MODELS: TtsModelDefinition[] = [ + { id: 'openai/gpt-4o-mini-tts', name: 'openai/gpt-4o-mini-tts' }, + { id: 'elevenlabs/eleven_multilingual_v2', name: 'elevenlabs/eleven_multilingual_v2' }, + { id: 'cartesia/sonic-3.5', name: 'cartesia/sonic-3.5' }, + { id: 'deepgram/aura-2', name: 'deepgram/aura-2' }, + { id: 'google/gemini-2.5-flash-preview-tts', name: 'google/gemini-2.5-flash-preview-tts' }, + { id: 'inworld/inworld-tts-1.5-max', name: 'inworld/inworld-tts-1.5-max' }, + { id: 'custom', name: 'Other' }, +]; const DEEPINFRA_API_VOICE_MODELS = new Set([ 'ResembleAI/chatterbox', 'Zyphra/Zonos-v0.1-hybrid', @@ -110,6 +120,40 @@ export const MINIMAX_SPEECH_VOICES = [ ] as const; export const QWEN3_TTS_VOICES = ['Aiden', 'Dylan'] as const; export const INWORLD_TTS_VOICES = ['Ashley', 'Dennis', 'Alex', 'Darlene'] as const; + +// Speech SDK direct-provider voices. ElevenLabs and Cartesia identify voices by +// opaque IDs; the listed ones are stable, publicly shared library voices. +export const ELEVENLABS_DEFAULT_VOICES = [ + 'JBFqnCBsd6RMkjVDRZzb', 'IKne3meq5aSn9XLyUdCD', 'XB0fDUnXU5powFXDhCwa', + 'Xb7hH8MSUJpSbSDYk0k2', 'iP95p4xoKVk53GoZ742B', 'nPczCjzI2devNBz1zQrb', + 'onwK4e9ZLuTAKqWW03F9', 'pFZP5JQG7iQjIQuC4Bku', 'pqHfZKP75CvOlQylNhV4', +] as const; +export const CARTESIA_DEFAULT_VOICES = [ + 'a0e99841-438c-4a64-b679-ae501e7d6091', '156fb8d2-335b-4950-9cb3-a2d33f0c0c2a', + '694f9389-aac1-45b6-b726-9d9369183238', '87748186-23bb-4571-8b8b-a73da9bf9c4f', + 'ee7ea9f8-c0c1-498c-9f62-dc2da49a6f98', '248be419-c632-4f23-adf1-5324ed7dbf1d', +] as const; +export const HUME_DEFAULT_VOICES = ['KORA', 'STELLA', 'DACHER'] as const; +export const DEEPGRAM_AURA2_DEFAULT_VOICES = [ + 'thalia-en', 'andromeda-en', 'helena-en', 'apollo-en', 'arcas-en', + 'aries-en', 'asteria-en', 'athena-en', 'atlas-en', 'aurora-en', +] as const; +export const INWORLD_DIRECT_DEFAULT_VOICES = ['Ashley', 'Dominus', 'Edward', 'Hades', 'Priya'] as const; +const SPEECH_SDK_DEFAULT_VOICES_BY_PREFIX: Record = { + openai: GPT4O_MINI_DEFAULT_VOICES, + elevenlabs: ELEVENLABS_DEFAULT_VOICES, + cartesia: CARTESIA_DEFAULT_VOICES, + hume: HUME_DEFAULT_VOICES, + deepgram: DEEPGRAM_AURA2_DEFAULT_VOICES, + google: GEMINI_FLASH_TTS_VOICES, + inworld: INWORLD_DIRECT_DEFAULT_VOICES, + minimax: MINIMAX_SPEECH_VOICES, +}; + +export function speechSdkProviderPrefix(model: string): string { + const slashIndex = model.indexOf('/'); + return slashIndex === -1 ? model : model.slice(0, slashIndex); +} const REPLICATE_DEFAULT_VOICES_BY_MODEL: Record = { [REPLICATE_KOKORO_82M_VERSIONED_MODEL]: KOKORO_DEFAULT_VOICES, 'google/gemini-3.1-flash-tts': GEMINI_FLASH_TTS_VOICES, @@ -151,6 +195,12 @@ export const TTS_PROVIDER_DEFINITIONS: TtsProviderDefinition[] = [ supportsCustomModel: false, models: () => OPENAI_MODELS, }, + { + id: 'speech-sdk', + name: 'Speech SDK', + supportsCustomModel: true, + models: () => SPEECH_SDK_MODELS, + }, ]; const BUILT_IN_PROVIDER_ID_SET: ReadonlySet = new Set( @@ -209,6 +259,12 @@ export function supportsNativeModelSpeed(provider: TtsProviderId, model: string return !REPLICATE_MODELS_WITHOUT_NATIVE_SPEED.has(model); } + // Speed support varies per underlying provider; play back at the requested + // rate client-side so cached audio stays reusable across speed changes. + if (provider === 'speech-sdk') { + return false; + } + return true; } @@ -247,6 +303,11 @@ export function getDefaultVoices(provider: TtsProviderId, model: string): string : [...CUSTOM_OPENAI_DEFAULT_VOICES]; } + if (provider === 'speech-sdk') { + const prefixVoices = SPEECH_SDK_DEFAULT_VOICES_BY_PREFIX[speechSdkProviderPrefix(model)]; + return prefixVoices ? [...prefixVoices] : ['default']; + } + return [...OPENAI_DEFAULT_VOICES]; } diff --git a/src/lib/shared/tts-provider-policy.ts b/src/lib/shared/tts-provider-policy.ts index 1301c0b..bd0c0f2 100644 --- a/src/lib/shared/tts-provider-policy.ts +++ b/src/lib/shared/tts-provider-policy.ts @@ -61,6 +61,7 @@ export function defaultModelForProviderType(providerType: TtsProviderId): string if (providerType === 'openai') return 'tts-1'; if (providerType === 'deepinfra') return 'hexgrad/Kokoro-82M'; if (providerType === 'replicate') return REPLICATE_KOKORO_82M_VERSIONED_MODEL; + if (providerType === 'speech-sdk') return 'openai/gpt-4o-mini-tts'; return 'kokoro'; } @@ -73,6 +74,7 @@ export function defaultBaseUrlForProviderType(providerType: TtsProviderId): stri export function defaultVoiceForProviderType(providerType: TtsProviderId): string { if (providerType === 'openai') return 'alloy'; if (providerType === 'deepinfra') return 'af_bella'; + if (providerType === 'speech-sdk') return 'alloy'; return 'af_sarah'; } diff --git a/tests/unit/tts-generate.vitest.spec.ts b/tests/unit/tts-generate.vitest.spec.ts index fe3dd99..ef40d55 100644 --- a/tests/unit/tts-generate.vitest.spec.ts +++ b/tests/unit/tts-generate.vitest.spec.ts @@ -4,6 +4,7 @@ import { buildReplicateInput, buildTTSCacheKey, extractReplicateAudioUrl, + generateTTSBuffer, resolveReplicateLanguageValue, } from '../../src/lib/server/tts/generate'; import { REPLICATE_KOKORO_82M_VERSIONED_MODEL } from '../../src/lib/shared/tts-provider-catalog'; @@ -136,3 +137,16 @@ describe('Replicate language schema values', () => { }); }); }); + +describe('speech-sdk request resolution', () => { + test('rejects unknown provider prefixes before any network call', async () => { + await expect(generateTTSBuffer({ + text: 'hello', + voice: 'alloy', + speed: 1, + model: 'doesnotexist/some-model', + provider: 'speech-sdk', + apiKey: 'unused', + })).rejects.toThrow('Unknown Speech SDK provider prefix'); + }); +}); diff --git a/tests/unit/tts-provider-catalog.vitest.spec.ts b/tests/unit/tts-provider-catalog.vitest.spec.ts index 3453cc1..3725bee 100644 --- a/tests/unit/tts-provider-catalog.vitest.spec.ts +++ b/tests/unit/tts-provider-catalog.vitest.spec.ts @@ -5,6 +5,7 @@ import { getDefaultVoices, providerSupportsCustomModel, resolveProviderModels, + speechSdkProviderPrefix, supportsNativeModelSpeed, supportsTtsInstructions, } from '../../src/lib/shared/tts-provider-catalog'; @@ -51,6 +52,23 @@ describe('tts provider catalog', () => { expect(providerSupportsCustomModel('deepinfra')).toBe(true); }); + test('resolves speech-sdk models, voices, and prefix parsing', () => { + expect(resolveProviderModels('speech-sdk').map((model) => model.id)).toContain('openai/gpt-4o-mini-tts'); + expect(providerSupportsCustomModel('speech-sdk')).toBe(true); + + expect(speechSdkProviderPrefix('elevenlabs/eleven_multilingual_v2')).toBe('elevenlabs'); + // fal model ids are path-style, so only the first slash delimits the provider. + expect(speechSdkProviderPrefix('fal-ai/kokoro/american-english')).toBe('fal-ai'); + + expect(getDefaultVoices('speech-sdk', 'openai/gpt-4o-mini-tts')).toContain('sage'); + expect(getDefaultVoices('speech-sdk', 'deepgram/aura-2')).toContain('thalia-en'); + // Providers without a static voice list fall back to the provider default. + expect(getDefaultVoices('speech-sdk', 'mistral/voxtral-mini-tts-2603')).toEqual(['default']); + + // Speed is applied client-side for speech-sdk so cached audio is speed-independent. + expect(supportsNativeModelSpeed('speech-sdk', 'openai/gpt-4o-mini-tts')).toBe(false); + }); + test('uses blocklist semantics for Replicate native speed support', () => { expect(supportsNativeModelSpeed('replicate', REPLICATE_KOKORO_82M_VERSIONED_MODEL)).toBe(true); expect(supportsNativeModelSpeed('replicate', 'acme/runtime-speed-model')).toBe(true);