danvics/openreader
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(tts): update Replicate model defaults and voice resolution

Browse source 
Align default Replicate model to the versioned Kokoro model across UI, docs,
and server logic. Refactor TTS settings merging for audiobooks to ensure
consistent normalization. Improve Retry-After header handling for upstream
rate limits. Expand Replicate voice resolution to use model schemas when
available, with test coverage for custom and built-in models.

Update documentation to reflect new Replicate defaults, model selection,
and configuration guidance.
This commit is contained in:
Richard R 2026-04-16 12:52:08 -06:00
parent d0c0ab7420
commit 893f74f038
13 changed files with 692 additions and 124 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -18,7 +18,7 @@ OpenReader is an open source, self-host-friendly text-to-speech document reader
## ✨ Highlights
- 🎯 **Multi-provider TTS** with OpenAI-compatible endpoints and cloud providers (OpenAI, Replicate, DeepInfra, Kokoro, KittenTTS-FastAPI, Orpheus, custom).
- 🎯 **Multi-provider TTS** with OpenAI-compatible endpoints and cloud providers (Kokoro-FastAPI, KittenTTS-FastAPI, Orpheus-FastAPI or OpenAI, Replicate, DeepInfra).
- 📖 **Read-along playback** for PDF/EPUB with sentence-aware narration.
- ⏱️ **Word-by-word highlighting** via optional `whisper.cpp` timestamps.
- 🛜 **Sync + library import** to bring docs across devices and from server-mounted folders.

5
docs-site/docs/configure/tts-provider-guides/replicate.md
View file

@ -11,7 +11,7 @@ Use Replicate's hosted TTS models as your provider.
```env
API_KEY=r8_...
NEXT_PUBLIC_DEFAULT_TTS_PROVIDER=replicate
NEXT_PUBLIC_DEFAULT_TTS_MODEL=google/gemini-3.1-flash-tts
NEXT_PUBLIC_DEFAULT_TTS_MODEL=alphanumericuser/kokoro-82m:89b6fa84e4fa2dd6bd3a96be3e1f12827a3516c9fda8fddbac7a0be131c9a6f5
```
**Or in-app via Settings -> TTS Provider:**
@ -25,11 +25,12 @@ Settings modal values override env vars. See [TTS Providers](../tts-providers) f
## Notes
- Built-in Replicate models:
- `alphanumericuser/kokoro-82m:89b6fa84e4fa2dd6bd3a96be3e1f12827a3516c9fda8fddbac7a0be131c9a6f5`
- `google/gemini-3.1-flash-tts`
- `minimax/speech-2.8-turbo`
- `qwen/qwen3-tts`
- `inworld/tts-1.5-mini`
- You can also choose `Other` and enter any Replicate model ID (for example `owner/model-name`).
- You can also choose `Other` and enter any Replicate model ID (for example `owner/model-name` or `owner/model-name:version`).
- Native model speed is not available on all Replicate models; OpenReader hides/disables native speed controls where unsupported.
- TTS requests are sent from the server, not the browser. The API key is never exposed to clients.

8
docs-site/docs/configure/tts-providers.md
View file

@ -16,16 +16,16 @@ Set env vars as deployment-level defaults. Users (or you, in a single-user setup
- **OpenAI**: Cloud. Base URL pre-filled (`https://api.openai.com/v1`). API key required.
- **Replicate**: Cloud. Base URL managed internally by OpenReader. API key required.
- **Deepinfra**: Cloud. Base URL pre-filled (`https://api.deepinfra.com/v1/openai`). API key required.
- **DeepInfra**: Cloud. Base URL pre-filled (`https://api.deepinfra.com/v1/openai`). API key required.
- **Custom OpenAI-Like**: Self-hosted or any custom endpoint. `API_BASE` must be set manually (typically ending in `/v1`). API key optional.
For `OpenAI`, `Deepinfra`, and `Replicate` you only need to supply an API key. For `Custom OpenAI-Like` you must also set `API_BASE`.
For `OpenAI`, `DeepInfra`, and `Replicate` you only need to supply an API key. For `Custom OpenAI-Like` you must also set `API_BASE`.
## Built-in model catalogs
- **Replicate** models: `google/gemini-3.1-flash-tts`, `minimax/speech-2.8-turbo`, `qwen/qwen3-tts`, `inworld/tts-1.5-mini` (or choose `Other` and enter any Replicate model ID)
- **Replicate** models: `alphanumericuser/kokoro-82m:89b6fa84e4fa2dd6bd3a96be3e1f12827a3516c9fda8fddbac7a0be131c9a6f5`, `google/gemini-3.1-flash-tts`, `minimax/speech-2.8-turbo`, `qwen/qwen3-tts`, `inworld/tts-1.5-mini` (or choose `Other` and enter any Replicate model ID, such as `owner/model` or `owner/model:version`)
- **OpenAI** models: `tts-1`, `tts-1-hd`, `gpt-4o-mini-tts`
- **Deepinfra** models: includes `hexgrad/Kokoro-82M` and additional hosted models (depending on API key / feature flags)
- **DeepInfra** models: includes `hexgrad/Kokoro-82M` and additional hosted models (depending on API key / feature flags)
## Custom provider requirements

9
docs-site/docs/deploy/vercel-deployment.md
View file

@ -18,8 +18,7 @@ This guide covers deploying OpenReader to Vercel with external Postgres and S3-c
Recommended production setup (auth enabled):
```bash
API_BASE=https://api.deepinfra.com/v1/openai
API_KEY=your_deepinfra_key
API_KEY=your_replicate_key
POSTGRES_URL=postgres://...
USE_EMBEDDED_WEED_MINI=false
S3_ACCESS_KEY_ID=...
@ -33,8 +32,8 @@ AUTH_SECRET=...
NEXT_PUBLIC_ENABLE_DOCX_CONVERSION=false
NEXT_PUBLIC_ENABLE_DESTRUCTIVE_DELETE_ACTIONS=false
NEXT_PUBLIC_ENABLE_TTS_PROVIDERS_TAB=false
NEXT_PUBLIC_DEFAULT_TTS_PROVIDER=deepinfra
NEXT_PUBLIC_DEFAULT_TTS_MODEL=hexgrad/Kokoro-82M
NEXT_PUBLIC_DEFAULT_TTS_PROVIDER=replicate
NEXT_PUBLIC_DEFAULT_TTS_MODEL=alphanumericuser/kokoro-82m:89b6fa84e4fa2dd6bd3a96be3e1f12827a3516c9fda8fddbac7a0be131c9a6f5
NEXT_PUBLIC_SHOW_ALL_DEEPINFRA_MODELS=false
NEXT_PUBLIC_ENABLE_AUDIOBOOK_EXPORT=true
NEXT_PUBLIC_ENABLE_WORD_HIGHLIGHT=false
@ -50,7 +49,7 @@ We recommend setting these defaults for a production-like environment:
- `NEXT_PUBLIC_ENABLE_DESTRUCTIVE_DELETE_ACTIONS=false`: Hides destructive "Delete All" actions
- `NEXT_PUBLIC_ENABLE_TTS_PROVIDERS_TAB=false`: Hides the Settings -> TTS Provider section
- `NEXT_PUBLIC_DEFAULT_TTS_PROVIDER=replicate`: Points default TTS to a scalable provider
- `NEXT_PUBLIC_DEFAULT_TTS_MODEL=google/gemini-3.1-flash-tts`: Uses a high-quality default model
- `NEXT_PUBLIC_DEFAULT_TTS_MODEL=alphanumericuser/kokoro-82m:89b6fa84e4fa2dd6bd3a96be3e1f12827a3516c9fda8fddbac7a0be131c9a6f5`: Uses a low-cost default model
- `NEXT_PUBLIC_SHOW_ALL_DEEPINFRA_MODELS=false`: Restricts usage to free models if no key is provided
- `NEXT_PUBLIC_ENABLE_AUDIOBOOK_EXPORT=true`: (Optional) Controls audiobook export UI
- `NEXT_PUBLIC_ENABLE_WORD_HIGHLIGHT=false`: (Optional) Controls word highlighting UI (requires timestamp backend)

2
docs-site/docs/introduction.md
View file

@ -18,7 +18,7 @@ It supports multiple TTS providers including OpenAI, Replicate, DeepInfra, and c
- [**Orpheus-FastAPI**](https://github.com/Lex-au/Orpheus-FastAPI)
- **Custom OpenAI-compatible**: any TTS API with `/v1/audio/voices` and `/v1/audio/speech` endpoints
- **Cloud TTS providers**:
- [**Replicate**](https://replicate.com/explore): `google/gemini-3.1-flash-tts`, `minimax/speech-2.8-turbo`, `qwen/qwen3-tts`, and `inworld/tts-1.5-mini`
- [**Replicate**](https://replicate.com/explore): includes a built-in catalog and supports any Replicate model ID via `Other`
- [**DeepInfra**](https://deepinfra.com/models/text-to-speech): Kokoro-82M and other hosted models
- [**OpenAI API**](https://platform.openai.com/docs/pricing#transcription-and-speech): `tts-1`, `tts-1-hd`, and `gpt-4o-mini-tts`
- 🛜 **Server-side Document Storage**

28
src/app/api/audiobook/chapter/route.ts
View file

@ -307,9 +307,16 @@ export async function POST(request: NextRequest) {
existingSettings = null;
}
const normalizedExistingSettings = existingSettings ? normalizeNativeSpeedForSettings(existingSettings) : undefined;
const incomingSettings = data.settings ? normalizeNativeSpeedForSettings(data.settings) : undefined;
if (existingSettings && hasChapters && incomingSettings) {
const normalizedExistingSettings = normalizeNativeSpeedForSettings(existingSettings);
const mergedSettings = (normalizedExistingSettings || incomingSettings)
? normalizeNativeSpeedForSettings({
...(normalizedExistingSettings || {}),
...(incomingSettings || {}),
} as AudiobookGenerationSettings)
: undefined;
if (normalizedExistingSettings && hasChapters && incomingSettings) {
const mismatch =
normalizedExistingSettings.ttsProvider !== incomingSettings.ttsProvider ||
normalizedExistingSettings.ttsModel !== incomingSettings.ttsModel ||
@ -330,10 +337,9 @@ export async function POST(request: NextRequest) {
const format: TTSAudiobookFormat =
(existingFormats.values().next().value as TTSAudiobookFormat | undefined) ??
existingSettings?.format ??
incomingSettings?.format ??
mergedSettings?.format ??
requestedFormat;
const rawPostSpeed = incomingSettings?.postSpeed ?? existingSettings?.postSpeed ?? 1;
const rawPostSpeed = mergedSettings?.postSpeed ?? 1;
const postSpeed = Number.isFinite(Number(rawPostSpeed)) ? Number(rawPostSpeed) : 1;
let chapterIndex: number;
@ -357,22 +363,20 @@ export async function POST(request: NextRequest) {
}
const provider = request.headers.get('x-tts-provider')
|| incomingSettings?.ttsProvider
|| existingSettings?.ttsProvider
|| mergedSettings?.ttsProvider
|| 'openai';
const openApiKey = request.headers.get('x-openai-key') || process.env.API_KEY || 'none';
const openApiBaseUrl = request.headers.get('x-openai-base-url') || process.env.API_BASE;
const model = incomingSettings?.ttsModel ?? existingSettings?.ttsModel;
const voice = incomingSettings?.voice
|| existingSettings?.voice
const model = mergedSettings?.ttsModel;
const voice = mergedSettings?.voice
|| (provider === 'openai'
? 'alloy'
: provider === 'deepinfra'
? 'af_bella'
: 'af_sarah');
const rawNativeSpeed = incomingSettings?.nativeSpeed ?? existingSettings?.nativeSpeed ?? 1;
const rawNativeSpeed = mergedSettings?.nativeSpeed ?? 1;
const nativeSpeed = Number.isFinite(Number(rawNativeSpeed)) ? Number(rawNativeSpeed) : 1;
const instructions = incomingSettings?.ttsInstructions ?? existingSettings?.ttsInstructions;
const instructions = mergedSettings?.ttsInstructions;
if (authEnabled && userId && isTtsRateLimitEnabled()) {
const isAnonymous = Boolean(user?.isAnonymous);

10
src/app/api/tts/route.ts
View file

@ -66,8 +66,14 @@ function getUpstreamRetryAfterSeconds(error: unknown): number | undefined {
const retryAfterHeader = response?.headers?.get?.('retry-after');
if (!retryAfterHeader) return undefined;
const parsed = Number(retryAfterHeader);
if (!Number.isFinite(parsed) || parsed <= 0) return undefined;
return Math.ceil(parsed);
if (Number.isFinite(parsed) && parsed > 0) {
return Math.ceil(parsed);
}
const parsedDateMs = Date.parse(retryAfterHeader);
if (!Number.isFinite(parsedDateMs)) return undefined;
const seconds = (parsedDateMs - Date.now()) / 1000;
if (seconds <= 0) return undefined;
return Math.ceil(seconds);
}
export async function POST(req: NextRequest) {

50
src/components/SettingsModal.tsx
View file

@ -36,7 +36,7 @@ import { deleteDocuments, mimeTypeForDoc, uploadDocuments } from '@/lib/client/a
import { cacheStoredDocumentFromBytes, clearDocumentCache } from '@/lib/client/cache/documents';
import { clearAllDocumentPreviewCaches, clearInMemoryDocumentPreviewCache } from '@/lib/client/cache/previews';
import { resolveTtsSettingsViewModel } from '@/lib/client/settings/tts-settings';
import { supportsTtsInstructions } from '@/lib/shared/tts-provider-catalog';
import { REPLICATE_KOKORO_82M_VERSIONED_MODEL, supportsTtsInstructions } from '@/lib/shared/tts-provider-catalog';
const enableDestructiveDelete = process.env.NEXT_PUBLIC_ENABLE_DESTRUCTIVE_DELETE_ACTIONS !== 'false';
const showAllDeepInfra = process.env.NEXT_PUBLIC_SHOW_ALL_DEEPINFRA_MODELS !== 'false';
@ -392,6 +392,12 @@ export function SettingsModal({ className = '' }: { className?: string }) {
const btnPrimary = `${btnBase} bg-accent text-background hover:bg-secondary-accent hover:scale-[1.04]`;
const btnSecondary = `${btnBase} bg-background text-foreground hover:bg-offbase hover:text-accent hover:scale-[1.04]`;
const btnOutline = `${btnBase} bg-background border border-offbase text-foreground hover:bg-offbase hover:text-accent hover:scale-[1.02]`;
const shouldShowBaseUrl = localTTSProvider !== 'replicate'
&& (localTTSProvider === 'custom-openai' || !localBaseUrl || localBaseUrl === '');
const selectedModel = ttsModels.find(m => m.id === selectedModelId) || ttsModels[0];
const selectedModelVersion = selectedModel?.id?.includes(':')
? selectedModel.id.slice(selectedModel.id.indexOf(':'))
: '';
return (
<>
@ -508,7 +514,7 @@ export function SettingsModal({ className = '' }: { className?: string }) {
setModelValue('kokoro');
setLocalBaseUrl('');
} else if (provider.id === 'replicate') {
setModelValue('google/gemini-3.1-flash-tts');
setModelValue(REPLICATE_KOKORO_82M_VERSIONED_MODEL);
setLocalBaseUrl('');
} else if (provider.id === 'deepinfra') {
setModelValue('hexgrad/Kokoro-82M');
@ -562,7 +568,7 @@ export function SettingsModal({ className = '' }: { className?: string }) {
</Listbox>
</div>
{localTTSProvider !== 'replicate' && (localTTSProvider === 'custom-openai' || !localBaseUrl || localBaseUrl === '') && (
{shouldShowBaseUrl && (
<div className="space-y-1.5">
<label className="block text-sm font-medium text-foreground">
API Base URL
@ -607,9 +613,20 @@ export function SettingsModal({ className = '' }: { className?: string }) {
}}
>
<ListboxButton className="relative w-full cursor-pointer rounded-lg bg-background py-2 pl-3 pr-10 text-left text-foreground shadow-sm focus:outline-none focus:ring-2 focus:ring-accent transform transition-transform duration-200 ease-in-out hover:scale-[1.005] hover:text-accent hover:bg-offbase">
<span className="block truncate">
{ttsModels.find(m => m.id === selectedModelId)?.name || 'Select Model'}
</span>
{selectedModel ? (
<span className="block">
<span className="block truncate">
{selectedModel.name}
</span>
{selectedModelVersion && (
<span className="block truncate text-xs text-muted">
{selectedModelVersion}
</span>
)}
</span>
) : (
<span className="block truncate">Select Model</span>
)}
<span className="pointer-events-none absolute inset-y-0 right-0 flex items-center pr-2">
<ChevronUpDownIcon className="h-5 w-5 text-muted" />
</span>
@ -631,14 +648,19 @@ export function SettingsModal({ className = '' }: { className?: string }) {
`relative cursor-pointer select-none py-2 pl-10 pr-4 ${active ? 'bg-offbase text-accent' : 'text-foreground'}`
}
value={model}
>
{({ selected }) => (
<>
<span className={`block truncate ${selected ? 'font-medium' : 'font-normal'}`}>
{model.name}
</span>
{selected && (
<span className="absolute inset-y-0 left-0 flex items-center pl-3 text-accent">
>
{({ selected }) => (
<>
<span className={`block ${selected ? 'font-medium' : 'font-normal'}`}>
<span className="block truncate">{model.name}</span>
{model.id.includes(':') && (
<span className="block truncate text-xs text-muted">
{model.id.slice(model.id.indexOf(':'))}
</span>
)}
</span>
{selected && (
<span className="absolute inset-y-0 left-0 flex items-center pl-3 text-accent">
<CheckIcon className="h-5 w-5" />
</span>
)}

86
src/lib/server/tts/generate.ts
View file

@ -2,7 +2,12 @@ import OpenAI from 'openai';
import Replicate from 'replicate';
import { SpeechCreateParams } from 'openai/resources/audio/speech.mjs';
import { isKokoroModel } from '@/lib/shared/kokoro';
import { supportsNativeModelSpeed, supportsTtsInstructions } from '@/lib/shared/tts-provider-catalog';
import {
REPLICATE_KOKORO_82M_VERSIONED_MODEL,
resolveReplicateVoiceInputKey,
supportsNativeModelSpeed,
supportsTtsInstructions,
} from '@/lib/shared/tts-provider-catalog';
import { LRUCache } from 'lru-cache';
import { createHash } from 'crypto';
import { access, readFile } from 'fs/promises';
@ -46,7 +51,6 @@ type InflightEntry = {
consumers: number;
};
let replicateQueue: Promise<void> = Promise.resolve();
let replicateBlockedUntilMs = 0;
const TTS_CACHE_MAX_SIZE_BYTES = Number(process.env.TTS_CACHE_MAX_SIZE_BYTES || 256 * 1024 * 1024); // 256MB
@ -106,8 +110,14 @@ function getUpstreamRetryAfterSeconds(error: unknown): number | undefined {
const retryAfterHeader = response?.headers?.get?.('retry-after');
if (!retryAfterHeader) return undefined;
const parsed = Number(retryAfterHeader);
if (!Number.isFinite(parsed) || parsed <= 0) return undefined;
return parsed;
if (Number.isFinite(parsed) && parsed > 0) {
return parsed;
}
const parsedDateMs = Date.parse(retryAfterHeader);
if (!Number.isFinite(parsedDateMs)) return undefined;
const seconds = (parsedDateMs - Date.now()) / 1000;
if (seconds <= 0) return undefined;
return Math.ceil(seconds);
}
function applyReplicateCooldown(cooldownMs: number) {
@ -117,33 +127,22 @@ function applyReplicateCooldown(cooldownMs: number) {
}
async function runWithReplicateGate<T>(signal: AbortSignal, operation: () => Promise<T>): Promise<T> {
let release: (() => void) | undefined;
const prev = replicateQueue;
replicateQueue = new Promise<void>((resolve) => {
release = resolve;
});
await prev;
try {
const waitMs = Math.max(0, replicateBlockedUntilMs - Date.now());
if (waitMs > 0) {
await sleepWithSignal(waitMs, signal);
}
return await operation();
} finally {
release?.();
const waitMs = Math.max(0, replicateBlockedUntilMs - Date.now());
if (waitMs > 0) {
await sleepWithSignal(waitMs, signal);
}
return operation();
}
function resolveTTSRequest(input: ServerTTSRequest): ResolvedServerTTSRequest {
const provider = input.provider || 'openai';
const rawModel = provider === 'deepinfra' && !input.model ? 'hexgrad/Kokoro-82M'
: provider === 'replicate' && !input.model ? 'google/gemini-3.1-flash-tts'
: provider === 'replicate' && !input.model ? REPLICATE_KOKORO_82M_VERSIONED_MODEL
: input.model;
const model = (rawModel ?? 'gpt-4o-mini-tts') as SpeechCreateParams['model'];
const normalizedVoice = (
!isKokoroModel(model as string) && input.voice.includes('+')
(provider === 'replicate' || !isKokoroModel(model as string)) && input.voice.includes('+')
? input.voice.split('+')[0].trim()
: input.voice
) as string;
@ -263,7 +262,7 @@ async function fetchTTSBufferWithRetry(
}
}
function buildReplicateInput(request: ResolvedServerTTSRequest): Record<string, unknown> {
async function buildReplicateInput(request: ResolvedServerTTSRequest): Promise<Record<string, unknown>> {
const model = request.model as string;
if (model === 'google/gemini-3.1-flash-tts') {
@ -313,12 +312,38 @@ function buildReplicateInput(request: ResolvedServerTTSRequest): Record<string,
return input;
}
return { text: request.text };
const input: Record<string, unknown> = { text: request.text };
const voiceInputKey = await resolveReplicateVoiceInputKey({
provider: 'replicate',
model,
apiKey: request.apiKey,
});
if (voiceInputKey) {
input[voiceInputKey] = request.voice;
} else {
input.voice = request.voice;
}
// Best-effort generic fields for custom models.
if (request.format !== 'mp3') {
input.audio_format = 'wav';
}
if (request.speed !== 1) {
input.speed = request.speed;
}
if (request.instructions) {
input.instructions = request.instructions;
}
return input;
}
async function runReplicateRequest(request: ResolvedServerTTSRequest, signal: AbortSignal): Promise<Buffer> {
const replicate = new Replicate({ auth: request.apiKey });
const input = buildReplicateInput(request);
const input = await buildReplicateInput(request);
const modelId = request.model as `${string}/${string}`;
return runWithReplicateGate(signal, async () => {
@ -333,7 +358,18 @@ async function runReplicateRequest(request: ResolvedServerTTSRequest, signal: Ab
const audioUrl = typeof output === 'string' ? output : String(output);
const audioResponse = await fetch(audioUrl, { signal });
if (!audioResponse.ok) {
throw new Error(`Failed to fetch Replicate audio: ${audioResponse.status}`);
const error = new Error(`Failed to fetch Replicate audio: ${audioResponse.status}`) as Error & {
status?: number;
statusCode?: number;
response?: { status: number; headers: Headers };
};
error.status = audioResponse.status;
error.statusCode = audioResponse.status;
error.response = {
status: audioResponse.status,
headers: audioResponse.headers,
};
throw error;
}
const buffer = await audioResponse.arrayBuffer();
return Buffer.from(buffer);

4
src/lib/shared/kokoro.ts
View file

@ -55,7 +55,9 @@ export const isKokoroModel = (modelName: string | undefined): boolean => {
export const getMaxVoicesForProvider = (provider: string, model: string): number => {
if (!isKokoroModel(model)) return 1;
// Deepinfra Kokoro does not support multiple voices
// Deepinfra and Replicate Kokoro do not support multiple voices.
// Replicate must always stay single-voice, even for Kokoro models.
if (provider === 'replicate') return 1;
if (provider === 'deepinfra') return 1;
// Other providers with Kokoro support unlimited voices

346
src/lib/shared/tts-provider-catalog.ts
View file

@ -1,7 +1,8 @@
import { isKokoroModel } from '@/lib/shared/kokoro';
export type TtsProviderId = 'custom-openai' | 'replicate' | 'deepinfra' | 'openai';
export type TtsVoiceSource = 'static' | 'deepinfra-api' | 'custom-openai-api';
export type TtsVoiceSource = 'static' | 'deepinfra-api' | 'custom-openai-api' | 'replicate-api';
export type ReplicateVoiceInputKey = 'voice' | 'voice_id' | 'speaker';
export interface TtsModelDefinition {
id: string;
@ -54,13 +55,26 @@ const DEEPINFRA_MODELS_LIMITED: TtsModelDefinition[] = [
{ id: 'hexgrad/Kokoro-82M', name: 'hexgrad/Kokoro-82M' },
];
export const REPLICATE_KOKORO_82M_VERSIONED_MODEL =
'alphanumericuser/kokoro-82m:89b6fa84e4fa2dd6bd3a96be3e1f12827a3516c9fda8fddbac7a0be131c9a6f5' as const;
const REPLICATE_MODELS: TtsModelDefinition[] = [
{ id: 'google/gemini-3.1-flash-tts', name: 'Gemini 3.1 Flash TTS' },
{ id: 'minimax/speech-2.8-turbo', name: 'MiniMax Speech 2.8 Turbo' },
{ id: 'qwen/qwen3-tts', name: 'Qwen3 TTS' },
{ id: 'inworld/tts-1.5-mini', name: 'Inworld TTS 1.5 Mini' },
{
id: REPLICATE_KOKORO_82M_VERSIONED_MODEL,
name: 'alphanumericuser/kokoro-82m',
},
{ id: 'google/gemini-3.1-flash-tts', name: 'google/gemini-3.1-flash-tts' },
{ id: 'minimax/speech-2.8-turbo', name: 'minimax/speech-2.8-turbo' },
{ id: 'qwen/qwen3-tts', name: 'qwen/qwen3-tts' },
{ id: 'inworld/tts-1.5-mini', name: 'inworld/tts-1.5-mini' },
{ id: 'custom', name: 'Other' },
];
const REPLICATE_BUILT_IN_MODELS = new Set(REPLICATE_MODELS.map((model) => model.id).filter((id) => id !== 'custom'));
const DEEPINFRA_API_VOICE_MODELS = new Set([
'ResembleAI/chatterbox',
'Zyphra/Zonos-v0.1-hybrid',
'Zyphra/Zonos-v0.1-transformer',
]);
const DEFAULT_MODELS: TtsModelDefinition[] = [{ id: 'tts-1', name: 'TTS-1' }];
@ -94,6 +108,24 @@ export const MINIMAX_SPEECH_VOICES = [
] as const;
export const QWEN3_TTS_VOICES = ['Aiden', 'Dylan'] as const;
export const INWORLD_TTS_VOICES = ['Ashley', 'Dennis', 'Alex', 'Darlene'] as const;
const REPLICATE_VOICE_KEYS: readonly ReplicateVoiceInputKey[] = ['voice', 'voice_id', 'speaker'];
const REPLICATE_DEFAULT_VOICES_BY_MODEL: Record<string, readonly string[]> = {
[REPLICATE_KOKORO_82M_VERSIONED_MODEL]: KOKORO_DEFAULT_VOICES,
'google/gemini-3.1-flash-tts': GEMINI_FLASH_TTS_VOICES,
'minimax/speech-2.8-turbo': MINIMAX_SPEECH_VOICES,
'qwen/qwen3-tts': QWEN3_TTS_VOICES,
'inworld/tts-1.5-mini': INWORLD_TTS_VOICES,
};
const DEEPINFRA_DEFAULT_VOICES_BY_MODEL: Record<string, readonly string[]> = {
'hexgrad/Kokoro-82M': KOKORO_DEFAULT_VOICES,
'canopylabs/orpheus-3b-0.1-ft': ORPHEUS_DEFAULT_VOICES,
'sesame/csm-1b': SESAME_DEFAULT_VOICES,
'ResembleAI/chatterbox': ['None'],
'Zyphra/Zonos-v0.1-hybrid': ['random'],
'Zyphra/Zonos-v0.1-transformer': ['random'],
};
const replicateVoiceInputKeyCache = new Map<string, ReplicateVoiceInputKey>();
const replicateOpenApiSchemaPromiseCache = new Map<string, Promise<unknown | null>>();
export const TTS_PROVIDER_DEFINITIONS: TtsProviderDefinition[] = [
{
@ -138,11 +170,6 @@ const REPLICATE_MODELS_WITHOUT_NATIVE_SPEED = new Set([
'qwen/qwen3-tts',
]);
const REPLICATE_MODELS_WITH_NATIVE_SPEED = new Set([
'minimax/speech-2.8-turbo',
'inworld/tts-1.5-mini',
]);
export function supportsTtsInstructions(model: string | null | undefined): boolean {
return !!model && MODELS_WITH_INSTRUCTIONS.has(model);
}
@ -153,11 +180,7 @@ export function supportsNativeModelSpeed(provider: string | null | undefined, mo
}
if (provider === 'replicate') {
if (REPLICATE_MODELS_WITHOUT_NATIVE_SPEED.has(model)) {
return false;
}
return REPLICATE_MODELS_WITH_NATIVE_SPEED.has(model);
return !REPLICATE_MODELS_WITHOUT_NATIVE_SPEED.has(model);
}
return true;
@ -177,66 +200,265 @@ export function providerSupportsCustomModel(provider: string | null | undefined)
export function getDefaultVoices(provider: string, model: string): string[] {
if (provider === 'openai') {
if (supportsTtsInstructions(model)) {
return [...GPT4O_MINI_DEFAULT_VOICES];
}
return [...OPENAI_DEFAULT_VOICES];
return supportsTtsInstructions(model) ? [...GPT4O_MINI_DEFAULT_VOICES] : [...OPENAI_DEFAULT_VOICES];
}
if (provider === 'custom-openai') {
if (isKokoroModel(model)) {
return [...KOKORO_DEFAULT_VOICES];
}
return [...CUSTOM_OPENAI_DEFAULT_VOICES];
return isKokoroModel(model) ? [...KOKORO_DEFAULT_VOICES] : [...CUSTOM_OPENAI_DEFAULT_VOICES];
}
if (provider === 'replicate') {
if (model === 'google/gemini-3.1-flash-tts') {
return [...GEMINI_FLASH_TTS_VOICES];
}
if (model === 'minimax/speech-2.8-turbo') {
return [...MINIMAX_SPEECH_VOICES];
}
if (model === 'qwen/qwen3-tts') {
return [...QWEN3_TTS_VOICES];
}
if (model === 'inworld/tts-1.5-mini') {
return [...INWORLD_TTS_VOICES];
}
return ['default'];
return REPLICATE_DEFAULT_VOICES_BY_MODEL[model] ? [...REPLICATE_DEFAULT_VOICES_BY_MODEL[model]] : ['default'];
}
if (provider === 'deepinfra') {
if (model === 'hexgrad/Kokoro-82M') {
return [...KOKORO_DEFAULT_VOICES];
}
if (model === 'canopylabs/orpheus-3b-0.1-ft') {
return [...ORPHEUS_DEFAULT_VOICES];
}
if (model === 'sesame/csm-1b') {
return [...SESAME_DEFAULT_VOICES];
}
if (model === 'ResembleAI/chatterbox') {
return ['None'];
}
if (model === 'Zyphra/Zonos-v0.1-hybrid' || model === 'Zyphra/Zonos-v0.1-transformer') {
return ['random'];
}
return [...CUSTOM_OPENAI_DEFAULT_VOICES];
return DEEPINFRA_DEFAULT_VOICES_BY_MODEL[model]
? [...DEEPINFRA_DEFAULT_VOICES_BY_MODEL[model]]
: [...CUSTOM_OPENAI_DEFAULT_VOICES];
}
return [...OPENAI_DEFAULT_VOICES];
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null;
}
function parseReplicateModelIdentifier(model: string): {
owner: string;
name: string;
version?: string;
} | null {
const [ref, version] = model.split(':', 2);
const segments = ref.split('/');
if (segments.length !== 2 || !segments[0] || !segments[1]) {
return null;
}
const parsed = {
owner: segments[0],
name: segments[1],
};
return version
? { ...parsed, version }
: parsed;
}
function extractSchemaStringEnums(schemaNode: unknown, seen = new Set<object>()): string[] {
if (!isRecord(schemaNode)) {
return [];
}
if (seen.has(schemaNode)) {
return [];
}
seen.add(schemaNode);
const values: string[] = [];
if (Array.isArray(schemaNode.enum)) {
values.push(...schemaNode.enum.filter((value): value is string => typeof value === 'string'));
}
if (typeof schemaNode.const === 'string') {
values.push(schemaNode.const);
}
for (const key of ['anyOf', 'allOf', 'oneOf'] as const) {
const branch = schemaNode[key];
if (!Array.isArray(branch)) continue;
for (const item of branch) {
values.push(...extractSchemaStringEnums(item, seen));
}
}
if (schemaNode.items) {
values.push(...extractSchemaStringEnums(schemaNode.items, seen));
}
return values;
}
function walkRecordGraph(root: unknown, visit: (node: Record<string, unknown>) => boolean | void): void {
if (!isRecord(root)) {
return;
}
const stack: Record<string, unknown>[] = [root];
const seen = new Set<object>();
while (stack.length > 0) {
const current = stack.pop();
if (!current) {
continue;
}
if (seen.has(current)) {
continue;
}
seen.add(current);
if (visit(current)) {
return;
}
for (const value of Object.values(current)) {
if (Array.isArray(value)) {
for (const item of value) {
if (isRecord(item)) {
stack.push(item);
}
}
} else if (isRecord(value)) {
stack.push(value);
}
}
}
}
function extractReplicateVoicesFromOpenApiSchema(openApiSchema: unknown): string[] {
const voices: string[] = [];
walkRecordGraph(openApiSchema, (node) => {
const properties = node.properties;
if (!isRecord(properties)) {
return;
}
for (const key of REPLICATE_VOICE_KEYS) {
if (!(key in properties)) continue;
voices.push(...extractSchemaStringEnums(properties[key]));
}
});
return Array.from(
new Set(
voices
.map((voice) => voice.trim())
.filter((voice) => voice.length > 0)
)
);
}
function extractReplicateVoiceInputKeyFromOpenApiSchema(openApiSchema: unknown): ReplicateVoiceInputKey | null {
let found: ReplicateVoiceInputKey | null = null;
walkRecordGraph(openApiSchema, (node) => {
const properties = node.properties;
if (!isRecord(properties)) {
return;
}
for (const key of REPLICATE_VOICE_KEYS) {
if (key in properties) {
found = key;
return true;
}
}
});
return found;
}
async function fetchReplicateOpenApiSchema(apiKey: string, model: string): Promise<unknown | null> {
const parsedModel = parseReplicateModelIdentifier(model);
if (!parsedModel) {
return null;
}
const controller = new AbortController();
const timeoutId = setTimeout(() => {
controller.abort();
}, 10_000);
try {
const endpoint = parsedModel.version
? `https://api.replicate.com/v1/models/${parsedModel.owner}/${parsedModel.name}/versions/${parsedModel.version}`
: `https://api.replicate.com/v1/models/${parsedModel.owner}/${parsedModel.name}`;
const response = await fetch(endpoint, {
signal: controller.signal,
headers: {
Authorization: `Bearer ${apiKey}`,
'Content-Type': 'application/json',
},
});
if (!response.ok) {
return null;
}
const data = await response.json();
let openApiSchema: unknown = null;
if (parsedModel.version) {
if (isRecord(data)) {
openApiSchema = data.openapi_schema;
}
} else if (isRecord(data) && isRecord(data.latest_version)) {
openApiSchema = data.latest_version.openapi_schema;
}
return openApiSchema;
} catch (error) {
if (error instanceof DOMException && error.name === 'AbortError') {
return null;
}
console.error('Error fetching Replicate model schema:', error);
return null;
} finally {
clearTimeout(timeoutId);
}
}
async function getReplicateOpenApiSchemaCached(apiKey: string, model: string): Promise<unknown | null> {
const cachedPromise = replicateOpenApiSchemaPromiseCache.get(model);
if (cachedPromise) {
return cachedPromise;
}
const fetchPromise = fetchReplicateOpenApiSchema(apiKey, model);
replicateOpenApiSchemaPromiseCache.set(model, fetchPromise);
const schema = await fetchPromise;
if (schema === null) {
replicateOpenApiSchemaPromiseCache.delete(model);
}
return schema;
}
async function fetchReplicateVoices(apiKey: string, model: string): Promise<string[] | null> {
const openApiSchema = await getReplicateOpenApiSchemaCached(apiKey, model);
const apiVoices = extractReplicateVoicesFromOpenApiSchema(openApiSchema);
return apiVoices.length > 0 ? apiVoices : null;
}
export async function resolveReplicateVoiceInputKey({
provider,
model,
apiKey = '',
}: ResolveVoicesOptions): Promise<ReplicateVoiceInputKey | null> {
if (provider !== 'replicate' || REPLICATE_BUILT_IN_MODELS.has(model) || !apiKey) {
return null;
}
const cached = replicateVoiceInputKeyCache.get(model);
if (cached) {
return cached;
}
const openApiSchema = await getReplicateOpenApiSchemaCached(apiKey, model);
const inputKey = extractReplicateVoiceInputKeyFromOpenApiSchema(openApiSchema);
if (inputKey) {
replicateVoiceInputKeyCache.set(model, inputKey);
}
return inputKey;
}
export function resolveVoiceSource(provider: string, model: string): TtsVoiceSource {
if (provider === 'deepinfra' && (
model === 'ResembleAI/chatterbox' ||
model === 'Zyphra/Zonos-v0.1-hybrid' ||
model === 'Zyphra/Zonos-v0.1-transformer'
)) {
if (provider === 'deepinfra' && DEEPINFRA_API_VOICE_MODELS.has(model)) {
return 'deepinfra-api';
}
if (provider === 'replicate' && parseReplicateModelIdentifier(model) !== null) {
return 'replicate-api';
}
if (provider === 'custom-openai') {
return 'custom-openai-api';
}
@ -335,5 +557,15 @@ export async function resolveVoices({ provider, model, apiKey = '', baseUrl = ''
}
}
if (voiceSource === 'replicate-api') {
if (!apiKey) {
return defaultVoices;
}
const apiVoices = await fetchReplicateVoices(apiKey, model);
if (apiVoices !== null) {
return apiVoices;
}
}
return defaultVoices;
}

21
tests/unit/kokoro.spec.ts Normal file
View file

@ -0,0 +1,21 @@
import { expect, test } from '@playwright/test';
import { getMaxVoicesForProvider } from '../../src/lib/shared/kokoro';
test.describe('kokoro voice limits', () => {
test('keeps Replicate single-voice even for Kokoro models', () => {
expect(getMaxVoicesForProvider('replicate', 'kokoro')).toBe(1);
expect(getMaxVoicesForProvider('replicate', 'hexgrad/Kokoro-82M')).toBe(1);
});
test('keeps Deepinfra single-voice and allows multi-voice elsewhere', () => {
expect(getMaxVoicesForProvider('deepinfra', 'hexgrad/Kokoro-82M')).toBe(1);
expect(getMaxVoicesForProvider('custom-openai', 'kokoro')).toBe(Infinity);
});
test('non-kokoro models are always single-voice', () => {
expect(getMaxVoicesForProvider('replicate', 'google/gemini-3.1-flash-tts')).toBe(1);
expect(getMaxVoicesForProvider('openai', 'gpt-4o-mini-tts')).toBe(1);
});
});

245
tests/unit/tts-provider-catalog.spec.ts
View file

@ -1,10 +1,13 @@
import { expect, test } from '@playwright/test';
import {
REPLICATE_KOKORO_82M_VERSIONED_MODEL,
getDefaultVoices,
providerSupportsCustomModel,
resolveReplicateVoiceInputKey,
resolveVoices,
resolveProviderModels,
supportsNativeModelSpeed,
supportsTtsInstructions,
} from '../../src/lib/shared/tts-provider-catalog';
import { applyConfigUpdate, getVoicePreferenceKey } from '../../src/lib/client/config/updates';
@ -50,6 +53,13 @@ test.describe('tts provider catalog', () => {
expect(providerSupportsCustomModel('deepinfra')).toBe(true);
});
test('uses blocklist semantics for Replicate native speed support', () => {
expect(supportsNativeModelSpeed('replicate', REPLICATE_KOKORO_82M_VERSIONED_MODEL)).toBe(true);
expect(supportsNativeModelSpeed('replicate', 'acme/runtime-speed-model')).toBe(true);
expect(supportsNativeModelSpeed('replicate', 'google/gemini-3.1-flash-tts')).toBe(false);
expect(supportsNativeModelSpeed('replicate', 'qwen/qwen3-tts')).toBe(false);
});
test('keeps explicit empty custom-openai voices response', async () => {
const originalFetch = globalThis.fetch;
globalThis.fetch = async () => ({
@ -68,6 +78,241 @@ test.describe('tts provider catalog', () => {
globalThis.fetch = originalFetch;
}
});
test('fetches custom Replicate model voices from model schema', async () => {
const originalFetch = globalThis.fetch;
let calls = 0;
let url = '';
globalThis.fetch = async (input) => {
calls += 1;
url = String(input);
return {
ok: true,
json: async () => ({
latest_version: {
openapi_schema: {
components: {
schemas: {
Input: {
type: 'object',
properties: {
voice: {
type: 'string',
enum: ['Narrator A', 'Narrator B'],
},
},
},
},
},
},
},
}),
} as Response;
};
try {
await expect(resolveVoices({
provider: 'replicate',
model: 'acme/my-tts-model-voices',
apiKey: 'r8_token',
})).resolves.toEqual(['Narrator A', 'Narrator B']);
expect(calls).toBe(1);
expect(url).toContain('/v1/models/acme/my-tts-model-voices');
} finally {
globalThis.fetch = originalFetch;
}
});
test('falls back to default voice for custom Replicate models when schema lookup fails', async () => {
const originalFetch = globalThis.fetch;
globalThis.fetch = async () => ({
ok: false,
status: 404,
json: async () => ({}),
}) as Response;
try {
await expect(resolveVoices({
provider: 'replicate',
model: 'acme/my-tts-model-fallback',
apiKey: 'r8_token',
})).resolves.toEqual(['default']);
} finally {
globalThis.fetch = originalFetch;
}
});
test('uses schema voices first for built-in Replicate models', async () => {
const originalFetch = globalThis.fetch;
let calls = 0;
globalThis.fetch = async () => {
calls += 1;
return {
ok: true,
json: async () => ({
latest_version: {
openapi_schema: {
components: {
schemas: {
Input: {
type: 'object',
properties: {
speaker: {
type: 'string',
enum: ['Schema Aiden', 'Schema Dylan'],
},
},
},
},
},
},
},
}),
} as Response;
};
try {
await expect(resolveVoices({
provider: 'replicate',
model: 'qwen/qwen3-tts',
apiKey: 'r8_token',
})).resolves.toEqual(['Schema Aiden', 'Schema Dylan']);
expect(calls).toBe(1);
} finally {
globalThis.fetch = originalFetch;
}
});
test('falls back to built-in Replicate static voices when schema has no voice enum', async () => {
const originalFetch = globalThis.fetch;
let calls = 0;
globalThis.fetch = async () => {
calls += 1;
return {
ok: true,
json: async () => ({
latest_version: {
openapi_schema: {
components: {
schemas: {
Input: {
type: 'object',
properties: {
speaker: { type: 'string' },
},
},
},
},
},
},
}),
} as Response;
};
try {
await expect(resolveVoices({
provider: 'replicate',
model: 'inworld/tts-1.5-mini',
apiKey: 'r8_token',
})).resolves.toEqual(['Ashley', 'Dennis', 'Alex', 'Darlene']);
expect(calls).toBe(1);
} finally {
globalThis.fetch = originalFetch;
}
});
test('resolves Replicate custom-model voice input key and caches it', async () => {
const originalFetch = globalThis.fetch;
let calls = 0;
globalThis.fetch = async () => {
calls += 1;
return {
ok: true,
json: async () => ({
latest_version: {
openapi_schema: {
components: {
schemas: {
Input: {
type: 'object',
properties: {
voice_id: { type: 'string' },
},
},
},
},
},
},
}),
} as Response;
};
try {
await expect(resolveReplicateVoiceInputKey({
provider: 'replicate',
model: 'acme/custom-voice-model',
apiKey: 'r8_token',
})).resolves.toBe('voice_id');
await expect(resolveReplicateVoiceInputKey({
provider: 'replicate',
model: 'acme/custom-voice-model',
apiKey: 'r8_token',
})).resolves.toBe('voice_id');
expect(calls).toBe(1);
} finally {
globalThis.fetch = originalFetch;
}
});
test('shares Replicate schema fetch between voices and voice-key resolution', async () => {
const originalFetch = globalThis.fetch;
let calls = 0;
globalThis.fetch = async () => {
calls += 1;
return {
ok: true,
json: async () => ({
latest_version: {
openapi_schema: {
components: {
schemas: {
Input: {
type: 'object',
properties: {
voice: {
type: 'string',
enum: ['V1', 'V2'],
},
},
},
},
},
},
},
}),
} as Response;
};
try {
await expect(resolveVoices({
provider: 'replicate',
model: 'acme/shared-schema-model',
apiKey: 'r8_token',
})).resolves.toEqual(['V1', 'V2']);
await expect(resolveReplicateVoiceInputKey({
provider: 'replicate',
model: 'acme/shared-schema-model',
apiKey: 'r8_token',
})).resolves.toBe('voice');
expect(calls).toBe(1);
} finally {
globalThis.fetch = originalFetch;
}
});
});
test.describe('config helpers', () => {