danvics/openreader
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #87 from richardr1126/replicate

Browse source 
replicate
This commit is contained in:
Richard R 2026-04-16 13:13:03 -06:00 committed by GitHub
commit 67097109ed
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
22 changed files with 1208 additions and 174 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -18,7 +18,7 @@ OpenReader is an open source, self-host-friendly text-to-speech document reader
## ✨ Highlights
- 🎯 **Multi-provider TTS** with OpenAI-compatible endpoints (OpenAI, DeepInfra, Kokoro, KittenTTS-FastAPI, Orpheus, custom).
- 🎯 **Multi-provider TTS** with OpenAI-compatible endpoints and cloud providers (Kokoro-FastAPI, KittenTTS-FastAPI, Orpheus-FastAPI or OpenAI, Replicate, DeepInfra).
- 📖 **Read-along playback** for PDF/EPUB with sentence-aware narration.
- ⏱️ **Word-by-word highlighting** via optional `whisper.cpp` timestamps.
- 🛜 **Sync + library import** to bring docs across devices and from server-mounted folders.

41
docs-site/docs/configure/tts-provider-guides/replicate.md Normal file
View file

@ -0,0 +1,41 @@
---
title: Replicate
---
Use Replicate's hosted TTS models as your provider.
## Setup
**Environment variables (recommended for deployment):**
```env
API_KEY=r8_...
NEXT_PUBLIC_DEFAULT_TTS_PROVIDER=replicate
NEXT_PUBLIC_DEFAULT_TTS_MODEL=alphanumericuser/kokoro-82m:89b6fa84e4fa2dd6bd3a96be3e1f12827a3516c9fda8fddbac7a0be131c9a6f5
```
**Or in-app via Settings -> TTS Provider:**
1. Set provider to `Replicate`.
2. Enter your `API_KEY`.
3. Choose a model and voice.
Settings modal values override env vars. See [TTS Providers](../tts-providers) for how the two layers interact.
## Notes
- Built-in Replicate models:
- `alphanumericuser/kokoro-82m:89b6fa84e4fa2dd6bd3a96be3e1f12827a3516c9fda8fddbac7a0be131c9a6f5`
- `google/gemini-3.1-flash-tts`
- `minimax/speech-2.8-turbo`
- `qwen/qwen3-tts`
- `inworld/tts-1.5-mini`
- You can also choose `Other` and enter any Replicate model ID (for example `owner/model-name` or `owner/model-name:version`).
- Native model speed is not available on all Replicate models; OpenReader hides/disables native speed controls where unsupported.
- TTS requests are sent from the server, not the browser. The API key is never exposed to clients.
## References
- [Replicate](https://replicate.com/explore)
- [TTS Providers](../tts-providers)
- [TTS Environment Variables](../../reference/environment-variables#tts-provider-and-request-behavior)

12
docs-site/docs/configure/tts-providers.md
View file

@ -15,10 +15,17 @@ Set env vars as deployment-level defaults. Users (or you, in a single-user setup
## Providers
- **OpenAI**: Cloud. Base URL pre-filled (`https://api.openai.com/v1`). API key required.
- **Deepinfra**: Cloud. Base URL pre-filled (`https://api.deepinfra.com/v1/openai`). API key required.
- **Replicate**: Cloud. Base URL managed internally by OpenReader. API key required.
- **DeepInfra**: Cloud. Base URL pre-filled (`https://api.deepinfra.com/v1/openai`). API key required.
- **Custom OpenAI-Like**: Self-hosted or any custom endpoint. `API_BASE` must be set manually (typically ending in `/v1`). API key optional.
For `OpenAI` and `Deepinfra` you only need to supply an API key. For `Custom OpenAI-Like` you must also set `API_BASE`.
For `OpenAI`, `DeepInfra`, and `Replicate` you only need to supply an API key. For `Custom OpenAI-Like` you must also set `API_BASE`.
## Built-in model catalogs
- **Replicate** models: `alphanumericuser/kokoro-82m:89b6fa84e4fa2dd6bd3a96be3e1f12827a3516c9fda8fddbac7a0be131c9a6f5`, `google/gemini-3.1-flash-tts`, `minimax/speech-2.8-turbo`, `qwen/qwen3-tts`, `inworld/tts-1.5-mini` (or choose `Other` and enter any Replicate model ID, such as `owner/model` or `owner/model:version`)
- **OpenAI** models: `tts-1`, `tts-1-hd`, `gpt-4o-mini-tts`
- **DeepInfra** models: includes `hexgrad/Kokoro-82M` and additional hosted models (depending on API key / feature flags)
## Custom provider requirements
@ -36,6 +43,7 @@ TTS requests originate from the **Next.js server**, not the browser. `API_BASE`
- [Kokoro-FastAPI](./tts-provider-guides/kokoro-fastapi)
- [KittenTTS-FastAPI](./tts-provider-guides/kitten-tts-fastapi)
- [Orpheus-FastAPI](./tts-provider-guides/orpheus-fastapi)
- [Replicate](./tts-provider-guides/replicate)
- [DeepInfra](./tts-provider-guides/deepinfra)
- [OpenAI](./tts-provider-guides/openai)
- [Other](./tts-provider-guides/other)

11
docs-site/docs/deploy/vercel-deployment.md
View file

@ -18,8 +18,7 @@ This guide covers deploying OpenReader to Vercel with external Postgres and S3-c
Recommended production setup (auth enabled):
```bash
API_BASE=https://api.deepinfra.com/v1/openai
API_KEY=your_deepinfra_key
API_KEY=your_replicate_key
POSTGRES_URL=postgres://...
USE_EMBEDDED_WEED_MINI=false
S3_ACCESS_KEY_ID=...
@ -33,8 +32,8 @@ AUTH_SECRET=...
NEXT_PUBLIC_ENABLE_DOCX_CONVERSION=false
NEXT_PUBLIC_ENABLE_DESTRUCTIVE_DELETE_ACTIONS=false
NEXT_PUBLIC_ENABLE_TTS_PROVIDERS_TAB=false
NEXT_PUBLIC_DEFAULT_TTS_PROVIDER=deepinfra
NEXT_PUBLIC_DEFAULT_TTS_MODEL=hexgrad/Kokoro-82M
NEXT_PUBLIC_DEFAULT_TTS_PROVIDER=replicate
NEXT_PUBLIC_DEFAULT_TTS_MODEL=alphanumericuser/kokoro-82m:89b6fa84e4fa2dd6bd3a96be3e1f12827a3516c9fda8fddbac7a0be131c9a6f5
NEXT_PUBLIC_SHOW_ALL_DEEPINFRA_MODELS=false
NEXT_PUBLIC_ENABLE_AUDIOBOOK_EXPORT=true
NEXT_PUBLIC_ENABLE_WORD_HIGHLIGHT=false
@ -49,8 +48,8 @@ We recommend setting these defaults for a production-like environment:
- `NEXT_PUBLIC_ENABLE_DOCX_CONVERSION=false`: Disables DOCX upload (requires external tools anyway)
- `NEXT_PUBLIC_ENABLE_DESTRUCTIVE_DELETE_ACTIONS=false`: Hides destructive "Delete All" actions
- `NEXT_PUBLIC_ENABLE_TTS_PROVIDERS_TAB=false`: Hides the Settings -> TTS Provider section
- `NEXT_PUBLIC_DEFAULT_TTS_PROVIDER=deepinfra`: Points default TTS to a scalable provider
- `NEXT_PUBLIC_DEFAULT_TTS_MODEL=hexgrad/Kokoro-82M`: Uses a high-quality default model
- `NEXT_PUBLIC_DEFAULT_TTS_PROVIDER=replicate`: Points default TTS to a scalable provider
- `NEXT_PUBLIC_DEFAULT_TTS_MODEL=alphanumericuser/kokoro-82m:89b6fa84e4fa2dd6bd3a96be3e1f12827a3516c9fda8fddbac7a0be131c9a6f5`: Uses a low-cost default model
- `NEXT_PUBLIC_SHOW_ALL_DEEPINFRA_MODELS=false`: Restricts usage to free models if no key is provided
- `NEXT_PUBLIC_ENABLE_AUDIOBOOK_EXPORT=true`: (Optional) Controls audiobook export UI
- `NEXT_PUBLIC_ENABLE_WORD_HIGHLIGHT=false`: (Optional) Controls word highlighting UI (requires timestamp backend)

2
docs-site/docs/docker-quick-start.md
View file

@ -8,7 +8,7 @@ import TabItem from '@theme/TabItem';
## Prerequisites
- A recent Docker version installed
- A TTS API server that OpenReader can reach (Kokoro-FastAPI, KittenTTS-FastAPI, Orpheus-FastAPI, DeepInfra, OpenAI, or equivalent)
- A TTS API server that OpenReader can reach (Kokoro-FastAPI, KittenTTS-FastAPI, Orpheus-FastAPI, Replicate, DeepInfra, OpenAI, or equivalent)
:::note
If you have suitable hardware, you can run Kokoro locally with Docker. See [Kokoro-FastAPI](./configure/tts-provider-guides/kokoro-fastapi).

3
docs-site/docs/introduction.md
View file

@ -8,7 +8,7 @@ OpenReader is an open source text-to-speech document reader built with Next.js.
> Previously named **OpenReader-WebUI**.
It supports multiple TTS providers including OpenAI, DeepInfra, and custom OpenAI-compatible endpoints such as [Kokoro-FastAPI](https://github.com/remsky/Kokoro-FastAPI), [KittenTTS-FastAPI](https://github.com/richardr1126/KittenTTS-FastAPI), and [Orpheus-FastAPI](https://github.com/Lex-au/Orpheus-FastAPI).
It supports multiple TTS providers including OpenAI, Replicate, DeepInfra, and custom OpenAI-compatible endpoints such as [Kokoro-FastAPI](https://github.com/remsky/Kokoro-FastAPI), [KittenTTS-FastAPI](https://github.com/richardr1126/KittenTTS-FastAPI), and [Orpheus-FastAPI](https://github.com/Lex-au/Orpheus-FastAPI).
## ✨ Highlights
@ -18,6 +18,7 @@ It supports multiple TTS providers including OpenAI, DeepInfra, and custom OpenA
- [**Orpheus-FastAPI**](https://github.com/Lex-au/Orpheus-FastAPI)
- **Custom OpenAI-compatible**: any TTS API with `/v1/audio/voices` and `/v1/audio/speech` endpoints
- **Cloud TTS providers**:
- [**Replicate**](https://replicate.com/explore): includes a built-in catalog and supports any Replicate model ID via `Other`
- [**DeepInfra**](https://deepinfra.com/models/text-to-speech): Kokoro-82M and other hosted models
- [**OpenAI API**](https://platform.openai.com/docs/pricing#transcription-and-speech): `tts-1`, `tts-1-hd`, and `gpt-4o-mini-tts`
- 🛜 **Server-side Document Storage**

2
docs-site/docs/reference/environment-variables.md
View file

@ -379,7 +379,7 @@ Controls whether the **TTS Provider** section appears in the Settings modal.
Sets the default TTS provider for new users.
- Default: `custom-openai`
- Example values: `deepinfra`, `openai`, `custom-openai`
- Example values: `replicate`, `deepinfra`, `openai`, `custom-openai`
### NEXT_PUBLIC_DEFAULT_TTS_MODEL

1
docs-site/sidebars.ts
View file

@ -23,6 +23,7 @@ const sidebars: SidebarsConfig = {
'configure/tts-provider-guides/kokoro-fastapi',
'configure/tts-provider-guides/kitten-tts-fastapi',
'configure/tts-provider-guides/orpheus-fastapi',
'configure/tts-provider-guides/replicate',
'configure/tts-provider-guides/deepinfra',
'configure/tts-provider-guides/openai',
'configure/tts-provider-guides/other',

1
package.json
View file

@ -59,6 +59,7 @@
"react-pdf": "^9.2.1",
"react-reader": "^2.0.15",
"remark-gfm": "^4.0.1",
"replicate": "^1.4.0",
"uuid": "^11.1.0"
},
"devDependencies": {

11
pnpm-lock.yaml
View file

@ -124,6 +124,9 @@ importers:
remark-gfm:
specifier: ^4.0.1
version: 4.0.1
replicate:
specifier: ^1.4.0
version: 1.4.0
uuid:
specifier: ^11.1.0
version: 11.1.0
@ -3588,6 +3591,10 @@ packages:
remark-stringify@11.0.0:
resolution: {integrity: sha512-1OSmLd3awB/t8qdoEOMazZkNsfVTeY4fTsgzcQFdXNq8ToTN4ZGwrMnlda4K6smTFKD+GRV6O48i6Z4iKgPPpw==}
replicate@1.4.0:
resolution: {integrity: sha512-1ufKejfUVz/azy+5TnzQP7U1+MHVWZ6psnQ06az8byUUnRhT+DZ/MvewzB1NQYBVMgNKR7xPDtTwlcP5nv/5+w==}
engines: {git: '>=2.11.0', node: '>=18.0.0', npm: '>=7.19.0', yarn: '>=1.7.0'}
resolve-from@4.0.0:
resolution: {integrity: sha512-pb/MYmXstAkysRFx8piNI1tGFNQIFA3vkE3Gq4EuA1dF6gHp/+vgZqsCGJapvy8N3Q+4o7FwvquPJcnZ7RYy4g==}
engines: {node: '>=4'}
@ -8028,6 +8035,10 @@ snapshots:
mdast-util-to-markdown: 2.1.2
unified: 11.0.5
replicate@1.4.0:
optionalDependencies:
readable-stream: 4.7.0
resolve-from@4.0.0: {}
resolve-pkg-maps@1.0.0: {}

104
src/app/api/audiobook/chapter/route.ts
View file

@ -29,6 +29,7 @@ import { getOpenReaderTestNamespace, getUnclaimedUserIdForNamespace } from '@/li
import { buildAllowedAudiobookUserIds, pickAudiobookOwner } from '@/lib/server/audiobooks/user-scope';
import { getFFmpegPath } from '@/lib/server/audiobooks/ffmpeg-bin';
import { generateTTSBuffer } from '@/lib/server/tts/generate';
import { supportsNativeModelSpeed } from '@/lib/shared/tts-provider-catalog';
import type { AudiobookGenerationSettings } from '@/types/client';
import type { TTSAudiobookFormat } from '@/types/tts';
@ -40,7 +41,7 @@ interface ConversionRequest {
bookId?: string;
format?: TTSAudiobookFormat;
chapterIndex?: number;
settings?: AudiobookGenerationSettings;
settings?: unknown;
}
type ChapterObject = {
@ -92,6 +93,35 @@ function s3NotConfiguredResponse(): NextResponse {
);
}
function normalizeNativeSpeedForSettings(settings: AudiobookGenerationSettings): AudiobookGenerationSettings {
return supportsNativeModelSpeed(settings.ttsProvider, settings.ttsModel)
? settings
: { ...settings, nativeSpeed: 1 };
}
function isFiniteNumber(value: unknown): value is number {
return typeof value === 'number' && Number.isFinite(value);
}
function isAudiobookFormat(value: unknown): value is TTSAudiobookFormat {
return value === 'mp3' || value === 'm4b';
}
function isAudiobookGenerationSettings(value: unknown): value is AudiobookGenerationSettings {
if (typeof value !== 'object' || value === null) {
return false;
}
const record = value as Record<string, unknown>;
return typeof record.ttsProvider === 'string'
&& typeof record.ttsModel === 'string'
&& typeof record.voice === 'string'
&& isFiniteNumber(record.nativeSpeed)
&& isFiniteNumber(record.postSpeed)
&& isAudiobookFormat(record.format)
&& (record.ttsInstructions === undefined || typeof record.ttsInstructions === 'string');
}
function chapterFileMimeType(format: TTSAudiobookFormat): string {
return format === 'mp3' ? 'audio/mpeg' : 'audio/mp4';
}
@ -290,28 +320,53 @@ export async function POST(request: NextRequest) {
const existingChapters = listChapterObjects(objectNames);
const hasChapters = existingChapters.length > 0;
let existingSettings: AudiobookGenerationSettings | null = null;
let normalizedExistingSettings: AudiobookGenerationSettings | undefined;
try {
existingSettings = JSON.parse(
const parsedSettings = JSON.parse(
(await getAudiobookObjectBuffer(bookId, storageUserId, 'audiobook.meta.json', testNamespace)).toString('utf8'),
) as AudiobookGenerationSettings;
) as unknown;
if (!isAudiobookGenerationSettings(parsedSettings)) {
console.error('Invalid audiobook.meta.json settings payload', { bookId, storageUserId });
return NextResponse.json({ error: 'Invalid audiobook metadata settings' }, { status: 500 });
}
normalizedExistingSettings = normalizeNativeSpeedForSettings(parsedSettings);
} catch (error) {
if (!isMissingBlobError(error)) throw error;
existingSettings = null;
normalizedExistingSettings = undefined;
}
const incomingSettings = data.settings;
if (existingSettings && hasChapters && incomingSettings) {
const incomingSettings = (() => {
if (data.settings === undefined) {
return undefined;
}
if (!isAudiobookGenerationSettings(data.settings)) {
return null;
}
return normalizeNativeSpeedForSettings(data.settings);
})();
if (incomingSettings === null) {
return NextResponse.json({ error: 'Invalid audiobook settings payload' }, { status: 400 });
}
const mergedSettings = normalizedExistingSettings && incomingSettings
? normalizeNativeSpeedForSettings({
...normalizedExistingSettings,
...incomingSettings,
})
: normalizedExistingSettings ?? incomingSettings;
if (normalizedExistingSettings && hasChapters && incomingSettings) {
const mismatch =
existingSettings.ttsProvider !== incomingSettings.ttsProvider ||
existingSettings.ttsModel !== incomingSettings.ttsModel ||
existingSettings.voice !== incomingSettings.voice ||
existingSettings.nativeSpeed !== incomingSettings.nativeSpeed ||
existingSettings.postSpeed !== incomingSettings.postSpeed ||
existingSettings.format !== incomingSettings.format ||
(existingSettings.ttsInstructions || '') !== (incomingSettings.ttsInstructions || '');
normalizedExistingSettings.ttsProvider !== incomingSettings.ttsProvider ||
normalizedExistingSettings.ttsModel !== incomingSettings.ttsModel ||
normalizedExistingSettings.voice !== incomingSettings.voice ||
normalizedExistingSettings.nativeSpeed !== incomingSettings.nativeSpeed ||
normalizedExistingSettings.postSpeed !== incomingSettings.postSpeed ||
normalizedExistingSettings.format !== incomingSettings.format ||
(normalizedExistingSettings.ttsInstructions || '') !== (incomingSettings.ttsInstructions || '');
if (mismatch) {
return NextResponse.json({ error: 'Audiobook settings mismatch', settings: existingSettings }, { status: 409 });
return NextResponse.json({ error: 'Audiobook settings mismatch', settings: normalizedExistingSettings }, { status: 409 });
}
}
@ -322,10 +377,9 @@ export async function POST(request: NextRequest) {
const format: TTSAudiobookFormat =
(existingFormats.values().next().value as TTSAudiobookFormat | undefined) ??
existingSettings?.format ??
incomingSettings?.format ??
mergedSettings?.format ??
requestedFormat;
const rawPostSpeed = incomingSettings?.postSpeed ?? existingSettings?.postSpeed ?? 1;
const rawPostSpeed = mergedSettings?.postSpeed ?? 1;
const postSpeed = Number.isFinite(Number(rawPostSpeed)) ? Number(rawPostSpeed) : 1;
let chapterIndex: number;
@ -349,22 +403,20 @@ export async function POST(request: NextRequest) {
}
const provider = request.headers.get('x-tts-provider')
|| incomingSettings?.ttsProvider
|| existingSettings?.ttsProvider
|| mergedSettings?.ttsProvider
|| 'openai';
const openApiKey = request.headers.get('x-openai-key') || process.env.API_KEY || 'none';
const openApiBaseUrl = request.headers.get('x-openai-base-url') || process.env.API_BASE;
const model = incomingSettings?.ttsModel ?? existingSettings?.ttsModel;
const voice = incomingSettings?.voice
|| existingSettings?.voice
const model = mergedSettings?.ttsModel;
const voice = mergedSettings?.voice
|| (provider === 'openai'
? 'alloy'
: provider === 'deepinfra'
? 'af_bella'
: 'af_sarah');
const rawNativeSpeed = incomingSettings?.nativeSpeed ?? existingSettings?.nativeSpeed ?? 1;
const rawNativeSpeed = mergedSettings?.nativeSpeed ?? 1;
const nativeSpeed = Number.isFinite(Number(rawNativeSpeed)) ? Number(rawNativeSpeed) : 1;
const instructions = incomingSettings?.ttsInstructions ?? existingSettings?.ttsInstructions;
const instructions = mergedSettings?.ttsInstructions;
if (authEnabled && userId && isTtsRateLimitEnabled()) {
const isAnonymous = Boolean(user?.isAnonymous);
@ -499,7 +551,7 @@ export async function POST(request: NextRequest) {
await deleteAudiobookObject(bookId, storageUserId, 'complete.mp3.manifest.json', testNamespace).catch(() => {});
await deleteAudiobookObject(bookId, storageUserId, 'complete.m4b.manifest.json', testNamespace).catch(() => {});
if (!existingSettings && incomingSettings) {
if (!normalizedExistingSettings && incomingSettings) {
await putAudiobookObject(
bookId,
storageUserId,

22
src/app/api/tts/route.ts
View file

@ -14,6 +14,7 @@ import {
getCachedTTSBuffer,
getTTSContentType,
} from '@/lib/server/tts/generate';
import { getUpstreamRetryAfterSeconds, getUpstreamStatus } from '@/lib/server/tts/upstream-response';
export const runtime = 'nodejs';
export const maxDuration = 60;
@ -49,14 +50,6 @@ function formatLimitForHint(limit: number): string {
return String(limit);
}
function getUpstreamStatus(error: unknown): number | undefined {
if (typeof error !== 'object' || error === null) return undefined;
const rec = error as Record<string, unknown>;
if (typeof rec.status === 'number') return rec.status;
if (typeof rec.statusCode === 'number') return rec.statusCode;
return undefined;
}
export async function POST(req: NextRequest) {
let providerForError: string | null = null;
let didCreateDeviceIdCookie = false;
@ -221,14 +214,18 @@ export async function POST(req: NextRequest) {
const upstreamStatus = getUpstreamStatus(error);
if (upstreamStatus === 429) {
const retryAfterSeconds = getUpstreamRetryAfterSeconds(error);
const problem: ProblemDetails = {
type: PROBLEM_TYPES.upstreamRateLimited,
title: 'Upstream rate limited',
status: 429,
detail: 'The TTS provider is rate limiting requests. Please try again shortly.',
detail: retryAfterSeconds
? `The TTS provider is rate limiting requests. Please retry in about ${retryAfterSeconds}s.`
: 'The TTS provider is rate limiting requests. Please try again shortly.',
code: 'UPSTREAM_RATE_LIMIT',
provider: providerForError ?? undefined,
upstreamStatus,
retryAfterSeconds,
instance: req.nextUrl.pathname,
};
@ -236,11 +233,16 @@ export async function POST(req: NextRequest) {
status: 429,
headers: {
'Content-Type': 'application/problem+json',
...(retryAfterSeconds ? { 'Retry-After': String(retryAfterSeconds) } : {}),
},
});
}
console.warn('Error generating TTS:', error);
const statusHint = getUpstreamStatus(error);
const errorMessage = error instanceof Error ? error.message : String(error);
console.warn(
`Error generating TTS${typeof statusHint === 'number' ? ` (upstream ${statusHint})` : ''}: ${errorMessage}`,
);
const errorBody: TTSError = {
code: 'TTS_GENERATION_FAILED',
message: 'Failed to generate audio',

76
src/components/AudiobookExportModal.tsx
View file

@ -11,7 +11,7 @@ import { LoadingSpinner } from '@/components/Spinner';
import { useConfig } from '@/contexts/ConfigContext';
import { useTTS } from '@/contexts/TTSContext';
import { VoicesControlBase } from '@/components/player/VoicesControlBase';
import { supportsTtsInstructions } from '@/lib/shared/tts-provider-catalog';
import { supportsNativeModelSpeed, supportsTtsInstructions } from '@/lib/shared/tts-provider-catalog';
import type { TTSAudiobookChapter, TTSAudiobookFormat } from '@/types/tts';
import {
getAudiobookStatus,
@ -73,6 +73,8 @@ export function AudiobookExportModal({
const formatSpeed = useCallback((speed: number) => {
return Number.isInteger(speed) ? speed.toString() : speed.toFixed(1);
}, []);
const nativeSpeedSupported = useMemo(() => supportsNativeModelSpeed(ttsProvider, ttsModel), [ttsProvider, ttsModel]);
const effectiveNativeSpeed = nativeSpeedSupported ? nativeSpeed : 1;
const hasExistingAudiobook = Boolean(bookId) || chapters.length > 0;
const isLegacyAudiobookMissingSettings = hasExistingAudiobook && savedSettings === null;
@ -113,12 +115,12 @@ export function AudiobookExportModal({
ttsProvider,
ttsModel,
voice: nextVoice,
nativeSpeed,
nativeSpeed: effectiveNativeSpeed,
postSpeed,
format,
ttsInstructions: supportsTtsInstructions(ttsModel) ? ttsInstructions : undefined,
};
}, [savedSettings, audiobookVoice, configVoice, availableVoices, ttsProvider, ttsModel, ttsInstructions, nativeSpeed, postSpeed, format]);
}, [savedSettings, audiobookVoice, configVoice, availableVoices, ttsProvider, ttsModel, ttsInstructions, effectiveNativeSpeed, postSpeed, format]);
const fetchExistingChapters = useCallback(async (soft: boolean = false) => {
if (soft) {
@ -529,11 +531,15 @@ export function AudiobookExportModal({
<div className="text-sm font-medium text-foreground">{savedSettings.format.toUpperCase()}</div>
</div>
</div>
<div className="grid grid-cols-2 gap-3">
<div className="rounded-lg bg-base p-3">
<div className="text-[11px] uppercase tracking-wider text-muted mb-1">Native speed</div>
<div className="text-sm font-medium text-foreground">{formatSpeed(savedSettings.nativeSpeed)}x</div>
</div>
<div className="grid grid-cols-2 gap-3">
<div className="rounded-lg bg-base p-3">
<div className="text-[11px] uppercase tracking-wider text-muted mb-1">Native speed</div>
<div className="text-sm font-medium text-foreground">
{supportsNativeModelSpeed(savedSettings.ttsProvider, savedSettings.ttsModel)
? `${formatSpeed(savedSettings.nativeSpeed)}x`
: 'Not supported'}
</div>
</div>
<div className="rounded-lg bg-base p-3">
<div className="text-[11px] uppercase tracking-wider text-muted mb-1">Post speed</div>
<div className="text-sm font-medium text-foreground">{formatSpeed(savedSettings.postSpeed)}x</div>
@ -616,29 +622,39 @@ export function AudiobookExportModal({
</div>
</div>
{/* Speed controls */}
<div className="rounded-lg bg-base p-3 space-y-3">
<div className="space-y-2">
<div className="flex items-center justify-between">
<label className="text-[11px] uppercase tracking-wider font-medium text-muted">Native model speed</label>
<span className="text-xs font-medium text-accent tabular-nums">{formatSpeed(nativeSpeed)}x</span>
</div>
<input
type="range"
min="0.5"
max="3"
step="0.1"
value={nativeSpeed}
onChange={(e) => setNativeSpeed(parseFloat(e.target.value))}
className="w-full h-1.5 bg-offbase rounded-full appearance-none cursor-pointer [&::-webkit-slider-runnable-track]:bg-offbase [&::-webkit-slider-runnable-track]:rounded-full [&::-webkit-slider-runnable-track]:h-1.5 [&::-webkit-slider-thumb]:appearance-none [&::-webkit-slider-thumb]:h-4 [&::-webkit-slider-thumb]:w-4 [&::-webkit-slider-thumb]:rounded-full [&::-webkit-slider-thumb]:bg-accent [&::-webkit-slider-thumb]:-mt-[5px] [&::-webkit-slider-thumb]:shadow-sm [&::-moz-range-track]:bg-offbase [&::-moz-range-track]:rounded-full [&::-moz-range-track]:h-1.5 [&::-moz-range-thumb]:appearance-none [&::-moz-range-thumb]:h-4 [&::-moz-range-thumb]:w-4 [&::-moz-range-thumb]:rounded-full [&::-moz-range-thumb]:bg-accent [&::-moz-range-thumb]:border-0"
/>
<div className="flex justify-between text-[10px] text-muted">
<span>0.5x</span>
<span>3x</span>
</div>
</div>
{/* Speed controls */}
<div className="rounded-lg bg-base p-3 space-y-3">
{!nativeSpeedSupported && (
<div className="rounded-md border border-offbase bg-background px-2 py-1.5 text-[11px] text-muted">
Native model speed is not available for this model.
</div>
)}
<div className="border-t border-offbase" />
{nativeSpeedSupported && (
<>
<div className="space-y-2">
<div className="flex items-center justify-between">
<label className="text-[11px] uppercase tracking-wider font-medium text-muted">Native model speed</label>
<span className="text-xs font-medium text-accent tabular-nums">{formatSpeed(nativeSpeed)}x</span>
</div>
<input
type="range"
min="0.5"
max="3"
step="0.1"
value={nativeSpeed}
onChange={(e) => setNativeSpeed(parseFloat(e.target.value))}
className="w-full h-1.5 bg-offbase rounded-full appearance-none cursor-pointer [&::-webkit-slider-runnable-track]:bg-offbase [&::-webkit-slider-runnable-track]:rounded-full [&::-webkit-slider-runnable-track]:h-1.5 [&::-webkit-slider-thumb]:appearance-none [&::-webkit-slider-thumb]:h-4 [&::-webkit-slider-thumb]:w-4 [&::-webkit-slider-thumb]:rounded-full [&::-webkit-slider-thumb]:bg-accent [&::-webkit-slider-thumb]:-mt-[5px] [&::-webkit-slider-thumb]:shadow-sm [&::-moz-range-track]:bg-offbase [&::-moz-range-track]:rounded-full [&::-moz-range-track]:h-1.5 [&::-moz-range-thumb]:appearance-none [&::-moz-range-thumb]:h-4 [&::-moz-range-thumb]:w-4 [&::-moz-range-thumb]:rounded-full [&::-moz-range-thumb]:bg-accent [&::-moz-range-thumb]:border-0"
/>
<div className="flex justify-between text-[10px] text-muted">
<span>0.5x</span>
<span>3x</span>
</div>
</div>
<div className="border-t border-offbase" />
</>
)}
<div className="space-y-2">
<div className="flex items-center justify-between">

63
src/components/SettingsModal.tsx
View file

@ -36,7 +36,7 @@ import { deleteDocuments, mimeTypeForDoc, uploadDocuments } from '@/lib/client/a
import { cacheStoredDocumentFromBytes, clearDocumentCache } from '@/lib/client/cache/documents';
import { clearAllDocumentPreviewCaches, clearInMemoryDocumentPreviewCache } from '@/lib/client/cache/previews';
import { resolveTtsSettingsViewModel } from '@/lib/client/settings/tts-settings';
import { supportsTtsInstructions } from '@/lib/shared/tts-provider-catalog';
import { REPLICATE_KOKORO_82M_VERSIONED_MODEL, supportsTtsInstructions } from '@/lib/shared/tts-provider-catalog';
const enableDestructiveDelete = process.env.NEXT_PUBLIC_ENABLE_DESTRUCTIVE_DELETE_ACTIONS !== 'false';
const showAllDeepInfra = process.env.NEXT_PUBLIC_SHOW_ALL_DEEPINFRA_MODELS !== 'false';
@ -392,6 +392,12 @@ export function SettingsModal({ className = '' }: { className?: string }) {
const btnPrimary = `${btnBase} bg-accent text-background hover:bg-secondary-accent hover:scale-[1.04]`;
const btnSecondary = `${btnBase} bg-background text-foreground hover:bg-offbase hover:text-accent hover:scale-[1.04]`;
const btnOutline = `${btnBase} bg-background border border-offbase text-foreground hover:bg-offbase hover:text-accent hover:scale-[1.02]`;
const shouldShowBaseUrl = localTTSProvider !== 'replicate'
&& (localTTSProvider === 'custom-openai' || !localBaseUrl || localBaseUrl === '');
const selectedModel = ttsModels.find(m => m.id === selectedModelId) || ttsModels[0];
const selectedModelVersion = selectedModel?.id?.includes(':')
? selectedModel.id.slice(selectedModel.id.indexOf(':'))
: '';
return (
<>
@ -491,7 +497,7 @@ export function SettingsModal({ className = '' }: { className?: string }) {
</nav>
{/* Content */}
<div className="flex-1 p-4 overflow-y-auto">
<div className="flex-1 min-w-0 p-4 overflow-y-auto">
{/* API Section */}
{activeSection === 'api' && (
<div className="space-y-4">
@ -507,6 +513,9 @@ export function SettingsModal({ className = '' }: { className?: string }) {
} else if (provider.id === 'custom-openai') {
setModelValue('kokoro');
setLocalBaseUrl('');
} else if (provider.id === 'replicate') {
setModelValue(REPLICATE_KOKORO_82M_VERSIONED_MODEL);
setLocalBaseUrl('');
} else if (provider.id === 'deepinfra') {
setModelValue('hexgrad/Kokoro-82M');
setLocalBaseUrl('https://api.deepinfra.com/v1/openai');
@ -528,7 +537,10 @@ export function SettingsModal({ className = '' }: { className?: string }) {
leaveFrom="opacity-100"
leaveTo="opacity-0"
>
<ListboxOptions className="absolute mt-1 w-full overflow-auto rounded-md bg-background py-1 shadow-lg ring-1 ring-offbase focus:outline-none z-50">
<ListboxOptions
anchor="bottom start"
className="z-50 w-[var(--button-width)] max-h-60 overflow-y-auto overscroll-contain rounded-md bg-background py-1 shadow-lg ring-1 ring-offbase focus:outline-none [--anchor-gap:0.25rem]"
>
{ttsProviders.map((provider) => (
<ListboxOption
key={provider.id}
@ -556,7 +568,7 @@ export function SettingsModal({ className = '' }: { className?: string }) {
</Listbox>
</div>
{(localTTSProvider === 'custom-openai' || !localBaseUrl || localBaseUrl === '') && (
{shouldShowBaseUrl && (
<div className="space-y-1.5">
<label className="block text-sm font-medium text-foreground">
API Base URL
@ -601,9 +613,20 @@ export function SettingsModal({ className = '' }: { className?: string }) {
}}
>
<ListboxButton className="relative w-full cursor-pointer rounded-lg bg-background py-2 pl-3 pr-10 text-left text-foreground shadow-sm focus:outline-none focus:ring-2 focus:ring-accent transform transition-transform duration-200 ease-in-out hover:scale-[1.005] hover:text-accent hover:bg-offbase">
<span className="block truncate">
{ttsModels.find(m => m.id === selectedModelId)?.name || 'Select Model'}
</span>
{selectedModel ? (
<span className="block">
<span className="block truncate">
{selectedModel.name}
</span>
{selectedModelVersion && (
<span className="block truncate text-xs text-muted">
{selectedModelVersion}
</span>
)}
</span>
) : (
<span className="block truncate">Select Model</span>
)}
<span className="pointer-events-none absolute inset-y-0 right-0 flex items-center pr-2">
<ChevronUpDownIcon className="h-5 w-5 text-muted" />
</span>
@ -614,7 +637,10 @@ export function SettingsModal({ className = '' }: { className?: string }) {
leaveFrom="opacity-100"
leaveTo="opacity-0"
>
<ListboxOptions className="absolute mt-1 w-full overflow-auto rounded-md bg-background py-1 shadow-lg ring-1 ring-offbase focus:outline-none z-50">
<ListboxOptions
anchor="bottom start"
className="z-50 w-[var(--button-width)] max-h-60 overflow-y-auto overscroll-contain rounded-md bg-background py-1 shadow-lg ring-1 ring-offbase focus:outline-none [--anchor-gap:0.25rem]"
>
{ttsModels.map((model) => (
<ListboxOption
key={model.id}
@ -622,14 +648,19 @@ export function SettingsModal({ className = '' }: { className?: string }) {
`relative cursor-pointer select-none py-2 pl-10 pr-4 ${active ? 'bg-offbase text-accent' : 'text-foreground'}`
}
value={model}
>
{({ selected }) => (
<>
<span className={`block truncate ${selected ? 'font-medium' : 'font-normal'}`}>
{model.name}
</span>
{selected && (
<span className="absolute inset-y-0 left-0 flex items-center pl-3 text-accent">
>
{({ selected }) => (
<>
<span className={`block ${selected ? 'font-medium' : 'font-normal'}`}>
<span className="block truncate">{model.name}</span>
{model.id.includes(':') && (
<span className="block truncate text-xs text-muted">
{model.id.slice(model.id.indexOf(':'))}
</span>
)}
</span>
{selected && (
<span className="absolute inset-y-0 left-0 flex items-center pl-3 text-accent">
<CheckIcon className="h-5 w-5" />
</span>
)}

66
src/components/player/SpeedControl.tsx
View file

@ -4,6 +4,7 @@ import { Input, Popover, PopoverButton, PopoverPanel } from '@headlessui/react';
import { ChevronUpDownIcon, SpeedometerIcon } from '@/components/icons/Icons';
import { useConfig } from '@/contexts/ConfigContext';
import { useCallback, useEffect, useMemo, useState } from 'react';
import { supportsNativeModelSpeed } from '@/lib/shared/tts-provider-catalog';
export const SpeedControl = ({
setSpeedAndRestart,
@ -12,7 +13,8 @@ export const SpeedControl = ({
setSpeedAndRestart: (speed: number) => void;
setAudioPlayerSpeedAndRestart: (speed: number) => void;
}) => {
const { voiceSpeed, audioPlayerSpeed } = useConfig();
const { voiceSpeed, audioPlayerSpeed, ttsProvider, ttsModel } = useConfig();
const nativeSpeedSupported = supportsNativeModelSpeed(ttsProvider, ttsModel);
const [localVoiceSpeed, setLocalVoiceSpeed] = useState(voiceSpeed);
const [localAudioSpeed, setLocalAudioSpeed] = useState(audioPlayerSpeed);
@ -53,28 +55,28 @@ export const SpeedControl = ({
const triggerLabel = useMemo(
() => {
const parts: string[] = [];
if (localVoiceSpeed !== 1.0) parts.push(`${formatSpeed(localVoiceSpeed, 1)}x`);
if (nativeSpeedSupported && localVoiceSpeed !== 1.0) parts.push(`${formatSpeed(localVoiceSpeed, 1)}x`);
if (localAudioSpeed !== 1.0) parts.push(`${formatSpeed(localAudioSpeed, 1)}x`);
return parts.length > 0 ? parts.join(' • ') : '1x';
},
[formatSpeed, localVoiceSpeed, localAudioSpeed]
[formatSpeed, localVoiceSpeed, localAudioSpeed, nativeSpeedSupported]
);
const compactTriggerLabel = useMemo(() => {
const voiceIsDefault = localVoiceSpeed === 1.0;
const voiceIsDefault = !nativeSpeedSupported || localVoiceSpeed === 1.0;
const audioIsDefault = localAudioSpeed === 1.0;
let combined = 1.0;
if (!voiceIsDefault && !audioIsDefault) {
if (nativeSpeedSupported && !voiceIsDefault && !audioIsDefault) {
combined = (localVoiceSpeed + localAudioSpeed) / 2;
} else if (!voiceIsDefault) {
} else if (nativeSpeedSupported && !voiceIsDefault) {
combined = localVoiceSpeed;
} else if (!audioIsDefault) {
combined = localAudioSpeed;
}
return `${formatSpeed(combined, 2)}x`;
}, [formatSpeed, localVoiceSpeed, localAudioSpeed]);
}, [formatSpeed, localVoiceSpeed, localAudioSpeed, nativeSpeedSupported]);
const min = 0.5;
const max = 3;
@ -90,28 +92,36 @@ export const SpeedControl = ({
</PopoverButton>
<PopoverPanel anchor="top" className="absolute z-50 bg-base p-3 rounded-md shadow-lg border border-offbase">
<div className="flex flex-col space-y-4">
<div className="flex flex-col space-y-2">
<div className="text-xs font-medium text-foreground">Native model speed</div>
<div className="flex justify-between">
<span className="text-xs">{min.toFixed(1)}x</span>
<span className="text-xs font-bold">
{Number.isInteger(localVoiceSpeed) ? localVoiceSpeed.toString() : localVoiceSpeed.toFixed(1)}x
</span>
<span className="text-xs">{max.toFixed(1)}x</span>
{!nativeSpeedSupported && (
<div className="rounded-md border border-offbase bg-background px-2 py-1.5 text-[11px] text-muted">
Native model speed is not available for this model.
</div>
<Input
type="range"
min={min}
max={max}
step={step}
value={localVoiceSpeed}
onChange={handleVoiceSpeedChange}
onMouseUp={handleVoiceSpeedChangeComplete}
onKeyUp={handleVoiceSpeedChangeComplete}
onTouchEnd={handleVoiceSpeedChangeComplete}
className="w-full bg-offbase rounded-lg appearance-none cursor-pointer accent-accent [&::-webkit-slider-runnable-track]:bg-offbase [&::-webkit-slider-runnable-track]:rounded-lg [&::-webkit-slider-thumb]:appearance-none [&::-webkit-slider-thumb]:h-4 [&::-webkit-slider-thumb]:w-4 [&::-webkit-slider-thumb]:rounded-full [&::-webkit-slider-thumb]:bg-accent [&::-moz-range-track]:bg-offbase [&::-moz-range-track]:rounded-lg [&::-moz-range-thumb]:appearance-none [&::-moz-range-thumb]:h-4 [&::-moz-range-thumb]:w-4 [&::-moz-range-thumb]:rounded-full [&::-moz-range-thumb]:bg-accent"
/>
</div>
)}
{nativeSpeedSupported && (
<div className="flex flex-col space-y-2">
<div className="text-xs font-medium text-foreground">Native model speed</div>
<div className="flex justify-between">
<span className="text-xs">{min.toFixed(1)}x</span>
<span className="text-xs font-bold">
{Number.isInteger(localVoiceSpeed) ? localVoiceSpeed.toString() : localVoiceSpeed.toFixed(1)}x
</span>
<span className="text-xs">{max.toFixed(1)}x</span>
</div>
<Input
type="range"
min={min}
max={max}
step={step}
value={localVoiceSpeed}
onChange={handleVoiceSpeedChange}
onMouseUp={handleVoiceSpeedChangeComplete}
onKeyUp={handleVoiceSpeedChangeComplete}
onTouchEnd={handleVoiceSpeedChangeComplete}
className="w-full bg-offbase rounded-lg appearance-none cursor-pointer accent-accent [&::-webkit-slider-runnable-track]:bg-offbase [&::-webkit-slider-runnable-track]:rounded-lg [&::-webkit-slider-thumb]:appearance-none [&::-webkit-slider-thumb]:h-4 [&::-webkit-slider-thumb]:w-4 [&::-webkit-slider-thumb]:rounded-full [&::-webkit-slider-thumb]:bg-accent [&::-moz-range-track]:bg-offbase [&::-moz-range-track]:rounded-lg [&::-moz-range-thumb]:appearance-none [&::-moz-range-thumb]:h-4 [&::-moz-range-thumb]:w-4 [&::-moz-range-thumb]:rounded-full [&::-moz-range-thumb]:bg-accent"
/>
</div>
)}
<div className="flex flex-col space-y-2">
<div className="text-xs font-medium text-foreground">Audio player speed</div>

22
src/contexts/TTSContext.tsx
View file

@ -39,7 +39,7 @@ import { getDocumentProgress, scheduleDocumentProgressSync } from '@/lib/client/
import { withRetry, generateTTS, alignAudio } from '@/lib/client/api/audiobooks';
import { preprocessSentenceForAudio, splitTextToTtsBlocks, splitTextToTtsBlocksEPUB } from '@/lib/shared/nlp';
import { isKokoroModel } from '@/lib/shared/kokoro';
import { supportsTtsInstructions } from '@/lib/shared/tts-provider-catalog';
import { supportsNativeModelSpeed, supportsTtsInstructions } from '@/lib/shared/tts-provider-catalog';
import { useAuthRateLimit } from '@/contexts/AuthRateLimitContext';
import type {
TTSLocation,
@ -386,6 +386,10 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
const [voice, setVoice] = useState(configVoice);
const [ttsModel, setTTSModel] = useState(configTTSModel);
const [ttsInstructions, setTTSInstructions] = useState(configTTSInstructions);
const effectiveNativeSpeed = useMemo(
() => (supportsNativeModelSpeed(configTTSProvider, ttsModel) ? speed : 1),
[configTTSProvider, ttsModel, speed],
);
// Track pending preload requests
const preloadRequests = useRef<Map<string, Promise<string>>>(new Map());
@ -1015,7 +1019,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
const alignmentKey = buildCacheKey(
sentence,
voice,
speed,
effectiveNativeSpeed,
configTTSProvider,
ttsModel,
);
@ -1053,7 +1057,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
const audioCacheKey = buildCacheKey(
sentence,
voice,
speed,
effectiveNativeSpeed,
configTTSProvider,
ttsModel,
);
@ -1097,7 +1101,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
const reqBody: TTSRequestPayload = {
text: sentence,
voice,
speed,
speed: effectiveNativeSpeed,
format: 'mp3',
model: ttsModel,
instructions: supportsTtsInstructions(ttsModel) ? ttsInstructions : undefined,
@ -1190,7 +1194,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
}
}, [
voice,
speed,
effectiveNativeSpeed,
ttsModel,
ttsInstructions,
audioCache,
@ -1532,7 +1536,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
const alignmentKey = buildCacheKey(
sentence,
voice,
speed,
effectiveNativeSpeed,
configTTSProvider,
ttsModel,
);
@ -1549,7 +1553,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
if (howl) {
howl.play();
}
}, [sentences, currentIndex, playSentenceWithHowl, voice, speed, configTTSProvider, ttsModel]);
}, [sentences, currentIndex, playSentenceWithHowl, voice, effectiveNativeSpeed, configTTSProvider, ttsModel]);
// Keep the current playback rate applied to the active Howl. Some browsers (notably
// iOS Safari with HTML5 audio) can reset playbackRate after initial load/play.
@ -1620,7 +1624,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
const nextKey = buildCacheKey(
nextSentence,
voice,
speed,
effectiveNativeSpeed,
configTTSProvider,
ttsModel,
);
@ -1673,7 +1677,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
} catch (error) {
console.error('Error initiating preload:', error);
}
}, [isAtLimit, currentIndex, sentences, audioCache, processSentence, voice, speed, configTTSProvider, ttsModel]);
}, [isAtLimit, currentIndex, sentences, audioCache, processSentence, voice, effectiveNativeSpeed, configTTSProvider, ttsModel]);
/**
* Main Playback Driver

199
src/lib/server/tts/generate.ts
View file

@ -1,7 +1,14 @@
import OpenAI from 'openai';
import Replicate from 'replicate';
import { SpeechCreateParams } from 'openai/resources/audio/speech.mjs';
import { isKokoroModel } from '@/lib/shared/kokoro';
import { supportsTtsInstructions } from '@/lib/shared/tts-provider-catalog';
import {
REPLICATE_KOKORO_82M_VERSIONED_MODEL,
resolveReplicateVoiceInputKey,
supportsNativeModelSpeed,
supportsTtsInstructions,
} from '@/lib/shared/tts-provider-catalog';
import { getUpstreamRetryAfterSeconds, getUpstreamStatus } from '@/lib/server/tts/upstream-response';
import { LRUCache } from 'lru-cache';
import { createHash } from 'crypto';
import { access, readFile } from 'fs/promises';
@ -45,6 +52,8 @@ type InflightEntry = {
consumers: number;
};
let replicateBlockedUntilMs = 0;
const TTS_CACHE_MAX_SIZE_BYTES = Number(process.env.TTS_CACHE_MAX_SIZE_BYTES || 256 * 1024 * 1024); // 256MB
const TTS_CACHE_TTL_MS = Number(process.env.TTS_CACHE_TTL_MS || 1000 * 60 * 30); // 30 minutes
@ -63,27 +72,58 @@ function sleep(ms: number) {
return new Promise((res) => setTimeout(res, ms));
}
function getUpstreamStatus(error: unknown): number | undefined {
if (typeof error !== 'object' || error === null) return undefined;
const rec = error as Record<string, unknown>;
if (typeof rec.status === 'number') return rec.status;
if (typeof rec.statusCode === 'number') return rec.statusCode;
return undefined;
function sleepWithSignal(ms: number, signal: AbortSignal): Promise<void> {
if (ms <= 0) return Promise.resolve();
if (signal.aborted) {
throw new DOMException('The operation was aborted.', 'AbortError');
}
return new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => {
signal.removeEventListener('abort', onAbort);
resolve();
}, ms);
const onAbort = () => {
clearTimeout(timer);
signal.removeEventListener('abort', onAbort);
reject(new DOMException('The operation was aborted.', 'AbortError'));
};
signal.addEventListener('abort', onAbort, { once: true });
});
}
function applyReplicateCooldown(cooldownMs: number) {
if (!Number.isFinite(cooldownMs) || cooldownMs <= 0) return;
const next = Date.now() + cooldownMs;
replicateBlockedUntilMs = Math.max(replicateBlockedUntilMs, next);
}
async function runWithReplicateGate<T>(signal: AbortSignal, operation: () => Promise<T>): Promise<T> {
const waitMs = Math.max(0, replicateBlockedUntilMs - Date.now());
if (waitMs > 0) {
await sleepWithSignal(waitMs, signal);
}
return operation();
}
function resolveTTSRequest(input: ServerTTSRequest): ResolvedServerTTSRequest {
const provider = input.provider || 'openai';
const rawModel = provider === 'deepinfra' && !input.model ? 'hexgrad/Kokoro-82M' : input.model;
const rawModel = provider === 'deepinfra' && !input.model ? 'hexgrad/Kokoro-82M'
: provider === 'replicate' && !input.model ? REPLICATE_KOKORO_82M_VERSIONED_MODEL
: input.model;
const model = (rawModel ?? 'gpt-4o-mini-tts') as SpeechCreateParams['model'];
const normalizedVoice = (
!isKokoroModel(model as string) && input.voice.includes('+')
(provider === 'replicate' || !isKokoroModel(model as string)) && input.voice.includes('+')
? input.voice.split('+')[0].trim()
: input.voice
) as string;
const format = input.format || 'mp3';
const speed = Number.isFinite(Number(input.speed)) ? Number(input.speed) : 1;
const requestedSpeed = Number.isFinite(Number(input.speed)) ? Number(input.speed) : 1;
const speed = supportsNativeModelSpeed(provider, model as string) ? requestedSpeed : 1;
const instructions = supportsTtsInstructions(model as string) && input.instructions
? input.instructions
: undefined;
@ -196,10 +236,149 @@ async function fetchTTSBufferWithRetry(
}
}
async function buildReplicateInput(request: ResolvedServerTTSRequest): Promise<Record<string, unknown>> {
const model = request.model as string;
if (model === 'google/gemini-3.1-flash-tts') {
const input: Record<string, unknown> = {
text: request.text,
voice: request.voice,
};
if (request.instructions) {
input.prompt = request.instructions;
}
return input;
}
if (model === 'minimax/speech-2.8-turbo') {
const input: Record<string, unknown> = {
text: request.text,
voice_id: request.voice,
audio_format: request.format === 'mp3' ? 'mp3' : 'wav',
};
if (request.speed !== 1) {
input.speed = Math.max(0.5, Math.min(2.0, request.speed));
}
return input;
}
if (model === 'qwen/qwen3-tts') {
const input: Record<string, unknown> = {
text: request.text,
mode: 'custom_voice',
speaker: request.voice,
};
if (request.instructions) {
input.style_instruction = request.instructions;
}
return input;
}
if (model === 'inworld/tts-1.5-mini') {
const input: Record<string, unknown> = {
text: request.text,
voice_id: request.voice,
audio_format: request.format === 'mp3' ? 'mp3' : 'wav',
};
if (request.speed !== 1) {
input.speaking_rate = request.speed;
}
return input;
}
const input: Record<string, unknown> = { text: request.text };
const voiceInputKey = await resolveReplicateVoiceInputKey({
provider: 'replicate',
model,
apiKey: request.apiKey,
});
if (voiceInputKey) {
input[voiceInputKey] = request.voice;
} else {
input.voice = request.voice;
}
// Best-effort generic fields for custom models.
if (request.format !== 'mp3') {
input.audio_format = 'wav';
}
if (request.speed !== 1) {
input.speed = request.speed;
}
if (request.instructions) {
input.instructions = request.instructions;
}
return input;
}
async function runReplicateRequest(request: ResolvedServerTTSRequest, signal: AbortSignal): Promise<Buffer> {
const replicate = new Replicate({ auth: request.apiKey });
const input = await buildReplicateInput(request);
const modelId = request.model as `${string}/${string}`;
return runWithReplicateGate(signal, async () => {
const maxRetries = Number(process.env.TTS_MAX_RETRIES ?? 2);
let attempt = 0;
for (; ;) {
try {
const output = await replicate.run(modelId, { input, signal }) as unknown;
// Output is a URI string pointing to the generated audio file
const audioUrl = typeof output === 'string' ? output : String(output);
const audioResponse = await fetch(audioUrl, { signal });
if (!audioResponse.ok) {
const error = new Error(`Failed to fetch Replicate audio: ${audioResponse.status}`) as Error & {
status?: number;
statusCode?: number;
response?: { status: number; headers: Headers };
};
error.status = audioResponse.status;
error.statusCode = audioResponse.status;
error.response = {
status: audioResponse.status,
headers: audioResponse.headers,
};
throw error;
}
const buffer = await audioResponse.arrayBuffer();
return Buffer.from(buffer);
} catch (error) {
if (signal.aborted || (error instanceof Error && error.name === 'AbortError')) {
throw error;
}
const status = getUpstreamStatus(error) ?? 0;
const retryable = status === 429 || status >= 500;
const retryAfterSeconds = status === 429 ? getUpstreamRetryAfterSeconds(error) : undefined;
const delay = retryAfterSeconds ? Math.max(retryAfterSeconds * 1000, 1000) : 10_000;
if (status === 429) {
applyReplicateCooldown(delay);
}
if (!retryable || attempt >= maxRetries) {
throw error;
}
await sleepWithSignal(delay, signal);
attempt += 1;
}
}
});
}
async function runProviderRequest(request: ResolvedServerTTSRequest, signal: AbortSignal): Promise<Buffer> {
const mockBuffer = await getTestMockTtsBuffer(request.testNamespace);
if (mockBuffer) return mockBuffer;
if (request.provider === 'replicate') {
return runReplicateRequest(request, signal);
}
const openai = new OpenAI({
apiKey: request.apiKey,
baseURL: request.baseUrl,

53
src/lib/server/tts/upstream-response.ts Normal file
View file

@ -0,0 +1,53 @@
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null;
}
export function getUpstreamStatus(error: unknown): number | undefined {
if (!isRecord(error)) return undefined;
if (typeof error.status === 'number') return error.status;
if (typeof error.statusCode === 'number') return error.statusCode;
const response = isRecord(error.response) ? error.response : undefined;
if (response && typeof response.status === 'number') return response.status;
return undefined;
}
function readRetryAfterHeader(error: unknown): string | undefined {
if (!isRecord(error)) return undefined;
const response = isRecord(error.response) ? error.response : undefined;
if (!response) return undefined;
const headers = response.headers;
if (isRecord(headers) && typeof headers.get === 'function') {
const value = headers.get('retry-after');
return typeof value === 'string' && value.length > 0 ? value : undefined;
}
if (isRecord(headers)) {
const lowerCaseValue = headers['retry-after'];
if (typeof lowerCaseValue === 'string' && lowerCaseValue.length > 0) {
return lowerCaseValue;
}
const canonicalValue = headers['Retry-After'];
if (typeof canonicalValue === 'string' && canonicalValue.length > 0) {
return canonicalValue;
}
}
return undefined;
}
export function getUpstreamRetryAfterSeconds(error: unknown): number | undefined {
const retryAfterHeader = readRetryAfterHeader(error);
if (!retryAfterHeader) return undefined;
const parsed = Number(retryAfterHeader);
if (Number.isFinite(parsed) && parsed > 0) {
return parsed;
}
const parsedDateMs = Date.parse(retryAfterHeader);
if (!Number.isFinite(parsedDateMs)) return undefined;
const seconds = (parsedDateMs - Date.now()) / 1000;
if (seconds <= 0) return undefined;
return Math.ceil(seconds);
}

4
src/lib/shared/kokoro.ts
View file

@ -55,7 +55,9 @@ export const isKokoroModel = (modelName: string | undefined): boolean => {
export const getMaxVoicesForProvider = (provider: string, model: string): number => {
if (!isKokoroModel(model)) return 1;
// Deepinfra Kokoro does not support multiple voices
// Deepinfra and Replicate Kokoro do not support multiple voices.
// Replicate must always stay single-voice, even for Kokoro models.
if (provider === 'replicate') return 1;
if (provider === 'deepinfra') return 1;
// Other providers with Kokoro support unlimited voices

421
src/lib/shared/tts-provider-catalog.ts
View file

@ -1,7 +1,8 @@
import { isKokoroModel } from '@/lib/shared/kokoro';
export type TtsProviderId = 'custom-openai' | 'deepinfra' | 'openai';
export type TtsVoiceSource = 'static' | 'deepinfra-api' | 'custom-openai-api';
export type TtsProviderId = 'custom-openai' | 'replicate' | 'deepinfra' | 'openai';
export type TtsVoiceSource = 'static' | 'deepinfra-api' | 'custom-openai-api' | 'replicate-api';
export type ReplicateVoiceInputKey = 'voice' | 'voice_id' | 'speaker';
export interface TtsModelDefinition {
id: string;
@ -54,6 +55,27 @@ const DEEPINFRA_MODELS_LIMITED: TtsModelDefinition[] = [
{ id: 'hexgrad/Kokoro-82M', name: 'hexgrad/Kokoro-82M' },
];
export const REPLICATE_KOKORO_82M_VERSIONED_MODEL =
'alphanumericuser/kokoro-82m:89b6fa84e4fa2dd6bd3a96be3e1f12827a3516c9fda8fddbac7a0be131c9a6f5' as const;
const REPLICATE_MODELS: TtsModelDefinition[] = [
{
id: REPLICATE_KOKORO_82M_VERSIONED_MODEL,
name: 'alphanumericuser/kokoro-82m',
},
{ id: 'google/gemini-3.1-flash-tts', name: 'google/gemini-3.1-flash-tts' },
{ id: 'minimax/speech-2.8-turbo', name: 'minimax/speech-2.8-turbo' },
{ id: 'qwen/qwen3-tts', name: 'qwen/qwen3-tts' },
{ id: 'inworld/tts-1.5-mini', name: 'inworld/tts-1.5-mini' },
{ id: 'custom', name: 'Other' },
];
const REPLICATE_BUILT_IN_MODELS = new Set(REPLICATE_MODELS.map((model) => model.id).filter((id) => id !== 'custom'));
const DEEPINFRA_API_VOICE_MODELS = new Set([
'ResembleAI/chatterbox',
'Zyphra/Zonos-v0.1-hybrid',
'Zyphra/Zonos-v0.1-transformer',
]);
const DEFAULT_MODELS: TtsModelDefinition[] = [{ id: 'tts-1', name: 'TTS-1' }];
export const OPENAI_DEFAULT_VOICES = ['alloy', 'echo', 'fable', 'onyx', 'nova', 'shimmer'] as const;
@ -71,6 +93,87 @@ export const KOKORO_DEFAULT_VOICES = [
export const ORPHEUS_DEFAULT_VOICES = ['tara', 'leah', 'jess', 'leo', 'dan', 'mia', 'zac'] as const;
export const SESAME_DEFAULT_VOICES = ['conversational_a', 'conversational_b', 'read_speech_a', 'read_speech_b', 'read_speech_c', 'read_speech_d', 'none'] as const;
// Replicate model voices
export const GEMINI_FLASH_TTS_VOICES = [
'Zephyr', 'Puck', 'Charon', 'Kore', 'Fenrir', 'Leda', 'Orus', 'Aoede',
'Callirrhoe', 'Autonoe', 'Enceladus', 'Iapetus', 'Umbriel', 'Algenib',
'Despina', 'Erinome', 'Laomedeia', 'Achernar', 'Algieba', 'Schedar',
'Gacrux', 'Pulcherrima', 'Achird', 'Zubenelgenubi', 'Vindemiatrix',
'Sadachbia', 'Sadaltager', 'Sulafat', 'Alnilam', 'Rasalgethi',
] as const;
export const MINIMAX_SPEECH_VOICES = [
'Deep_Voice_Man', 'Imposing_Manner', 'Elegant_Man', 'Casual_Guy',
'Friendly_Person', 'Decent_Boy', 'Lively_Girl', 'Exuberant_Girl',
'Inspirational_girl', 'Young_Knight', 'Abbess', 'Wise_Woman',
] as const;
export const QWEN3_TTS_VOICES = ['Aiden', 'Dylan'] as const;
export const INWORLD_TTS_VOICES = ['Ashley', 'Dennis', 'Alex', 'Darlene'] as const;
const REPLICATE_VOICE_KEYS: readonly ReplicateVoiceInputKey[] = ['voice', 'voice_id', 'speaker'];
const REPLICATE_DEFAULT_VOICES_BY_MODEL: Record<string, readonly string[]> = {
[REPLICATE_KOKORO_82M_VERSIONED_MODEL]: KOKORO_DEFAULT_VOICES,
'google/gemini-3.1-flash-tts': GEMINI_FLASH_TTS_VOICES,
'minimax/speech-2.8-turbo': MINIMAX_SPEECH_VOICES,
'qwen/qwen3-tts': QWEN3_TTS_VOICES,
'inworld/tts-1.5-mini': INWORLD_TTS_VOICES,
};
const DEEPINFRA_DEFAULT_VOICES_BY_MODEL: Record<string, readonly string[]> = {
'hexgrad/Kokoro-82M': KOKORO_DEFAULT_VOICES,
'canopylabs/orpheus-3b-0.1-ft': ORPHEUS_DEFAULT_VOICES,
'sesame/csm-1b': SESAME_DEFAULT_VOICES,
'ResembleAI/chatterbox': ['None'],
'Zyphra/Zonos-v0.1-hybrid': ['random'],
'Zyphra/Zonos-v0.1-transformer': ['random'],
};
class LRUMap<K, V> {
private readonly maxEntries: number;
private readonly store = new Map<K, V>();
constructor(maxEntries: number) {
this.maxEntries = Math.max(1, maxEntries);
}
get(key: K): V | undefined {
const value = this.store.get(key);
if (value === undefined) {
return undefined;
}
this.store.delete(key);
this.store.set(key, value);
return value;
}
set(key: K, value: V): this {
if (this.store.has(key)) {
this.store.delete(key);
}
this.store.set(key, value);
if (this.store.size > this.maxEntries) {
const oldestKey = this.store.keys().next().value as K | undefined;
if (oldestKey !== undefined) {
this.store.delete(oldestKey);
}
}
return this;
}
delete(key: K): boolean {
return this.store.delete(key);
}
}
const REPLICATE_VOICE_INPUT_KEY_CACHE_MAX_ENTRIES = 128;
const REPLICATE_OPENAPI_SCHEMA_PROMISE_CACHE_MAX_ENTRIES = 128;
const replicateVoiceInputKeyCache = new LRUMap<string, ReplicateVoiceInputKey>(
REPLICATE_VOICE_INPUT_KEY_CACHE_MAX_ENTRIES,
);
const replicateOpenApiSchemaPromiseCache = new LRUMap<string, Promise<unknown | null>>(
REPLICATE_OPENAPI_SCHEMA_PROMISE_CACHE_MAX_ENTRIES,
);
export const TTS_PROVIDER_DEFINITIONS: TtsProviderDefinition[] = [
{
id: 'custom-openai',
@ -78,6 +181,12 @@ export const TTS_PROVIDER_DEFINITIONS: TtsProviderDefinition[] = [
supportsCustomModel: true,
models: () => CUSTOM_OPENAI_MODELS,
},
{
id: 'replicate',
name: 'Replicate',
supportsCustomModel: true,
models: () => REPLICATE_MODELS,
},
{
id: 'deepinfra',
name: 'Deepinfra',
@ -97,8 +206,31 @@ export const TTS_PROVIDER_DEFINITIONS: TtsProviderDefinition[] = [
},
];
const MODELS_WITH_INSTRUCTIONS = new Set([
'gpt-4o-mini-tts',
'google/gemini-3.1-flash-tts',
'qwen/qwen3-tts',
]);
const REPLICATE_MODELS_WITHOUT_NATIVE_SPEED = new Set([
'google/gemini-3.1-flash-tts',
'qwen/qwen3-tts',
]);
export function supportsTtsInstructions(model: string | null | undefined): boolean {
return model === 'gpt-4o-mini-tts';
return !!model && MODELS_WITH_INSTRUCTIONS.has(model);
}
export function supportsNativeModelSpeed(provider: string | null | undefined, model: string | null | undefined): boolean {
if (!model) {
return true;
}
if (provider === 'replicate') {
return !REPLICATE_MODELS_WITHOUT_NATIVE_SPEED.has(model);
}
return true;
}
export function getProviderDefinition(provider: string | null | undefined): TtsProviderDefinition | undefined {
@ -115,50 +247,265 @@ export function providerSupportsCustomModel(provider: string | null | undefined)
export function getDefaultVoices(provider: string, model: string): string[] {
if (provider === 'openai') {
if (supportsTtsInstructions(model)) {
return [...GPT4O_MINI_DEFAULT_VOICES];
}
return [...OPENAI_DEFAULT_VOICES];
return supportsTtsInstructions(model) ? [...GPT4O_MINI_DEFAULT_VOICES] : [...OPENAI_DEFAULT_VOICES];
}
if (provider === 'custom-openai') {
if (isKokoroModel(model)) {
return [...KOKORO_DEFAULT_VOICES];
}
return [...CUSTOM_OPENAI_DEFAULT_VOICES];
return isKokoroModel(model) ? [...KOKORO_DEFAULT_VOICES] : [...CUSTOM_OPENAI_DEFAULT_VOICES];
}
if (provider === 'replicate') {
return REPLICATE_DEFAULT_VOICES_BY_MODEL[model] ? [...REPLICATE_DEFAULT_VOICES_BY_MODEL[model]] : ['default'];
}
if (provider === 'deepinfra') {
if (model === 'hexgrad/Kokoro-82M') {
return [...KOKORO_DEFAULT_VOICES];
}
if (model === 'canopylabs/orpheus-3b-0.1-ft') {
return [...ORPHEUS_DEFAULT_VOICES];
}
if (model === 'sesame/csm-1b') {
return [...SESAME_DEFAULT_VOICES];
}
if (model === 'ResembleAI/chatterbox') {
return ['None'];
}
if (model === 'Zyphra/Zonos-v0.1-hybrid' || model === 'Zyphra/Zonos-v0.1-transformer') {
return ['random'];
}
return [...CUSTOM_OPENAI_DEFAULT_VOICES];
return DEEPINFRA_DEFAULT_VOICES_BY_MODEL[model]
? [...DEEPINFRA_DEFAULT_VOICES_BY_MODEL[model]]
: [...CUSTOM_OPENAI_DEFAULT_VOICES];
}
return [...OPENAI_DEFAULT_VOICES];
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null;
}
function parseReplicateModelIdentifier(model: string): {
owner: string;
name: string;
version?: string;
} | null {
const [ref, version] = model.split(':', 2);
const segments = ref.split('/');
if (segments.length !== 2 || !segments[0] || !segments[1]) {
return null;
}
const parsed = {
owner: segments[0],
name: segments[1],
};
return version
? { ...parsed, version }
: parsed;
}
function extractSchemaStringEnums(schemaNode: unknown, seen = new Set<object>()): string[] {
if (!isRecord(schemaNode)) {
return [];
}
if (seen.has(schemaNode)) {
return [];
}
seen.add(schemaNode);
const values: string[] = [];
if (Array.isArray(schemaNode.enum)) {
values.push(...schemaNode.enum.filter((value): value is string => typeof value === 'string'));
}
if (typeof schemaNode.const === 'string') {
values.push(schemaNode.const);
}
for (const key of ['anyOf', 'allOf', 'oneOf'] as const) {
const branch = schemaNode[key];
if (!Array.isArray(branch)) continue;
for (const item of branch) {
values.push(...extractSchemaStringEnums(item, seen));
}
}
if (schemaNode.items) {
values.push(...extractSchemaStringEnums(schemaNode.items, seen));
}
return values;
}
function walkRecordGraph(root: unknown, visit: (node: Record<string, unknown>) => boolean | void): void {
if (!isRecord(root)) {
return;
}
const stack: Record<string, unknown>[] = [root];
const seen = new Set<object>();
while (stack.length > 0) {
const current = stack.pop();
if (!current) {
continue;
}
if (seen.has(current)) {
continue;
}
seen.add(current);
if (visit(current)) {
return;
}
for (const value of Object.values(current)) {
if (Array.isArray(value)) {
for (const item of value) {
if (isRecord(item)) {
stack.push(item);
}
}
} else if (isRecord(value)) {
stack.push(value);
}
}
}
}
function extractReplicateVoicesFromOpenApiSchema(openApiSchema: unknown): string[] {
const voices: string[] = [];
walkRecordGraph(openApiSchema, (node) => {
const properties = node.properties;
if (!isRecord(properties)) {
return;
}
for (const key of REPLICATE_VOICE_KEYS) {
if (!(key in properties)) continue;
voices.push(...extractSchemaStringEnums(properties[key]));
}
});
return Array.from(
new Set(
voices
.map((voice) => voice.trim())
.filter((voice) => voice.length > 0)
)
);
}
function extractReplicateVoiceInputKeyFromOpenApiSchema(openApiSchema: unknown): ReplicateVoiceInputKey | null {
let found: ReplicateVoiceInputKey | null = null;
walkRecordGraph(openApiSchema, (node) => {
const properties = node.properties;
if (!isRecord(properties)) {
return;
}
for (const key of REPLICATE_VOICE_KEYS) {
if (key in properties) {
found = key;
return true;
}
}
});
return found;
}
async function fetchReplicateOpenApiSchema(apiKey: string, model: string): Promise<unknown | null> {
const parsedModel = parseReplicateModelIdentifier(model);
if (!parsedModel) {
return null;
}
const controller = new AbortController();
const timeoutId = setTimeout(() => {
controller.abort();
}, 10_000);
try {
const endpoint = parsedModel.version
? `https://api.replicate.com/v1/models/${parsedModel.owner}/${parsedModel.name}/versions/${parsedModel.version}`
: `https://api.replicate.com/v1/models/${parsedModel.owner}/${parsedModel.name}`;
const response = await fetch(endpoint, {
signal: controller.signal,
headers: {
Authorization: `Bearer ${apiKey}`,
'Content-Type': 'application/json',
},
});
if (!response.ok) {
return null;
}
const data = await response.json();
let openApiSchema: unknown = null;
if (parsedModel.version) {
if (isRecord(data)) {
openApiSchema = data.openapi_schema;
}
} else if (isRecord(data) && isRecord(data.latest_version)) {
openApiSchema = data.latest_version.openapi_schema;
}
return openApiSchema;
} catch (error) {
if (error instanceof DOMException && error.name === 'AbortError') {
return null;
}
console.error('Error fetching Replicate model schema:', error);
return null;
} finally {
clearTimeout(timeoutId);
}
}
async function getReplicateOpenApiSchemaCached(apiKey: string, model: string): Promise<unknown | null> {
const cachedPromise = replicateOpenApiSchemaPromiseCache.get(model);
if (cachedPromise) {
return cachedPromise;
}
const fetchPromise = fetchReplicateOpenApiSchema(apiKey, model);
replicateOpenApiSchemaPromiseCache.set(model, fetchPromise);
const schema = await fetchPromise;
if (schema === null) {
replicateOpenApiSchemaPromiseCache.delete(model);
}
return schema;
}
async function fetchReplicateVoices(apiKey: string, model: string): Promise<string[] | null> {
const openApiSchema = await getReplicateOpenApiSchemaCached(apiKey, model);
const apiVoices = extractReplicateVoicesFromOpenApiSchema(openApiSchema);
return apiVoices.length > 0 ? apiVoices : null;
}
export async function resolveReplicateVoiceInputKey({
provider,
model,
apiKey = '',
}: ResolveVoicesOptions): Promise<ReplicateVoiceInputKey | null> {
if (provider !== 'replicate' || REPLICATE_BUILT_IN_MODELS.has(model) || !apiKey) {
return null;
}
const cached = replicateVoiceInputKeyCache.get(model);
if (cached) {
return cached;
}
const openApiSchema = await getReplicateOpenApiSchemaCached(apiKey, model);
const inputKey = extractReplicateVoiceInputKeyFromOpenApiSchema(openApiSchema);
if (inputKey) {
replicateVoiceInputKeyCache.set(model, inputKey);
}
return inputKey;
}
export function resolveVoiceSource(provider: string, model: string): TtsVoiceSource {
if (provider === 'deepinfra' && (
model === 'ResembleAI/chatterbox' ||
model === 'Zyphra/Zonos-v0.1-hybrid' ||
model === 'Zyphra/Zonos-v0.1-transformer'
)) {
if (provider === 'deepinfra' && DEEPINFRA_API_VOICE_MODELS.has(model)) {
return 'deepinfra-api';
}
if (provider === 'replicate' && parseReplicateModelIdentifier(model) !== null) {
return 'replicate-api';
}
if (provider === 'custom-openai') {
return 'custom-openai-api';
}
@ -257,5 +604,15 @@ export async function resolveVoices({ provider, model, apiKey = '', baseUrl = ''
}
}
if (voiceSource === 'replicate-api') {
if (!apiKey) {
return defaultVoices;
}
const apiVoices = await fetchReplicateVoices(apiKey, model);
if (apiVoices !== null) {
return apiVoices;
}
}
return defaultVoices;
}

21
tests/unit/kokoro.spec.ts Normal file
View file

@ -0,0 +1,21 @@
import { expect, test } from '@playwright/test';
import { getMaxVoicesForProvider } from '../../src/lib/shared/kokoro';
test.describe('kokoro voice limits', () => {
test('keeps Replicate single-voice even for Kokoro models', () => {
expect(getMaxVoicesForProvider('replicate', 'kokoro')).toBe(1);
expect(getMaxVoicesForProvider('replicate', 'hexgrad/Kokoro-82M')).toBe(1);
});
test('keeps Deepinfra single-voice and allows multi-voice elsewhere', () => {
expect(getMaxVoicesForProvider('deepinfra', 'hexgrad/Kokoro-82M')).toBe(1);
expect(getMaxVoicesForProvider('custom-openai', 'kokoro')).toBe(Infinity);
});
test('non-kokoro models are always single-voice', () => {
expect(getMaxVoicesForProvider('replicate', 'google/gemini-3.1-flash-tts')).toBe(1);
expect(getMaxVoicesForProvider('openai', 'gpt-4o-mini-tts')).toBe(1);
});
});

245
tests/unit/tts-provider-catalog.spec.ts
View file

@ -1,10 +1,13 @@
import { expect, test } from '@playwright/test';
import {
REPLICATE_KOKORO_82M_VERSIONED_MODEL,
getDefaultVoices,
providerSupportsCustomModel,
resolveReplicateVoiceInputKey,
resolveVoices,
resolveProviderModels,
supportsNativeModelSpeed,
supportsTtsInstructions,
} from '../../src/lib/shared/tts-provider-catalog';
import { applyConfigUpdate, getVoicePreferenceKey } from '../../src/lib/client/config/updates';
@ -50,6 +53,13 @@ test.describe('tts provider catalog', () => {
expect(providerSupportsCustomModel('deepinfra')).toBe(true);
});
test('uses blocklist semantics for Replicate native speed support', () => {
expect(supportsNativeModelSpeed('replicate', REPLICATE_KOKORO_82M_VERSIONED_MODEL)).toBe(true);
expect(supportsNativeModelSpeed('replicate', 'acme/runtime-speed-model')).toBe(true);
expect(supportsNativeModelSpeed('replicate', 'google/gemini-3.1-flash-tts')).toBe(false);
expect(supportsNativeModelSpeed('replicate', 'qwen/qwen3-tts')).toBe(false);
});
test('keeps explicit empty custom-openai voices response', async () => {
const originalFetch = globalThis.fetch;
globalThis.fetch = async () => ({
@ -68,6 +78,241 @@ test.describe('tts provider catalog', () => {
globalThis.fetch = originalFetch;
}
});
test('fetches custom Replicate model voices from model schema', async () => {
const originalFetch = globalThis.fetch;
let calls = 0;
let url = '';
globalThis.fetch = async (input) => {
calls += 1;
url = String(input);
return {
ok: true,
json: async () => ({
latest_version: {
openapi_schema: {
components: {
schemas: {
Input: {
type: 'object',
properties: {
voice: {
type: 'string',
enum: ['Narrator A', 'Narrator B'],
},
},
},
},
},
},
},
}),
} as Response;
};
try {
await expect(resolveVoices({
provider: 'replicate',
model: 'acme/my-tts-model-voices',
apiKey: 'r8_token',
})).resolves.toEqual(['Narrator A', 'Narrator B']);
expect(calls).toBe(1);
expect(url).toContain('/v1/models/acme/my-tts-model-voices');
} finally {
globalThis.fetch = originalFetch;
}
});
test('falls back to default voice for custom Replicate models when schema lookup fails', async () => {
const originalFetch = globalThis.fetch;
globalThis.fetch = async () => ({
ok: false,
status: 404,
json: async () => ({}),
}) as Response;
try {
await expect(resolveVoices({
provider: 'replicate',
model: 'acme/my-tts-model-fallback',
apiKey: 'r8_token',
})).resolves.toEqual(['default']);
} finally {
globalThis.fetch = originalFetch;
}
});
test('uses schema voices first for built-in Replicate models', async () => {
const originalFetch = globalThis.fetch;
let calls = 0;
globalThis.fetch = async () => {
calls += 1;
return {
ok: true,
json: async () => ({
latest_version: {
openapi_schema: {
components: {
schemas: {
Input: {
type: 'object',
properties: {
speaker: {
type: 'string',
enum: ['Schema Aiden', 'Schema Dylan'],
},
},
},
},
},
},
},
}),
} as Response;
};
try {
await expect(resolveVoices({
provider: 'replicate',
model: 'qwen/qwen3-tts',
apiKey: 'r8_token',
})).resolves.toEqual(['Schema Aiden', 'Schema Dylan']);
expect(calls).toBe(1);
} finally {
globalThis.fetch = originalFetch;
}
});
test('falls back to built-in Replicate static voices when schema has no voice enum', async () => {
const originalFetch = globalThis.fetch;
let calls = 0;
globalThis.fetch = async () => {
calls += 1;
return {
ok: true,
json: async () => ({
latest_version: {
openapi_schema: {
components: {
schemas: {
Input: {
type: 'object',
properties: {
speaker: { type: 'string' },
},
},
},
},
},
},
}),
} as Response;
};
try {
await expect(resolveVoices({
provider: 'replicate',
model: 'inworld/tts-1.5-mini',
apiKey: 'r8_token',
})).resolves.toEqual(['Ashley', 'Dennis', 'Alex', 'Darlene']);
expect(calls).toBe(1);
} finally {
globalThis.fetch = originalFetch;
}
});
test('resolves Replicate custom-model voice input key and caches it', async () => {
const originalFetch = globalThis.fetch;
let calls = 0;
globalThis.fetch = async () => {
calls += 1;
return {
ok: true,
json: async () => ({
latest_version: {
openapi_schema: {
components: {
schemas: {
Input: {
type: 'object',
properties: {
voice_id: { type: 'string' },
},
},
},
},
},
},
}),
} as Response;
};
try {
await expect(resolveReplicateVoiceInputKey({
provider: 'replicate',
model: 'acme/custom-voice-model',
apiKey: 'r8_token',
})).resolves.toBe('voice_id');
await expect(resolveReplicateVoiceInputKey({
provider: 'replicate',
model: 'acme/custom-voice-model',
apiKey: 'r8_token',
})).resolves.toBe('voice_id');
expect(calls).toBe(1);
} finally {
globalThis.fetch = originalFetch;
}
});
test('shares Replicate schema fetch between voices and voice-key resolution', async () => {
const originalFetch = globalThis.fetch;
let calls = 0;
globalThis.fetch = async () => {
calls += 1;
return {
ok: true,
json: async () => ({
latest_version: {
openapi_schema: {
components: {
schemas: {
Input: {
type: 'object',
properties: {
voice: {
type: 'string',
enum: ['V1', 'V2'],
},
},
},
},
},
},
},
}),
} as Response;
};
try {
await expect(resolveVoices({
provider: 'replicate',
model: 'acme/shared-schema-model',
apiKey: 'r8_token',
})).resolves.toEqual(['V1', 'V2']);
await expect(resolveReplicateVoiceInputKey({
provider: 'replicate',
model: 'acme/shared-schema-model',
apiKey: 'r8_token',
})).resolves.toBe('voice');
expect(calls).toBe(1);
} finally {
globalThis.fetch = originalFetch;
}
});
});
test.describe('config helpers', () => {