feat(tts): add smart sentence continuation

introduce configurable smart sentence splitting with persisted state,
UI toggle, and EPUB/PDF contexts that send continuation metadata.
enhance TTS pipeline to merge cross-page sentences, manage carryover,
and trigger visual page changes during playback for smoother narration.

update README with kokoro quick-start guidance, remove the legacy
issues mapping doc, and add deterministic TTS mocks plus sample audio
for Playwright tests.
This commit is contained in:
Richard Roberson 2025-11-14 12:00:03 -07:00
parent d7ef0fa8bb
commit e3799e40aa
14 changed files with 641 additions and 197 deletions

View file

@ -36,9 +36,12 @@ OpenReader WebUI is a document reader with Text-to-Speech capabilities, offering
- Recent version of Docker installed on your machine - Recent version of Docker installed on your machine
- A TTS API server (Kokoro-FastAPI, Orpheus-FastAPI, Deepinfra, OpenAI, etc.) running and accessible - A TTS API server (Kokoro-FastAPI, Orpheus-FastAPI, Deepinfra, OpenAI, etc.) running and accessible
> **Note:** If you have good hardware, you can run [Kokoro-FastAPI with Docker locally](#🗣️-local-kokoro-fastapi-quick-start-cpu-or-gpu) (see below).
### 1. 🐳 Start the Docker container: ### 1. 🐳 Start the Docker container:
```bash ```bash
docker run --name openreader-webui \ docker run --name openreader-webui \
--restart unless-stopped \
-p 3003:3003 \ -p 3003:3003 \
-v openreader_docstore:/app/docstore \ -v openreader_docstore:/app/docstore \
ghcr.io/richardr1126/openreader-webui:latest ghcr.io/richardr1126/openreader-webui:latest
@ -47,6 +50,7 @@ OpenReader WebUI is a document reader with Text-to-Speech capabilities, offering
(Optionally): Set the TTS `API_BASE` URL and/or `API_KEY` to be default for all devices (Optionally): Set the TTS `API_BASE` URL and/or `API_KEY` to be default for all devices
```bash ```bash
docker run --name openreader-webui \ docker run --name openreader-webui \
--restart unless-stopped \
-e API_KEY=none \ -e API_KEY=none \
-e API_BASE=http://host.docker.internal:8880/v1 \ -e API_BASE=http://host.docker.internal:8880/v1 \
-p 3003:3003 \ -p 3003:3003 \
@ -72,36 +76,48 @@ docker rm openreader-webui && \
docker pull ghcr.io/richardr1126/openreader-webui:latest docker pull ghcr.io/richardr1126/openreader-webui:latest
``` ```
### (Alternate) 🐳 Configuration with Docker Compose and Kokoro-FastAPI ### 🗣️ Local Kokoro-FastAPI Quick-start (CPU or GPU)
A complete example docker-compose file with Kokoro-FastAPI and OpenReader WebUI is available in [`docs/examples/docker-compose.yml`](docs/examples/docker-compose.yml). You can download and use it: You can run the Kokoro TTS API server directly with Docker. **We are not responsible for issues with Kokoro-FastAPI.** For best performance, use an NVIDIA GPU (for GPU version) or Apple Silicon (for CPU version).
> **Note:** When using these, set the `API_BASE` env var to `http://host.docker.internal:8880/v1` or `http://kokoro-tts:8880/v1`.
> You can also use the example `docker-compose.yml` in `examples/docker-compose.yml` if you prefer Docker Compose.
**CPU Version:**
```bash ```bash
# Download example docker-compose.yml docker run -d \
curl --create-dirs -L -o openreader-compose/docker-compose.yml https://raw.githubusercontent.com/richardr1126/OpenReader-WebUI/main/docs/examples/docker-compose.yml --name kokoro-tts \
--restart unless-stopped \
cd openreader-compose -p 8880:8880 \
docker compose up -d -e ONNX_NUM_THREADS=8 \
-e ONNX_INTER_OP_THREADS=4 \
-e ONNX_EXECUTION_MODE=parallel \
-e ONNX_OPTIMIZATION_LEVEL=all \
-e ONNX_MEMORY_PATTERN=true \
-e ONNX_ARENA_EXTEND_STRATEGY=kNextPowerOfTwo \
-e API_LOG_LEVEL=DEBUG \
ghcr.io/remsky/kokoro-fastapi-cpu:v0.2.4
``` ```
Or add OpenReader WebUI to your existing `docker-compose.yml`: **GPU Version:**
```yaml ```bash
services: docker run -d \
openreader-webui: --name kokoro-tts \
container_name: openreader-webui --gpus all \
image: ghcr.io/richardr1126/openreader-webui:latest --user 1001:1001 \
environment: --restart unless-stopped \
- API_BASE=http://host.docker.internal:8880/v1 -p 8880:8880 \
ports: -e USE_GPU=true \
- "3003:3003" -e PYTHONUNBUFFERED=1 \
volumes: -e API_LOG_LEVEL=DEBUG \
- docstore:/app/docstore ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.4
restart: unless-stopped
volumes:
docstore:
``` ```
> **Note:**
> - These commands are for running the Kokoro TTS API server only. For issues or support, see the [Kokoro-FastAPI repository](https://github.com/remsky/Kokoro-FastAPI).
> - The GPU version requires NVIDIA Docker support and works best with NVIDIA GPUs. The CPU version works best on Apple Silicon or modern x86 CPUs.
> - Adjust environment variables as needed for your hardware and use case.
## Dev Installation ## Dev Installation
### Prerequisites ### Prerequisites

View file

@ -1,100 +0,0 @@
# OpenReader Issue Triage and Mapping
Repository: https://github.com/richardr1126/OpenReader-WebUI/issues
Reviewed via gh at 2025-11-10.
Summary of open items included:
- #59 Feature: Chapter-Based MP3 Export
- #48 Bug: Failed to export after complete render with Kokoro
- #47 Feature: Combine voices (Kokoro “plus” syntax)
- #44 Bug: Dialog not chunked together
- #40 Bug: PDF left/right extraction margins not working
Global guardrails
- Streaming-first playback (replace Howler with HTMLAudioElement/MSE)
- Dexie DB replacing vanilla IndexedDB
- Single engine cut-over (no dual engines)
- Keep audiobook (m4b) and server-side sync
Issue #59 — Chapter-Based MP3 Export
Type: Feature
Hypothesis / intent:
- Users want one-mp3-per-chapter output (besides full-book m4b).
Ideas:
- Add chapterized MP3 pipeline that chunks by adapter “chapter” units.
- Provide ZIP export for many MP3 files (streamed).
API design:
- POST /api/audio/convert?mode=chapters&format=mp3 -> returns stream of a ZIP.
Logging to add:
- Chapter boundaries, byte sizes, cumulative progress in [src/app/api/audio/convert/route.ts](src/app/api/audio/convert/route.ts:1).
Acceptance:
- A multi-chapter EPUB produces N mp3 parts named “NN - Chapter Title.mp3”; total duration ~ sum of parts; ZIP streamed without timeouts in Docker.
Issue #48 — Failed export after complete render with Kokoro
Type: Bug (large-book export)
Observations:
- UI reaches 100% then resets, no download.
- Docker shows “fin … undefined”, likely final transfer problem (not TTS).
Likely root causes:
- Final m4b delivery uses single huge arrayBuffer; browser memory/timeout.
- Missing Content-Disposition/Range; no resumable download.
- Temp-file lifecycle cleanup racing with response.
Ideas for remediation:
- Serve final artifact as file on disk with streaming and Range:
- New endpoint: GET /api/audio/convert/download?bookId=… that streams file with Accept-Ranges.
- UI performs streamed download; no arrayBuffer buffering.
- Optionally support S3-compatible offload in future.
Touched modules:
- [src/app/api/audio/convert/route.ts](src/app/api/audio/convert/route.ts:1)
Instrumentation to add:
- Book ID, file size on disk, stream chunk counts, and client-abort detection.
- Add explicit log/error surfaces at:
- [src/app/api/audio/convert/route.ts](src/app/api/audio/convert/route.ts:1)
Acceptance:
- 12 GB m4b exports download in Docker without UI reset; bookId persists until deletion; status 206 Range works.
Issue #47 — Kokoro combined voices (“+” syntax)
Type: Feature
Intent:
- Support voice strings like “bf_emma+af_heart” when provider is Kokoro/FastAPI or DeepInfra Kokoro.
Ideas for implementation:
- Allow free-form voice string entry and pass-through if not in known set.
- Validate known providers; if “+” present, skip voice validation list.
Logging:
- Emit provider, model, raw voice string in request (no PII).
Tests:
- “a+b” voices produce audible combined output; works via DeepInfra pass-through and Kokoro-FastAPI; voice dropdown offers “Custom voice…” input.
Issue #44 — Dialog not being chunked together
Type: Bug (NLP splitting)
Observations:
- Dialog lines split mid-quote; needs grouping.
Proposed improvements:
- Extend [splitIntoSentences()](src/utils/nlp.ts:34) to apply quote-aware grouping.
- When a sentence begins with an opening quote and the next ends with a closing quote, join them before MAX_BLOCK_LENGTH checks.
Ideas for remediation:
- Provide composable splitter strategy wrapping existing utility.
- Add “dialog-preserve” flag toggled in settings.
Logging to add:
- Count of quote-joined sentences; per-page example of before/after lengths.
Acceptance:
- Quoted dialog flows as single units unless exceeding MAX_BLOCK_LENGTH by > X%; regression-safe for non-dialog text.
Issue #40 — PDF left/right extraction margins not working
Type: Bug (PDF extraction)
Observations:
- Current filter uses transform[4]/[5] with width heuristics. Edge cases: missing width, skew, page scale differences.
Likely causes:
- Some pdf.js TextItem miss width; horizontal margins computed but not applied due to scale mismatch.
- Defaults in [src/contexts/ConfigContext.tsx](src/contexts/ConfigContext.tsx:216) set left/right to '0.0' on first run; UX confusion.
Remediation:
- Compute glyph bbox width when width absent using transform matrix.
- Normalize x to [0..1] by dividing by pageWidth; compare to margins reliably.
- Add visual debug overlay (dev mode) to draw margin boxes while extracting.
Code touchpoints:
- [src/utils/pdf.ts](src/utils/pdf.ts:60) extractTextFromPDF(): margin math and width fallback.
Diagnostics to add:
- Per-page: kept/filtered counts, extremes of x positions, computed margins.
Acceptance:
- Test PDFs show left/right trimming correctly; e2e highlight still robust.

View file

@ -1,13 +1,5 @@
import { defineConfig, devices } from '@playwright/test'; import { defineConfig, devices } from '@playwright/test';
/**
* Read environment variables from file.
* https://github.com/motdotla/dotenv
*/
// import dotenv from 'dotenv';
// import path from 'path';
// dotenv.config({ path: path.resolve(__dirname, '.env') });
/** /**
* See https://playwright.dev/docs/test-configuration. * See https://playwright.dev/docs/test-configuration.
*/ */
@ -15,14 +7,11 @@ export default defineConfig({
testDir: './tests', testDir: './tests',
timeout: 30 * 1000, timeout: 30 * 1000,
outputDir: './tests/results', outputDir: './tests/results',
/* Run tests in files in parallel */ fullyParallel: false,
fullyParallel: true,
/* Fail the build on CI if you accidentally left test.only in the source code. */ /* Fail the build on CI if you accidentally left test.only in the source code. */
forbidOnly: !!process.env.CI, forbidOnly: !!process.env.CI,
/* Retry on CI only */
retries: process.env.CI ? 2 : 0, retries: process.env.CI ? 2 : 0,
/* Opt out of parallel tests on CI. */ workers: process.env.CI ? '100%' : '75%',
workers: undefined,
/* Reporter to use. See https://playwright.dev/docs/test-reporters */ /* Reporter to use. See https://playwright.dev/docs/test-reporters */
reporter: 'html', reporter: 'html',
/* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */ /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */

View file

@ -217,7 +217,7 @@ export async function POST(req: NextRequest) {
console.log('TTS in-flight JOIN for key:', cacheKey.slice(0, 8)); console.log('TTS in-flight JOIN for key:', cacheKey.slice(0, 8));
existing.consumers += 1; existing.consumers += 1;
const onAbort = (_evt: Event) => { const onAbort = () => {
existing.consumers = Math.max(0, existing.consumers - 1); existing.consumers = Math.max(0, existing.consumers - 1);
if (existing.consumers === 0) { if (existing.consumers === 0) {
existing.controller.abort(); existing.controller.abort();
@ -262,7 +262,7 @@ export async function POST(req: NextRequest) {
inflightRequests.set(cacheKey, entry); inflightRequests.set(cacheKey, entry);
const onAbort = (_evt: Event) => { const onAbort = () => {
entry.consumers = Math.max(0, entry.consumers - 1); entry.consumers = Math.max(0, entry.consumers - 1);
if (entry.consumers === 0) { if (entry.consumers === 0) {
entry.controller.abort(); entry.controller.abort();

View file

@ -27,6 +27,7 @@ export function DocumentSettings({ isOpen, setIsOpen, epub, html }: {
viewType, viewType,
skipBlank, skipBlank,
epubTheme, epubTheme,
smartSentenceSplitting,
headerMargin, headerMargin,
footerMargin, footerMargin,
leftMargin, leftMargin,
@ -308,6 +309,24 @@ export function DocumentSettings({ isOpen, setIsOpen, epub, html }: {
Automatically skip pages with no text content Automatically skip pages with no text content
</p> </p>
</div>} </div>}
{!html && (
<div className="space-y-1">
<label className="flex items-center space-x-2">
<input
type="checkbox"
checked={smartSentenceSplitting}
onChange={(e) => updateConfigKey('smartSentenceSplitting', e.target.checked)}
className="form-checkbox h-4 w-4 text-accent rounded border-muted"
/>
<span className="text-sm font-medium text-foreground">
Smart sentence splitting
</span>
</label>
<p className="text-sm text-muted pl-6">
Merge sentences across page or section breaks for smoother TTS.
</p>
</div>
)}
{epub && ( {epub && (
<div className="space-y-1"> <div className="space-y-1">
<label className="flex items-center space-x-2"> <label className="flex items-center space-x-2">

View file

@ -29,6 +29,7 @@ type ConfigValues = {
ttsModel: string; ttsModel: string;
ttsInstructions: string; ttsInstructions: string;
savedVoices: SavedVoices; savedVoices: SavedVoices;
smartSentenceSplitting: boolean;
}; };
/** Interface defining the configuration context shape and functionality */ /** Interface defining the configuration context shape and functionality */
@ -41,6 +42,7 @@ interface ConfigContextType {
voice: string; voice: string;
skipBlank: boolean; skipBlank: boolean;
epubTheme: boolean; epubTheme: boolean;
smartSentenceSplitting: boolean;
headerMargin: number; headerMargin: number;
footerMargin: number; footerMargin: number;
leftMargin: number; leftMargin: number;
@ -73,6 +75,7 @@ export function ConfigProvider({ children }: { children: ReactNode }) {
const [voice, setVoice] = useState<string>('af_sarah'); const [voice, setVoice] = useState<string>('af_sarah');
const [skipBlank, setSkipBlank] = useState<boolean>(true); const [skipBlank, setSkipBlank] = useState<boolean>(true);
const [epubTheme, setEpubTheme] = useState<boolean>(false); const [epubTheme, setEpubTheme] = useState<boolean>(false);
const [smartSentenceSplitting, setSmartSentenceSplitting] = useState<boolean>(true);
const [headerMargin, setHeaderMargin] = useState<number>(0.07); const [headerMargin, setHeaderMargin] = useState<number>(0.07);
const [footerMargin, setFooterMargin] = useState<number>(0.07); const [footerMargin, setFooterMargin] = useState<number>(0.07);
const [leftMargin, setLeftMargin] = useState<number>(0.07); const [leftMargin, setLeftMargin] = useState<number>(0.07);
@ -103,6 +106,7 @@ export function ConfigProvider({ children }: { children: ReactNode }) {
const cachedAudioPlayerSpeed = await getItem('audioPlayerSpeed'); const cachedAudioPlayerSpeed = await getItem('audioPlayerSpeed');
const cachedSkipBlank = await getItem('skipBlank'); const cachedSkipBlank = await getItem('skipBlank');
const cachedEpubTheme = await getItem('epubTheme'); const cachedEpubTheme = await getItem('epubTheme');
const cachedSmartSentenceSplitting = await getItem('smartSentenceSplitting');
const cachedHeaderMargin = await getItem('headerMargin'); const cachedHeaderMargin = await getItem('headerMargin');
const cachedFooterMargin = await getItem('footerMargin'); const cachedFooterMargin = await getItem('footerMargin');
const cachedLeftMargin = await getItem('leftMargin'); const cachedLeftMargin = await getItem('leftMargin');
@ -187,6 +191,7 @@ export function ConfigProvider({ children }: { children: ReactNode }) {
setAudioPlayerSpeed(parseFloat(cachedAudioPlayerSpeed || '1')); setAudioPlayerSpeed(parseFloat(cachedAudioPlayerSpeed || '1'));
setSkipBlank(cachedSkipBlank === 'false' ? false : true); setSkipBlank(cachedSkipBlank === 'false' ? false : true);
setEpubTheme(cachedEpubTheme === 'true'); setEpubTheme(cachedEpubTheme === 'true');
setSmartSentenceSplitting(cachedSmartSentenceSplitting === 'false' ? false : true);
setHeaderMargin(parseFloat(cachedHeaderMargin || '0.07')); setHeaderMargin(parseFloat(cachedHeaderMargin || '0.07'));
setFooterMargin(parseFloat(cachedFooterMargin || '0.07')); setFooterMargin(parseFloat(cachedFooterMargin || '0.07'));
setLeftMargin(parseFloat(cachedLeftMargin || '0.07')); setLeftMargin(parseFloat(cachedLeftMargin || '0.07'));
@ -211,6 +216,9 @@ export function ConfigProvider({ children }: { children: ReactNode }) {
if (cachedEpubTheme === null) { if (cachedEpubTheme === null) {
await setItem('epubTheme', 'false'); await setItem('epubTheme', 'false');
} }
if (cachedSmartSentenceSplitting === null) {
await setItem('smartSentenceSplitting', 'true');
}
if (cachedHeaderMargin === null) await setItem('headerMargin', '0.07'); if (cachedHeaderMargin === null) await setItem('headerMargin', '0.07');
if (cachedFooterMargin === null) await setItem('footerMargin', '0.07'); if (cachedFooterMargin === null) await setItem('footerMargin', '0.07');
if (cachedLeftMargin === null) await setItem('leftMargin', '0.0'); if (cachedLeftMargin === null) await setItem('leftMargin', '0.0');
@ -353,6 +361,9 @@ export function ConfigProvider({ children }: { children: ReactNode }) {
case 'epubTheme': case 'epubTheme':
setEpubTheme(value as boolean); setEpubTheme(value as boolean);
break; break;
case 'smartSentenceSplitting':
setSmartSentenceSplitting(value as boolean);
break;
case 'headerMargin': case 'headerMargin':
setHeaderMargin(value as number); setHeaderMargin(value as number);
break; break;
@ -388,6 +399,7 @@ export function ConfigProvider({ children }: { children: ReactNode }) {
voice, voice,
skipBlank, skipBlank,
epubTheme, epubTheme,
smartSentenceSplitting,
headerMargin, headerMargin,
footerMargin, footerMargin,
leftMargin, leftMargin,

View file

@ -8,7 +8,7 @@ import {
useCallback, useCallback,
useMemo, useMemo,
useRef, useRef,
RefObject, RefObject
} from 'react'; } from 'react';
import { indexedDBService } from '@/utils/indexedDB'; import { indexedDBService } from '@/utils/indexedDB';
import { useTTS } from '@/contexts/TTSContext'; import { useTTS } from '@/contexts/TTSContext';
@ -43,6 +43,81 @@ interface EPUBContextType {
const EPUBContext = createContext<EPUBContextType | undefined>(undefined); const EPUBContext = createContext<EPUBContextType | undefined>(undefined);
const EPUB_CONTINUATION_CHARS = 600;
const stepToNextNode = (node: Node | null, root: Node): Node | null => {
if (!node) return null;
if (node.firstChild) {
return node.firstChild;
}
let current: Node | null = node;
while (current) {
if (current === root) {
return null;
}
if (current.nextSibling) {
return current.nextSibling;
}
current = current.parentNode;
}
return null;
};
const getNextTextNode = (node: Node | null, root: Node): Text | null => {
let next = stepToNextNode(node, root);
while (next) {
if (next.nodeType === Node.TEXT_NODE) {
return next as Text;
}
next = stepToNextNode(next, root);
}
return null;
};
const collectContinuationFromRange = (range: Range | null | undefined, limit = EPUB_CONTINUATION_CHARS): string => {
if (typeof window === 'undefined' || !range) {
return '';
}
const root = range.commonAncestorContainer;
if (!root) {
return '';
}
const parts: string[] = [];
let remaining = limit;
const appendFromTextNode = (textNode: Text, offset: number) => {
if (remaining <= 0) return;
const textContent = textNode.textContent || '';
if (offset >= textContent.length) return;
const slice = textContent.slice(offset, offset + remaining);
if (slice) {
parts.push(slice);
remaining -= slice.length;
}
};
if (range.endContainer.nodeType === Node.TEXT_NODE) {
appendFromTextNode(range.endContainer as Text, range.endOffset);
let nextNode = getNextTextNode(range.endContainer, root);
while (nextNode && remaining > 0) {
appendFromTextNode(nextNode, 0);
nextNode = getNextTextNode(nextNode, root);
}
} else {
let nextNode = getNextTextNode(range.endContainer, root);
while (nextNode && remaining > 0) {
appendFromTextNode(nextNode, 0);
nextNode = getNextTextNode(nextNode, root);
}
}
return parts.join(' ').replace(/\s+/g, ' ').trim();
};
/** /**
* Provider component for EPUB functionality * Provider component for EPUB functionality
* Manages the state and operations for EPUB document handling * Manages the state and operations for EPUB document handling
@ -61,6 +136,7 @@ export function EPUBProvider({ children }: { children: ReactNode }) {
ttsProvider, ttsProvider,
ttsModel, ttsModel,
ttsInstructions, ttsInstructions,
smartSentenceSplitting,
} = useConfig(); } = useConfig();
// Current document state // Current document state
const [currDocData, setCurrDocData] = useState<ArrayBuffer>(); const [currDocData, setCurrDocData] = useState<ArrayBuffer>();
@ -141,8 +217,23 @@ export function EPUBProvider({ children }: { children: ReactNode }) {
return ''; return '';
} }
const textContent = range.toString().trim(); const textContent = range.toString().trim();
const continuationPreview = collectContinuationFromRange(range);
setTTSText(textContent, shouldPause); if (smartSentenceSplitting) {
setTTSText(textContent, {
shouldPause,
location: start.cfi,
nextLocation: end.cfi,
nextText: continuationPreview
});
} else {
// When smart splitting is disabled, behave like the original implementation:
// send only the current page/location text without any continuation preview.
setTTSText(textContent, {
shouldPause,
location: start.cfi,
});
}
setCurrDocText(textContent); setCurrDocText(textContent);
return textContent; return textContent;
@ -150,7 +241,7 @@ export function EPUBProvider({ children }: { children: ReactNode }) {
console.error('Error extracting EPUB text:', error); console.error('Error extracting EPUB text:', error);
return ''; return '';
} }
}, [setTTSText]); }, [setTTSText, smartSentenceSplitting]);
/** /**
* Extracts text content from the entire EPUB book * Extracts text content from the entire EPUB book

View file

@ -22,6 +22,7 @@ import {
useCallback, useCallback,
useMemo, useMemo,
RefObject, RefObject,
useRef,
} from 'react'; } from 'react';
import { indexedDBService } from '@/utils/indexedDB'; import { indexedDBService } from '@/utils/indexedDB';
@ -33,6 +34,7 @@ import {
clearHighlights, clearHighlights,
handleTextClick, handleTextClick,
} from '@/utils/pdf'; } from '@/utils/pdf';
import { processTextToSentences } from '@/utils/nlp';
import type { PDFDocumentProxy } from 'pdfjs-dist'; import type { PDFDocumentProxy } from 'pdfjs-dist';
import { withRetry } from '@/utils/audio'; import { withRetry } from '@/utils/audio';
@ -70,6 +72,8 @@ interface PDFContextType {
// Create the context // Create the context
const PDFContext = createContext<PDFContextType | undefined>(undefined); const PDFContext = createContext<PDFContextType | undefined>(undefined);
const CONTINUATION_PREVIEW_CHARS = 600;
/** /**
* PDFProvider Component * PDFProvider Component
* *
@ -83,10 +87,11 @@ export function PDFProvider({ children }: { children: ReactNode }) {
const { const {
setText: setTTSText, setText: setTTSText,
stop, stop,
currDocPageNumber: currDocPage, currDocPageNumber,
currDocPages, currDocPages,
setCurrDocPages, setCurrDocPages,
setIsEPUB setIsEPUB,
registerVisualPageChangeHandler,
} = useTTS(); } = useTTS();
const { const {
headerMargin, headerMargin,
@ -100,6 +105,7 @@ export function PDFProvider({ children }: { children: ReactNode }) {
ttsProvider, ttsProvider,
ttsModel, ttsModel,
ttsInstructions, ttsInstructions,
smartSentenceSplitting,
} = useConfig(); } = useConfig();
// Current document state // Current document state
@ -108,6 +114,12 @@ export function PDFProvider({ children }: { children: ReactNode }) {
const [currDocText, setCurrDocText] = useState<string>(); const [currDocText, setCurrDocText] = useState<string>();
const [pdfDocument, setPdfDocument] = useState<PDFDocumentProxy>(); const [pdfDocument, setPdfDocument] = useState<PDFDocumentProxy>();
const [isAudioCombining] = useState(false); const [isAudioCombining] = useState(false);
const pageTextCacheRef = useRef<Map<number, string>>(new Map());
const [currDocPage, setCurrDocPage] = useState<number>(currDocPageNumber);
useEffect(() => {
setCurrDocPage(currDocPageNumber);
}, [currDocPageNumber]);
/** /**
* Handles successful PDF document load * Handles successful PDF document load
@ -129,22 +141,60 @@ export function PDFProvider({ children }: { children: ReactNode }) {
const loadCurrDocText = useCallback(async () => { const loadCurrDocText = useCallback(async () => {
try { try {
if (!pdfDocument) return; if (!pdfDocument) return;
const text = await extractTextFromPDF(pdfDocument, currDocPage, {
const margins = {
header: headerMargin, header: headerMargin,
footer: footerMargin, footer: footerMargin,
left: leftMargin, left: leftMargin,
right: rightMargin right: rightMargin
}); };
// Only update TTS text if the content has actually changed
// This prevents unnecessary resets of the sentence index const getPageText = async (pageNumber: number, shouldCache = false): Promise<string> => {
if (pageTextCacheRef.current.has(pageNumber)) {
const cached = pageTextCacheRef.current.get(pageNumber)!;
if (!shouldCache) {
pageTextCacheRef.current.delete(pageNumber);
}
return cached;
}
const extracted = await extractTextFromPDF(pdfDocument, pageNumber, margins);
if (shouldCache) {
pageTextCacheRef.current.set(pageNumber, extracted);
}
return extracted;
};
const totalPages = currDocPages ?? pdfDocument.numPages;
const nextPageNumber = currDocPageNumber < totalPages ? currDocPageNumber + 1 : undefined;
const [text, nextText] = await Promise.all([
getPageText(currDocPageNumber),
nextPageNumber ? getPageText(nextPageNumber, true) : Promise.resolve<string | undefined>(undefined),
]);
if (text !== currDocText || text === '') { if (text !== currDocText || text === '') {
setCurrDocText(text); setCurrDocText(text);
setTTSText(text); setTTSText(text, {
location: currDocPageNumber,
nextLocation: nextPageNumber,
nextText: nextText?.slice(0, CONTINUATION_PREVIEW_CHARS),
});
} }
} catch (error) { } catch (error) {
console.error('Error loading PDF text:', error); console.error('Error loading PDF text:', error);
} }
}, [pdfDocument, currDocPage, setTTSText, currDocText, headerMargin, footerMargin, leftMargin, rightMargin]); }, [
pdfDocument,
currDocPageNumber,
currDocPages,
setTTSText,
currDocText,
headerMargin,
footerMargin,
leftMargin,
rightMargin,
]);
/** /**
* Effect hook to update document text when the page changes * Effect hook to update document text when the page changes
@ -154,7 +204,7 @@ export function PDFProvider({ children }: { children: ReactNode }) {
if (currDocData) { if (currDocData) {
loadCurrDocText(); loadCurrDocText();
} }
}, [currDocPage, currDocData, loadCurrDocText]); }, [currDocPageNumber, currDocData, loadCurrDocText]);
/** /**
* Sets the current document based on its ID * Sets the current document based on its ID
@ -185,6 +235,7 @@ export function PDFProvider({ children }: { children: ReactNode }) {
setCurrDocText(undefined); setCurrDocText(undefined);
setCurrDocPages(undefined); setCurrDocPages(undefined);
setPdfDocument(undefined); setPdfDocument(undefined);
pageTextCacheRef.current.clear();
stop(); stop();
}, [setCurrDocPages, stop]); }, [setCurrDocPages, stop]);
@ -212,16 +263,20 @@ export function PDFProvider({ children }: { children: ReactNode }) {
let totalLength = 0; let totalLength = 0;
for (let pageNum = 1; pageNum <= pdfDocument.numPages; pageNum++) { for (let pageNum = 1; pageNum <= pdfDocument.numPages; pageNum++) {
const text = await extractTextFromPDF(pdfDocument, pageNum, { const rawText = await extractTextFromPDF(pdfDocument, pageNum, {
header: headerMargin, header: headerMargin,
footer: footerMargin, footer: footerMargin,
left: leftMargin, left: leftMargin,
right: rightMargin right: rightMargin
}); });
const trimmedText = text.trim(); const trimmedText = rawText.trim();
if (trimmedText) { if (trimmedText) {
textPerPage.push(trimmedText); const processedText = smartSentenceSplitting
totalLength += trimmedText.length; ? processTextToSentences(trimmedText).join(' ')
: trimmedText;
textPerPage.push(processedText);
totalLength += processedText.length;
} }
} }
@ -405,7 +460,7 @@ export function PDFProvider({ children }: { children: ReactNode }) {
console.error('Error creating audiobook:', error); console.error('Error creating audiobook:', error);
throw error; throw error;
} }
}, [pdfDocument, headerMargin, footerMargin, leftMargin, rightMargin, apiKey, baseUrl, voice, voiceSpeed, ttsProvider, ttsModel, ttsInstructions]); }, [pdfDocument, headerMargin, footerMargin, leftMargin, rightMargin, apiKey, baseUrl, voice, voiceSpeed, ttsProvider, ttsModel, ttsInstructions, smartSentenceSplitting]);
/** /**
* Regenerates a specific chapter (page) of the PDF audiobook * Regenerates a specific chapter (page) of the PDF audiobook
@ -444,18 +499,22 @@ export function PDFProvider({ children }: { children: ReactNode }) {
const pageNum = nonEmptyPages[chapterIndex]; const pageNum = nonEmptyPages[chapterIndex];
// Extract text from the mapped page // Extract text from the mapped page
const text = await extractTextFromPDF(pdfDocument, pageNum, { const rawText = await extractTextFromPDF(pdfDocument, pageNum, {
header: headerMargin, header: headerMargin,
footer: footerMargin, footer: footerMargin,
left: leftMargin, left: leftMargin,
right: rightMargin right: rightMargin
}); });
const trimmedText = text.trim(); const trimmedText = rawText.trim();
if (!trimmedText) { if (!trimmedText) {
throw new Error('No text content found on page'); throw new Error('No text content found on page');
} }
const textForTTS = smartSentenceSplitting
? processTextToSentences(trimmedText).join(' ')
: trimmedText;
// Use logical chapter numbering (index + 1) to match original generation titles // Use logical chapter numbering (index + 1) to match original generation titles
const chapterTitle = `Page ${chapterIndex + 1}`; const chapterTitle = `Page ${chapterIndex + 1}`;
@ -475,7 +534,7 @@ export function PDFProvider({ children }: { children: ReactNode }) {
'x-tts-provider': ttsProvider, 'x-tts-provider': ttsProvider,
}, },
body: JSON.stringify({ body: JSON.stringify({
text: trimmedText, text: textForTTS,
voice: voice || (ttsProvider === 'openai' ? 'alloy' : (ttsProvider === 'deepinfra' ? 'af_bella' : 'af_sarah')), voice: voice || (ttsProvider === 'openai' ? 'alloy' : (ttsProvider === 'deepinfra' ? 'af_bella' : 'af_sarah')),
speed: voiceSpeed, speed: voiceSpeed,
format: 'mp3', format: 'mp3',
@ -549,7 +608,7 @@ export function PDFProvider({ children }: { children: ReactNode }) {
console.error('Error regenerating page:', error); console.error('Error regenerating page:', error);
throw error; throw error;
} }
}, [pdfDocument, headerMargin, footerMargin, leftMargin, rightMargin, apiKey, baseUrl, voice, voiceSpeed, ttsProvider, ttsModel, ttsInstructions]); }, [pdfDocument, headerMargin, footerMargin, leftMargin, rightMargin, apiKey, baseUrl, voice, voiceSpeed, ttsProvider, ttsModel, ttsInstructions, smartSentenceSplitting]);
/** /**
* Effect hook to initialize TTS as non-EPUB mode * Effect hook to initialize TTS as non-EPUB mode
@ -558,6 +617,16 @@ export function PDFProvider({ children }: { children: ReactNode }) {
setIsEPUB(false); setIsEPUB(false);
}, [setIsEPUB]); }, [setIsEPUB]);
useEffect(() => {
registerVisualPageChangeHandler(location => {
if (typeof location !== 'number') return;
if (!pdfDocument) return;
const totalPages = currDocPages ?? pdfDocument.numPages;
const clamped = Math.min(Math.max(location, 1), totalPages);
setCurrDocPage(clamped);
});
}, [registerVisualPageChangeHandler, currDocPages, pdfDocument]);
// Context value memoization // Context value memoization
const contextValue = useMemo( const contextValue = useMemo(
() => ({ () => ({

View file

@ -37,7 +37,7 @@ import { useAudioContext } from '@/hooks/audio/useAudioContext';
import { getLastDocumentLocation, setLastDocumentLocation } from '@/utils/indexedDB'; import { getLastDocumentLocation, setLastDocumentLocation } from '@/utils/indexedDB';
import { useBackgroundState } from '@/hooks/audio/useBackgroundState'; import { useBackgroundState } from '@/hooks/audio/useBackgroundState';
import { withRetry } from '@/utils/audio'; import { withRetry } from '@/utils/audio';
import { processTextToSentences } from '@/utils/nlp'; import { preprocessSentenceForAudio, processTextToSentences } from '@/utils/nlp';
import { isKokoroModel } from '@/utils/voice'; import { isKokoroModel } from '@/utils/voice';
// Media globals // Media globals
@ -72,16 +72,185 @@ interface TTSContextType {
pause: () => void; pause: () => void;
stop: () => void; stop: () => void;
stopAndPlayFromIndex: (index: number) => void; stopAndPlayFromIndex: (index: number) => void;
setText: (text: string, shouldPause?: boolean) => void; setText: (text: string, options?: boolean | SetTextOptions) => void;
setCurrDocPages: (num: number | undefined) => void; setCurrDocPages: (num: number | undefined) => void;
setSpeedAndRestart: (speed: number) => void; setSpeedAndRestart: (speed: number) => void;
setAudioPlayerSpeedAndRestart: (speed: number) => void; setAudioPlayerSpeedAndRestart: (speed: number) => void;
setVoiceAndRestart: (voice: string) => void; setVoiceAndRestart: (voice: string) => void;
skipToLocation: (location: string | number, shouldPause?: boolean) => void; skipToLocation: (location: string | number, shouldPause?: boolean) => void;
registerLocationChangeHandler: (handler: (location: string | number) => void) => void; // EPUB-only: Handles chapter navigation registerLocationChangeHandler: (handler: (location: string | number) => void) => void; // EPUB-only: Handles chapter navigation
registerVisualPageChangeHandler: (handler: (location: string | number) => void) => void;
setIsEPUB: (isEPUB: boolean) => void; setIsEPUB: (isEPUB: boolean) => void;
} }
interface SetTextOptions {
shouldPause?: boolean;
location?: string | number;
nextLocation?: string | number;
nextText?: string;
}
interface ContinuationMergeResult {
text: string;
carried: string;
}
interface PageTurnEstimate {
location: string | number;
sentenceIndex: number;
fraction: number;
}
const CONTINUATION_LOOKAHEAD = 600;
const SENTENCE_ENDING = /[.?!…]["'”’)\]]*\s*$/;
const normalizeLocationKey = (location: string | number) =>
typeof location === 'number' ? `num:${location}` : `str:${location}`;
const isWhitespaceChar = (char: string) => /\s/.test(char);
const skipWhitespace = (source: string, start: number) => {
let index = start;
while (index < source.length && isWhitespaceChar(source[index])) {
index++;
}
return index;
};
const matchNormalizedPrefixLength = (text: string, prefix: string): number | null => {
let textIndex = 0;
let prefixIndex = 0;
while (prefixIndex < prefix.length) {
const prefixChar = prefix[prefixIndex];
if (isWhitespaceChar(prefixChar)) {
prefixIndex = skipWhitespace(prefix, prefixIndex);
textIndex = skipWhitespace(text, textIndex);
continue;
}
if (textIndex >= text.length) {
return null;
}
const textChar = text[textIndex];
if (textChar === prefixChar || textChar.toLowerCase() === prefixChar.toLowerCase()) {
textIndex++;
prefixIndex++;
continue;
}
return null;
}
return textIndex;
};
const needsSentenceContinuation = (text: string) => {
const trimmed = text.trim();
if (!trimmed) {
return false;
}
return !SENTENCE_ENDING.test(trimmed);
};
const stripContinuationPrefix = (text: string, prefix: string) => {
if (!prefix) return { text, removed: false };
// Try literal match first since PDF text is normalized already
if (text.startsWith(prefix)) {
return {
text: text.slice(prefix.length).trimStart(),
removed: true,
};
}
const trimmedPrefix = prefix.trimStart();
const trimmedText = text.trimStart();
if (trimmedText.startsWith(trimmedPrefix)) {
const offset = text.length - trimmedText.length;
return {
text: text.slice(offset + trimmedPrefix.length).trimStart(),
removed: true,
};
}
const matchedLength = matchNormalizedPrefixLength(text, prefix);
if (matchedLength !== null) {
return {
text: text.slice(matchedLength).trimStart(),
removed: true,
};
}
return { text, removed: false };
};
const extractContinuationSlice = (nextText: string): ContinuationMergeResult | null => {
if (!nextText?.trim()) {
return null;
}
const snippet = nextText.trim().slice(0, CONTINUATION_LOOKAHEAD);
let boundaryIndex = -1;
for (let i = 0; i < snippet.length; i++) {
const char = snippet[i];
if (/[.?!…]/.test(char)) {
let j = i + 1;
while (j < snippet.length && /["'”’)\]]/.test(snippet[j])) {
j++;
}
while (j < snippet.length && /\s/.test(snippet[j])) {
j++;
}
boundaryIndex = j;
break;
}
}
if (boundaryIndex === -1) {
return null;
}
const rawSlice = snippet.slice(0, boundaryIndex);
const addition = rawSlice.trim();
if (!addition) {
return null;
}
return {
text: addition,
carried: rawSlice,
};
};
const mergeContinuation = (text: string, nextText: string): ContinuationMergeResult | null => {
if (!needsSentenceContinuation(text)) {
return null;
}
const slice = extractContinuationSlice(nextText);
if (!slice) {
return null;
}
const trimmed = text.trimEnd();
const endsWithHyphen = trimmed.endsWith('-');
const base = endsWithHyphen ? trimmed.slice(0, -1) : trimmed;
const joiner = endsWithHyphen ? '' : (base ? ' ' : '');
const mergedText = `${base}${joiner}${slice.text}`.trim();
return {
text: mergedText,
carried: slice.carried,
};
};
// Create the context // Create the context
const TTSContext = createContext<TTSContextType | undefined>(undefined); const TTSContext = createContext<TTSContextType | undefined>(undefined);
@ -107,6 +276,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
ttsInstructions: configTTSInstructions, ttsInstructions: configTTSInstructions,
updateConfigKey, updateConfigKey,
skipBlank, skipBlank,
smartSentenceSplitting,
} = useConfig(); } = useConfig();
// Remove OpenAI client reference as it's no longer needed // Remove OpenAI client reference as it's no longer needed
@ -116,6 +286,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
// Add ref for location change handler // Add ref for location change handler
const locationChangeHandlerRef = useRef<((location: string | number) => void) | null>(null); const locationChangeHandlerRef = useRef<((location: string | number) => void) | null>(null);
const visualPageChangeHandlerRef = useRef<((location: string | number) => void) | null>(null);
/** /**
* Registers a handler function for location changes in EPUB documents * Registers a handler function for location changes in EPUB documents
@ -127,6 +298,10 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
locationChangeHandlerRef.current = handler; locationChangeHandlerRef.current = handler;
}, []); }, []);
const registerVisualPageChangeHandler = useCallback((handler: (location: string | number) => void) => {
visualPageChangeHandlerRef.current = handler;
}, []);
// Get document ID from URL params // Get document ID from URL params
const { id } = useParams(); const { id } = useParams();
@ -158,6 +333,11 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
const [pendingRestoreIndex, setPendingRestoreIndex] = useState<number | null>(null); const [pendingRestoreIndex, setPendingRestoreIndex] = useState<number | null>(null);
// Guard to coalesce rapid restarts and only resume the latest change // Guard to coalesce rapid restarts and only resume the latest change
const restartSeqRef = useRef(0); const restartSeqRef = useRef(0);
// Track continuation slices for PDF/EPUB page transitions
const continuationCarryRef = useRef<Map<string, string>>(new Map());
const epubContinuationRef = useRef<string | null>(null);
const pageTurnEstimateRef = useRef<PageTurnEstimate | null>(null);
const pageTurnTimeoutRef = useRef<ReturnType<typeof setTimeout> | null>(null);
/** /**
* Processes text into sentences using the shared NLP utility * Processes text into sentences using the shared NLP utility
@ -181,20 +361,22 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
const abortAudio = useCallback((clearPending = false) => { const abortAudio = useCallback((clearPending = false) => {
if (activeHowl) { if (activeHowl) {
activeHowl.stop(); activeHowl.stop();
activeHowl.unload(); // Ensure Howl instance is fully cleaned up activeHowl.unload();
setActiveHowl(null); setActiveHowl(null);
} }
if (clearPending) { if (clearPending) {
// Abort all active TTS requests
console.log('Aborting active TTS requests');
activeAbortControllers.current.forEach(controller => { activeAbortControllers.current.forEach(controller => {
controller.abort(); controller.abort();
}); });
activeAbortControllers.current.clear(); activeAbortControllers.current.clear();
// Clear any pending preload requests
preloadRequests.current.clear(); preloadRequests.current.clear();
} }
if (pageTurnTimeoutRef.current) {
clearTimeout(pageTurnTimeoutRef.current);
pageTurnTimeoutRef.current = null;
}
}, [activeHowl]); }, [activeHowl]);
/** /**
@ -211,7 +393,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
* Works for both PDF pages and EPUB locations * Works for both PDF pages and EPUB locations
* *
* @param {string | number} location - The target location to navigate to * @param {string | number} location - The target location to navigate to
* @param {boolean} keepPlaying - Whether to maintain playback state * @param {boolean} shouldPause - Whether to pause playback
*/ */
const skipToLocation = useCallback((location: string | number, shouldPause = false) => { const skipToLocation = useCallback((location: string | number, shouldPause = false) => {
// Reset state for new content in correct order // Reset state for new content in correct order
@ -233,7 +415,6 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
// Handle within current page bounds // Handle within current page bounds
if (nextIndex < sentences.length && nextIndex >= 0) { if (nextIndex < sentences.length && nextIndex >= 0) {
console.log('isEPUB', isEPUB!);
setCurrentIndex(nextIndex); setCurrentIndex(nextIndex);
return; return;
} }
@ -297,9 +478,65 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
* *
* @param {string} text - The text to be processed * @param {string} text - The text to be processed
*/ */
const setText = useCallback((text: string, shouldPause = false) => { const setText = useCallback((text: string, options?: boolean | SetTextOptions) => {
// Check for blank section first const normalizedOptions: SetTextOptions = typeof options === 'boolean'
if (handleBlankSection(text)) return; ? { shouldPause: options }
: (options || {});
let workingText = text;
// Apply or clear sentence continuation logic based on config
let continuationCarried: string | undefined;
if (smartSentenceSplitting) {
if (isEPUB && epubContinuationRef.current) {
const { text: strippedText, removed } = stripContinuationPrefix(workingText, epubContinuationRef.current);
workingText = strippedText;
if (removed) {
epubContinuationRef.current = null;
}
}
if (!isEPUB && normalizedOptions.location !== undefined) {
const key = normalizeLocationKey(normalizedOptions.location);
const carried = continuationCarryRef.current.get(key);
if (carried) {
const { text: strippedText, removed } = stripContinuationPrefix(workingText, carried);
workingText = strippedText;
if (removed) {
continuationCarryRef.current.delete(key);
}
}
}
if (normalizedOptions.nextText) {
const merged = mergeContinuation(workingText, normalizedOptions.nextText);
if (merged) {
workingText = merged.text;
continuationCarried = merged.carried;
}
}
if (continuationCarried) {
if (isEPUB) {
epubContinuationRef.current = continuationCarried;
} else if (normalizedOptions.nextLocation !== undefined) {
continuationCarryRef.current.set(
normalizeLocationKey(normalizedOptions.nextLocation),
continuationCarried
);
}
}
} else {
// When disabled, clear any stale continuation state
epubContinuationRef.current = null;
continuationCarryRef.current.clear();
pageTurnEstimateRef.current = null;
}
// Check for blank section after adjustments
if (handleBlankSection(workingText)) return;
const shouldPause = normalizedOptions.shouldPause ?? false;
// Keep track of previous state and pause playback // Keep track of previous state and pause playback
const wasPlaying = isPlaying; const wasPlaying = isPlaying;
@ -307,8 +544,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
abortAudio(true); // Clear pending requests since text is changing abortAudio(true); // Clear pending requests since text is changing
setIsProcessing(true); // Set processing state before text processing starts setIsProcessing(true); // Set processing state before text processing starts
console.log('Setting text:', text); processTextToSentencesLocal(workingText)
processTextToSentencesLocal(text)
.then(newSentences => { .then(newSentences => {
if (newSentences.length === 0) { if (newSentences.length === 0) {
console.warn('No sentences found in text'); console.warn('No sentences found in text');
@ -329,6 +565,41 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
setCurrentIndex(0); setCurrentIndex(0);
} }
// Compute auto page-turn estimate for PDFs when we have a continuation
if (smartSentenceSplitting && !isEPUB && continuationCarried && normalizedOptions.nextLocation !== undefined) {
const continuationNormalized = preprocessSentenceForAudio(continuationCarried);
if (continuationNormalized) {
let bestEstimate: PageTurnEstimate | null = null;
newSentences.forEach((sentence, index) => {
const normalizedSentence = preprocessSentenceForAudio(sentence);
if (!normalizedSentence) return;
if (!normalizedSentence.toLowerCase().endsWith(continuationNormalized.toLowerCase())) return;
const totalLength = normalizedSentence.length;
const continuationLength = continuationNormalized.length;
if (totalLength <= continuationLength) return;
const baseLength = totalLength - continuationLength;
const fraction = baseLength / totalLength;
if (fraction <= 0 || fraction >= 1) return;
bestEstimate = {
location: normalizedOptions.nextLocation!,
sentenceIndex: index,
fraction,
};
});
pageTurnEstimateRef.current = bestEstimate;
} else {
pageTurnEstimateRef.current = null;
}
} else {
pageTurnEstimateRef.current = null;
}
setIsProcessing(false); setIsProcessing(false);
// Restore playback state if needed // Restore playback state if needed
@ -347,7 +618,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
duration: 3000, duration: 3000,
}); });
}); });
}, [isPlaying, handleBlankSection, abortAudio, processTextToSentencesLocal, pendingRestoreIndex, isEPUB]); }, [isPlaying, handleBlankSection, abortAudio, processTextToSentencesLocal, pendingRestoreIndex, isEPUB, smartSentenceSplitting]);
/** /**
* Toggles the playback state between playing and paused * Toggles the playback state between playing and paused
@ -576,7 +847,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
* *
* @param {string} sentence - The sentence to play * @param {string} sentence - The sentence to play
*/ */
const playSentenceWithHowl = useCallback(async (sentence: string) => { const playSentenceWithHowl = useCallback(async (sentence: string, sentenceIndex: number) => {
if (!sentence) { if (!sentence) {
console.log('No sentence to play'); console.log('No sentence to play');
setIsProcessing(false); setIsProcessing(false);
@ -606,13 +877,35 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
preload: true, preload: true,
pool: 5, pool: 5,
rate: audioSpeed, rate: audioSpeed,
onload: function (this: Howl) {
const estimate = pageTurnEstimateRef.current;
if (!estimate || estimate.sentenceIndex !== sentenceIndex) return;
if (!visualPageChangeHandlerRef.current) return;
const duration = this.duration();
if (!duration || !Number.isFinite(duration)) return;
const delayMs = duration * estimate.fraction * 1000;
if (delayMs <= 0 || delayMs >= duration * 1000) return;
if (pageTurnTimeoutRef.current) {
clearTimeout(pageTurnTimeoutRef.current);
}
pageTurnTimeoutRef.current = setTimeout(() => {
if (!isPlaying) return;
const currentEstimate = pageTurnEstimateRef.current;
if (!currentEstimate || currentEstimate.sentenceIndex !== sentenceIndex) return;
visualPageChangeHandlerRef.current?.(currentEstimate.location);
}, delayMs);
},
onplay: () => { onplay: () => {
setIsProcessing(false); setIsProcessing(false);
if ('mediaSession' in navigator) { if ('mediaSession' in navigator) {
navigator.mediaSession.playbackState = 'playing'; navigator.mediaSession.playbackState = 'playing';
} }
}, },
onplayerror: function(this: Howl, error) { onplayerror: function (this: Howl, error) {
console.warn('Howl playback error:', error); console.warn('Howl playback error:', error);
// Try to recover by forcing HTML5 audio mode // Try to recover by forcing HTML5 audio mode
if (this.state() === 'loaded') { if (this.state() === 'loaded') {
@ -623,7 +916,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
this.load(); this.load();
} }
}, },
onloaderror: async function(this: Howl, error) { onloaderror: async function (this: Howl, error) {
console.warn(`Error loading audio (attempt ${retryCount + 1}/${MAX_RETRIES}):`, error); console.warn(`Error loading audio (attempt ${retryCount + 1}/${MAX_RETRIES}):`, error);
if (retryCount < MAX_RETRIES) { if (retryCount < MAX_RETRIES) {
@ -659,14 +952,18 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
advance(); advance();
} }
}, },
onend: function(this: Howl) { onend: function (this: Howl) {
this.unload(); this.unload();
setActiveHowl(null); setActiveHowl(null);
if (pageTurnTimeoutRef.current) {
clearTimeout(pageTurnTimeoutRef.current);
pageTurnTimeoutRef.current = null;
}
if (isPlaying) { if (isPlaying) {
advance(); advance();
} }
}, },
onstop: function(this: Howl) { onstop: function (this: Howl) {
setIsProcessing(false); setIsProcessing(false);
this.unload(); this.unload();
} }
@ -705,7 +1002,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
}, [isPlaying, advance, activeHowl, processSentence, audioSpeed]); }, [isPlaying, advance, activeHowl, processSentence, audioSpeed]);
const playAudio = useCallback(async () => { const playAudio = useCallback(async () => {
const howl = await playSentenceWithHowl(sentences[currentIndex]); const howl = await playSentenceWithHowl(sentences[currentIndex], currentIndex);
if (howl) { if (howl) {
howl.play(); howl.play();
} }
@ -777,6 +1074,8 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
// Cancel any ongoing request // Cancel any ongoing request
abortAudio(); abortAudio();
locationChangeHandlerRef.current = null; locationChangeHandlerRef.current = null;
epubContinuationRef.current = null;
continuationCarryRef.current.clear();
setIsPlaying(false); setIsPlaying(false);
setCurrentIndex(0); setCurrentIndex(0);
setSentences([]); setSentences([]);
@ -921,6 +1220,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
setVoiceAndRestart, setVoiceAndRestart,
skipToLocation, skipToLocation,
registerLocationChangeHandler, registerLocationChangeHandler,
registerVisualPageChangeHandler,
setIsEPUB setIsEPUB
}), [ }), [
isPlaying, isPlaying,
@ -945,6 +1245,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
setVoiceAndRestart, setVoiceAndRestart,
skipToLocation, skipToLocation,
registerLocationChangeHandler, registerLocationChangeHandler,
registerVisualPageChangeHandler,
setIsEPUB setIsEPUB
]); ]);

View file

@ -1,6 +1,6 @@
import { pdfjs } from 'react-pdf'; import { pdfjs } from 'react-pdf';
import type { TextItem } from 'pdfjs-dist/types/src/display/api'; import type { TextItem } from 'pdfjs-dist/types/src/display/api';
import type { PDFDocumentProxy } from 'pdfjs-dist'; import { type PDFDocumentProxy, TextLayer } from 'pdfjs-dist';
import "core-js/proposals/promise-with-resolvers"; import "core-js/proposals/promise-with-resolvers";
import { processTextToSentences } from '@/utils/nlp'; import { processTextToSentences } from '@/utils/nlp';
import { CmpStr } from 'cmpstr'; import { CmpStr } from 'cmpstr';
@ -52,6 +52,26 @@ function initPDFWorker() {
// Initialize the worker // Initialize the worker
initPDFWorker(); initPDFWorker();
// Patch TextLayer.render to treat cancelled renders as non-errors
try {
const textLayerProto = TextLayer?.prototype;
const originalRender = textLayerProto?.render;
if (typeof originalRender === 'function') {
textLayerProto.render = async function patchedRender(...args) {
const task = originalRender.apply(this, args);
if (!task || typeof task.then !== 'function') return task;
return task.catch((error) => {
if (error && (error.name === 'AbortException' || error.name === 'RenderingCancelledException')) {
return;
}
throw error;
});
};
}
} catch (e) {
console.error('Error patching TextLayer.render:', e);
}
interface TextMatch { interface TextMatch {
elements: HTMLElement[]; elements: HTMLElement[];
rating: number; rating: number;

BIN
tests/files/sample.mp3 Normal file

Binary file not shown.

View file

@ -1,6 +1,30 @@
import { Page, expect } from '@playwright/test'; import { Page, expect } from '@playwright/test';
import fs from 'fs';
import path from 'path';
const DIR = './tests/files/'; const DIR = './tests/files/';
const TTS_MOCK_PATH = path.join(__dirname, 'files', 'sample.mp3');
let ttsMockBuffer: Buffer | null = null;
async function ensureTtsRouteMock(page: Page) {
if (!ttsMockBuffer) {
ttsMockBuffer = fs.readFileSync(TTS_MOCK_PATH);
}
await page.route('**/api/tts', async (route) => {
// Only mock the POST TTS generation calls; let anything else pass through.
if (route.request().method().toUpperCase() !== 'POST') {
return route.continue();
}
await route.fulfill({
status: 200,
contentType: 'audio/mpeg',
body: ttsMockBuffer as Buffer,
});
});
}
// Small util to safely use filenames inside regex patterns // Small util to safely use filenames inside regex patterns
function escapeRegExp(input: string) { function escapeRegExp(input: string) {
return input.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); return input.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
@ -79,6 +103,9 @@ export async function pauseTTSAndVerify(page: Page) {
* Common test setup function * Common test setup function
*/ */
export async function setupTest(page: Page) { export async function setupTest(page: Page) {
// Mock the TTS API so tests don't hit the real TTS service.
await ensureTtsRouteMock(page);
// Navigate to the home page before each test // Navigate to the home page before each test
await page.goto('/'); await page.goto('/');
await page.waitForLoadState('networkidle'); await page.waitForLoadState('networkidle');
@ -341,7 +368,7 @@ export async function expectMediaState(page: Page, state: 'playing' | 'paused')
} catch { } catch {
return false; return false;
} }
}, state, { timeout: 25000 }); }, state);
} }
// Use Navigator to go to a specific page number (PDF) // Use Navigator to go to a specific page number (PDF)