feat(tts): add smart sentence continuation
introduce configurable smart sentence splitting with persisted state, UI toggle, and EPUB/PDF contexts that send continuation metadata. enhance TTS pipeline to merge cross-page sentences, manage carryover, and trigger visual page changes during playback for smoother narration. update README with kokoro quick-start guidance, remove the legacy issues mapping doc, and add deterministic TTS mocks plus sample audio for Playwright tests.
This commit is contained in:
parent
d7ef0fa8bb
commit
e3799e40aa
14 changed files with 641 additions and 197 deletions
62
README.md
62
README.md
|
|
@ -36,9 +36,12 @@ OpenReader WebUI is a document reader with Text-to-Speech capabilities, offering
|
|||
- Recent version of Docker installed on your machine
|
||||
- A TTS API server (Kokoro-FastAPI, Orpheus-FastAPI, Deepinfra, OpenAI, etc.) running and accessible
|
||||
|
||||
> **Note:** If you have good hardware, you can run [Kokoro-FastAPI with Docker locally](#🗣️-local-kokoro-fastapi-quick-start-cpu-or-gpu) (see below).
|
||||
|
||||
### 1. 🐳 Start the Docker container:
|
||||
```bash
|
||||
docker run --name openreader-webui \
|
||||
--restart unless-stopped \
|
||||
-p 3003:3003 \
|
||||
-v openreader_docstore:/app/docstore \
|
||||
ghcr.io/richardr1126/openreader-webui:latest
|
||||
|
|
@ -47,6 +50,7 @@ OpenReader WebUI is a document reader with Text-to-Speech capabilities, offering
|
|||
(Optionally): Set the TTS `API_BASE` URL and/or `API_KEY` to be default for all devices
|
||||
```bash
|
||||
docker run --name openreader-webui \
|
||||
--restart unless-stopped \
|
||||
-e API_KEY=none \
|
||||
-e API_BASE=http://host.docker.internal:8880/v1 \
|
||||
-p 3003:3003 \
|
||||
|
|
@ -72,36 +76,48 @@ docker rm openreader-webui && \
|
|||
docker pull ghcr.io/richardr1126/openreader-webui:latest
|
||||
```
|
||||
|
||||
### (Alternate) 🐳 Configuration with Docker Compose and Kokoro-FastAPI
|
||||
### 🗣️ Local Kokoro-FastAPI Quick-start (CPU or GPU)
|
||||
|
||||
A complete example docker-compose file with Kokoro-FastAPI and OpenReader WebUI is available in [`docs/examples/docker-compose.yml`](docs/examples/docker-compose.yml). You can download and use it:
|
||||
You can run the Kokoro TTS API server directly with Docker. **We are not responsible for issues with Kokoro-FastAPI.** For best performance, use an NVIDIA GPU (for GPU version) or Apple Silicon (for CPU version).
|
||||
|
||||
> **Note:** When using these, set the `API_BASE` env var to `http://host.docker.internal:8880/v1` or `http://kokoro-tts:8880/v1`.
|
||||
> You can also use the example `docker-compose.yml` in `examples/docker-compose.yml` if you prefer Docker Compose.
|
||||
|
||||
**CPU Version:**
|
||||
```bash
|
||||
# Download example docker-compose.yml
|
||||
curl --create-dirs -L -o openreader-compose/docker-compose.yml https://raw.githubusercontent.com/richardr1126/OpenReader-WebUI/main/docs/examples/docker-compose.yml
|
||||
|
||||
cd openreader-compose
|
||||
docker compose up -d
|
||||
docker run -d \
|
||||
--name kokoro-tts \
|
||||
--restart unless-stopped \
|
||||
-p 8880:8880 \
|
||||
-e ONNX_NUM_THREADS=8 \
|
||||
-e ONNX_INTER_OP_THREADS=4 \
|
||||
-e ONNX_EXECUTION_MODE=parallel \
|
||||
-e ONNX_OPTIMIZATION_LEVEL=all \
|
||||
-e ONNX_MEMORY_PATTERN=true \
|
||||
-e ONNX_ARENA_EXTEND_STRATEGY=kNextPowerOfTwo \
|
||||
-e API_LOG_LEVEL=DEBUG \
|
||||
ghcr.io/remsky/kokoro-fastapi-cpu:v0.2.4
|
||||
```
|
||||
|
||||
Or add OpenReader WebUI to your existing `docker-compose.yml`:
|
||||
```yaml
|
||||
services:
|
||||
openreader-webui:
|
||||
container_name: openreader-webui
|
||||
image: ghcr.io/richardr1126/openreader-webui:latest
|
||||
environment:
|
||||
- API_BASE=http://host.docker.internal:8880/v1
|
||||
ports:
|
||||
- "3003:3003"
|
||||
volumes:
|
||||
- docstore:/app/docstore
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
docstore:
|
||||
**GPU Version:**
|
||||
```bash
|
||||
docker run -d \
|
||||
--name kokoro-tts \
|
||||
--gpus all \
|
||||
--user 1001:1001 \
|
||||
--restart unless-stopped \
|
||||
-p 8880:8880 \
|
||||
-e USE_GPU=true \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
-e API_LOG_LEVEL=DEBUG \
|
||||
ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.4
|
||||
```
|
||||
|
||||
> **Note:**
|
||||
> - These commands are for running the Kokoro TTS API server only. For issues or support, see the [Kokoro-FastAPI repository](https://github.com/remsky/Kokoro-FastAPI).
|
||||
> - The GPU version requires NVIDIA Docker support and works best with NVIDIA GPUs. The CPU version works best on Apple Silicon or modern x86 CPUs.
|
||||
> - Adjust environment variables as needed for your hardware and use case.
|
||||
|
||||
## Dev Installation
|
||||
|
||||
### Prerequisites
|
||||
|
|
|
|||
|
|
@ -1,100 +0,0 @@
|
|||
# OpenReader Issue Triage and Mapping
|
||||
|
||||
Repository: https://github.com/richardr1126/OpenReader-WebUI/issues
|
||||
Reviewed via gh at 2025-11-10.
|
||||
|
||||
Summary of open items included:
|
||||
- #59 Feature: Chapter-Based MP3 Export
|
||||
- #48 Bug: Failed to export after complete render with Kokoro
|
||||
- #47 Feature: Combine voices (Kokoro “plus” syntax)
|
||||
- #44 Bug: Dialog not chunked together
|
||||
- #40 Bug: PDF left/right extraction margins not working
|
||||
|
||||
Global guardrails
|
||||
- Streaming-first playback (replace Howler with HTMLAudioElement/MSE)
|
||||
- Dexie DB replacing vanilla IndexedDB
|
||||
- Single engine cut-over (no dual engines)
|
||||
- Keep audiobook (m4b) and server-side sync
|
||||
|
||||
Issue #59 — Chapter-Based MP3 Export
|
||||
Type: Feature
|
||||
Hypothesis / intent:
|
||||
- Users want one-mp3-per-chapter output (besides full-book m4b).
|
||||
Ideas:
|
||||
- Add chapterized MP3 pipeline that chunks by adapter “chapter” units.
|
||||
- Provide ZIP export for many MP3 files (streamed).
|
||||
API design:
|
||||
- POST /api/audio/convert?mode=chapters&format=mp3 -> returns stream of a ZIP.
|
||||
Logging to add:
|
||||
- Chapter boundaries, byte sizes, cumulative progress in [src/app/api/audio/convert/route.ts](src/app/api/audio/convert/route.ts:1).
|
||||
Acceptance:
|
||||
- A multi-chapter EPUB produces N mp3 parts named “NN - Chapter Title.mp3”; total duration ~ sum of parts; ZIP streamed without timeouts in Docker.
|
||||
|
||||
Issue #48 — Failed export after complete render with Kokoro
|
||||
Type: Bug (large-book export)
|
||||
Observations:
|
||||
- UI reaches 100% then resets, no download.
|
||||
- Docker shows “fin … undefined”, likely final transfer problem (not TTS).
|
||||
Likely root causes:
|
||||
- Final m4b delivery uses single huge arrayBuffer; browser memory/timeout.
|
||||
- Missing Content-Disposition/Range; no resumable download.
|
||||
- Temp-file lifecycle cleanup racing with response.
|
||||
Ideas for remediation:
|
||||
- Serve final artifact as file on disk with streaming and Range:
|
||||
- New endpoint: GET /api/audio/convert/download?bookId=… that streams file with Accept-Ranges.
|
||||
- UI performs streamed download; no arrayBuffer buffering.
|
||||
- Optionally support S3-compatible offload in future.
|
||||
Touched modules:
|
||||
- [src/app/api/audio/convert/route.ts](src/app/api/audio/convert/route.ts:1)
|
||||
Instrumentation to add:
|
||||
- Book ID, file size on disk, stream chunk counts, and client-abort detection.
|
||||
- Add explicit log/error surfaces at:
|
||||
- [src/app/api/audio/convert/route.ts](src/app/api/audio/convert/route.ts:1)
|
||||
Acceptance:
|
||||
- 1–2 GB m4b exports download in Docker without UI reset; bookId persists until deletion; status 206 Range works.
|
||||
|
||||
Issue #47 — Kokoro combined voices (“+” syntax)
|
||||
Type: Feature
|
||||
Intent:
|
||||
- Support voice strings like “bf_emma+af_heart” when provider is Kokoro/FastAPI or DeepInfra Kokoro.
|
||||
Ideas for implementation:
|
||||
- Allow free-form voice string entry and pass-through if not in known set.
|
||||
- Validate known providers; if “+” present, skip voice validation list.
|
||||
|
||||
Logging:
|
||||
- Emit provider, model, raw voice string in request (no PII).
|
||||
Tests:
|
||||
- “a+b” voices produce audible combined output; works via DeepInfra pass-through and Kokoro-FastAPI; voice dropdown offers “Custom voice…” input.
|
||||
|
||||
Issue #44 — Dialog not being chunked together
|
||||
Type: Bug (NLP splitting)
|
||||
Observations:
|
||||
- Dialog lines split mid-quote; needs grouping.
|
||||
Proposed improvements:
|
||||
- Extend [splitIntoSentences()](src/utils/nlp.ts:34) to apply quote-aware grouping.
|
||||
- When a sentence begins with an opening quote and the next ends with a closing quote, join them before MAX_BLOCK_LENGTH checks.
|
||||
Ideas for remediation:
|
||||
- Provide composable splitter strategy wrapping existing utility.
|
||||
- Add “dialog-preserve” flag toggled in settings.
|
||||
Logging to add:
|
||||
- Count of quote-joined sentences; per-page example of before/after lengths.
|
||||
Acceptance:
|
||||
- Quoted dialog flows as single units unless exceeding MAX_BLOCK_LENGTH by > X%; regression-safe for non-dialog text.
|
||||
|
||||
Issue #40 — PDF left/right extraction margins not working
|
||||
Type: Bug (PDF extraction)
|
||||
Observations:
|
||||
- Current filter uses transform[4]/[5] with width heuristics. Edge cases: missing width, skew, page scale differences.
|
||||
Likely causes:
|
||||
- Some pdf.js TextItem miss width; horizontal margins computed but not applied due to scale mismatch.
|
||||
- Defaults in [src/contexts/ConfigContext.tsx](src/contexts/ConfigContext.tsx:216) set left/right to '0.0' on first run; UX confusion.
|
||||
Remediation:
|
||||
- Compute glyph bbox width when width absent using transform matrix.
|
||||
- Normalize x to [0..1] by dividing by pageWidth; compare to margins reliably.
|
||||
- Add visual debug overlay (dev mode) to draw margin boxes while extracting.
|
||||
Code touchpoints:
|
||||
- [src/utils/pdf.ts](src/utils/pdf.ts:60) extractTextFromPDF(): margin math and width fallback.
|
||||
Diagnostics to add:
|
||||
- Per-page: kept/filtered counts, extremes of x positions, computed margins.
|
||||
Acceptance:
|
||||
- Test PDFs show left/right trimming correctly; e2e highlight still robust.
|
||||
|
|
@ -1,13 +1,5 @@
|
|||
import { defineConfig, devices } from '@playwright/test';
|
||||
|
||||
/**
|
||||
* Read environment variables from file.
|
||||
* https://github.com/motdotla/dotenv
|
||||
*/
|
||||
// import dotenv from 'dotenv';
|
||||
// import path from 'path';
|
||||
// dotenv.config({ path: path.resolve(__dirname, '.env') });
|
||||
|
||||
/**
|
||||
* See https://playwright.dev/docs/test-configuration.
|
||||
*/
|
||||
|
|
@ -15,14 +7,11 @@ export default defineConfig({
|
|||
testDir: './tests',
|
||||
timeout: 30 * 1000,
|
||||
outputDir: './tests/results',
|
||||
/* Run tests in files in parallel */
|
||||
fullyParallel: true,
|
||||
fullyParallel: false,
|
||||
/* Fail the build on CI if you accidentally left test.only in the source code. */
|
||||
forbidOnly: !!process.env.CI,
|
||||
/* Retry on CI only */
|
||||
retries: process.env.CI ? 2 : 0,
|
||||
/* Opt out of parallel tests on CI. */
|
||||
workers: undefined,
|
||||
workers: process.env.CI ? '100%' : '75%',
|
||||
/* Reporter to use. See https://playwright.dev/docs/test-reporters */
|
||||
reporter: 'html',
|
||||
/* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
|
||||
|
|
|
|||
|
|
@ -217,7 +217,7 @@ export async function POST(req: NextRequest) {
|
|||
console.log('TTS in-flight JOIN for key:', cacheKey.slice(0, 8));
|
||||
existing.consumers += 1;
|
||||
|
||||
const onAbort = (_evt: Event) => {
|
||||
const onAbort = () => {
|
||||
existing.consumers = Math.max(0, existing.consumers - 1);
|
||||
if (existing.consumers === 0) {
|
||||
existing.controller.abort();
|
||||
|
|
@ -262,7 +262,7 @@ export async function POST(req: NextRequest) {
|
|||
|
||||
inflightRequests.set(cacheKey, entry);
|
||||
|
||||
const onAbort = (_evt: Event) => {
|
||||
const onAbort = () => {
|
||||
entry.consumers = Math.max(0, entry.consumers - 1);
|
||||
if (entry.consumers === 0) {
|
||||
entry.controller.abort();
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ export function DocumentSettings({ isOpen, setIsOpen, epub, html }: {
|
|||
viewType,
|
||||
skipBlank,
|
||||
epubTheme,
|
||||
smartSentenceSplitting,
|
||||
headerMargin,
|
||||
footerMargin,
|
||||
leftMargin,
|
||||
|
|
@ -308,6 +309,24 @@ export function DocumentSettings({ isOpen, setIsOpen, epub, html }: {
|
|||
Automatically skip pages with no text content
|
||||
</p>
|
||||
</div>}
|
||||
{!html && (
|
||||
<div className="space-y-1">
|
||||
<label className="flex items-center space-x-2">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={smartSentenceSplitting}
|
||||
onChange={(e) => updateConfigKey('smartSentenceSplitting', e.target.checked)}
|
||||
className="form-checkbox h-4 w-4 text-accent rounded border-muted"
|
||||
/>
|
||||
<span className="text-sm font-medium text-foreground">
|
||||
Smart sentence splitting
|
||||
</span>
|
||||
</label>
|
||||
<p className="text-sm text-muted pl-6">
|
||||
Merge sentences across page or section breaks for smoother TTS.
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
{epub && (
|
||||
<div className="space-y-1">
|
||||
<label className="flex items-center space-x-2">
|
||||
|
|
|
|||
|
|
@ -239,4 +239,4 @@ export function PDFViewer({ zoomLevel }: PDFViewerProps) {
|
|||
</Document>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -29,6 +29,7 @@ type ConfigValues = {
|
|||
ttsModel: string;
|
||||
ttsInstructions: string;
|
||||
savedVoices: SavedVoices;
|
||||
smartSentenceSplitting: boolean;
|
||||
};
|
||||
|
||||
/** Interface defining the configuration context shape and functionality */
|
||||
|
|
@ -41,6 +42,7 @@ interface ConfigContextType {
|
|||
voice: string;
|
||||
skipBlank: boolean;
|
||||
epubTheme: boolean;
|
||||
smartSentenceSplitting: boolean;
|
||||
headerMargin: number;
|
||||
footerMargin: number;
|
||||
leftMargin: number;
|
||||
|
|
@ -73,6 +75,7 @@ export function ConfigProvider({ children }: { children: ReactNode }) {
|
|||
const [voice, setVoice] = useState<string>('af_sarah');
|
||||
const [skipBlank, setSkipBlank] = useState<boolean>(true);
|
||||
const [epubTheme, setEpubTheme] = useState<boolean>(false);
|
||||
const [smartSentenceSplitting, setSmartSentenceSplitting] = useState<boolean>(true);
|
||||
const [headerMargin, setHeaderMargin] = useState<number>(0.07);
|
||||
const [footerMargin, setFooterMargin] = useState<number>(0.07);
|
||||
const [leftMargin, setLeftMargin] = useState<number>(0.07);
|
||||
|
|
@ -103,6 +106,7 @@ export function ConfigProvider({ children }: { children: ReactNode }) {
|
|||
const cachedAudioPlayerSpeed = await getItem('audioPlayerSpeed');
|
||||
const cachedSkipBlank = await getItem('skipBlank');
|
||||
const cachedEpubTheme = await getItem('epubTheme');
|
||||
const cachedSmartSentenceSplitting = await getItem('smartSentenceSplitting');
|
||||
const cachedHeaderMargin = await getItem('headerMargin');
|
||||
const cachedFooterMargin = await getItem('footerMargin');
|
||||
const cachedLeftMargin = await getItem('leftMargin');
|
||||
|
|
@ -187,6 +191,7 @@ export function ConfigProvider({ children }: { children: ReactNode }) {
|
|||
setAudioPlayerSpeed(parseFloat(cachedAudioPlayerSpeed || '1'));
|
||||
setSkipBlank(cachedSkipBlank === 'false' ? false : true);
|
||||
setEpubTheme(cachedEpubTheme === 'true');
|
||||
setSmartSentenceSplitting(cachedSmartSentenceSplitting === 'false' ? false : true);
|
||||
setHeaderMargin(parseFloat(cachedHeaderMargin || '0.07'));
|
||||
setFooterMargin(parseFloat(cachedFooterMargin || '0.07'));
|
||||
setLeftMargin(parseFloat(cachedLeftMargin || '0.07'));
|
||||
|
|
@ -211,6 +216,9 @@ export function ConfigProvider({ children }: { children: ReactNode }) {
|
|||
if (cachedEpubTheme === null) {
|
||||
await setItem('epubTheme', 'false');
|
||||
}
|
||||
if (cachedSmartSentenceSplitting === null) {
|
||||
await setItem('smartSentenceSplitting', 'true');
|
||||
}
|
||||
if (cachedHeaderMargin === null) await setItem('headerMargin', '0.07');
|
||||
if (cachedFooterMargin === null) await setItem('footerMargin', '0.07');
|
||||
if (cachedLeftMargin === null) await setItem('leftMargin', '0.0');
|
||||
|
|
@ -353,6 +361,9 @@ export function ConfigProvider({ children }: { children: ReactNode }) {
|
|||
case 'epubTheme':
|
||||
setEpubTheme(value as boolean);
|
||||
break;
|
||||
case 'smartSentenceSplitting':
|
||||
setSmartSentenceSplitting(value as boolean);
|
||||
break;
|
||||
case 'headerMargin':
|
||||
setHeaderMargin(value as number);
|
||||
break;
|
||||
|
|
@ -388,6 +399,7 @@ export function ConfigProvider({ children }: { children: ReactNode }) {
|
|||
voice,
|
||||
skipBlank,
|
||||
epubTheme,
|
||||
smartSentenceSplitting,
|
||||
headerMargin,
|
||||
footerMargin,
|
||||
leftMargin,
|
||||
|
|
@ -417,4 +429,4 @@ export function useConfig() {
|
|||
throw new Error('useConfig must be used within a ConfigProvider');
|
||||
}
|
||||
return context;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ import {
|
|||
useCallback,
|
||||
useMemo,
|
||||
useRef,
|
||||
RefObject,
|
||||
RefObject
|
||||
} from 'react';
|
||||
import { indexedDBService } from '@/utils/indexedDB';
|
||||
import { useTTS } from '@/contexts/TTSContext';
|
||||
|
|
@ -43,6 +43,81 @@ interface EPUBContextType {
|
|||
|
||||
const EPUBContext = createContext<EPUBContextType | undefined>(undefined);
|
||||
|
||||
const EPUB_CONTINUATION_CHARS = 600;
|
||||
|
||||
const stepToNextNode = (node: Node | null, root: Node): Node | null => {
|
||||
if (!node) return null;
|
||||
if (node.firstChild) {
|
||||
return node.firstChild;
|
||||
}
|
||||
|
||||
let current: Node | null = node;
|
||||
while (current) {
|
||||
if (current === root) {
|
||||
return null;
|
||||
}
|
||||
if (current.nextSibling) {
|
||||
return current.nextSibling;
|
||||
}
|
||||
current = current.parentNode;
|
||||
}
|
||||
|
||||
return null;
|
||||
};
|
||||
|
||||
const getNextTextNode = (node: Node | null, root: Node): Text | null => {
|
||||
let next = stepToNextNode(node, root);
|
||||
while (next) {
|
||||
if (next.nodeType === Node.TEXT_NODE) {
|
||||
return next as Text;
|
||||
}
|
||||
next = stepToNextNode(next, root);
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
const collectContinuationFromRange = (range: Range | null | undefined, limit = EPUB_CONTINUATION_CHARS): string => {
|
||||
if (typeof window === 'undefined' || !range) {
|
||||
return '';
|
||||
}
|
||||
|
||||
const root = range.commonAncestorContainer;
|
||||
if (!root) {
|
||||
return '';
|
||||
}
|
||||
|
||||
const parts: string[] = [];
|
||||
let remaining = limit;
|
||||
|
||||
const appendFromTextNode = (textNode: Text, offset: number) => {
|
||||
if (remaining <= 0) return;
|
||||
const textContent = textNode.textContent || '';
|
||||
if (offset >= textContent.length) return;
|
||||
const slice = textContent.slice(offset, offset + remaining);
|
||||
if (slice) {
|
||||
parts.push(slice);
|
||||
remaining -= slice.length;
|
||||
}
|
||||
};
|
||||
|
||||
if (range.endContainer.nodeType === Node.TEXT_NODE) {
|
||||
appendFromTextNode(range.endContainer as Text, range.endOffset);
|
||||
let nextNode = getNextTextNode(range.endContainer, root);
|
||||
while (nextNode && remaining > 0) {
|
||||
appendFromTextNode(nextNode, 0);
|
||||
nextNode = getNextTextNode(nextNode, root);
|
||||
}
|
||||
} else {
|
||||
let nextNode = getNextTextNode(range.endContainer, root);
|
||||
while (nextNode && remaining > 0) {
|
||||
appendFromTextNode(nextNode, 0);
|
||||
nextNode = getNextTextNode(nextNode, root);
|
||||
}
|
||||
}
|
||||
|
||||
return parts.join(' ').replace(/\s+/g, ' ').trim();
|
||||
};
|
||||
|
||||
/**
|
||||
* Provider component for EPUB functionality
|
||||
* Manages the state and operations for EPUB document handling
|
||||
|
|
@ -61,6 +136,7 @@ export function EPUBProvider({ children }: { children: ReactNode }) {
|
|||
ttsProvider,
|
||||
ttsModel,
|
||||
ttsInstructions,
|
||||
smartSentenceSplitting,
|
||||
} = useConfig();
|
||||
// Current document state
|
||||
const [currDocData, setCurrDocData] = useState<ArrayBuffer>();
|
||||
|
|
@ -141,8 +217,23 @@ export function EPUBProvider({ children }: { children: ReactNode }) {
|
|||
return '';
|
||||
}
|
||||
const textContent = range.toString().trim();
|
||||
const continuationPreview = collectContinuationFromRange(range);
|
||||
|
||||
setTTSText(textContent, shouldPause);
|
||||
if (smartSentenceSplitting) {
|
||||
setTTSText(textContent, {
|
||||
shouldPause,
|
||||
location: start.cfi,
|
||||
nextLocation: end.cfi,
|
||||
nextText: continuationPreview
|
||||
});
|
||||
} else {
|
||||
// When smart splitting is disabled, behave like the original implementation:
|
||||
// send only the current page/location text without any continuation preview.
|
||||
setTTSText(textContent, {
|
||||
shouldPause,
|
||||
location: start.cfi,
|
||||
});
|
||||
}
|
||||
setCurrDocText(textContent);
|
||||
|
||||
return textContent;
|
||||
|
|
@ -150,7 +241,7 @@ export function EPUBProvider({ children }: { children: ReactNode }) {
|
|||
console.error('Error extracting EPUB text:', error);
|
||||
return '';
|
||||
}
|
||||
}, [setTTSText]);
|
||||
}, [setTTSText, smartSentenceSplitting]);
|
||||
|
||||
/**
|
||||
* Extracts text content from the entire EPUB book
|
||||
|
|
@ -657,4 +748,4 @@ export function useEPUB() {
|
|||
throw new Error('useEPUB must be used within an EPUBProvider');
|
||||
}
|
||||
return context;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -22,6 +22,7 @@ import {
|
|||
useCallback,
|
||||
useMemo,
|
||||
RefObject,
|
||||
useRef,
|
||||
} from 'react';
|
||||
|
||||
import { indexedDBService } from '@/utils/indexedDB';
|
||||
|
|
@ -33,6 +34,7 @@ import {
|
|||
clearHighlights,
|
||||
handleTextClick,
|
||||
} from '@/utils/pdf';
|
||||
import { processTextToSentences } from '@/utils/nlp';
|
||||
|
||||
import type { PDFDocumentProxy } from 'pdfjs-dist';
|
||||
import { withRetry } from '@/utils/audio';
|
||||
|
|
@ -70,6 +72,8 @@ interface PDFContextType {
|
|||
// Create the context
|
||||
const PDFContext = createContext<PDFContextType | undefined>(undefined);
|
||||
|
||||
const CONTINUATION_PREVIEW_CHARS = 600;
|
||||
|
||||
/**
|
||||
* PDFProvider Component
|
||||
*
|
||||
|
|
@ -83,10 +87,11 @@ export function PDFProvider({ children }: { children: ReactNode }) {
|
|||
const {
|
||||
setText: setTTSText,
|
||||
stop,
|
||||
currDocPageNumber: currDocPage,
|
||||
currDocPageNumber,
|
||||
currDocPages,
|
||||
setCurrDocPages,
|
||||
setIsEPUB
|
||||
setIsEPUB,
|
||||
registerVisualPageChangeHandler,
|
||||
} = useTTS();
|
||||
const {
|
||||
headerMargin,
|
||||
|
|
@ -100,6 +105,7 @@ export function PDFProvider({ children }: { children: ReactNode }) {
|
|||
ttsProvider,
|
||||
ttsModel,
|
||||
ttsInstructions,
|
||||
smartSentenceSplitting,
|
||||
} = useConfig();
|
||||
|
||||
// Current document state
|
||||
|
|
@ -108,6 +114,12 @@ export function PDFProvider({ children }: { children: ReactNode }) {
|
|||
const [currDocText, setCurrDocText] = useState<string>();
|
||||
const [pdfDocument, setPdfDocument] = useState<PDFDocumentProxy>();
|
||||
const [isAudioCombining] = useState(false);
|
||||
const pageTextCacheRef = useRef<Map<number, string>>(new Map());
|
||||
const [currDocPage, setCurrDocPage] = useState<number>(currDocPageNumber);
|
||||
|
||||
useEffect(() => {
|
||||
setCurrDocPage(currDocPageNumber);
|
||||
}, [currDocPageNumber]);
|
||||
|
||||
/**
|
||||
* Handles successful PDF document load
|
||||
|
|
@ -129,22 +141,60 @@ export function PDFProvider({ children }: { children: ReactNode }) {
|
|||
const loadCurrDocText = useCallback(async () => {
|
||||
try {
|
||||
if (!pdfDocument) return;
|
||||
const text = await extractTextFromPDF(pdfDocument, currDocPage, {
|
||||
|
||||
const margins = {
|
||||
header: headerMargin,
|
||||
footer: footerMargin,
|
||||
left: leftMargin,
|
||||
right: rightMargin
|
||||
});
|
||||
// Only update TTS text if the content has actually changed
|
||||
// This prevents unnecessary resets of the sentence index
|
||||
};
|
||||
|
||||
const getPageText = async (pageNumber: number, shouldCache = false): Promise<string> => {
|
||||
if (pageTextCacheRef.current.has(pageNumber)) {
|
||||
const cached = pageTextCacheRef.current.get(pageNumber)!;
|
||||
if (!shouldCache) {
|
||||
pageTextCacheRef.current.delete(pageNumber);
|
||||
}
|
||||
return cached;
|
||||
}
|
||||
|
||||
const extracted = await extractTextFromPDF(pdfDocument, pageNumber, margins);
|
||||
if (shouldCache) {
|
||||
pageTextCacheRef.current.set(pageNumber, extracted);
|
||||
}
|
||||
return extracted;
|
||||
};
|
||||
|
||||
const totalPages = currDocPages ?? pdfDocument.numPages;
|
||||
const nextPageNumber = currDocPageNumber < totalPages ? currDocPageNumber + 1 : undefined;
|
||||
|
||||
const [text, nextText] = await Promise.all([
|
||||
getPageText(currDocPageNumber),
|
||||
nextPageNumber ? getPageText(nextPageNumber, true) : Promise.resolve<string | undefined>(undefined),
|
||||
]);
|
||||
|
||||
if (text !== currDocText || text === '') {
|
||||
setCurrDocText(text);
|
||||
setTTSText(text);
|
||||
setTTSText(text, {
|
||||
location: currDocPageNumber,
|
||||
nextLocation: nextPageNumber,
|
||||
nextText: nextText?.slice(0, CONTINUATION_PREVIEW_CHARS),
|
||||
});
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Error loading PDF text:', error);
|
||||
}
|
||||
}, [pdfDocument, currDocPage, setTTSText, currDocText, headerMargin, footerMargin, leftMargin, rightMargin]);
|
||||
}, [
|
||||
pdfDocument,
|
||||
currDocPageNumber,
|
||||
currDocPages,
|
||||
setTTSText,
|
||||
currDocText,
|
||||
headerMargin,
|
||||
footerMargin,
|
||||
leftMargin,
|
||||
rightMargin,
|
||||
]);
|
||||
|
||||
/**
|
||||
* Effect hook to update document text when the page changes
|
||||
|
|
@ -154,7 +204,7 @@ export function PDFProvider({ children }: { children: ReactNode }) {
|
|||
if (currDocData) {
|
||||
loadCurrDocText();
|
||||
}
|
||||
}, [currDocPage, currDocData, loadCurrDocText]);
|
||||
}, [currDocPageNumber, currDocData, loadCurrDocText]);
|
||||
|
||||
/**
|
||||
* Sets the current document based on its ID
|
||||
|
|
@ -185,6 +235,7 @@ export function PDFProvider({ children }: { children: ReactNode }) {
|
|||
setCurrDocText(undefined);
|
||||
setCurrDocPages(undefined);
|
||||
setPdfDocument(undefined);
|
||||
pageTextCacheRef.current.clear();
|
||||
stop();
|
||||
}, [setCurrDocPages, stop]);
|
||||
|
||||
|
|
@ -212,16 +263,20 @@ export function PDFProvider({ children }: { children: ReactNode }) {
|
|||
let totalLength = 0;
|
||||
|
||||
for (let pageNum = 1; pageNum <= pdfDocument.numPages; pageNum++) {
|
||||
const text = await extractTextFromPDF(pdfDocument, pageNum, {
|
||||
const rawText = await extractTextFromPDF(pdfDocument, pageNum, {
|
||||
header: headerMargin,
|
||||
footer: footerMargin,
|
||||
left: leftMargin,
|
||||
right: rightMargin
|
||||
});
|
||||
const trimmedText = text.trim();
|
||||
const trimmedText = rawText.trim();
|
||||
if (trimmedText) {
|
||||
textPerPage.push(trimmedText);
|
||||
totalLength += trimmedText.length;
|
||||
const processedText = smartSentenceSplitting
|
||||
? processTextToSentences(trimmedText).join(' ')
|
||||
: trimmedText;
|
||||
|
||||
textPerPage.push(processedText);
|
||||
totalLength += processedText.length;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -405,7 +460,7 @@ export function PDFProvider({ children }: { children: ReactNode }) {
|
|||
console.error('Error creating audiobook:', error);
|
||||
throw error;
|
||||
}
|
||||
}, [pdfDocument, headerMargin, footerMargin, leftMargin, rightMargin, apiKey, baseUrl, voice, voiceSpeed, ttsProvider, ttsModel, ttsInstructions]);
|
||||
}, [pdfDocument, headerMargin, footerMargin, leftMargin, rightMargin, apiKey, baseUrl, voice, voiceSpeed, ttsProvider, ttsModel, ttsInstructions, smartSentenceSplitting]);
|
||||
|
||||
/**
|
||||
* Regenerates a specific chapter (page) of the PDF audiobook
|
||||
|
|
@ -444,18 +499,22 @@ export function PDFProvider({ children }: { children: ReactNode }) {
|
|||
const pageNum = nonEmptyPages[chapterIndex];
|
||||
|
||||
// Extract text from the mapped page
|
||||
const text = await extractTextFromPDF(pdfDocument, pageNum, {
|
||||
const rawText = await extractTextFromPDF(pdfDocument, pageNum, {
|
||||
header: headerMargin,
|
||||
footer: footerMargin,
|
||||
left: leftMargin,
|
||||
right: rightMargin
|
||||
});
|
||||
|
||||
const trimmedText = text.trim();
|
||||
const trimmedText = rawText.trim();
|
||||
if (!trimmedText) {
|
||||
throw new Error('No text content found on page');
|
||||
}
|
||||
|
||||
const textForTTS = smartSentenceSplitting
|
||||
? processTextToSentences(trimmedText).join(' ')
|
||||
: trimmedText;
|
||||
|
||||
// Use logical chapter numbering (index + 1) to match original generation titles
|
||||
const chapterTitle = `Page ${chapterIndex + 1}`;
|
||||
|
||||
|
|
@ -475,7 +534,7 @@ export function PDFProvider({ children }: { children: ReactNode }) {
|
|||
'x-tts-provider': ttsProvider,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
text: trimmedText,
|
||||
text: textForTTS,
|
||||
voice: voice || (ttsProvider === 'openai' ? 'alloy' : (ttsProvider === 'deepinfra' ? 'af_bella' : 'af_sarah')),
|
||||
speed: voiceSpeed,
|
||||
format: 'mp3',
|
||||
|
|
@ -549,7 +608,7 @@ export function PDFProvider({ children }: { children: ReactNode }) {
|
|||
console.error('Error regenerating page:', error);
|
||||
throw error;
|
||||
}
|
||||
}, [pdfDocument, headerMargin, footerMargin, leftMargin, rightMargin, apiKey, baseUrl, voice, voiceSpeed, ttsProvider, ttsModel, ttsInstructions]);
|
||||
}, [pdfDocument, headerMargin, footerMargin, leftMargin, rightMargin, apiKey, baseUrl, voice, voiceSpeed, ttsProvider, ttsModel, ttsInstructions, smartSentenceSplitting]);
|
||||
|
||||
/**
|
||||
* Effect hook to initialize TTS as non-EPUB mode
|
||||
|
|
@ -558,6 +617,16 @@ export function PDFProvider({ children }: { children: ReactNode }) {
|
|||
setIsEPUB(false);
|
||||
}, [setIsEPUB]);
|
||||
|
||||
useEffect(() => {
|
||||
registerVisualPageChangeHandler(location => {
|
||||
if (typeof location !== 'number') return;
|
||||
if (!pdfDocument) return;
|
||||
const totalPages = currDocPages ?? pdfDocument.numPages;
|
||||
const clamped = Math.min(Math.max(location, 1), totalPages);
|
||||
setCurrDocPage(clamped);
|
||||
});
|
||||
}, [registerVisualPageChangeHandler, currDocPages, pdfDocument]);
|
||||
|
||||
// Context value memoization
|
||||
const contextValue = useMemo(
|
||||
() => ({
|
||||
|
|
@ -613,4 +682,4 @@ export function usePDF() {
|
|||
throw new Error('usePDF must be used within a PDFProvider');
|
||||
}
|
||||
return context;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ import { useAudioContext } from '@/hooks/audio/useAudioContext';
|
|||
import { getLastDocumentLocation, setLastDocumentLocation } from '@/utils/indexedDB';
|
||||
import { useBackgroundState } from '@/hooks/audio/useBackgroundState';
|
||||
import { withRetry } from '@/utils/audio';
|
||||
import { processTextToSentences } from '@/utils/nlp';
|
||||
import { preprocessSentenceForAudio, processTextToSentences } from '@/utils/nlp';
|
||||
import { isKokoroModel } from '@/utils/voice';
|
||||
|
||||
// Media globals
|
||||
|
|
@ -72,16 +72,185 @@ interface TTSContextType {
|
|||
pause: () => void;
|
||||
stop: () => void;
|
||||
stopAndPlayFromIndex: (index: number) => void;
|
||||
setText: (text: string, shouldPause?: boolean) => void;
|
||||
setText: (text: string, options?: boolean | SetTextOptions) => void;
|
||||
setCurrDocPages: (num: number | undefined) => void;
|
||||
setSpeedAndRestart: (speed: number) => void;
|
||||
setAudioPlayerSpeedAndRestart: (speed: number) => void;
|
||||
setVoiceAndRestart: (voice: string) => void;
|
||||
skipToLocation: (location: string | number, shouldPause?: boolean) => void;
|
||||
registerLocationChangeHandler: (handler: (location: string | number) => void) => void; // EPUB-only: Handles chapter navigation
|
||||
registerVisualPageChangeHandler: (handler: (location: string | number) => void) => void;
|
||||
setIsEPUB: (isEPUB: boolean) => void;
|
||||
}
|
||||
|
||||
interface SetTextOptions {
|
||||
shouldPause?: boolean;
|
||||
location?: string | number;
|
||||
nextLocation?: string | number;
|
||||
nextText?: string;
|
||||
}
|
||||
|
||||
interface ContinuationMergeResult {
|
||||
text: string;
|
||||
carried: string;
|
||||
}
|
||||
|
||||
interface PageTurnEstimate {
|
||||
location: string | number;
|
||||
sentenceIndex: number;
|
||||
fraction: number;
|
||||
}
|
||||
|
||||
const CONTINUATION_LOOKAHEAD = 600;
|
||||
const SENTENCE_ENDING = /[.?!…]["'”’)\]]*\s*$/;
|
||||
|
||||
const normalizeLocationKey = (location: string | number) =>
|
||||
typeof location === 'number' ? `num:${location}` : `str:${location}`;
|
||||
|
||||
const isWhitespaceChar = (char: string) => /\s/.test(char);
|
||||
|
||||
const skipWhitespace = (source: string, start: number) => {
|
||||
let index = start;
|
||||
while (index < source.length && isWhitespaceChar(source[index])) {
|
||||
index++;
|
||||
}
|
||||
return index;
|
||||
};
|
||||
|
||||
const matchNormalizedPrefixLength = (text: string, prefix: string): number | null => {
|
||||
let textIndex = 0;
|
||||
let prefixIndex = 0;
|
||||
|
||||
while (prefixIndex < prefix.length) {
|
||||
const prefixChar = prefix[prefixIndex];
|
||||
|
||||
if (isWhitespaceChar(prefixChar)) {
|
||||
prefixIndex = skipWhitespace(prefix, prefixIndex);
|
||||
textIndex = skipWhitespace(text, textIndex);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (textIndex >= text.length) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const textChar = text[textIndex];
|
||||
|
||||
if (textChar === prefixChar || textChar.toLowerCase() === prefixChar.toLowerCase()) {
|
||||
textIndex++;
|
||||
prefixIndex++;
|
||||
continue;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
return textIndex;
|
||||
};
|
||||
|
||||
const needsSentenceContinuation = (text: string) => {
|
||||
const trimmed = text.trim();
|
||||
if (!trimmed) {
|
||||
return false;
|
||||
}
|
||||
return !SENTENCE_ENDING.test(trimmed);
|
||||
};
|
||||
|
||||
const stripContinuationPrefix = (text: string, prefix: string) => {
|
||||
if (!prefix) return { text, removed: false };
|
||||
|
||||
// Try literal match first since PDF text is normalized already
|
||||
if (text.startsWith(prefix)) {
|
||||
return {
|
||||
text: text.slice(prefix.length).trimStart(),
|
||||
removed: true,
|
||||
};
|
||||
}
|
||||
|
||||
const trimmedPrefix = prefix.trimStart();
|
||||
const trimmedText = text.trimStart();
|
||||
|
||||
if (trimmedText.startsWith(trimmedPrefix)) {
|
||||
const offset = text.length - trimmedText.length;
|
||||
return {
|
||||
text: text.slice(offset + trimmedPrefix.length).trimStart(),
|
||||
removed: true,
|
||||
};
|
||||
}
|
||||
|
||||
const matchedLength = matchNormalizedPrefixLength(text, prefix);
|
||||
if (matchedLength !== null) {
|
||||
return {
|
||||
text: text.slice(matchedLength).trimStart(),
|
||||
removed: true,
|
||||
};
|
||||
}
|
||||
|
||||
return { text, removed: false };
|
||||
};
|
||||
|
||||
const extractContinuationSlice = (nextText: string): ContinuationMergeResult | null => {
|
||||
if (!nextText?.trim()) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const snippet = nextText.trim().slice(0, CONTINUATION_LOOKAHEAD);
|
||||
let boundaryIndex = -1;
|
||||
|
||||
for (let i = 0; i < snippet.length; i++) {
|
||||
const char = snippet[i];
|
||||
if (/[.?!…]/.test(char)) {
|
||||
let j = i + 1;
|
||||
while (j < snippet.length && /["'”’)\]]/.test(snippet[j])) {
|
||||
j++;
|
||||
}
|
||||
while (j < snippet.length && /\s/.test(snippet[j])) {
|
||||
j++;
|
||||
}
|
||||
boundaryIndex = j;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (boundaryIndex === -1) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const rawSlice = snippet.slice(0, boundaryIndex);
|
||||
const addition = rawSlice.trim();
|
||||
|
||||
if (!addition) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return {
|
||||
text: addition,
|
||||
carried: rawSlice,
|
||||
};
|
||||
};
|
||||
|
||||
const mergeContinuation = (text: string, nextText: string): ContinuationMergeResult | null => {
|
||||
if (!needsSentenceContinuation(text)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const slice = extractContinuationSlice(nextText);
|
||||
if (!slice) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const trimmed = text.trimEnd();
|
||||
const endsWithHyphen = trimmed.endsWith('-');
|
||||
const base = endsWithHyphen ? trimmed.slice(0, -1) : trimmed;
|
||||
const joiner = endsWithHyphen ? '' : (base ? ' ' : '');
|
||||
const mergedText = `${base}${joiner}${slice.text}`.trim();
|
||||
|
||||
return {
|
||||
text: mergedText,
|
||||
carried: slice.carried,
|
||||
};
|
||||
};
|
||||
|
||||
// Create the context
|
||||
const TTSContext = createContext<TTSContextType | undefined>(undefined);
|
||||
|
||||
|
|
@ -107,6 +276,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
ttsInstructions: configTTSInstructions,
|
||||
updateConfigKey,
|
||||
skipBlank,
|
||||
smartSentenceSplitting,
|
||||
} = useConfig();
|
||||
|
||||
// Remove OpenAI client reference as it's no longer needed
|
||||
|
|
@ -116,6 +286,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
|
||||
// Add ref for location change handler
|
||||
const locationChangeHandlerRef = useRef<((location: string | number) => void) | null>(null);
|
||||
const visualPageChangeHandlerRef = useRef<((location: string | number) => void) | null>(null);
|
||||
|
||||
/**
|
||||
* Registers a handler function for location changes in EPUB documents
|
||||
|
|
@ -127,6 +298,10 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
locationChangeHandlerRef.current = handler;
|
||||
}, []);
|
||||
|
||||
const registerVisualPageChangeHandler = useCallback((handler: (location: string | number) => void) => {
|
||||
visualPageChangeHandlerRef.current = handler;
|
||||
}, []);
|
||||
|
||||
// Get document ID from URL params
|
||||
const { id } = useParams();
|
||||
|
||||
|
|
@ -158,6 +333,11 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
const [pendingRestoreIndex, setPendingRestoreIndex] = useState<number | null>(null);
|
||||
// Guard to coalesce rapid restarts and only resume the latest change
|
||||
const restartSeqRef = useRef(0);
|
||||
// Track continuation slices for PDF/EPUB page transitions
|
||||
const continuationCarryRef = useRef<Map<string, string>>(new Map());
|
||||
const epubContinuationRef = useRef<string | null>(null);
|
||||
const pageTurnEstimateRef = useRef<PageTurnEstimate | null>(null);
|
||||
const pageTurnTimeoutRef = useRef<ReturnType<typeof setTimeout> | null>(null);
|
||||
|
||||
/**
|
||||
* Processes text into sentences using the shared NLP utility
|
||||
|
|
@ -181,20 +361,22 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
const abortAudio = useCallback((clearPending = false) => {
|
||||
if (activeHowl) {
|
||||
activeHowl.stop();
|
||||
activeHowl.unload(); // Ensure Howl instance is fully cleaned up
|
||||
activeHowl.unload();
|
||||
setActiveHowl(null);
|
||||
}
|
||||
|
||||
if (clearPending) {
|
||||
// Abort all active TTS requests
|
||||
console.log('Aborting active TTS requests');
|
||||
activeAbortControllers.current.forEach(controller => {
|
||||
controller.abort();
|
||||
});
|
||||
activeAbortControllers.current.clear();
|
||||
// Clear any pending preload requests
|
||||
preloadRequests.current.clear();
|
||||
}
|
||||
|
||||
if (pageTurnTimeoutRef.current) {
|
||||
clearTimeout(pageTurnTimeoutRef.current);
|
||||
pageTurnTimeoutRef.current = null;
|
||||
}
|
||||
}, [activeHowl]);
|
||||
|
||||
/**
|
||||
|
|
@ -211,7 +393,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
* Works for both PDF pages and EPUB locations
|
||||
*
|
||||
* @param {string | number} location - The target location to navigate to
|
||||
* @param {boolean} keepPlaying - Whether to maintain playback state
|
||||
* @param {boolean} shouldPause - Whether to pause playback
|
||||
*/
|
||||
const skipToLocation = useCallback((location: string | number, shouldPause = false) => {
|
||||
// Reset state for new content in correct order
|
||||
|
|
@ -233,7 +415,6 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
|
||||
// Handle within current page bounds
|
||||
if (nextIndex < sentences.length && nextIndex >= 0) {
|
||||
console.log('isEPUB', isEPUB!);
|
||||
setCurrentIndex(nextIndex);
|
||||
return;
|
||||
}
|
||||
|
|
@ -297,9 +478,65 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
*
|
||||
* @param {string} text - The text to be processed
|
||||
*/
|
||||
const setText = useCallback((text: string, shouldPause = false) => {
|
||||
// Check for blank section first
|
||||
if (handleBlankSection(text)) return;
|
||||
const setText = useCallback((text: string, options?: boolean | SetTextOptions) => {
|
||||
const normalizedOptions: SetTextOptions = typeof options === 'boolean'
|
||||
? { shouldPause: options }
|
||||
: (options || {});
|
||||
|
||||
let workingText = text;
|
||||
|
||||
// Apply or clear sentence continuation logic based on config
|
||||
let continuationCarried: string | undefined;
|
||||
if (smartSentenceSplitting) {
|
||||
if (isEPUB && epubContinuationRef.current) {
|
||||
const { text: strippedText, removed } = stripContinuationPrefix(workingText, epubContinuationRef.current);
|
||||
workingText = strippedText;
|
||||
if (removed) {
|
||||
epubContinuationRef.current = null;
|
||||
}
|
||||
}
|
||||
|
||||
if (!isEPUB && normalizedOptions.location !== undefined) {
|
||||
const key = normalizeLocationKey(normalizedOptions.location);
|
||||
const carried = continuationCarryRef.current.get(key);
|
||||
if (carried) {
|
||||
const { text: strippedText, removed } = stripContinuationPrefix(workingText, carried);
|
||||
workingText = strippedText;
|
||||
if (removed) {
|
||||
continuationCarryRef.current.delete(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (normalizedOptions.nextText) {
|
||||
const merged = mergeContinuation(workingText, normalizedOptions.nextText);
|
||||
if (merged) {
|
||||
workingText = merged.text;
|
||||
continuationCarried = merged.carried;
|
||||
}
|
||||
}
|
||||
|
||||
if (continuationCarried) {
|
||||
if (isEPUB) {
|
||||
epubContinuationRef.current = continuationCarried;
|
||||
} else if (normalizedOptions.nextLocation !== undefined) {
|
||||
continuationCarryRef.current.set(
|
||||
normalizeLocationKey(normalizedOptions.nextLocation),
|
||||
continuationCarried
|
||||
);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// When disabled, clear any stale continuation state
|
||||
epubContinuationRef.current = null;
|
||||
continuationCarryRef.current.clear();
|
||||
pageTurnEstimateRef.current = null;
|
||||
}
|
||||
|
||||
// Check for blank section after adjustments
|
||||
if (handleBlankSection(workingText)) return;
|
||||
|
||||
const shouldPause = normalizedOptions.shouldPause ?? false;
|
||||
|
||||
// Keep track of previous state and pause playback
|
||||
const wasPlaying = isPlaying;
|
||||
|
|
@ -307,8 +544,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
abortAudio(true); // Clear pending requests since text is changing
|
||||
setIsProcessing(true); // Set processing state before text processing starts
|
||||
|
||||
console.log('Setting text:', text);
|
||||
processTextToSentencesLocal(text)
|
||||
processTextToSentencesLocal(workingText)
|
||||
.then(newSentences => {
|
||||
if (newSentences.length === 0) {
|
||||
console.warn('No sentences found in text');
|
||||
|
|
@ -318,7 +554,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
|
||||
// Set all state updates in a predictable order
|
||||
setSentences(newSentences);
|
||||
|
||||
|
||||
// Check if we have a pending restore index for PDF
|
||||
if (pendingRestoreIndex !== null && !isEPUB) {
|
||||
const restoreIndex = Math.min(pendingRestoreIndex, newSentences.length - 1);
|
||||
|
|
@ -328,7 +564,42 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
} else {
|
||||
setCurrentIndex(0);
|
||||
}
|
||||
|
||||
|
||||
// Compute auto page-turn estimate for PDFs when we have a continuation
|
||||
if (smartSentenceSplitting && !isEPUB && continuationCarried && normalizedOptions.nextLocation !== undefined) {
|
||||
const continuationNormalized = preprocessSentenceForAudio(continuationCarried);
|
||||
if (continuationNormalized) {
|
||||
let bestEstimate: PageTurnEstimate | null = null;
|
||||
|
||||
newSentences.forEach((sentence, index) => {
|
||||
const normalizedSentence = preprocessSentenceForAudio(sentence);
|
||||
if (!normalizedSentence) return;
|
||||
|
||||
if (!normalizedSentence.toLowerCase().endsWith(continuationNormalized.toLowerCase())) return;
|
||||
|
||||
const totalLength = normalizedSentence.length;
|
||||
const continuationLength = continuationNormalized.length;
|
||||
if (totalLength <= continuationLength) return;
|
||||
|
||||
const baseLength = totalLength - continuationLength;
|
||||
const fraction = baseLength / totalLength;
|
||||
if (fraction <= 0 || fraction >= 1) return;
|
||||
|
||||
bestEstimate = {
|
||||
location: normalizedOptions.nextLocation!,
|
||||
sentenceIndex: index,
|
||||
fraction,
|
||||
};
|
||||
});
|
||||
|
||||
pageTurnEstimateRef.current = bestEstimate;
|
||||
} else {
|
||||
pageTurnEstimateRef.current = null;
|
||||
}
|
||||
} else {
|
||||
pageTurnEstimateRef.current = null;
|
||||
}
|
||||
|
||||
setIsProcessing(false);
|
||||
|
||||
// Restore playback state if needed
|
||||
|
|
@ -347,7 +618,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
duration: 3000,
|
||||
});
|
||||
});
|
||||
}, [isPlaying, handleBlankSection, abortAudio, processTextToSentencesLocal, pendingRestoreIndex, isEPUB]);
|
||||
}, [isPlaying, handleBlankSection, abortAudio, processTextToSentencesLocal, pendingRestoreIndex, isEPUB, smartSentenceSplitting]);
|
||||
|
||||
/**
|
||||
* Toggles the playback state between playing and paused
|
||||
|
|
@ -547,7 +818,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
try {
|
||||
const audioBuffer = await getAudio(sentence);
|
||||
if (!audioBuffer) throw new Error('No audio data generated');
|
||||
|
||||
|
||||
// Convert to base64 data URI
|
||||
const bytes = new Uint8Array(audioBuffer);
|
||||
const binaryString = bytes.reduce((acc, byte) => acc + String.fromCharCode(byte), '');
|
||||
|
|
@ -576,7 +847,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
*
|
||||
* @param {string} sentence - The sentence to play
|
||||
*/
|
||||
const playSentenceWithHowl = useCallback(async (sentence: string) => {
|
||||
const playSentenceWithHowl = useCallback(async (sentence: string, sentenceIndex: number) => {
|
||||
if (!sentence) {
|
||||
console.log('No sentence to play');
|
||||
setIsProcessing(false);
|
||||
|
|
@ -606,13 +877,35 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
preload: true,
|
||||
pool: 5,
|
||||
rate: audioSpeed,
|
||||
onload: function (this: Howl) {
|
||||
const estimate = pageTurnEstimateRef.current;
|
||||
if (!estimate || estimate.sentenceIndex !== sentenceIndex) return;
|
||||
if (!visualPageChangeHandlerRef.current) return;
|
||||
|
||||
const duration = this.duration();
|
||||
if (!duration || !Number.isFinite(duration)) return;
|
||||
|
||||
const delayMs = duration * estimate.fraction * 1000;
|
||||
if (delayMs <= 0 || delayMs >= duration * 1000) return;
|
||||
|
||||
if (pageTurnTimeoutRef.current) {
|
||||
clearTimeout(pageTurnTimeoutRef.current);
|
||||
}
|
||||
|
||||
pageTurnTimeoutRef.current = setTimeout(() => {
|
||||
if (!isPlaying) return;
|
||||
const currentEstimate = pageTurnEstimateRef.current;
|
||||
if (!currentEstimate || currentEstimate.sentenceIndex !== sentenceIndex) return;
|
||||
visualPageChangeHandlerRef.current?.(currentEstimate.location);
|
||||
}, delayMs);
|
||||
},
|
||||
onplay: () => {
|
||||
setIsProcessing(false);
|
||||
if ('mediaSession' in navigator) {
|
||||
navigator.mediaSession.playbackState = 'playing';
|
||||
}
|
||||
},
|
||||
onplayerror: function(this: Howl, error) {
|
||||
onplayerror: function (this: Howl, error) {
|
||||
console.warn('Howl playback error:', error);
|
||||
// Try to recover by forcing HTML5 audio mode
|
||||
if (this.state() === 'loaded') {
|
||||
|
|
@ -623,17 +916,17 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
this.load();
|
||||
}
|
||||
},
|
||||
onloaderror: async function(this: Howl, error) {
|
||||
onloaderror: async function (this: Howl, error) {
|
||||
console.warn(`Error loading audio (attempt ${retryCount + 1}/${MAX_RETRIES}):`, error);
|
||||
|
||||
|
||||
if (retryCount < MAX_RETRIES) {
|
||||
// Calculate exponential backoff delay
|
||||
const delay = INITIAL_RETRY_DELAY * Math.pow(2, retryCount);
|
||||
console.log(`Retrying in ${delay}ms...`);
|
||||
|
||||
|
||||
// Wait for the delay
|
||||
await new Promise(resolve => setTimeout(resolve, delay));
|
||||
|
||||
|
||||
// Try to create a new Howl instance
|
||||
const retryHowl = await createHowl(retryCount + 1);
|
||||
if (retryHowl) {
|
||||
|
|
@ -646,7 +939,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
setActiveHowl(null);
|
||||
this.unload();
|
||||
setIsPlaying(false);
|
||||
|
||||
|
||||
toast.error('Audio loading failed after retries. Moving to next sentence...', {
|
||||
id: 'audio-load-error',
|
||||
style: {
|
||||
|
|
@ -655,18 +948,22 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
},
|
||||
duration: 2000,
|
||||
});
|
||||
|
||||
|
||||
advance();
|
||||
}
|
||||
},
|
||||
onend: function(this: Howl) {
|
||||
onend: function (this: Howl) {
|
||||
this.unload();
|
||||
setActiveHowl(null);
|
||||
if (pageTurnTimeoutRef.current) {
|
||||
clearTimeout(pageTurnTimeoutRef.current);
|
||||
pageTurnTimeoutRef.current = null;
|
||||
}
|
||||
if (isPlaying) {
|
||||
advance();
|
||||
}
|
||||
},
|
||||
onstop: function(this: Howl) {
|
||||
onstop: function (this: Howl) {
|
||||
setIsProcessing(false);
|
||||
this.unload();
|
||||
}
|
||||
|
|
@ -705,7 +1002,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
}, [isPlaying, advance, activeHowl, processSentence, audioSpeed]);
|
||||
|
||||
const playAudio = useCallback(async () => {
|
||||
const howl = await playSentenceWithHowl(sentences[currentIndex]);
|
||||
const howl = await playSentenceWithHowl(sentences[currentIndex], currentIndex);
|
||||
if (howl) {
|
||||
howl.play();
|
||||
}
|
||||
|
|
@ -777,6 +1074,8 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
// Cancel any ongoing request
|
||||
abortAudio();
|
||||
locationChangeHandlerRef.current = null;
|
||||
epubContinuationRef.current = null;
|
||||
continuationCarryRef.current.clear();
|
||||
setIsPlaying(false);
|
||||
setCurrentIndex(0);
|
||||
setSentences([]);
|
||||
|
|
@ -921,6 +1220,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
setVoiceAndRestart,
|
||||
skipToLocation,
|
||||
registerLocationChangeHandler,
|
||||
registerVisualPageChangeHandler,
|
||||
setIsEPUB
|
||||
}), [
|
||||
isPlaying,
|
||||
|
|
@ -945,6 +1245,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
setVoiceAndRestart,
|
||||
skipToLocation,
|
||||
registerLocationChangeHandler,
|
||||
registerVisualPageChangeHandler,
|
||||
setIsEPUB
|
||||
]);
|
||||
|
||||
|
|
@ -961,7 +1262,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
getLastDocumentLocation(id as string).then(lastLocation => {
|
||||
if (lastLocation) {
|
||||
console.log('Setting last location:', lastLocation);
|
||||
|
||||
|
||||
if (isEPUB && locationChangeHandlerRef.current) {
|
||||
// For EPUB documents, use the location change handler
|
||||
locationChangeHandlerRef.current(lastLocation);
|
||||
|
|
@ -971,7 +1272,7 @@ export function TTSProvider({ children }: { children: ReactNode }): ReactElement
|
|||
const [pageStr, sentenceIndexStr] = lastLocation.split(':');
|
||||
const page = parseInt(pageStr, 10);
|
||||
const sentenceIndex = parseInt(sentenceIndexStr, 10);
|
||||
|
||||
|
||||
if (!isNaN(page) && !isNaN(sentenceIndex)) {
|
||||
console.log(`Restoring PDF position: page ${page}, sentence ${sentenceIndex}`);
|
||||
// Skip to the page first, then the sentence index will be restored when setText is called
|
||||
|
|
@ -1029,4 +1330,4 @@ export function useTTS() {
|
|||
throw new Error('useTTS must be used within a TTSProvider');
|
||||
}
|
||||
return context;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import { pdfjs } from 'react-pdf';
|
||||
import type { TextItem } from 'pdfjs-dist/types/src/display/api';
|
||||
import type { PDFDocumentProxy } from 'pdfjs-dist';
|
||||
import { type PDFDocumentProxy, TextLayer } from 'pdfjs-dist';
|
||||
import "core-js/proposals/promise-with-resolvers";
|
||||
import { processTextToSentences } from '@/utils/nlp';
|
||||
import { CmpStr } from 'cmpstr';
|
||||
|
|
@ -52,6 +52,26 @@ function initPDFWorker() {
|
|||
// Initialize the worker
|
||||
initPDFWorker();
|
||||
|
||||
// Patch TextLayer.render to treat cancelled renders as non-errors
|
||||
try {
|
||||
const textLayerProto = TextLayer?.prototype;
|
||||
const originalRender = textLayerProto?.render;
|
||||
if (typeof originalRender === 'function') {
|
||||
textLayerProto.render = async function patchedRender(...args) {
|
||||
const task = originalRender.apply(this, args);
|
||||
if (!task || typeof task.then !== 'function') return task;
|
||||
return task.catch((error) => {
|
||||
if (error && (error.name === 'AbortException' || error.name === 'RenderingCancelledException')) {
|
||||
return;
|
||||
}
|
||||
throw error;
|
||||
});
|
||||
};
|
||||
}
|
||||
} catch (e) {
|
||||
console.error('Error patching TextLayer.render:', e);
|
||||
}
|
||||
|
||||
interface TextMatch {
|
||||
elements: HTMLElement[];
|
||||
rating: number;
|
||||
|
|
|
|||
BIN
tests/files/sample.mp3
Normal file
BIN
tests/files/sample.mp3
Normal file
Binary file not shown.
|
|
@ -1,6 +1,30 @@
|
|||
import { Page, expect } from '@playwright/test';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
|
||||
const DIR = './tests/files/';
|
||||
const TTS_MOCK_PATH = path.join(__dirname, 'files', 'sample.mp3');
|
||||
let ttsMockBuffer: Buffer | null = null;
|
||||
|
||||
async function ensureTtsRouteMock(page: Page) {
|
||||
if (!ttsMockBuffer) {
|
||||
ttsMockBuffer = fs.readFileSync(TTS_MOCK_PATH);
|
||||
}
|
||||
|
||||
await page.route('**/api/tts', async (route) => {
|
||||
// Only mock the POST TTS generation calls; let anything else pass through.
|
||||
if (route.request().method().toUpperCase() !== 'POST') {
|
||||
return route.continue();
|
||||
}
|
||||
|
||||
await route.fulfill({
|
||||
status: 200,
|
||||
contentType: 'audio/mpeg',
|
||||
body: ttsMockBuffer as Buffer,
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
// Small util to safely use filenames inside regex patterns
|
||||
function escapeRegExp(input: string) {
|
||||
return input.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
|
|
@ -79,6 +103,9 @@ export async function pauseTTSAndVerify(page: Page) {
|
|||
* Common test setup function
|
||||
*/
|
||||
export async function setupTest(page: Page) {
|
||||
// Mock the TTS API so tests don't hit the real TTS service.
|
||||
await ensureTtsRouteMock(page);
|
||||
|
||||
// Navigate to the home page before each test
|
||||
await page.goto('/');
|
||||
await page.waitForLoadState('networkidle');
|
||||
|
|
@ -341,7 +368,7 @@ export async function expectMediaState(page: Page, state: 'playing' | 'paused')
|
|||
} catch {
|
||||
return false;
|
||||
}
|
||||
}, state, { timeout: 25000 });
|
||||
}, state);
|
||||
}
|
||||
|
||||
// Use Navigator to go to a specific page number (PDF)
|
||||
|
|
@ -402,4 +429,4 @@ export async function waitForDocumentListHintPersist(page: Page, expected: boole
|
|||
return false;
|
||||
}
|
||||
}, expected, { timeout: 5000 });
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue