danvics/openreader
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
openreader/docs/v1/todo.md
Richard Roberson 452bc2adfe feat: Add issue triage and mapping documentation for OpenReader v1 
- Created `issues-to-components.md` to summarize open issues and their mappings to components for v1 development.
- Documented key features, bugs, and proposed solutions for issues #59, #48, #47, #44, and #40.
- Outlined global guardrails and cross-cutting improvements for the v1 architecture.

chore: Establish v1 todo and planning framework

- Created `todo.md` to capture the 1.0 rewrite plan, including scope, architecture overview, phased milestones, and a master checklist.
- Defined action items, dependency ordering, and issue mapping alignment for efficient development.

chore: Update pnpm workspace configuration

- Added `ignoredBuiltDependencies` for `canvas` in `pnpm-workspace.yaml` to prevent build issues.
2025-11-10 13:19:45 -07:00

35 KiB
Raw Blame History

OpenReader v1 Todo and Plan

This document captures the 1.0 rewrite plan, prioritized for a clean engine replacement and Dexie adoption. No dual engines during cutover.

See triage mapping: docs/v1/issues-to-components.md

Scope and guardrails

  • Streaming first playback with HTMLAudioElement plus MediaSource, blob fallback
  • Replace Howler across codebase
  • Dexie.js for client storage, with one time migration
  • Keep audiobook m4b and add chapter based MP3 export
  • Keep server side document sync
  • Browsers: Chrome, Firefox, Edge, Safari 16+

Architecture overview

flowchart TD
  Viewers[PDF viewer  EPUB viewer  HTML viewer] --> Adapters[Document adapters]
  Adapters --> Splitter[Sentence splitter and mapping]
  Splitter --> Queue[Sentence queue and preloader]
  Queue --> Engine[Playback engine state machine]
  Engine --> Media[Media controller HTMLAudioElement MSE]
  Media --> Output[Audio output  media session  background handling]
  Engine --> Cache[Audio cache Dexie]
  Engine --> TTS[TTS providers  OpenAI  Deepinfra  Custom]

Phased milestones

  • 1.0.0 alpha: skeleton, Dexie, engine minimal wired to HTML viewer, streaming route
  • 1.0.0 beta: PDF and EPUB adapters, highlighting, chapter MP3, migration UI
  • 1.0.0: m4b stability, server sync, remove legacy, tests and a11y

Master checklist

Dependency ordering

  • Dexie schema and repos precede engine cache features
  • Engine and MediaController precede viewers
  • Adapters precede highlighting and resume position
  • Streaming route precedes engine integration tests

Issue mapping alignment

  • #59 Chapter MP3 export -> audiobook.ts stream ZIP
  • #48 Large m4b download -> range enabled download streaming response bookId lifecycle
  • #47 Voice combo plus -> allow free form voice strings and pass through for Kokoro
  • #44 Dialog chunking -> quote aware grouping in splitter strategy
  • #40 PDF margins -> normalize x width and add debug overlay

Decision record outline

  • Engine is single source of truth for playback
  • Media controller abstracts HTMLAudioElement and MSE
  • Providers are stateless except for voice lists
  • Dexie stores documents configs audio cache positions

Checkpoint gates

  • Alpha: streaming route returns audio engine plays cached sentences HTML viewer wired
  • Beta: EPUB and PDF adapters stable highlighting chapter MP3 migration UI
  • GA: m4b and server sync robust e2e tests green legacy removed

Open questions

  • Do we support true streaming from all providers or progressive only for some
  • Do we want optional S3 offload for very large m4b files in Docker

Notes

  • Use strict abort controller scoping on every request to avoid double requests
  • Use unique cache keys text hash plus voice plus speed plus model
  • Ensure token budgets by chunk length not chars only for providers with limits

Expanded planning details

To make the v1 implementation predictable and fast to execute, this section adds concrete specifications (without code) for schemas, contracts, state machine events, milestones, risks, and acceptance criteria.

Milestone breakdown and exit criteria

Alpha (focus: Dexie + Engine + HTML adapter + streaming route)

  • Deliverables:
    • Dexie schema + repositories
    • Playback engine (core states and cancellation)
    • MediaController with MSE and blob fallback baseline
    • HTML viewer wired to adapters and engine
    • TTS streaming route (chunked where possible, progressive fallback)
  • Acceptance:
    • Start-to-speech under 700ms on cached blocks
    • Play/Pause/Skip stable with single in-flight cancellation
    • Voice change cancels and resumes with one request wave
    • Basic resume position works for HTML

Beta (focus: PDF/EPUB adapters + highlighting + chapter MP3 + migration UI)

  • Deliverables:
    • PDF adapter + highlighter; EPUB adapter + highlighter
    • Quote-aware sentence grouping toggle in splitter
    • Chapter-based MP3 export (ZIP streamed)
    • First-run migration UI (legacy to Dexie)
  • Acceptance:
    • Highlight tracks current sentence reliably in PDF and EPUB
    • Chapter ZIP exports on sample EPUBs with correct file naming
    • Migration completes with progress and resumability
    • PDF margins trimming reproducible across sample PDFs

GA (focus: m4b stability + server sync + tests + removal of legacy)

  • Deliverables:
    • Stable m4b creation with range-enabled download endpoint
    • Dexie-backed client sync using existing server route
    • Full test suite and a11y pass; remove legacy modules and Howler
  • Acceptance:
    • Large (1–2 GB) m4b downloads complete in Docker with Range support
    • Sync round-trips documents reliably (basic CRUD and bulk)
    • E2E tests for PDF/EPUB/HTML, streaming, exports all green
    • Legacy removed and version bumped to 1.0.0

Dexie schema (draft, no code)

Tables and primary keys:

  • documents
    • id (string, pk), type (enum pdf|epub|html), name, size, lastModified, dataRef (blob pointer or chunk refs), createdAt, updatedAt
  • config
    • key (string, pk), value (string), updatedAt
  • audioCache
    • key (string, pk: hash of text+voice+speed+model+format), createdAt, expiresAt, size, bytesRef (blob pointer or chunk refs), provider, model, voice, speed
  • positions
    • id (auto), docId (string, idx), locationToken (string), sentenceIndex (number), updatedAt
  • voices
    • id (auto), provider (string, idx), model (string, idx), voices (json), updatedAt

Repository responsibilities:

  • DocumentsRepo: CRUD, list, size budgeting, typed loaders for different doc types.
  • ConfigRepo: get/set/clear keys, transactional updates for related keys (e.g., provider+model+savedVoices).
  • AudioCacheRepo: TTL+LRU enforcement, size budgets (e.g., 100–250MB), purge policies.
  • VoicesRepo: cache list per provider-model; freshness TTL.

Migration notes:

  • One-time importer reads via legacy helpers in src/utils/indexedDB.ts and writes to Dexie with conversion of binary arrays to blobs/references.
  • Idempotent: re-runs skip if target record exists with same hash.
  • Progress UI surfaces counts and bytes processed.

Playback engine state machine (events and guards)

States:

  • idle, preparing, buffering, playing, paused, stopping, error

Key events:

  • playRequested (with queue head present)
  • pauseRequested
  • stopRequested
  • skipForwardRequested, skipBackwardRequested
  • configChanged (voice/model/speed) -> must produce a new request token and cancel in-flight
  • dataAvailable (buffer threshold met)
  • underflowDetected (drop to buffering)
  • errorOccurred (carry error type; decide recoverable vs terminal)

Guards:

  • When configChanged, any previously created network or decode work is cancelled using an AbortController keyed by the current session token.
  • Single active playback pipeline: next preloads are concurrent but bounded (e.g., N=1–2 lookahead).

Effects:

  • Queue manages sentence blocks; emits prefetch intents; respects AudioCacheRepo.
  • MediaController mediates HTMLAudioElement load/append and playbackRate changes (decoupled from TTS voice speed).
  • PositionStore persists docId+locationToken+sentenceIndex at debounced intervals (e.g., 800–1200ms) to positions table.

MediaController behavior (MSE-first, blob fallback)

Capabilities:

  • Prefer MSE with audio/mpeg or audio/aac SourceBuffer where supported (Chrome, Edge, Firefox).
  • Safari 16+ fallback path: small blob segment chaining to minimize gaps.
  • PlaybackRate applies to the element (audio player speed). Voice speed remains a synth-time parameter.
  • Integrate Media Session actions (play, pause, next, previous).
  • Visibility changes: pause when hidden (configurable), resume on foreground if engine state is playing.

Error handling:

  • MSE append errors cause a controlled transition to error or fallback to blob-segment mode, depending on browser capability checks.
  • Network stalls (no chunks for T ms) trigger buffering state.

TTS provider interface expectations (descriptive)

Responsibilities:

  • Accept text block, provider, model, voice (free-form for Kokoro plus syntax), speed, and format (mp3|aac).
  • Return either:
    • A stream of audio bytes (preferred), or
    • A complete audio buffer (fallback)
  • Voice listing may return default sets or pass-through if API supports dynamic retrieval.

Provider notes:

  • Deepinfra/Kokoro: permit plus syntax in voice; skip strict validation if “+” in string.
  • Custom OpenAI-compatible: try /audio/voices endpoint for list; fallback to default.
  • OpenAI: standard tts-1 or gpt-4o-mini-tts with optional instructions (for supported models).

Streaming route contract (descriptive)

Route: POST /api/tts/stream

  • Request: json body includes text, provider, model, voice, speed, response_format (mp3|aac).
  • Behavior:
    • If upstream supports chunked streaming, proxy as chunked response to client.
    • Else, progressively stream the full buffer to client without buffering entire file in memory when possible.
  • Headers:
    • Content-Type audio/mpeg|audio/aac
    • Prefer Transfer-Encoding chunked if supported.
  • Cancellation:
    • Must respect AbortSignal to cancel upstream and downstream.

Document adapters and location tokens

Requirements:

  • Adapters yield tuples { textBlock, locationToken, rawMap } where:
    • textBlock is the processed snippet the engine will synthesize/play.
    • locationToken is a stable opaque token for resume and navigation (for PDFs: page:sentence; for EPUB: cfi-based; for HTML: line-offset or logical chunk index).
    • rawMap enables highlight alignment (processed->raw mapping).
  • Navigation:
    • next/prev logic is adapter-specific (paginate for PDF, spine/section for EPUB, block index for HTML).
  • PDF margins:
    • Normalize x in [0..1] by dividing by pageWidth and compare to left/right margins.
    • Compute fallback width when TextItem.width is missing; exclude header/footer using normalized y.

Splitter strategy (dialog-aware grouping)

Behavior:

  • MAX_BLOCK_LENGTH remains bounded (e.g., 300 characters target).
  • Quoted dialog grouping heuristic:
    • When a sentence begins with an opening quote and the next ends with a closing quote, join unless combined length exceeds threshold; allow small overruns (e.g., +10%) to preserve dialog continuity.
  • Toggle in settings to enable/disable dialog grouping; default enabled for EPUB and HTML, opt-in for PDFs.

Exports: chapter MP3 and m4b improvements

Chapter MP3:

  • Emitter builds per-chapter tasks using EPUB table of contents mapping; fallback to “Unknown Section - N” when absent.
  • ZIP stream response for many files; return progressively to client; ensure Content-Disposition triggers download reliably.

m4b improvements:

  • Post-processing retains bookId and serves artifact via a dedicated GET endpoint that supports Accept-Ranges.
  • UI downloads via streamed response, not loading entire buffer in memory.
  • Clean-up policy configurable (e.g., keep artifacts for X hours/days).

Server-side sync plan (client-driven)

Client sync service uses repositories to:

  • Read all documents and upload in batches to src/app/api/documents/route.ts.
  • Download and write back to Dexie (convert numeric arrays to ArrayBuffer/Blob).
  • Provide progress events to UI with percentage and status text.

Testing matrix and acceptance

Unit (engine and adapters):

  • Engine: transitions, cancellation on config change, underflow to buffering, single resume path.
  • Adapters: PDF margin trimming, EPUB section mapping, HTML block indexing, dialog grouping on/off.

Integration:

  • Streaming playback path: start-to-speech latency, skip forward/back, voice switch.
  • Chapter MP3 export on sample EPUB: counts and filenames match TOC.
  • m4b large export streaming with Range download in Docker; verify file size and integrity.

E2E:

  • Upload and open PDF/EPUB/HTML; play with highlight; persist/resume position.
  • Settings changes (provider/model/voice/speeds) do not cause double requests or deadlocks.

Risks and mitigations

  • Provider streaming variability:
    • Route capability detection and fallback; progressive responses when chunked not possible.
  • Safari inconsistencies:
    • Proactive blob-fallback and small segment chaining; limit segment durations to reduce gaps.
  • Cache growth and quota:
    • Dexie TTL/size budgets; purge on startup and opportunistically; surface approximate disk usage to users in settings.
  • Large artifacts:
    • Range-enabled endpoints; avoid buffering full files in memory.

Work sequencing (implementation order)

  1. Dexie foundation
  • Schema + client + repositories
  • One-time migration script and Progress UI
  1. Engine baseline
  • State, queue, cancellation, MediaController baseline
  • HTML adapter and streaming route
  • Hook facade for React
  1. PDF/EPUB adapters + highlighting
  • PDF first, then EPUB (navigation and CFI)
  • Splitter dialog-aware grouping + toggle
  1. Exports and downloads
  • Chapter MP3 (ZIP), m4b GET download with Range
  1. Sync + polish
  • Client sync service
  • A11y, performance, tests, legacy removal, version bump

Acceptance gates (per milestone)

  • Alpha gate:
    • Play cached text in HTML with streaming route; skip/voice switch stable
  • Beta gate:
    • PDF and EPUB highlight track sentences; chapter export functional; migration works reliably
  • GA gate:
    • m4b Range download stable for large files; server sync; E2E suite green; legacy removed; version bumped

Scaffold and naming (authoritative file list)

The following v1 files will be created verbatim to avoid dual-engine complexity. Each path is the single source of truth for its concern.

Playback and media

TTS providers

Adapters and highlighting

NLP

Dexie database

React contexts and components

API routes

Tests

Sequenced next actions (implementation-ready)

  1. Documentation and dependencies
  1. Dexie foundation
  1. Migration and UI
  1. Playback engine baseline
  1. HTML path to first audio
  1. PDF and EPUB adapters
  1. NLP dialog grouping
  1. Exports
  1. Sync
  1. Switch-over and removal

Acceptance tests to author (explicit)

Playback engine

  • Single inflight: issuing toggle or voice change cancels prior token and yields exactly one fresh request wave
  • Underflow causes buffering and recovers on next data
  • Background/foreground preserves intended state

Streaming route

  • Chunked path sends Content-Type audio/mpeg|aac; progressive fallback functions when chunking is not supported
  • Client AbortSignal cancels upstream

Adapters and highlight

  • PDF: left/right margins trim deterministically across sample PDFs; highlight aligns with playing sentence
  • EPUB: section navigation next/prev works; highlight aligns; resume from location token
  • HTML: block indexing stable and resume works

NLP grouping

  • Dialog passages grouped unless size exceeds threshold; toggle off reverts to baseline splitting

Exports

  • Chapter MP3 ZIP produces files named NN - Chapter Title.mp3; ZIP streams without buffering entire archive
  • m4b Range download stable for 1–2 GB files under Docker

Sync and migration

  • First-run migration completes with retriable steps and progress
  • Sync uploads/downloads all docs with correct counts and bytes

Risks checklist and mitigations

  • Provider streaming variability -> route capability checks and progressive response fallback
  • Safari MSE variance -> short blob segment chaining; segment duration caps
  • Cache quota pressure -> Dexie TTL/LRU budgets and purge on startup
  • Large artifact memory -> Range endpoints; avoid in-memory buffers; disk-backed streams
  • Race conditions -> session tokens and abort scoping for network and decode paths

Notes on versioning and flags

  • Feature flags: dialog grouping default on for EPUB/HTML, off for PDF
  • Schema version pin in src/v1/db/client.ts with explicit upgrade function
  • Saved voices map by provider:model in ConfigRepo

Done criteria for 1.0.0

  • Legacy engine removed; all routes/viewers wired to v1
  • Exports (m4b and chapter MP3) robust under Docker and local
  • E2E tests cover PDF/EPUB/HTML reading, streaming playback, skip, voice switch, resume, exports
  • Docs updated: docs/v1/README.md, README.md, changelog updated and version bumped in package.json

Capability matrix and detection

Provider streaming capability (planned handling)

  • OpenAI (audio.speech): treat as progressively readable stream from the Response body; prefer chunked transfer in src/app/api/tts/stream/route.ts
  • Deepinfra: prefer progressive streaming when supported; fallback to full buffer proxy
  • Custom OpenAI-compatible: probe endpoint with a tiny synth call in a capability check, then cache provider features in src/v1/tts/voices.ts or src/v1/tts/Provider.ts

Browser feature checks (performed once per session)

Decision rules

  • If MSE and provider stream: MSE pipeline
  • If MSE but provider returns full buffer only: still MSE if chunking the buffer is practical; else blob fallback
  • If no MSE: blob fallback with small segments to minimize gaps

Migration plan (step-by-step)

Pre-migration checks

  • Detect legacy stores via src/utils/indexedDB.ts
  • Estimate total bytes and number of documents; show plan to user

Execution steps (idempotent)

  1. Create Dexie stores in src/v1/db/client.ts
  2. Copy documents from legacy to Dexie in batches; convert numeric arrays to ArrayBuffer/Blob
  3. Copy config items; normalize provider, model, baseUrl; write to src/v1/db/repositories/ConfigRepo.ts
  4. Copy last-location metadata into src/v1/playback/positionStore.ts
  5. Report progress and allow retry on failure (resume from last completed batch)

Post-migration

  • Mark legacy as migrated (config flag) to avoid future runs
  • Option to purge legacy data (explicit user action)

UI

Chapter MP3 export plan

Source of chapter boundaries

  • EPUB: TOC mapping in adapter; chapter per TOC item, merge small sections to avoid micro-chapters
  • PDF: future enhancement (out of initial scope unless PDF outlines/TOC present)

Export path

  • Build per-chapter audio buffers using providers via the same synth path as playback (no separate engine)
  • Stream a ZIP response with ZIP64 support for large aggregate sizes through src/app/api/audio/convert/route.ts with a mode switch
  • Names: NN - Chapter Title.mp3 with zero-padded NN
  • Progress events (client-side): show current chapter index, bytes accumulated

Acceptance criteria

  • A 50+ chapter EPUB exports as a streamed ZIP without UI stalls or memory spikes
  • ZIP integrity verified by unzip on client

Large m4b export stability (range-enabled)

Server

  • Add GET download path on src/app/api/audio/convert/route.ts to stream artifacts from disk via Accept-Ranges with 206 responses
  • Keep temp artifacts by bookId; implement cleanup policy (e.g., TTL-based) and a management UI later

Client

  • Perform HTTP range-enabled download; avoid buffering into memory; use streams to file via browser download APIs

Acceptance criteria

  • 1–2 GB m4b downloads complete in Docker with stable throughput; resume on flaky connections is acceptable via range retries

Highlight alignment and sentence mapping

PDF

  • Normalize x to [0..1]; apply left/right margins strictly
  • When width is missing, estimate from glyph metrics; exclude header/footer using normalized y
  • Highlighter replays processed-to-raw mapping from adapter map in src/v1/highlight/pdfHighlighter.ts

EPUB

HTML

  • Use block index mapping; highlighting simpler (no layout transforms)

Performance and budgets

Audio cache (Dexie)

  • Budget target: 100–250 MB, configurable
  • Eviction: TTL and LRU; purge on startup and opportunistically when budgets exceeded
  • Keys: hash(text + provider + model + voice + speed + format)

Queue and prefetch

  • Lookahead of 1–2 blocks, adaptive to latency
  • Abort prefetch on config change or navigation

Target latencies

  • Cached block start-to-speech < 500–700 ms
  • Non-cached with stream: first audible audio chunk < 1.5–2.5 s (provider-dependent)

Accessibility checklist

Player

  • Buttons keyboard accessible, ARIA labels, focus ring
  • Space/Enter to toggle play/pause; left/right to skip; up/down for speed adjustments

Viewers

  • Highlight announced via ARIA-live optional verbosity setting
  • Navigator provides input with label and validation errors announced

Work package owners (placeholders)

  • Dexie schema and repos: assign to Core
  • Engine and MediaController: assign to Core
  • HTML adapter path to first audio: assign to Core
  • PDF adapter + highlighting: assign to PDF
  • EPUB adapter + highlighting: assign to EPUB
  • Chapter MP3 export and m4b Range path: assign to Audio
  • Migration UI: assign to UI
  • Tests (unit, integration, e2e): assign to QA

Timeline (tentative, after approval)

  • Week 1–2: Dexie + Engine baseline + HTML path + stream route (Alpha)
  • Week 3–4: PDF + EPUB adapters + highlighting + dialog grouping + migration UI (Beta)
  • Week 5–6: Chapter MP3 + m4b Range + sync + tests + a11y + legacy removal (GA)

Open questions (to resolve before coding)

  • Exact provider streaming capabilities to lock in API adapter behavior in src/app/api/tts/stream/route.ts
  • Target audio formats default (mp3 vs aac) per provider for best MSE compatibility
  • Chapter detection for PDFs: scope to outlines only in v1.0 or defer to 1.1

Planning links