Commit graph

431 commits

Author SHA1 Message Date
Richard R
3f17874fe5 tts ensure: skip whisper alignment in test namespaces 2026-05-24 20:58:48 -06:00
Richard R
b2517bc91d pdf parse: dedupe shared jobs and stop enqueue-on-read 2026-05-24 20:58:34 -06:00
Richard R
12c586d7be refactor(core,server): migrate contracts to api-contracts and modularize job handlers
Move shared type contracts from `contracts` to new `api-contracts` module in `compute/core` for clearer API boundaries. Remove legacy `contracts`, `pdf`, and `whisper` index files. Update all imports to use `api-contracts`. Refactor server job logic by replacing `parsePdfJob.ts` with modular `user-pdf-layout-job.ts` and `user-whisper-align-job.ts`, updating all relevant API routes and compute integration. This improves maintainability and separation of concerns across compute and server layers.
2026-05-24 11:39:25 -06:00
Richard R
e75114a943 refactor(onboarding): introduce onboarding state coordinator and central registry
Add useOnboardingCoordinator hook and onboarding-state registry to manage
onboarding-related state such as privacy acceptance and first-visit tracking.
Refactor SettingsModal to utilize the new onboarding state abstractions,
removing legacy privacy gating logic and improving maintainability.

Includes unit tests for onboarding-state registry to ensure correctness.
2026-05-23 17:07:50 -06:00
Richard R
df321b8832 fix(jobs): make TTS segment cache invalidation non-blocking in parsePdfJob
Move TTS segment cache invalidation to a fire-and-forget promise after
document update to prevent cache errors from blocking parse readiness.
Log warnings for both cache invalidation failures and warnings, but do not
interrupt the main job flow. This improves job robustness and avoids
unnecessary failures due to cache issues.
2026-05-23 05:55:22 -06:00
Richard R
65d25e3ce5 refactor(ui): centralize privacy gating logic for settings modal
Consolidate privacy acceptance checks into reusable hooks and callbacks within
SettingsModal. Replace scattered gating with a single entry point for opening
settings or changelog views, ensuring consistent enforcement. Update test
helpers to robustly dismiss onboarding and settings overlays by avoiding
state race conditions.
2026-05-23 05:50:54 -06:00
Richard R
b6be71e3b1 refactor(types,api): centralize PDF and TTS type exports to compute-core/types
Move and re-export all PDF parsing and TTS-related types from
@openreader/compute-core/types, eliminating local type duplication.
Update imports throughout the codebase to use the new centralized type
module. Remove obsolete worker-contract file and update tsconfig paths
and compute-core exports for the new types entry point. Refactor API
routes and job logic to use the new type imports and shared parse-state
healing utility. This streamlines type management and improves
consistency between compute and app layers.
2026-05-22 01:51:51 -06:00
Richard R
09e8898683 feat(pdf,config): implement stale parse-state healing and unify op stale window config
Introduce automatic healing for stale PDF parse states in API routes. Add
`isDocumentParseStateStale` utility and use a shared `COMPUTE_OP_STALE_MS`
environment variable (with config getter) to control the stale window for both
worker op replacement and app-side parse-state healing. Update documentation and
environment variable references to reflect new config.

Enables automatic marking of stuck parses as failed, improving reliability and
retry behavior in distributed compute environments.
2026-05-22 01:21:18 -06:00
Richard R
1d4a301a5d fix(pdf): correct parse status handling and loader display logic
Update PDF viewer and document hook to improve parse status management:
- Set parseStatus to null on clear instead of 'pending' for accurate state
- Explicitly set initial parseStatus and reset parseProgress when loading a new document
- Adjust loader display logic to check for resolved parse status before showing expanded loader

Prevents UI inconsistencies and ensures parse progress is reflected correctly during document transitions.
2026-05-22 00:17:15 -06:00
Richard R
37b6999c2c refactor(compute-core): collapse to root/local-runtime entrypoints 2026-05-21 21:30:44 -06:00
Richard R
37d11bf9b8 refactor(compute): remove legacy app whisper duplicate and unify core whisper boundary 2026-05-21 21:15:10 -06:00
Richard R
1612e6694b refactor(compute-core): capability-boundary exports for runtime and pdf-layout 2026-05-21 21:11:28 -06:00
Richard R
754fbee227 refactor(compute-core): centralize runtime cpu config exports 2026-05-21 20:54:22 -06:00
Richard R
384369f48b refactor: fix tests + remove deprecated PDF layout processing files and update imports
- Deleted mergeTextWithRegions.ts, LICENSE.txt, manifest.json, parsePdf.ts, renderPage.ts, runLayoutModel.ts, stitchCrossPageBlocks.ts, and types.ts as they are no longer needed.
- Updated unit tests to reflect new import paths from @openreader/compute-core.
- Adjusted tsconfig.json to include new paths for pdf-layout module.
2026-05-21 18:48:17 -06:00
Richard R
3ebcb3e6fd tts: make ensure route abort-aware and reduce abort log noise 2026-05-21 16:46:47 -06:00
Richard R
3c25c93e8b tts: remove redundant route-level alignment timeout wrappers 2026-05-21 16:33:55 -06:00
Richard R
c9dad9c23c compute: reuse shared timeout helpers across worker and local 2026-05-21 16:32:24 -06:00
Richard R
e384b66b7d compute: align local and worker-client timeout behavior 2026-05-21 16:17:47 -06:00
Richard R
f8672b073e refactor(compute): unify job concurrency and CPU thread allocation logic
Standardize compute job concurrency configuration by introducing a shared
COMPUTE_JOB_CONCURRENCY setting, replacing separate PDF and Whisper concurrency
controls. Add cross-platform CPU core detection and thread budgeting utilities,
and update both ONNX model execution and concurrency limiters to use dynamic
thread allocation per job. Refactor environment variable docs and examples to
reflect unified concurrency management. Streamline PDF.js font path resolution
for improved reliability across environments.
2026-05-21 15:58:29 -06:00
Richard R
671d5c6adb feat(pdf,worker): enhance PDF layout parsing UX and add robust timeout controls
Introduce idle and hard cap timeouts for PDF layout parsing in the compute worker,
ensuring long-running jobs are capped and progress is tracked with explicit
timeouts. Refactor NATS JetStream client initialization to use a configurable
API timeout. Update PDF viewer UI to treat parse state as 'pending' by default,
improving clarity during document preparation and parsing. Refine parse loader
labels and progress logic for more accurate user feedback.
2026-05-21 13:50:40 -06:00
Richard R
a2c714c2a5 refactor(worker,tts): unify PDF and Whisper retry config, add alignment timeouts
Update compute worker to use separate environment variable for PDF job attempts
(`COMPUTE_PDF_JOB_ATTEMPTS`) and set Whisper job max deliveries to 1, clarifying
and separating retry logic between job types. Adjust `.env.example` and deployment
docs to comment out defaults and document new/renamed variables for advanced tuning.

In TTS segment ensure route, introduce explicit timeouts for Whisper alignment
operations, with configurable durations for local and worker modes, improving
robustness and error handling for long-running alignments.
2026-05-21 12:37:21 -06:00
Richard R
fe9104e8fc docs(pdf,spinner,config): update parse state UI, spinner theming, and worker env docs
Revise PDF viewer to introduce an explicit 'unknown' parse state and improve
loader and status messaging for better user clarity. Refactor Spinner to use
'currentColor' for border styling and simplify conditional rendering based on
className. Expand compute worker .env.example with explicit concurrency,
timeout, and job attempt variables for enhanced configuration transparency.
2026-05-21 11:59:12 -06:00
Richard R
bde8d74a97 refactor(pdf): improve parse loader UX and PDF viewer readiness signaling
Enhance PDF parse loader transitions by introducing a delayed expansion for detailed progress, reducing UI flicker during parse-to-render handoff. Refactor PDFViewer to ensure document readiness is signaled only once per load, preventing redundant callbacks and improving synchronization between parsing and rendering states.
2026-05-21 10:54:07 -06:00
Richard R
dd494d50aa Add force-reparse confirmation flow in PDF UI 2026-05-21 10:34:29 -06:00
Richard R
10748c7fcd Add force-token cache bust for worker PDF op keys 2026-05-21 10:32:18 -06:00
Richard R
3aad8a51e4 Add local compute concurrency limiter with tests 2026-05-21 10:30:36 -06:00
Richard R
4497f610c0 feat(pdf): implement granular PDF parse progress tracking and migrate to parseState
Introduce detailed progress reporting for PDF layout parsing, exposing phase and page-level updates to clients. Replace legacy parseStatus with a structured parseState field in the database schema, updating all relevant backend and API logic. Add SSE endpoint for real-time parse progress updates. Update client hooks and UI to reflect granular progress and improve user feedback during document parsing. Includes migration scripts and new parse-state utility module.
2026-05-21 10:27:16 -06:00
Richard R
a7b955e1ca refactor(worker): adopt operation-based API and opKey deduplication for compute flows
Migrate worker and backend contract to an operation-centric model, introducing opKey-based deduplication for Whisper alignment and PDF layout parsing. Add new operation request/state types and update worker endpoints to accept generic operation requests and expose operation status via SSE. Refactor client and server logic to support operation flows. Update deployment docs, environment examples, and Dockerfile to reflect new architecture and dependency management.
2026-05-21 09:05:26 -06:00
Richard R
8190b57ff2 feat(documents): support parsed document retrieval by key and configurable worker wait timeout
Add ability to fetch parsed documents using a stored S3 key, enabling more flexible blob access patterns. Update API route to prefer `parsedJsonKey` if present, falling back to legacy lookup. Introduce `COMPUTE_WORKER_WAIT_TIMEOUT_MS` environment variable for customizing worker job wait time, with a new default of 120 seconds. Update documentation and examples to reflect these changes.
2026-05-20 18:07:32 -06:00
Richard R
086329505f fix(worker): improve environment variable handling and URL normalization
Update worker server to prioritize platform-specific PORT environment variable
when setting the compute worker port. Add robust normalization for
COMPUTE_WORKER_URL, ensuring proper scheme and validation, and trim trailing
slashes for consistency. This enhances reliability in diverse deployment
environments.
2026-05-20 17:01:14 -06:00
Richard R
91d9fe47a5 refactor(compute): standardize compute mode detection for backend initialization
Update backend creation logic to consistently determine compute mode using
a local variable, ensuring correct handling when the mode is undefined. This
improves clarity and robustness in backend selection.
2026-05-20 16:39:25 -06:00
Richard R
944b8246e5 refactor(context): expose provider config type and instruction flags in TTS value
Update TTS context to include configProviderType, supportsInstructions, and
ttsInstructions in the provided value, improving downstream access to provider
capabilities and instruction handling.
2026-05-20 15:42:45 -06:00
Richard R
88809b826b fix(compute): unify PDF layout result handling with object key support
Update compute backend interfaces and job logic to consistently handle parsed
PDF layout results as either direct data or S3 object key references. Refactor
result types, local and worker backends, and job processing to support this
pattern. Improves flexibility for large document parsing and object storage
integration.
2026-05-20 15:38:24 -06:00
Richard R
1bd13c02fe refactor(compute): modularize core contracts and runtime, enable flexible backend resolution
Restructure compute core by extracting job contracts and runtime logic into separate
modules (`contracts.ts`, `local-runtime.ts`) for improved modularity and clearer
API boundaries. Update exports and imports across worker, server, and local
compute backends to use the new modules. Enhance Next.js config to support
dynamic backend selection via build-time constants and aliasing, enabling
seamless switching between local and worker compute modes. Adjust TypeScript
paths and Dockerfile entrypoint to align with the new structure.
2026-05-20 15:07:22 -06:00
Richard R
502c98f11f refactor(tts): add segment cache clearing and improve sidebar scroll handling
Introduce clearSegmentCaches to TTS context for explicit cache invalidation after segment clearing. Enhance SegmentsSidebar scroll logic to better distinguish user-initiated and programmatic scrolls, improving UX during segment updates.
2026-05-20 04:59:09 -06:00
Richard R
b679bf736c chore(config): remove smartSentenceSplitting option and related code paths
Eliminate the smartSentenceSplitting configuration flag from application state, UI, and all TTS and audiobook adapter logic. Consolidate TTS segment planning to always include context units, streamlining code and reducing conditional branches. Update tests and type definitions to reflect the removal of this feature toggle.
2026-05-20 04:51:10 -06:00
Richard R
bb3eb40966 refactor(tts): extend segment prefetching to support source unit arrays and enhance cache key granularity
Update TTS segment prefetching logic to handle arrays of CanonicalTtsSourceUnit for upcoming and next locations, enabling more precise TTS segment mapping. Expand cache key construction to include providerType and instructions, improving cache differentiation for TTS requests. Adjust related types and usages to support richer segment metadata propagation.
2026-05-20 04:44:47 -06:00
Richard R
9d1f9d5707 fix(pdf): readd source unit extraction for TTS integration in page parsing
Introduce sourceUnitsFromParsedPage helper to extract CanonicalTtsSourceUnit
objects from parsed PDF pages, filtering by block kind and text content. Update
TTS state setter to include extracted source units when available, enabling
improved TTS segment mapping and future extensibility.
2026-05-20 03:53:25 -06:00
Richard R
84c3b22e5a refactor(pdf-layout): streamline viewer layout settling and harden pdf document loading
Unify and improve layout settling logic across EPUB, HTML, and PDF viewers by
adding delayed recomputation and guarding against zero-height container during
transient states. Refactor PDF document loading to include retry logic with
success detection, ensuring robust handling of transient failures. Update
Playwright test helpers to provide more reliable PDF viewer readiness checks
and introduce request retry utilities for backend API calls. Add targeted unit
tests for single-section chapter fallback in the PDF audiobook adapter. Update
test files and helpers to align with new layout and export logic.
2026-05-19 23:02:41 -06:00
Richard R
01cb95d8e7 fix(compute): migrate compute backend initialization to async and update consumers
Refactor compute backend initialization to use asynchronous loading and promise caching, replacing previous synchronous singleton pattern. Update all consumers—including TTS segment alignment and PDF layout parsing jobs—to await compute backend resolution before invoking methods. This change improves compatibility with dynamic imports and future-proofs backend selection logic.
2026-05-19 19:38:36 -06:00
Richard R
e52b754106 refactor(pdf): remove legacy text extraction and update parse status normalization
Eliminate unused PDF text extraction logic and margin-based filtering from the client library. Refactor API routes to generalize parse status normalization, removing explicit references to deprecated "unsupported" status and type variants. Update usages to accept string parse statuses for improved flexibility. Simplify document state dependency tracking in the PDF document hook.
2026-05-19 19:11:51 -06:00
Richard R
f5d7408f17 refactor(pdf-layout): remove compute "none" mode and unsupported parse status
Eliminate the "none" compute mode and all related code paths, including the NoneComputeBackend, "unsupported" parse status, and PDF margin settings. Parsing is now always available if the app starts successfully, and configuration is limited to "local" or "worker" compute modes. Update types, API routes, client adapters, and documentation to reflect this simplification.

BREAKING CHANGE: "none" is no longer a valid COMPUTE_MODE; only "local" and "worker" are supported. "unsupported" parse status and PDF margin settings are removed.
2026-05-19 18:25:27 -06:00
Richard R
bab3416a36 refactor(compute): add timing metrics to worker job results and status
Introduce WorkerJobTiming interface to capture queue wait, S3 fetch, and compute
durations for compute worker jobs. Update job result and status response types
to include timing data. Enhance worker implementation to record and attach
timing metrics for alignment jobs, improving observability of job processing
latency. Update exports and type usage to support new timing fields.
2026-05-19 15:55:35 -06:00
Richard R
f1aa1c3e3b feat(compute): add external compute worker backend and integration
Introduce support for external compute worker mode (`COMPUTE_MODE=worker`) using a new `WorkerComputeBackend`. This enables offloading heavy ONNX Whisper alignment and PDF layout parsing to a standalone worker service (Redis + BullMQ), improving scalability and compatibility with serverless/limited environments.

- Add `@openreader/compute-core` as a shared package for ONNX inference and PDF parsing logic.
- Implement `WorkerComputeBackend` and worker contract/types for remote job execution.
- Update compute backend selection logic and remove previous worker mode guards.
- Extend `WhisperAlignInput` and `PdfLayoutInput` types to support object keys for remote data access.
- Refactor local compute backend to use `@openreader/compute-core` and support both buffer and object key inputs.
- Update job runner, TTS segment alignment, and PDF layout parsing flows to use new compute backend APIs.
- Add scripts, Docker workflow, and documentation for deploying and running the compute worker.
- Update environment variable docs and examples for worker mode, including storage requirements and configuration.
- Document published images and stack changes to reflect the new compute worker architecture.

BREAKING CHANGE: `COMPUTE_MODE=worker` now requires an external compute worker service and S3-compatible object storage. Embedded SeaweedFS (`weed mini`) is not supported in worker mode. See the new documentation for deployment and configuration details.
2026-05-19 15:21:25 -06:00
Richard R
3a21f2a5f5 refactor(config): migrate compute and runtime env vars to new naming scheme
- Replace `OPENREADER_COMPUTE_MODE` and related variables with `COMPUTE_MODE`
- Replace `OPENREADER_*` PDF/Whisper model URLs with `PDF_LAYOUT_MODEL_BASE_URL` and `WHISPER_MODEL_BASE_URL`
- Remove legacy `NEXT_PUBLIC_*` runtime config seeds in favor of `RUNTIME_SEED_*`
- Update documentation, code, and environment references to match new variable names
- Remove deprecated `scripts/fetch-models.mjs` and related npm script
- Update runtime config SSR injection from `window.__OPENREADER_RUNTIME_CONFIG__` to `window.__RUNTIME_CONFIG__`
- Adjust Next.js config to use new compute mode env var and optimize output file tracing for ONNX dependencies

BREAKING CHANGE: Environment variable names for compute mode, model URLs, and runtime config seeding have changed. Update `.env` files and deployment configs to use `COMPUTE_MODE`, `PDF_LAYOUT_MODEL_BASE_URL`, `WHISPER_MODEL_BASE_URL`, and `RUNTIME_SEED_*` as appropriate. Legacy `OPENREADER_*` and `NEXT_PUBLIC_*` variables are no longer supported.
2026-05-19 13:27:07 -06:00
Richard R
874e5ef359 refactor(whisper): migrate word alignment to ONNX backend and remove whisper.cpp integration
Replace the previous whisper.cpp-based word alignment with a fully ONNX-based
implementation using onnxruntime-node and @huggingface/tokenizers. Add new
Whisper ONNX model management, alignment mapping, and spectral analysis modules.
Remove all code and documentation referencing whisper.cpp, update environment
variables, Dockerfile, and docs to reflect ONNX-only alignment. Add unit tests
for alignment and ONNX model logic.
2026-05-19 13:00:21 -06:00
Richard R
6ab5c230c2 feat(pdf): improve cross-page block stitching and enforce TTS segment source boundaries
Enhance PDF cross-page block stitching to move only sentence continuations and
preserve remaining text on the following page. Update TTS segment planning to
support strict source boundary enforcement, ensuring segments do not cross
block or paragraph-title boundaries. Add new tests for both features and
refactor PDF text item normalization to filter out skewed or rotated runs.
2026-05-18 13:21:43 -06:00
Richard R
90e7caac43 refactor(previews): streamline preview generation and update image variant to 400px
Remove temporary working directory usage and in-place file writes from preview
generation, simplifying logic for both PDF and EPUB previews. Change default
preview image variant from 480px to 400px width, updating related constants and
cache schema version to maintain consistency across client and server code.
2026-05-18 12:37:26 -06:00
Richard R
8af4e857b7 feat(pdf): add block geometry-based highlighting using parsed document and locator
Implement block-level highlighting in PDF viewer by leveraging parsed document
structure and TTS segment locator. Extend highlightPattern API to accept options
for parsedDocument, locator, and block geometry mode. Update PDFViewer to use
block geometry highlighting when available, improving accuracy and alignment of
sentence and word highlights with ONNX-based DocLayoutV3 parsing. Refactor
highlighting logic to support both legacy span-based and new geometry-based
approaches.
2026-05-18 12:25:41 -06:00
Richard R
1544a2d969 refactor(previews): update preview variant defaults, cache schema, and image handling
Remove hardcoded defaults for preview variant and width in database schemas to allow more flexible preview generation. Bump preview cache schema version and add versioning to cache rows for consistency. Switch preview image generation to 480px JPEGs with updated file naming and content type. Refactor PDF preview rendering to support configurable output format and quality, and update all preview consumers to use unified "image" buffer property instead of "png". Add migration scripts and update metadata for both Postgres and SQLite.

BREAKING CHANGE: Preview variant and width defaults removed from database; preview cache and image handling updated across system.
2026-05-18 05:20:20 -06:00
Richard R
37a90b734d feat(pdf): migrate to PP-DocLayoutV3 ONNX model and update PDF block taxonomy
Switch PDF layout parsing to use the PP-DocLayoutV3 ONNX model, replacing the previous Docling-based approach. Update all environment variables, model fetching, and manifest handling for the new model and its artifacts. Refactor block kind taxonomy throughout the codebase, tests, and UI to align with PP-DocLayoutV3 labels, including expanded and renamed block types. Revise document settings, block filtering, and stitching logic to support the new set of block kinds. Update documentation and environment variable references to reflect the model transition.

BREAKING CHANGE: PDF parsing now requires PP-DocLayoutV3 ONNX model and updated environment variables; block kind names and settings have changed throughout the system.
2026-05-18 04:26:39 -06:00
Richard R
766c04d08d refactor(pdf): implement ONNX-based Docling layout parsing and block-level TTS for PDFs
Migrate PDF parsing to use ONNX Docling layout model for structured block extraction, enabling improved TTS segmentation and chaptering. Add compute backend abstraction for heavy tasks (alignment, layout parsing) with configuration via `OPENREADER_COMPUTE_MODE`. Integrate block-level locators and document settings for per-document PDF parsing options. Update S3 storage for parsed PDF JSON, add migration and schema changes, and extend client and server APIs for parsed data and settings. Remove legacy word highlight flag in favor of compute capability detection.

- Add ONNX model download, local/none compute modes, and `onnxruntime-node` dependency
- Update PDF viewer and TTS pipeline to use parsed blocks and block-level locators
- Add document settings storage and APIs for per-document PDF options
- Update S3 storage layout for parsed PDF JSON
- Add admin/config/docs updates for new compute and parsing features
- Remove obsolete word highlight runtime flag and UI

BREAKING CHANGE: PDF parsing and word highlighting now require `OPENREADER_COMPUTE_MODE=local` and ONNX model; document settings and S3 layout updated; legacy word highlight flag removed.
2026-05-17 21:18:51 -06:00
Richard R
d90e48aaf3 feat(html): add HTML/TXT/MD sentence and word highlighting, settings, and audiobook export support
Introduce sentence and word highlighting for HTML/TXT/MD documents, including new config options (`htmlHighlightEnabled`, `htmlWordHighlightEnabled`) and synced user preferences. Update the HTML viewer to support these highlights using custom spans, and add corresponding CSS styles. Extend document settings to allow toggling these features. Implement block-level locator sorting and display improvements. Add audiobook export support for HTML documents, including adapter and pipeline integration.

- Add `htmlHighlightEnabled` and `htmlWordHighlightEnabled` to config, user state, and Dexie storage
- Update HTML viewer to apply and manage highlights for sentences and words
- Add CSS for HTML highlight classes and scroll margin
- Extend document settings UI for HTML highlight toggles
- Support HTML audiobook export in modal and pipeline
- Improve block locator sorting and sidebar display for HTML
- Add HTML audiobook adapter and block parsing utilities
2026-05-17 11:14:02 -06:00
Richard R
64182e6c8d chore(workspace): streamline monorepo and CI with pnpm workspace, dependency, and workflow updates
Update GitHub Actions workflows to latest action versions and improve pnpm usage by introducing per-package workspace configs. Add docs-site/pnpm-workspace.yaml and update Dockerfile/package.json for better monorepo support. Upgrade dependencies across main and docs-site packages for compatibility and security. Refine TypeScript types and hooks for improved type safety and maintainability.
2026-05-14 15:44:22 -06:00
Richard R
ca4228970b fix(server): enforce server-only usage for runtime config resolvers
Add explicit runtime checks to prevent runtime config functions from being called on the client. This guards against accidental client-side invocation by throwing errors if executed outside the server environment.
2026-05-14 13:07:43 -06:00
Richard R
fb39723089 fix(config): extract RSC-compatible runtime config resolver
Move runtime config resolution logic to a new `runtime-config-rsc.ts` module for use in React Server Components. Update public layout and landing page to consume the RSC-specific resolver, improving compatibility and separation of concerns.
2026-05-14 12:50:16 -06:00
Richard R
522540452c feat(auth): add runtime toggle for user sign-ups and enforce signup policy
Introduce `enableUserSignups` runtime setting to allow administrators to control
whether new accounts can be created. Update environment variable and documentation
references to support this feature. UI elements for account creation are now
conditionally rendered based on this flag. Signup attempts are blocked server-side
when disabled, including email, OAuth, and anonymous upgrades.

Add `assertUserSignupAllowed` utility for consistent enforcement and corresponding
unit tests to verify policy behavior.
2026-05-14 12:47:05 -06:00
Richard R
fef3775ad4 style(ui): update modal gradient backgrounds and adjust settings header layout
Refine radial gradient backgrounds in Settings and DocumentSettings modals to use
CSS color-mix with accent variable for improved theming. Adjust Settings modal
header spacing and layout for better alignment and consistency.
2026-05-14 12:14:55 -06:00
Richard R
c3fb3af3d3 refactor(components): unify button styling using buttonClass utility across multiple components 2026-05-14 12:06:21 -06:00
Richard R
1f548f71ea feat(audiobook): integrate settings coercion utility and migrate metadata handling
Refactor chapter API to use new coerceAudiobookGenerationSettings utility for
validating and migrating audiobook metadata. Add src/lib/server/audiobooks/settings.ts
with shared logic and introduce corresponding unit tests to ensure correct
settings migration and validation.
2026-05-14 11:34:16 -06:00
Richard R
0787d2e88a refactor(settings): extract changelog check logic to reusable utility
Move changelog version check effect from SettingsModal to a new
scheduleChangelogCheck utility for improved modularity and testability.
Add unit tests for changelog check scheduling behavior.
2026-05-14 10:58:32 -06:00
Richard R
beb854cb39 feat(preferences): add changelog version tracking and payload meta support
Integrate changelog version check on settings open, triggering the changelog
panel when a new app version is detected. Refactor user preferences storage to
support metadata via payload utilities, enabling version tracking and future
extensibility. Add API endpoint for changelog version check and related tests
for changelog and preferences payload logic.
2026-05-14 10:53:38 -06:00
Richard R
5efb29cda9 refactor(ui): restructure settings modal navigation and changelog panel logic
Update SettingsModal to use conditional rendering for the changelog panel,
displaying the main settings navigation only when the changelog is closed.
Refactor mobile and desktop navigation layout to improve maintainability.
Add missing ChevronRightIcon import for icon consistency.
2026-05-14 10:01:51 -06:00
Richard R
9e09f80067 feat(changelog): add changelog feed support and settings integration
Introduce changelog feed manifest support, including client and shared
utilities for fetching and parsing changelog data. Add a new
Settings modal panel for viewing changelog entries, with version
detection based on the current app version. Expose a configurable
changelog feed URL in both environment variables and admin panel
runtime settings. Update documentation and deployment workflow to
support changelog feed generation and consumption. Include unit tests
for changelog utilities.
2026-05-14 09:45:04 -06:00
Richard R
f300dfb01f Retry errored TTS segments with cooldown and retry-after 2026-05-14 08:31:40 -06:00
Richard R
df841210a3 Remove dead segment audio presign route and legacy helpers 2026-05-14 08:01:53 -06:00
Richard R
513128b802 Stream fallback segment audio and support byte-range requests 2026-05-14 07:55:48 -06:00
Richard R
da629b3ecd Enable private caching headers for segment audio routes 2026-05-13 23:44:13 -06:00
Richard R
ef2ee09064 Warm segment audio during preload before playback 2026-05-13 23:39:29 -06:00
Richard R
16bf52f889 Use direct signed segment audio URLs in ensure/manifest 2026-05-13 23:33:00 -06:00
Richard R
89391410da refactor(ui): replace loading text with skeleton placeholders across panels
Update multiple UI panels to use skeleton components instead of loading text or spinners.
Standardize loading states in AudiobookExportModal, AdminFeaturesPanel, AdminProvidersPanel,
DocumentSelectionModal, and SegmentsSidebar for improved visual consistency and user feedback.
2026-05-13 23:17:47 -06:00
Richard R
d5eebd3b11 chore(config): remove Deepinfra model gating and related admin setting
Eliminate the showAllDeepInfraModels runtime/admin config and all code paths
that allowed restricting Deepinfra's model catalog. Deepinfra now always
shows the full model list regardless of API key or environment variable.
Update documentation, environment examples, admin panels, runtime config,
provider catalog logic, and tests to reflect this change.
2026-05-13 18:42:40 -06:00
Richard R
e27d20419d refactor(ui): unify form primitives and simplify document selection modal
Move shared button, input, and listbox styles to a new formPrimitives module
used across admin panels, settings, and document settings. Update admin and
settings components to import these styles from the new location. Refactor
DocumentSelectionModal to use a direct files prop with loading and error
states, removing fetcher logic and internal fetch state. Add a custom hook
for library document queries. Clean up legacy admin/ui primitives and
standardize segmented controls and section layouts.
2026-05-13 18:42:31 -06:00
Richard R
a296d6fb13 feat(reader,settings): add react-query caching for library import and TTS segments
Integrate @tanstack/react-query in SettingsModal to cache and prefetch library
documents, improving import modal responsiveness. Refactor SegmentsSidebar to
use infiniteQuery for TTS segments manifest, enabling pagination, cache
invalidation, and consistent data fetching. Remove legacy fetch state logic in
favor of react-query state management.
2026-05-13 17:39:04 -06:00
Richard R
fdeeb60f7e refactor epub reader hooks and add unit coverage 2026-05-13 16:56:13 -06:00
Richard R
fd6a06e467 refactor(epub): move reader state to route-local hook 2026-05-13 14:22:24 -06:00
Richard R
2f067c0a90 refactor(pdf): move reader state to route-local hook 2026-05-13 14:08:47 -06:00
Richard R
41036e1341 refactor(html): move reader state to route-local hook 2026-05-13 13:40:17 -06:00
Richard R
3a0ed999ac refactor(reader,epub,providers): unify layout structure and improve resize handling
Adopt new layout files for app, epub, html, and pdf routes to standardize
Next.js nested layout structure. Refactor SegmentsSidebar to accept an explicit
epubBookRef prop, decoupling it from internal context. Enhance EPUB resize hook
to ignore initial baseline rect and avoid unnecessary TTS interruptions on load.
Simplify Providers by removing legacy context nesting and conditional logic.
2026-05-13 13:32:37 -06:00
Richard R
c8a35e505f feat(admin,providers,contexts): integrate react-query for admin panels and shared provider state
Adopt @tanstack/react-query for data fetching and cache management in admin
settings, provider management, shared provider hooks, and context providers.
Replace legacy useState/useEffect data loading with react-query's useQuery and
useMutation patterns. Refactor document, rate-limit, and shared provider
contexts to use query keys and cache invalidation for consistent state across
the app. Add QueryClientProvider to root providers. Update package.json to
include react-query dependency.
2026-05-13 12:40:58 -06:00
Richard R
913c6d5d76 refactor(tts,config,admin): update provider resolution logic and ordering for consistency
Remove fallbackProviderRef usage from settings and config context to streamline
provider resolution. Adjust normalizeLegacyProviderRef to retain 'default-openai'
unless a non-default fallback is explicitly provided. Update provider listing to
enforce deterministic ordering by updatedAt, createdAt, and slug. Expand unit tests
to verify provider ordering, legacy ref normalization, and selection logic.
2026-05-13 12:08:39 -06:00
Richard R
b4f4d43d6a feat(tts,config,types): migrate to providerRef/providerType model and add centralized TTS provider policy
Transition all TTS-related logic, types, and UI to use the new providerRef/providerType model in place of legacy ttsProvider fields. Introduce a centralized tts-provider-policy module to encapsulate provider/model capability checks, default value resolution, and compatibility logic. Update all API routes, contexts, hooks, components, and tests to use providerRef and providerType, ensuring consistent handling of built-in and shared TTS providers. Remove legacy defaultTtsModel config in favor of per-provider defaults and shared provider admin control. Add the showAllProviderModels runtime flag to restrict users to provider default models when desired.

BREAKING CHANGE: ttsProvider fields are replaced by providerRef/providerType throughout the codebase; defaultTtsModel config is removed in favor of per-provider defaults.
2026-05-13 10:11:05 -06:00
Richard R
876ca7d774 feat(admin,config,ui,db): introduce runtime-editable site config and admin-managed TTS providers
Add database-backed runtime configuration for feature flags and TTS provider credentials, editable via a new admin UI panel. Replace static NEXT_PUBLIC_* and TTS provider env vars with admin-managed settings stored in the database and injected at SSR for client access. Implement admin-only panels for managing shared TTS provider credentials (with encrypted API keys) and live site feature flags. Add schema migrations, API routes, React contexts, and hooks for SSR-injected runtime config and live updates. Update client and server logic to resolve configuration from the database at runtime, enforcing admin restrictions and supporting migration from legacy env-based config.

BREAKING CHANGE: Feature flags and TTS provider credentials are now managed at runtime via the admin UI. Environment variables are only used for initial seeding and are ignored after first boot. Existing deployments must migrate configuration to the admin panel.
2026-05-12 22:39:24 -06:00
Richard R
c648d2bf6a refactor(api,tts,types): remove legacy /api/tts endpoint and update error handling
Eliminate the deprecated `/api/tts` route and its client usage, consolidating TTS generation logic under chapter and segment endpoints. Update client API utilities to remove direct TTS buffer fetching and improve error propagation for chapter generation failures. Extend TTS segment error metadata to include upstream rate limit and retry details, and standardize error typing in shared types. Adjust tests and helpers to reflect the new TTS mocking approach and endpoint structure. This streamlines TTS API surface and enhances error reporting consistency.
2026-05-12 18:40:46 -06:00
Richard R
d7e6a7b7ba fix(api,rate-limit): centralize daily quota exceeded response and update imports
Move daily quota exceeded problem response logic to a dedicated utility in `lib/server/rate-limit/problem-response`. Refactor API routes to use the new builder and remove duplicated formatting code. Update all relevant imports for consistency. This improves maintainability and ensures uniform error responses across TTS endpoints.
2026-05-12 18:24:39 -06:00
Richard R
08d9218ce4 refactor(client,tts,html): migrate EPUB TTS utilities to client and unify HTML progress logic
Move `epub-word-highlight`, `tts-epub-handoff`, and `tts-epub-preload` from shared to client/epub to clarify their usage and improve bundling. Update all imports to use new client paths. Extend TTS and sidebar logic to support HTML document progress tracking and restoration, introducing a new location format for HTML and consistent handling for non-EPUB readers.

BREAKING CHANGE: EPUB TTS utilities relocated to `lib/client/epub`; update import paths. HTML progress persistence now uses a new format incompatible with previous versions.
2026-05-12 17:51:17 -06:00
Richard R
8488ad37e1 refactor(tts,db,manifest): split tts_segments into normalized entry and variant tables
Redesign TTS segment storage by decomposing the `tts_segments` table into two normalized tables: `tts_segment_entries` for unique segment identity and locator projection, and `tts_segment_variants` for per-settings audio variants. Update schema, queries, and API routes to use the new structure, including manifest pagination and segment resolution logic. Refactor segment locator handling to use stable projections and manifest cursors. Migrate tests and data cleanup utilities to support the new model.

BREAKING CHANGE: Database schema for TTS segments is now split into entry and variant tables; all code and consumers must use the new structure. Persisted data and APIs relying on the old `tts_segments` table are incompatible.
2026-05-12 17:44:57 -06:00
Richard R
f9adba791d feat(tts,epub,db): enforce stable EPUB segment locators and add segmentKey identity
Implement strict validation for EPUB TTS segment locators, requiring stable spine coordinates (`spineHref`, `spineIndex`, `charOffset`) and rejecting legacy CFI-only locators. Introduce `segmentKey` as a normalized identity derived from segment text, used for robust merging and deduplication of synthesized and persisted segments. Update database schemas, API routes, and manifest logic to support the new locator format and segmentKey. Add helpers for locator resolution and segmentKey construction, with comprehensive unit tests for identity, normalization, and manifest grouping.

BREAKING CHANGE: Persisted EPUB TTS segment locators must now include stable spine coordinates; legacy CFI-only locators are no longer accepted. Consumers must handle the new `segmentKey` field for segment identity and merging.
2026-05-12 16:18:36 -06:00
Richard R
e8d4c11434 refactor(tts,epub,locator): introduce canonical segment keys and locator comparison utilities
Refactor TTS segment identification to support canonical segment keys, decoupling segment IDs from locator and index for more robust deduplication and handoff. Add `segmentKey` to segment types, manifest, and ensure route. Implement natural sorting for EPUB CFI locations and segment locators using new comparison utilities. Update sidebar and manifest helpers to use locator-aware sorting. Add foundational modules for EPUB word highlighting, TTS segment planning, and EPUB handoff logic. Expand types and tests to cover new canonical segment and locator behaviors.

BREAKING CHANGE: TTS segment IDs now support canonical segment keys; APIs and manifest consumers must handle `segmentKey` and updated locator comparison logic.
2026-05-11 05:17:41 -06:00
Richard R
265e1cb56a feat(tts,config,ui): add advanced segment preloading and block length controls
Introduce configurable segment preloading depth, sentence lookahead, and TTS segment max block length for both PDF and EPUB readers. Add new settings to user preferences, config context, and document settings UI. Refactor TTS segment splitting logic to support per-user max block length and propagate these options through PDF/EPUB adapters and TTS segment generation. Update API, manifest pagination, and segment deduplication for improved performance and correctness. Add tests for new NLP options and manifest logic.

BREAKING CHANGE: TTS segment splitting and manifest APIs now require max block length and preloading parameters; user config schema updated.
2026-05-07 12:26:08 -06:00
Richard R
3862d0b0ad feat(tts): add leading context support for smart sentence splitting
Introduce logic to provide previous text context for TTS sentence splitting,
enabling more accurate prefix stripping and improved handling of segment
boundaries. EPUB and PDF providers now supply previous segment text, and
TTSContext leverages this context to avoid redundant continuation logic.
This enhances TTS playback continuity and reduces repeated or truncated
sentences across segment transitions.
2026-05-04 20:59:14 -06:00
Richard R
45ae2f5000 fix(tts): improve segment grouping and sidebar selection for locator variants
Update segment manifest API to group segments by both index and locator
attributes, ensuring distinct segment variants are not merged incorrectly.
Adjust SegmentsSidebar logic to select the most relevant segment per index
based on locator match and recency, improving sidebar accuracy when
multiple locator variants exist for a segment.
2026-05-04 20:08:51 -06:00
Richard R
4cb482c3f2 refactor(tts): enhance pause handling and centralize TTS test helpers
Update TTSContext to track manual pause epochs, ensuring explicit user
pauses are respected during async sentence splitting and playback state
restoration. Adjust TTSPlayer button disabling logic to allow pausing
while processing. Move and improve pauseTTSAndVerify helper for robust
cross-browser TTS pause assertions, replacing local test implementations.
Update expectDocumentListed to use polling for document link visibility.
2026-05-04 19:38:23 -06:00
Richard R
62103ea752 style(ui): update aria-labels for clarity and consistency in sidebar controls
Standardize aria-labels in DocumentHeaderMenu and ReaderSidebarShell to use
"Hide" and "Close" phrasing for improved accessibility and user experience.
2026-05-04 15:23:48 -06:00
Richard R
71e1472650 feat(reader,db): add segments sidebar and manifest API; update TTS segment schema
Introduce SegmentsSidebar component and supporting reader UI for segment-level
navigation and inspection. Add TTS segments manifest and clear API endpoints to
enable efficient retrieval and management of segment variants. Update TTS
segment schema to include settings_hash for improved lookup and indexing,
and adjust indexes to use settings_hash instead of settings_json. Update
contexts and types to support segment variant display and interaction.

This enhances document navigation and TTS segment management for end users.
2026-05-04 15:02:05 -06:00
Richard R
71cef7b540 refactor(db): migrate tts_segments to settings_json and update indexes
Switch tts_segments schema to use settings_json (jsonb for Postgres, text for SQLite)
instead of settings_hash. Update related indexes and migration SQL. Regenerate
Drizzle snapshots to reflect the new structure. This improves flexibility for
storing TTS settings and enables more robust segment lookups.
2026-05-04 13:22:58 -06:00
Richard R
222d221e81 fix(tts): prevent stale playback by tracking playback run state in TTS context
Introduce `playbackRunIdRef` and `invalidatePlaybackRun` to ensure that playback
state is correctly reset on abort and content changes. Update abort and navigation
handlers to call `invalidatePlaybackRun`, preventing duplicate or stale playback
runs, especially in scenarios with rapid restarts or browser-specific audio event
quirks. This addresses issues where playback could become stuck or behave
unexpectedly due to outdated in-flight guards.
2026-05-04 12:26:29 -06:00
Richard R
7d7e933323 refactor(tts): replace manifest and audio endpoints with presigned and fallback URLs
Remove legacy TTS segment manifest and audio endpoints in favor of new presigned and fallback audio URL approach. Update segment manifest structure to include `audioPresignUrl` and `audioFallbackUrl` fields, deprecating the previous `audioUrl`. Refactor segment ensure logic and TTS context to utilize the new playback source model. Add server utilities for generating presigned S3 URLs and introduce new API routes for presign and fallback audio retrieval. Clean up obsolete client API functions and types to reflect the new manifest format.
2026-05-04 12:04:24 -06:00
Richard R
9999cafec1 feat(tts): add TTS segments API, schema, storage, and manifest endpoints
Introduce TTS segment support with new database tables and migrations for both Postgres and SQLite. Implement segment ensure, audio, and manifest API endpoints, segment storage in S3, and segment authorization logic. Update TTS context and client API to use segment-based synthesis and retrieval. Add server-side helpers for segment ID, settings hash, locator normalization, and audio probing. Extend user data cleanup and teardown logic to remove TTS segment objects. Add unit tests for segment helpers.

BREAKING CHANGE: TTS synthesis now uses segment-based storage and APIs; database migrations required.
2026-05-04 10:18:56 -06:00
Richard R
21030f28eb refactor: update document section title from 'Local Documents' to 'Your Documents' 2026-04-16 20:13:01 -06:00