Commit graph

498 commits

Author SHA1 Message Date
Richard R
fb32dea73c build(config): adjust output and file tracing for Vercel and local compute modes
Update Next.js configuration to conditionally set 'output' only when not on
Vercel, and always define 'outputFileTracingExcludes' with dynamic includes
based on compute mode. This ensures compatibility with Vercel deployments and
improves local/server build behavior.
2026-05-20 15:53:26 -06:00
Richard R
944b8246e5 refactor(context): expose provider config type and instruction flags in TTS value
Update TTS context to include configProviderType, supportsInstructions, and
ttsInstructions in the provided value, improving downstream access to provider
capabilities and instruction handling.
2026-05-20 15:42:45 -06:00
Richard R
88809b826b fix(compute): unify PDF layout result handling with object key support
Update compute backend interfaces and job logic to consistently handle parsed
PDF layout results as either direct data or S3 object key references. Refactor
result types, local and worker backends, and job processing to support this
pattern. Improves flexibility for large document parsing and object storage
integration.
2026-05-20 15:38:24 -06:00
Richard R
4ddedee8ec docs(worker): document and expose JetStream storage limits for jobs and KV state
Add `COMPUTE_JOBS_STREAM_MAX_BYTES` and `COMPUTE_JOB_STATES_MAX_BYTES` to
environment example, worker server configuration, and deployment docs. These
variables allow tuning JetStream resource caps for job queue and state storage
to better fit deployment requirements.
2026-05-20 15:34:17 -06:00
Richard R
1bd13c02fe refactor(compute): modularize core contracts and runtime, enable flexible backend resolution
Restructure compute core by extracting job contracts and runtime logic into separate
modules (`contracts.ts`, `local-runtime.ts`) for improved modularity and clearer
API boundaries. Update exports and imports across worker, server, and local
compute backends to use the new modules. Enhance Next.js config to support
dynamic backend selection via build-time constants and aliasing, enabling
seamless switching between local and worker compute modes. Adjust TypeScript
paths and Dockerfile entrypoint to align with the new structure.
2026-05-20 15:07:22 -06:00
Richard R
8310091816 docs(deploy): update compute worker and Vercel deployment guides for Synadia + Railway
Add a comprehensive Synadia Cloud and Railway setup guide to the compute worker
deployment docs, including step-by-step environment variable configuration and
example image tags. Update Vercel deployment instructions with a quick start for
external compute worker integration via Railway and Synadia NGS. Document new
`.env.prod` pattern in .gitignore for production environment management.
2026-05-20 14:17:07 -06:00
Richard R
3a1dd41d91 refactor(build): update Dockerfiles for improved dependency isolation and workspace handling
Remove legacy .npmrc config. Revise root and worker Dockerfiles to enhance
layer caching, explicitly copy only required build artifacts, and streamline
runtime dependencies. Switch to pnpm workspace-aware install for compute worker
to ensure correct dependency resolution. Set Next.js output to standalone mode
for optimized server builds. Update docker-compose sync targets for consistency.
2026-05-20 13:21:12 -06:00
Richard R
b30304a119 docs(worker): document NATS credentials options and update authentication support
Expand compute worker documentation to explain NATS authentication using
either a credentials file or raw credentials string, including Synadia Cloud
integration steps. Update environment variable examples and server connection
logic to support both `NATS_CREDS_FILE` and `NATS_CREDS`. Add `.creds` to
.gitignore to avoid accidental credential leaks.

This improves deployment flexibility for cloud and on-prem environments by
supporting multiple authentication methods for NATS JetStream.
2026-05-20 12:25:13 -06:00
Richard R
4b26531172 chore(worker): migrate compute worker from Redis/BullMQ to NATS JetStream
Replace Redis and BullMQ with NATS JetStream and KV for job queuing and state management
in the compute worker service. Update environment variables, Docker Compose, and documentation
to reflect the new message queue backend. Adjust dependencies and code to use NATS-based
work queue and key-value storage for durable job processing.

This change improves scalability and reliability of distributed compute workloads by leveraging
NATS JetStream's work queue and persistence features. Documentation and configuration examples
are updated to guide deployments using the new backend.
2026-05-20 07:38:03 -06:00
Richard R
502c98f11f refactor(tts): add segment cache clearing and improve sidebar scroll handling
Introduce clearSegmentCaches to TTS context for explicit cache invalidation after segment clearing. Enhance SegmentsSidebar scroll logic to better distinguish user-initiated and programmatic scrolls, improving UX during segment updates.
2026-05-20 04:59:09 -06:00
Richard R
b679bf736c chore(config): remove smartSentenceSplitting option and related code paths
Eliminate the smartSentenceSplitting configuration flag from application state, UI, and all TTS and audiobook adapter logic. Consolidate TTS segment planning to always include context units, streamlining code and reducing conditional branches. Update tests and type definitions to reflect the removal of this feature toggle.
2026-05-20 04:51:10 -06:00
Richard R
bb3eb40966 refactor(tts): extend segment prefetching to support source unit arrays and enhance cache key granularity
Update TTS segment prefetching logic to handle arrays of CanonicalTtsSourceUnit for upcoming and next locations, enabling more precise TTS segment mapping. Expand cache key construction to include providerType and instructions, improving cache differentiation for TTS requests. Adjust related types and usages to support richer segment metadata propagation.
2026-05-20 04:44:47 -06:00
Richard R
9d1f9d5707 fix(pdf): readd source unit extraction for TTS integration in page parsing
Introduce sourceUnitsFromParsedPage helper to extract CanonicalTtsSourceUnit
objects from parsed PDF pages, filtering by block kind and text content. Update
TTS state setter to include extracted source units when available, enabling
improved TTS segment mapping and future extensibility.
2026-05-20 03:53:25 -06:00
Richard R
0798cd845f refactor(export): extract download trigger helper and add retry logic to full download
Move download logic into reusable downloadViaTrigger function. Add retries and
button enabled checks to downloadFullAudiobook for improved robustness against
transient UI or backend delays.
2026-05-20 03:44:09 -06:00
Richard R
8439216dd8 test(pdf-upload): update assertions for PDF viewer heading and page navigation
Replace text-based assertion for PDF title with role-based heading check and
add assertion for visible page navigation button. Improves test robustness
against UI changes and ensures accessibility compliance.
2026-05-19 23:19:53 -06:00
Richard R
84c3b22e5a refactor(pdf-layout): streamline viewer layout settling and harden pdf document loading
Unify and improve layout settling logic across EPUB, HTML, and PDF viewers by
adding delayed recomputation and guarding against zero-height container during
transient states. Refactor PDF document loading to include retry logic with
success detection, ensuring robust handling of transient failures. Update
Playwright test helpers to provide more reliable PDF viewer readiness checks
and introduce request retry utilities for backend API calls. Add targeted unit
tests for single-section chapter fallback in the PDF audiobook adapter. Update
test files and helpers to align with new layout and export logic.
2026-05-19 23:02:41 -06:00
Richard R
01cb95d8e7 fix(compute): migrate compute backend initialization to async and update consumers
Refactor compute backend initialization to use asynchronous loading and promise caching, replacing previous synchronous singleton pattern. Update all consumers—including TTS segment alignment and PDF layout parsing jobs—to await compute backend resolution before invoking methods. This change improves compatibility with dynamic imports and future-proofs backend selection logic.
2026-05-19 19:38:36 -06:00
Richard R
e52b754106 refactor(pdf): remove legacy text extraction and update parse status normalization
Eliminate unused PDF text extraction logic and margin-based filtering from the client library. Refactor API routes to generalize parse status normalization, removing explicit references to deprecated "unsupported" status and type variants. Update usages to accept string parse statuses for improved flexibility. Simplify document state dependency tracking in the PDF document hook.
2026-05-19 19:11:51 -06:00
Richard R
f5d7408f17 refactor(pdf-layout): remove compute "none" mode and unsupported parse status
Eliminate the "none" compute mode and all related code paths, including the NoneComputeBackend, "unsupported" parse status, and PDF margin settings. Parsing is now always available if the app starts successfully, and configuration is limited to "local" or "worker" compute modes. Update types, API routes, client adapters, and documentation to reflect this simplification.

BREAKING CHANGE: "none" is no longer a valid COMPUTE_MODE; only "local" and "worker" are supported. "unsupported" parse status and PDF margin settings are removed.
2026-05-19 18:25:27 -06:00
Richard R
bab3416a36 refactor(compute): add timing metrics to worker job results and status
Introduce WorkerJobTiming interface to capture queue wait, S3 fetch, and compute
durations for compute worker jobs. Update job result and status response types
to include timing data. Enhance worker implementation to record and attach
timing metrics for alignment jobs, improving observability of job processing
latency. Update exports and type usage to support new timing fields.
2026-05-19 15:55:35 -06:00
Richard R
ea5611c882 docs(intro): update highlights and backend details in docs and README
Revise project highlights in README and introduction to emphasize ONNX-based
PDF layout parsing (PP-DocLayoutV3), segment-based TTS playback, and flexible
backend/storage options. Clarify compute modes and self-hosting features.
Update TTS provider descriptions for accuracy and conciseness.

Also expand tsconfig exclude list to omit compute worker sources from main build.
2026-05-19 15:32:18 -06:00
Richard R
f1aa1c3e3b feat(compute): add external compute worker backend and integration
Introduce support for external compute worker mode (`COMPUTE_MODE=worker`) using a new `WorkerComputeBackend`. This enables offloading heavy ONNX Whisper alignment and PDF layout parsing to a standalone worker service (Redis + BullMQ), improving scalability and compatibility with serverless/limited environments.

- Add `@openreader/compute-core` as a shared package for ONNX inference and PDF parsing logic.
- Implement `WorkerComputeBackend` and worker contract/types for remote job execution.
- Update compute backend selection logic and remove previous worker mode guards.
- Extend `WhisperAlignInput` and `PdfLayoutInput` types to support object keys for remote data access.
- Refactor local compute backend to use `@openreader/compute-core` and support both buffer and object key inputs.
- Update job runner, TTS segment alignment, and PDF layout parsing flows to use new compute backend APIs.
- Add scripts, Docker workflow, and documentation for deploying and running the compute worker.
- Update environment variable docs and examples for worker mode, including storage requirements and configuration.
- Document published images and stack changes to reflect the new compute worker architecture.

BREAKING CHANGE: `COMPUTE_MODE=worker` now requires an external compute worker service and S3-compatible object storage. Embedded SeaweedFS (`weed mini`) is not supported in worker mode. See the new documentation for deployment and configuration details.
2026-05-19 15:21:25 -06:00
Richard R
3a21f2a5f5 refactor(config): migrate compute and runtime env vars to new naming scheme
- Replace `OPENREADER_COMPUTE_MODE` and related variables with `COMPUTE_MODE`
- Replace `OPENREADER_*` PDF/Whisper model URLs with `PDF_LAYOUT_MODEL_BASE_URL` and `WHISPER_MODEL_BASE_URL`
- Remove legacy `NEXT_PUBLIC_*` runtime config seeds in favor of `RUNTIME_SEED_*`
- Update documentation, code, and environment references to match new variable names
- Remove deprecated `scripts/fetch-models.mjs` and related npm script
- Update runtime config SSR injection from `window.__OPENREADER_RUNTIME_CONFIG__` to `window.__RUNTIME_CONFIG__`
- Adjust Next.js config to use new compute mode env var and optimize output file tracing for ONNX dependencies

BREAKING CHANGE: Environment variable names for compute mode, model URLs, and runtime config seeding have changed. Update `.env` files and deployment configs to use `COMPUTE_MODE`, `PDF_LAYOUT_MODEL_BASE_URL`, `WHISPER_MODEL_BASE_URL`, and `RUNTIME_SEED_*` as appropriate. Legacy `OPENREADER_*` and `NEXT_PUBLIC_*` variables are no longer supported.
2026-05-19 13:27:07 -06:00
Richard R
874e5ef359 refactor(whisper): migrate word alignment to ONNX backend and remove whisper.cpp integration
Replace the previous whisper.cpp-based word alignment with a fully ONNX-based
implementation using onnxruntime-node and @huggingface/tokenizers. Add new
Whisper ONNX model management, alignment mapping, and spectral analysis modules.
Remove all code and documentation referencing whisper.cpp, update environment
variables, Dockerfile, and docs to reflect ONNX-only alignment. Add unit tests
for alignment and ONNX model logic.
2026-05-19 13:00:21 -06:00
Richard R
6ab5c230c2 feat(pdf): improve cross-page block stitching and enforce TTS segment source boundaries
Enhance PDF cross-page block stitching to move only sentence continuations and
preserve remaining text on the following page. Update TTS segment planning to
support strict source boundary enforcement, ensuring segments do not cross
block or paragraph-title boundaries. Add new tests for both features and
refactor PDF text item normalization to filter out skewed or rotated runs.
2026-05-18 13:21:43 -06:00
Richard R
90e7caac43 refactor(previews): streamline preview generation and update image variant to 400px
Remove temporary working directory usage and in-place file writes from preview
generation, simplifying logic for both PDF and EPUB previews. Change default
preview image variant from 480px to 400px width, updating related constants and
cache schema version to maintain consistency across client and server code.
2026-05-18 12:37:26 -06:00
Richard R
8af4e857b7 feat(pdf): add block geometry-based highlighting using parsed document and locator
Implement block-level highlighting in PDF viewer by leveraging parsed document
structure and TTS segment locator. Extend highlightPattern API to accept options
for parsedDocument, locator, and block geometry mode. Update PDFViewer to use
block geometry highlighting when available, improving accuracy and alignment of
sentence and word highlights with ONNX-based DocLayoutV3 parsing. Refactor
highlighting logic to support both legacy span-based and new geometry-based
approaches.
2026-05-18 12:25:41 -06:00
Richard R
1544a2d969 refactor(previews): update preview variant defaults, cache schema, and image handling
Remove hardcoded defaults for preview variant and width in database schemas to allow more flexible preview generation. Bump preview cache schema version and add versioning to cache rows for consistency. Switch preview image generation to 480px JPEGs with updated file naming and content type. Refactor PDF preview rendering to support configurable output format and quality, and update all preview consumers to use unified "image" buffer property instead of "png". Add migration scripts and update metadata for both Postgres and SQLite.

BREAKING CHANGE: Preview variant and width defaults removed from database; preview cache and image handling updated across system.
2026-05-18 05:20:20 -06:00
Richard R
37a90b734d feat(pdf): migrate to PP-DocLayoutV3 ONNX model and update PDF block taxonomy
Switch PDF layout parsing to use the PP-DocLayoutV3 ONNX model, replacing the previous Docling-based approach. Update all environment variables, model fetching, and manifest handling for the new model and its artifacts. Refactor block kind taxonomy throughout the codebase, tests, and UI to align with PP-DocLayoutV3 labels, including expanded and renamed block types. Revise document settings, block filtering, and stitching logic to support the new set of block kinds. Update documentation and environment variable references to reflect the model transition.

BREAKING CHANGE: PDF parsing now requires PP-DocLayoutV3 ONNX model and updated environment variables; block kind names and settings have changed throughout the system.
2026-05-18 04:26:39 -06:00
Richard R
766c04d08d refactor(pdf): implement ONNX-based Docling layout parsing and block-level TTS for PDFs
Migrate PDF parsing to use ONNX Docling layout model for structured block extraction, enabling improved TTS segmentation and chaptering. Add compute backend abstraction for heavy tasks (alignment, layout parsing) with configuration via `OPENREADER_COMPUTE_MODE`. Integrate block-level locators and document settings for per-document PDF parsing options. Update S3 storage for parsed PDF JSON, add migration and schema changes, and extend client and server APIs for parsed data and settings. Remove legacy word highlight flag in favor of compute capability detection.

- Add ONNX model download, local/none compute modes, and `onnxruntime-node` dependency
- Update PDF viewer and TTS pipeline to use parsed blocks and block-level locators
- Add document settings storage and APIs for per-document PDF options
- Update S3 storage layout for parsed PDF JSON
- Add admin/config/docs updates for new compute and parsing features
- Remove obsolete word highlight runtime flag and UI

BREAKING CHANGE: PDF parsing and word highlighting now require `OPENREADER_COMPUTE_MODE=local` and ONNX model; document settings and S3 layout updated; legacy word highlight flag removed.
2026-05-17 21:18:51 -06:00
Richard R
d90e48aaf3 feat(html): add HTML/TXT/MD sentence and word highlighting, settings, and audiobook export support
Introduce sentence and word highlighting for HTML/TXT/MD documents, including new config options (`htmlHighlightEnabled`, `htmlWordHighlightEnabled`) and synced user preferences. Update the HTML viewer to support these highlights using custom spans, and add corresponding CSS styles. Extend document settings to allow toggling these features. Implement block-level locator sorting and display improvements. Add audiobook export support for HTML documents, including adapter and pipeline integration.

- Add `htmlHighlightEnabled` and `htmlWordHighlightEnabled` to config, user state, and Dexie storage
- Update HTML viewer to apply and manage highlights for sentences and words
- Add CSS for HTML highlight classes and scroll margin
- Extend document settings UI for HTML highlight toggles
- Support HTML audiobook export in modal and pipeline
- Improve block locator sorting and sidebar display for HTML
- Add HTML audiobook adapter and block parsing utilities
2026-05-17 11:14:02 -06:00
Richard R
92762fe858 ci(docs): separate release-triggered deploy and enhance changelog feed sync
Extract release-based deploy triggers to new workflow for clarity and
reliability. Update docs-deploy.yml to streamline dispatch inputs and
remove release/workflow_run triggers. Extend build-changelog-feed.mjs to
support incremental changelog sync via workflow_dispatch with release tag
and action, enabling precise upserts and deletions of changelog entries.
2026-05-14 17:14:43 -06:00
github-actions[bot]
3645b7a327 docs: snapshot v3.0.0 2026-05-14 22:32:52 +00:00
Richard R
ff62cc702e chore(config): downgrade pnpm version and refine allowBuilds exclusions
Some checks failed
Create and publish Docker images / prepare (push) Has been cancelled
Version Docs on Tag / version-docs (push) Has been cancelled
Create and publish Docker images / build (amd64, linux/amd64, ubuntu-24.04) (push) Has been cancelled
Create and publish Docker images / build (arm64, linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Create and publish Docker images / merge (push) Has been cancelled
Revert packageManager to pnpm@10.33.4 in package.json for compatibility.
Expand allowBuilds in pnpm-workspace.yaml to explicitly exclude canvas,
core-js, es5-ext, esbuild, sharp, and unrs-resolver from workspace builds.
These adjustments enhance build reliability and prevent issues with
problematic dependencies.
2026-05-14 16:22:07 -06:00
Richard R
64182e6c8d chore(workspace): streamline monorepo and CI with pnpm workspace, dependency, and workflow updates
Update GitHub Actions workflows to latest action versions and improve pnpm usage by introducing per-package workspace configs. Add docs-site/pnpm-workspace.yaml and update Dockerfile/package.json for better monorepo support. Upgrade dependencies across main and docs-site packages for compatibility and security. Refine TypeScript types and hooks for improved type safety and maintainability.
2026-05-14 15:44:22 -06:00
Richard R
288af0f370 build(deps): update pnpm to v11.1.2 and enforce workspace settings
Upgrade pnpm version to 11.1.2 across Dockerfile, workflows, and package.json.
Add pnpm-workspace.yaml to enable workspace features. Update all docs-related
scripts and CI jobs to use --ignore-workspace for isolated docs-site installs.
Set packageManager field in package.json for explicit version control.

This ensures consistent dependency management and prepares for multi-package
workspace support.
2026-05-14 14:25:29 -06:00
Richard R
80f9386a6c docs(admin-panel): clarify admin provider key handling and AUTH_SECRET rotation
Expand admin panel documentation to detail the transition from live env var
reading to one-shot seeding for TTS providers in v3.0.0. Add warnings about
the impact of rotating AUTH_SECRET on encrypted admin provider keys and the
required manual steps after rotation.

docs(migrations): add schema migration history table

Document the sequence and purpose of each migration introduced in v3.0.0,
clarifying upgrade behavior from v2.2.0 and summarizing schema changes.

chore(env): remove commented default TTS model example from .env.example
2026-05-14 13:56:53 -06:00
Richard R
ca4228970b fix(server): enforce server-only usage for runtime config resolvers
Add explicit runtime checks to prevent runtime config functions from being called on the client. This guards against accidental client-side invocation by throwing errors if executed outside the server environment.
2026-05-14 13:07:43 -06:00
Richard R
fb39723089 fix(config): extract RSC-compatible runtime config resolver
Move runtime config resolution logic to a new `runtime-config-rsc.ts` module for use in React Server Components. Update public layout and landing page to consume the RSC-specific resolver, improving compatibility and separation of concerns.
2026-05-14 12:50:16 -06:00
Richard R
522540452c feat(auth): add runtime toggle for user sign-ups and enforce signup policy
Introduce `enableUserSignups` runtime setting to allow administrators to control
whether new accounts can be created. Update environment variable and documentation
references to support this feature. UI elements for account creation are now
conditionally rendered based on this flag. Signup attempts are blocked server-side
when disabled, including email, OAuth, and anonymous upgrades.

Add `assertUserSignupAllowed` utility for consistent enforcement and corresponding
unit tests to verify policy behavior.
2026-05-14 12:47:05 -06:00
Richard R
fef3775ad4 style(ui): update modal gradient backgrounds and adjust settings header layout
Refine radial gradient backgrounds in Settings and DocumentSettings modals to use
CSS color-mix with accent variable for improved theming. Adjust Settings modal
header spacing and layout for better alignment and consistency.
2026-05-14 12:14:55 -06:00
Richard R
c3fb3af3d3 refactor(components): unify button styling using buttonClass utility across multiple components 2026-05-14 12:06:21 -06:00
Richard R
205d8f37e4 fix(tests): enhance onboarding modal dismissal logic in helpers 2026-05-14 12:06:12 -06:00
Richard R
1f548f71ea feat(audiobook): integrate settings coercion utility and migrate metadata handling
Refactor chapter API to use new coerceAudiobookGenerationSettings utility for
validating and migrating audiobook metadata. Add src/lib/server/audiobooks/settings.ts
with shared logic and introduce corresponding unit tests to ensure correct
settings migration and validation.
2026-05-14 11:34:16 -06:00
Richard R
0dee3c7b60 chore(release): bump version to 3.0.0
Update package version to 3.0.0 to reflect recent feature additions and
refactorings.
2026-05-14 11:05:47 -06:00
Richard R
0787d2e88a refactor(settings): extract changelog check logic to reusable utility
Move changelog version check effect from SettingsModal to a new
scheduleChangelogCheck utility for improved modularity and testability.
Add unit tests for changelog check scheduling behavior.
2026-05-14 10:58:32 -06:00
Richard R
beb854cb39 feat(preferences): add changelog version tracking and payload meta support
Integrate changelog version check on settings open, triggering the changelog
panel when a new app version is detected. Refactor user preferences storage to
support metadata via payload utilities, enabling version tracking and future
extensibility. Add API endpoint for changelog version check and related tests
for changelog and preferences payload logic.
2026-05-14 10:53:38 -06:00
Richard R
5efb29cda9 refactor(ui): restructure settings modal navigation and changelog panel logic
Update SettingsModal to use conditional rendering for the changelog panel,
displaying the main settings navigation only when the changelog is closed.
Refactor mobile and desktop navigation layout to improve maintainability.
Add missing ChevronRightIcon import for icon consistency.
2026-05-14 10:01:51 -06:00
Richard R
9e09f80067 feat(changelog): add changelog feed support and settings integration
Introduce changelog feed manifest support, including client and shared
utilities for fetching and parsing changelog data. Add a new
Settings modal panel for viewing changelog entries, with version
detection based on the current app version. Expose a configurable
changelog feed URL in both environment variables and admin panel
runtime settings. Update documentation and deployment workflow to
support changelog feed generation and consumption. Include unit tests
for changelog utilities.
2026-05-14 09:45:04 -06:00
Richard R
f300dfb01f Retry errored TTS segments with cooldown and retry-after 2026-05-14 08:31:40 -06:00