Commit graph

22 commits

Author SHA1 Message Date
Richard R
f5d7408f17 refactor(pdf-layout): remove compute "none" mode and unsupported parse status
Eliminate the "none" compute mode and all related code paths, including the NoneComputeBackend, "unsupported" parse status, and PDF margin settings. Parsing is now always available if the app starts successfully, and configuration is limited to "local" or "worker" compute modes. Update types, API routes, client adapters, and documentation to reflect this simplification.

BREAKING CHANGE: "none" is no longer a valid COMPUTE_MODE; only "local" and "worker" are supported. "unsupported" parse status and PDF margin settings are removed.
2026-05-19 18:25:27 -06:00
Richard R
f1aa1c3e3b feat(compute): add external compute worker backend and integration
Introduce support for external compute worker mode (`COMPUTE_MODE=worker`) using a new `WorkerComputeBackend`. This enables offloading heavy ONNX Whisper alignment and PDF layout parsing to a standalone worker service (Redis + BullMQ), improving scalability and compatibility with serverless/limited environments.

- Add `@openreader/compute-core` as a shared package for ONNX inference and PDF parsing logic.
- Implement `WorkerComputeBackend` and worker contract/types for remote job execution.
- Update compute backend selection logic and remove previous worker mode guards.
- Extend `WhisperAlignInput` and `PdfLayoutInput` types to support object keys for remote data access.
- Refactor local compute backend to use `@openreader/compute-core` and support both buffer and object key inputs.
- Update job runner, TTS segment alignment, and PDF layout parsing flows to use new compute backend APIs.
- Add scripts, Docker workflow, and documentation for deploying and running the compute worker.
- Update environment variable docs and examples for worker mode, including storage requirements and configuration.
- Document published images and stack changes to reflect the new compute worker architecture.

BREAKING CHANGE: `COMPUTE_MODE=worker` now requires an external compute worker service and S3-compatible object storage. Embedded SeaweedFS (`weed mini`) is not supported in worker mode. See the new documentation for deployment and configuration details.
2026-05-19 15:21:25 -06:00
Richard R
3a21f2a5f5 refactor(config): migrate compute and runtime env vars to new naming scheme
- Replace `OPENREADER_COMPUTE_MODE` and related variables with `COMPUTE_MODE`
- Replace `OPENREADER_*` PDF/Whisper model URLs with `PDF_LAYOUT_MODEL_BASE_URL` and `WHISPER_MODEL_BASE_URL`
- Remove legacy `NEXT_PUBLIC_*` runtime config seeds in favor of `RUNTIME_SEED_*`
- Update documentation, code, and environment references to match new variable names
- Remove deprecated `scripts/fetch-models.mjs` and related npm script
- Update runtime config SSR injection from `window.__OPENREADER_RUNTIME_CONFIG__` to `window.__RUNTIME_CONFIG__`
- Adjust Next.js config to use new compute mode env var and optimize output file tracing for ONNX dependencies

BREAKING CHANGE: Environment variable names for compute mode, model URLs, and runtime config seeding have changed. Update `.env` files and deployment configs to use `COMPUTE_MODE`, `PDF_LAYOUT_MODEL_BASE_URL`, `WHISPER_MODEL_BASE_URL`, and `RUNTIME_SEED_*` as appropriate. Legacy `OPENREADER_*` and `NEXT_PUBLIC_*` variables are no longer supported.
2026-05-19 13:27:07 -06:00
Richard R
874e5ef359 refactor(whisper): migrate word alignment to ONNX backend and remove whisper.cpp integration
Replace the previous whisper.cpp-based word alignment with a fully ONNX-based
implementation using onnxruntime-node and @huggingface/tokenizers. Add new
Whisper ONNX model management, alignment mapping, and spectral analysis modules.
Remove all code and documentation referencing whisper.cpp, update environment
variables, Dockerfile, and docs to reflect ONNX-only alignment. Add unit tests
for alignment and ONNX model logic.
2026-05-19 13:00:21 -06:00
Richard R
37a90b734d feat(pdf): migrate to PP-DocLayoutV3 ONNX model and update PDF block taxonomy
Switch PDF layout parsing to use the PP-DocLayoutV3 ONNX model, replacing the previous Docling-based approach. Update all environment variables, model fetching, and manifest handling for the new model and its artifacts. Refactor block kind taxonomy throughout the codebase, tests, and UI to align with PP-DocLayoutV3 labels, including expanded and renamed block types. Revise document settings, block filtering, and stitching logic to support the new set of block kinds. Update documentation and environment variable references to reflect the model transition.

BREAKING CHANGE: PDF parsing now requires PP-DocLayoutV3 ONNX model and updated environment variables; block kind names and settings have changed throughout the system.
2026-05-18 04:26:39 -06:00
Richard R
766c04d08d refactor(pdf): implement ONNX-based Docling layout parsing and block-level TTS for PDFs
Migrate PDF parsing to use ONNX Docling layout model for structured block extraction, enabling improved TTS segmentation and chaptering. Add compute backend abstraction for heavy tasks (alignment, layout parsing) with configuration via `OPENREADER_COMPUTE_MODE`. Integrate block-level locators and document settings for per-document PDF parsing options. Update S3 storage for parsed PDF JSON, add migration and schema changes, and extend client and server APIs for parsed data and settings. Remove legacy word highlight flag in favor of compute capability detection.

- Add ONNX model download, local/none compute modes, and `onnxruntime-node` dependency
- Update PDF viewer and TTS pipeline to use parsed blocks and block-level locators
- Add document settings storage and APIs for per-document PDF options
- Update S3 storage layout for parsed PDF JSON
- Add admin/config/docs updates for new compute and parsing features
- Remove obsolete word highlight runtime flag and UI

BREAKING CHANGE: PDF parsing and word highlighting now require `OPENREADER_COMPUTE_MODE=local` and ONNX model; document settings and S3 layout updated; legacy word highlight flag removed.
2026-05-17 21:18:51 -06:00
Richard R
522540452c feat(auth): add runtime toggle for user sign-ups and enforce signup policy
Introduce `enableUserSignups` runtime setting to allow administrators to control
whether new accounts can be created. Update environment variable and documentation
references to support this feature. UI elements for account creation are now
conditionally rendered based on this flag. Signup attempts are blocked server-side
when disabled, including email, OAuth, and anonymous upgrades.

Add `assertUserSignupAllowed` utility for consistent enforcement and corresponding
unit tests to verify policy behavior.
2026-05-14 12:47:05 -06:00
Richard R
d5eebd3b11 chore(config): remove Deepinfra model gating and related admin setting
Eliminate the showAllDeepInfraModels runtime/admin config and all code paths
that allowed restricting Deepinfra's model catalog. Deepinfra now always
shows the full model list regardless of API key or environment variable.
Update documentation, environment examples, admin panels, runtime config,
provider catalog logic, and tests to reflect this change.
2026-05-13 18:42:40 -06:00
Richard R
b4f4d43d6a feat(tts,config,types): migrate to providerRef/providerType model and add centralized TTS provider policy
Transition all TTS-related logic, types, and UI to use the new providerRef/providerType model in place of legacy ttsProvider fields. Introduce a centralized tts-provider-policy module to encapsulate provider/model capability checks, default value resolution, and compatibility logic. Update all API routes, contexts, hooks, components, and tests to use providerRef and providerType, ensuring consistent handling of built-in and shared TTS providers. Remove legacy defaultTtsModel config in favor of per-provider defaults and shared provider admin control. Add the showAllProviderModels runtime flag to restrict users to provider default models when desired.

BREAKING CHANGE: ttsProvider fields are replaced by providerRef/providerType throughout the codebase; defaultTtsModel config is removed in favor of per-provider defaults.
2026-05-13 10:11:05 -06:00
Richard R
dae97b2afc docs(admin,config,tts): document admin panel for runtime TTS provider and feature config
Add a dedicated Admin Panel documentation page detailing management of shared TTS providers and site features via the admin UI. Update all TTS provider guides, environment variable references, and deployment docs to clarify the new runtime configuration model: environment variables serve as first-boot seeds only, with ongoing management handled through the admin interface. Revise sidebars and cross-links to include the new admin panel docs and clarify the distinction between legacy env-based and admin-managed configuration.
2026-05-12 22:39:37 -06:00
Richard R
0a7d392d77 docs(tts,storage,dev): update documentation for TTS segment storage, schema, and local setup
Expand and clarify documentation for TTS segment storage, including key layout, cleanup, and CLI access in object storage. Add details on the `tts_segments` schema and migration process. Revise local development and Docker quick start guides with improved prerequisites, platform-specific setup instructions, and updated TTS provider references. Document new usage of `AUTH_SECRET` for TTS segment text fingerprinting.
2026-05-04 11:29:02 -06:00
Richard R
5ea9aa7474 docs: add warning about SeaweedFS 4.19 upload issues and pinning to 4.18
Document observed S3 `InternalError` regressions with SeaweedFS 4.19 in
object storage, local development, and Docker quick start guides. Clarify that
version 4.18 is pinned in CI and Docker builds for reliability while
compatibility is investigated.
2026-04-16 17:13:24 -06:00
Richard R
893f74f038 refactor(tts): update Replicate model defaults and voice resolution
Align default Replicate model to the versioned Kokoro model across UI, docs,
and server logic. Refactor TTS settings merging for audiobooks to ensure
consistent normalization. Improve Retry-After header handling for upstream
rate limits. Expand Replicate voice resolution to use model schemas when
available, with test coverage for custom and built-in models.

Update documentation to reflect new Replicate defaults, model selection,
and configuration guidance.
2026-04-16 12:52:08 -06:00
Richard R
9da9232d39 feat: add support for Replicate TTS provider and models
- Updated environment variables documentation to include Replicate as a TTS provider option.
- Added Replicate to the sidebar for TTS provider guides.
- Included Replicate as a dependency in package.json and pnpm-lock.yaml.
- Enhanced audiobook chapter generation to normalize native speed settings based on the TTS provider.
- Improved error handling in TTS API routes to provide retry information for rate-limited responses.
- Updated AudiobookExportModal to reflect native speed support for Replicate models.
- Modified SettingsModal to set default model for Replicate.
- Enhanced SpeedControl component to conditionally render native speed controls based on provider support.
- Updated TTSContext to utilize effective native speed for TTS requests.
- Implemented Replicate request handling in the TTS generation logic.
- Added new documentation for configuring Replicate as a TTS provider.
2026-04-16 11:20:38 -06:00
Richard R
b2a623ba2a feat(ui): add feature flag for TTS Providers tab and update related documentation 2026-03-19 12:25:42 -06:00
Richard R
7482a4265e feat: Enhance authentication handling by adding error logging and improving session checks in the API and ClaimDataModal component 2026-02-19 02:13:43 -07:00
Richard R
a0cc189a0f refactor: rename project from OpenReader WebUI to OpenReader across all relevant files. 2026-02-18 02:49:47 -07:00
Richard R
8f5e87ed4a feat: Replace NEXT_PUBLIC_NODE_ENV with explicit feature flags for granular control over DOCX conversion, destructive actions, TTS defaults, and DeepInfra model visibility. 2026-02-17 18:26:03 -07:00
Richard R
deaacbd6f0 feat(auth): add conditional GitHub authentication support
Implement a check for GitHub OAuth credentials to conditionally render
the GitHub sign-in button. This ensures the UI accurately reflects
available authentication methods based on environment configuration.

- Add `isGithubAuthEnabled` utility to verify server-side credentials
- Propagate GitHub auth status through context providers to the UI
- Conditionally display the GitHub sign-in button in the sign-in page
- Update documentation and examples to use hex encoding for `AUTH_SECRET`
- Add unit tests for the new GitHub authentication configuration logic
2026-02-16 14:21:25 -07:00
Richard R
fa406a8cb2 refactor(deps): remove ffprobe-static and fix playwright flakiness
Replace ffprobe-static with ffmpeg-static for metadata extraction in
migration scripts. This removes the requirement for ffprobe binaries
across Docker, CI, and Vercel environments, simplifying the dependency
graph.
2026-02-16 01:41:54 -07:00
Richard R
4d13b5b821 refactor: improve async state and session reliability
Implement AbortController in ConfigContext, PDFContext, and preference sync to prevent race conditions during session changes or rapid navigation. Refactor document preview routes into shared utilities and add LRU caching for text previews to optimize memory. Update documentation for Better Auth schema ownership and migration workflows. Fix issues with Postgres system user seeding and better-sqlite3 configuration.
2026-02-15 15:13:25 -07:00
Richard R
852092c558 docs: reorganize documentation structure and fix word highlight behavior 2026-02-11 06:03:51 -07:00