Commit graph

120 commits

Author SHA1 Message Date
Richard R
943e547836 test: fix JetStream mock typing for tsc 2026-05-26 16:39:40 -06:00
Richard R
f77ee8cfd4 hard-cut batch B: worker-proxy parsed SSE and remove app progress bus 2026-05-26 15:58:14 -06:00
Richard R
df05a7d7a3 refactor(control-plane): hard-cut worker events and shared SSE helpers 2026-05-26 11:41:33 -06:00
Richard R
084cbdfac1 refactor(worker-control-plane): route ops through core orchestrator 2026-05-26 11:16:54 -06:00
Richard R
00fc2d5e36 feat(control-plane): add shared orchestrator and in-memory adapters 2026-05-26 11:09:43 -06:00
Richard R
2df49b54de feat(parse-progress): add local in-memory realtime bus 2026-05-25 21:06:18 -06:00
Richard R
e70b3619e0 tests: harden pdf readiness waits and extend tts/pdf timeouts 2026-05-24 20:58:57 -06:00
Richard R
e75114a943 refactor(onboarding): introduce onboarding state coordinator and central registry
Add useOnboardingCoordinator hook and onboarding-state registry to manage
onboarding-related state such as privacy acceptance and first-visit tracking.
Refactor SettingsModal to utilize the new onboarding state abstractions,
removing legacy privacy gating logic and improving maintainability.

Includes unit tests for onboarding-state registry to ensure correctness.
2026-05-23 17:07:50 -06:00
Richard R
65d25e3ce5 refactor(ui): centralize privacy gating logic for settings modal
Consolidate privacy acceptance checks into reusable hooks and callbacks within
SettingsModal. Replace scattered gating with a single entry point for opening
settings or changelog views, ensuring consistent enforcement. Update test
helpers to robustly dismiss onboarding and settings overlays by avoiding
state race conditions.
2026-05-23 05:50:54 -06:00
Richard R
50c4330ab1 refactor(core,config,tests): update ONNX model manifest loading and test imports
- Replace static JSON imports with runtime manifest loading in pdf and whisper model modules for compatibility with ESM and bundlers
- Refactor Next.js config to clarify compute mode logic and improve worker bundling conditions
- Update test imports to use package entrypoints instead of relative paths
- Remove redundant whisper alignment/model tests now covered elsewhere or by integration
- No breaking changes to public API or model handling logic
2026-05-21 23:49:56 -06:00
Richard R
b2bb36911a test(whisper): fix playwright ESM loading via dynamic imports 2026-05-21 21:53:55 -06:00
Richard R
2fd9f05652 fix(config): load webpack DefinePlugin via createRequire 2026-05-21 21:49:03 -06:00
Richard R
7a49ff9896 chore(lint): fix eslint violations in worker config and e2e tests 2026-05-21 21:44:58 -06:00
Richard R
55acf05ad6 refactor(compute-core): rename pdf and whisper modules to clean capability layout 2026-05-21 21:41:37 -06:00
Richard R
37b6999c2c refactor(compute-core): collapse to root/local-runtime entrypoints 2026-05-21 21:30:44 -06:00
Richard R
37d11bf9b8 refactor(compute): remove legacy app whisper duplicate and unify core whisper boundary 2026-05-21 21:15:10 -06:00
Richard R
1612e6694b refactor(compute-core): capability-boundary exports for runtime and pdf-layout 2026-05-21 21:11:28 -06:00
Richard R
4d12c6913a test(export): streamline export cleanup and improve test reliability
Remove unnecessary UI reload and modal interactions during audiobook export
cleanup to reduce flakiness under load. Set explicit timeout for long-running
MP3 export test to ensure consistent execution. These changes enhance test
stability and maintainability.
2026-05-21 18:50:11 -06:00
Richard R
384369f48b refactor: fix tests + remove deprecated PDF layout processing files and update imports
- Deleted mergeTextWithRegions.ts, LICENSE.txt, manifest.json, parsePdf.ts, renderPage.ts, runLayoutModel.ts, stitchCrossPageBlocks.ts, and types.ts as they are no longer needed.
- Updated unit tests to reflect new import paths from @openreader/compute-core.
- Adjusted tsconfig.json to include new paths for pdf-layout module.
2026-05-21 18:48:17 -06:00
Richard R
dd494d50aa Add force-reparse confirmation flow in PDF UI 2026-05-21 10:34:29 -06:00
Richard R
10748c7fcd Add force-token cache bust for worker PDF op keys 2026-05-21 10:32:18 -06:00
Richard R
3aad8a51e4 Add local compute concurrency limiter with tests 2026-05-21 10:30:36 -06:00
Richard R
4497f610c0 feat(pdf): implement granular PDF parse progress tracking and migrate to parseState
Introduce detailed progress reporting for PDF layout parsing, exposing phase and page-level updates to clients. Replace legacy parseStatus with a structured parseState field in the database schema, updating all relevant backend and API logic. Add SSE endpoint for real-time parse progress updates. Update client hooks and UI to reflect granular progress and improve user feedback during document parsing. Includes migration scripts and new parse-state utility module.
2026-05-21 10:27:16 -06:00
Richard R
b679bf736c chore(config): remove smartSentenceSplitting option and related code paths
Eliminate the smartSentenceSplitting configuration flag from application state, UI, and all TTS and audiobook adapter logic. Consolidate TTS segment planning to always include context units, streamlining code and reducing conditional branches. Update tests and type definitions to reflect the removal of this feature toggle.
2026-05-20 04:51:10 -06:00
Richard R
bb3eb40966 refactor(tts): extend segment prefetching to support source unit arrays and enhance cache key granularity
Update TTS segment prefetching logic to handle arrays of CanonicalTtsSourceUnit for upcoming and next locations, enabling more precise TTS segment mapping. Expand cache key construction to include providerType and instructions, improving cache differentiation for TTS requests. Adjust related types and usages to support richer segment metadata propagation.
2026-05-20 04:44:47 -06:00
Richard R
0798cd845f refactor(export): extract download trigger helper and add retry logic to full download
Move download logic into reusable downloadViaTrigger function. Add retries and
button enabled checks to downloadFullAudiobook for improved robustness against
transient UI or backend delays.
2026-05-20 03:44:09 -06:00
Richard R
8439216dd8 test(pdf-upload): update assertions for PDF viewer heading and page navigation
Replace text-based assertion for PDF title with role-based heading check and
add assertion for visible page navigation button. Improves test robustness
against UI changes and ensures accessibility compliance.
2026-05-19 23:19:53 -06:00
Richard R
84c3b22e5a refactor(pdf-layout): streamline viewer layout settling and harden pdf document loading
Unify and improve layout settling logic across EPUB, HTML, and PDF viewers by
adding delayed recomputation and guarding against zero-height container during
transient states. Refactor PDF document loading to include retry logic with
success detection, ensuring robust handling of transient failures. Update
Playwright test helpers to provide more reliable PDF viewer readiness checks
and introduce request retry utilities for backend API calls. Add targeted unit
tests for single-section chapter fallback in the PDF audiobook adapter. Update
test files and helpers to align with new layout and export logic.
2026-05-19 23:02:41 -06:00
Richard R
f5d7408f17 refactor(pdf-layout): remove compute "none" mode and unsupported parse status
Eliminate the "none" compute mode and all related code paths, including the NoneComputeBackend, "unsupported" parse status, and PDF margin settings. Parsing is now always available if the app starts successfully, and configuration is limited to "local" or "worker" compute modes. Update types, API routes, client adapters, and documentation to reflect this simplification.

BREAKING CHANGE: "none" is no longer a valid COMPUTE_MODE; only "local" and "worker" are supported. "unsupported" parse status and PDF margin settings are removed.
2026-05-19 18:25:27 -06:00
Richard R
874e5ef359 refactor(whisper): migrate word alignment to ONNX backend and remove whisper.cpp integration
Replace the previous whisper.cpp-based word alignment with a fully ONNX-based
implementation using onnxruntime-node and @huggingface/tokenizers. Add new
Whisper ONNX model management, alignment mapping, and spectral analysis modules.
Remove all code and documentation referencing whisper.cpp, update environment
variables, Dockerfile, and docs to reflect ONNX-only alignment. Add unit tests
for alignment and ONNX model logic.
2026-05-19 13:00:21 -06:00
Richard R
6ab5c230c2 feat(pdf): improve cross-page block stitching and enforce TTS segment source boundaries
Enhance PDF cross-page block stitching to move only sentence continuations and
preserve remaining text on the following page. Update TTS segment planning to
support strict source boundary enforcement, ensuring segments do not cross
block or paragraph-title boundaries. Add new tests for both features and
refactor PDF text item normalization to filter out skewed or rotated runs.
2026-05-18 13:21:43 -06:00
Richard R
37a90b734d feat(pdf): migrate to PP-DocLayoutV3 ONNX model and update PDF block taxonomy
Switch PDF layout parsing to use the PP-DocLayoutV3 ONNX model, replacing the previous Docling-based approach. Update all environment variables, model fetching, and manifest handling for the new model and its artifacts. Refactor block kind taxonomy throughout the codebase, tests, and UI to align with PP-DocLayoutV3 labels, including expanded and renamed block types. Revise document settings, block filtering, and stitching logic to support the new set of block kinds. Update documentation and environment variable references to reflect the model transition.

BREAKING CHANGE: PDF parsing now requires PP-DocLayoutV3 ONNX model and updated environment variables; block kind names and settings have changed throughout the system.
2026-05-18 04:26:39 -06:00
Richard R
766c04d08d refactor(pdf): implement ONNX-based Docling layout parsing and block-level TTS for PDFs
Migrate PDF parsing to use ONNX Docling layout model for structured block extraction, enabling improved TTS segmentation and chaptering. Add compute backend abstraction for heavy tasks (alignment, layout parsing) with configuration via `OPENREADER_COMPUTE_MODE`. Integrate block-level locators and document settings for per-document PDF parsing options. Update S3 storage for parsed PDF JSON, add migration and schema changes, and extend client and server APIs for parsed data and settings. Remove legacy word highlight flag in favor of compute capability detection.

- Add ONNX model download, local/none compute modes, and `onnxruntime-node` dependency
- Update PDF viewer and TTS pipeline to use parsed blocks and block-level locators
- Add document settings storage and APIs for per-document PDF options
- Update S3 storage layout for parsed PDF JSON
- Add admin/config/docs updates for new compute and parsing features
- Remove obsolete word highlight runtime flag and UI

BREAKING CHANGE: PDF parsing and word highlighting now require `OPENREADER_COMPUTE_MODE=local` and ONNX model; document settings and S3 layout updated; legacy word highlight flag removed.
2026-05-17 21:18:51 -06:00
Richard R
d90e48aaf3 feat(html): add HTML/TXT/MD sentence and word highlighting, settings, and audiobook export support
Introduce sentence and word highlighting for HTML/TXT/MD documents, including new config options (`htmlHighlightEnabled`, `htmlWordHighlightEnabled`) and synced user preferences. Update the HTML viewer to support these highlights using custom spans, and add corresponding CSS styles. Extend document settings to allow toggling these features. Implement block-level locator sorting and display improvements. Add audiobook export support for HTML documents, including adapter and pipeline integration.

- Add `htmlHighlightEnabled` and `htmlWordHighlightEnabled` to config, user state, and Dexie storage
- Update HTML viewer to apply and manage highlights for sentences and words
- Add CSS for HTML highlight classes and scroll margin
- Extend document settings UI for HTML highlight toggles
- Support HTML audiobook export in modal and pipeline
- Improve block locator sorting and sidebar display for HTML
- Add HTML audiobook adapter and block parsing utilities
2026-05-17 11:14:02 -06:00
Richard R
522540452c feat(auth): add runtime toggle for user sign-ups and enforce signup policy
Introduce `enableUserSignups` runtime setting to allow administrators to control
whether new accounts can be created. Update environment variable and documentation
references to support this feature. UI elements for account creation are now
conditionally rendered based on this flag. Signup attempts are blocked server-side
when disabled, including email, OAuth, and anonymous upgrades.

Add `assertUserSignupAllowed` utility for consistent enforcement and corresponding
unit tests to verify policy behavior.
2026-05-14 12:47:05 -06:00
Richard R
205d8f37e4 fix(tests): enhance onboarding modal dismissal logic in helpers 2026-05-14 12:06:12 -06:00
Richard R
1f548f71ea feat(audiobook): integrate settings coercion utility and migrate metadata handling
Refactor chapter API to use new coerceAudiobookGenerationSettings utility for
validating and migrating audiobook metadata. Add src/lib/server/audiobooks/settings.ts
with shared logic and introduce corresponding unit tests to ensure correct
settings migration and validation.
2026-05-14 11:34:16 -06:00
Richard R
0787d2e88a refactor(settings): extract changelog check logic to reusable utility
Move changelog version check effect from SettingsModal to a new
scheduleChangelogCheck utility for improved modularity and testability.
Add unit tests for changelog check scheduling behavior.
2026-05-14 10:58:32 -06:00
Richard R
9e09f80067 feat(changelog): add changelog feed support and settings integration
Introduce changelog feed manifest support, including client and shared
utilities for fetching and parsing changelog data. Add a new
Settings modal panel for viewing changelog entries, with version
detection based on the current app version. Expose a configurable
changelog feed URL in both environment variables and admin panel
runtime settings. Update documentation and deployment workflow to
support changelog feed generation and consumption. Include unit tests
for changelog utilities.
2026-05-14 09:45:04 -06:00
Richard R
f300dfb01f Retry errored TTS segments with cooldown and retry-after 2026-05-14 08:31:40 -06:00
Richard R
df841210a3 Remove dead segment audio presign route and legacy helpers 2026-05-14 08:01:53 -06:00
Richard R
513128b802 Stream fallback segment audio and support byte-range requests 2026-05-14 07:55:48 -06:00
Richard R
da629b3ecd Enable private caching headers for segment audio routes 2026-05-13 23:44:13 -06:00
Richard R
ef2ee09064 Warm segment audio during preload before playback 2026-05-13 23:39:29 -06:00
Richard R
16bf52f889 Use direct signed segment audio URLs in ensure/manifest 2026-05-13 23:33:00 -06:00
Richard R
d5eebd3b11 chore(config): remove Deepinfra model gating and related admin setting
Eliminate the showAllDeepInfraModels runtime/admin config and all code paths
that allowed restricting Deepinfra's model catalog. Deepinfra now always
shows the full model list regardless of API key or environment variable.
Update documentation, environment examples, admin panels, runtime config,
provider catalog logic, and tests to reflect this change.
2026-05-13 18:42:40 -06:00
Richard R
fdeeb60f7e refactor epub reader hooks and add unit coverage 2026-05-13 16:56:13 -06:00
Richard R
913c6d5d76 refactor(tts,config,admin): update provider resolution logic and ordering for consistency
Remove fallbackProviderRef usage from settings and config context to streamline
provider resolution. Adjust normalizeLegacyProviderRef to retain 'default-openai'
unless a non-default fallback is explicitly provided. Update provider listing to
enforce deterministic ordering by updatedAt, createdAt, and slug. Expand unit tests
to verify provider ordering, legacy ref normalization, and selection logic.
2026-05-13 12:08:39 -06:00
Richard R
b4f4d43d6a feat(tts,config,types): migrate to providerRef/providerType model and add centralized TTS provider policy
Transition all TTS-related logic, types, and UI to use the new providerRef/providerType model in place of legacy ttsProvider fields. Introduce a centralized tts-provider-policy module to encapsulate provider/model capability checks, default value resolution, and compatibility logic. Update all API routes, contexts, hooks, components, and tests to use providerRef and providerType, ensuring consistent handling of built-in and shared TTS providers. Remove legacy defaultTtsModel config in favor of per-provider defaults and shared provider admin control. Add the showAllProviderModels runtime flag to restrict users to provider default models when desired.

BREAKING CHANGE: ttsProvider fields are replaced by providerRef/providerType throughout the codebase; defaultTtsModel config is removed in favor of per-provider defaults.
2026-05-13 10:11:05 -06:00
Richard R
876ca7d774 feat(admin,config,ui,db): introduce runtime-editable site config and admin-managed TTS providers
Add database-backed runtime configuration for feature flags and TTS provider credentials, editable via a new admin UI panel. Replace static NEXT_PUBLIC_* and TTS provider env vars with admin-managed settings stored in the database and injected at SSR for client access. Implement admin-only panels for managing shared TTS provider credentials (with encrypted API keys) and live site feature flags. Add schema migrations, API routes, React contexts, and hooks for SSR-injected runtime config and live updates. Update client and server logic to resolve configuration from the database at runtime, enforcing admin restrictions and supporting migration from legacy env-based config.

BREAKING CHANGE: Feature flags and TTS provider credentials are now managed at runtime via the admin UI. Environment variables are only used for initial seeding and are ignored after first boot. Existing deployments must migrate configuration to the admin panel.
2026-05-12 22:39:24 -06:00