Commit graph

50 commits

Author SHA1 Message Date
Richard R
358a56cc9a refactor: implement modular monorepo packages
- Extract database schema and migrations into @openreader/database

- Extract startup and orchestration scripts into @openreader/bootstrap

- Move compute-worker into packages/compute-worker

- Remove runtime dependency on drizzle-kit

- Update Dockerfile to deploy isolated packages without merging node_modules
2026-06-13 13:51:18 -06:00
Richard R
a56aaa2be9 Refactor PDF parsing and rendering logic; migrate to compute-worker protocol
- Removed deprecated functions related to document parsing and blob storage in blobstore.ts.
- Introduced new PDF rendering logic in pdf-preview-renderer.ts and pdf-preview-pdfjs-runtime.ts.
- Updated previews-render.ts to utilize the new PDF rendering functions.
- Refactored user-whisper-align-job.ts to use the compute-worker client for alignment requests.
- Enhanced artifact.ts and operation.ts to validate parsed PDF artifacts and resolve current PDF parses.
- Updated snapshot.ts to align with new worker operation types.
- Adjusted runtime-config.ts to check for compute-worker availability.
- Modified types in parsed-pdf.ts and tts.ts to reflect changes in the compute-worker protocol.
- Added unit tests for PDF artifact validation and compute-worker client contract.
- Removed obsolete pdf-op-key.vitest.spec.ts test file.
2026-06-12 13:43:33 -06:00
Richard R
328dbbbb62 ci(playwright): simplify workflow by removing test matrix for browsers
Eliminate the Playwright test matrix to run all browser projects in a single job.
Consolidate report artifact naming and streamline workflow configuration for
faster and more maintainable CI execution.
2026-06-05 10:40:45 -06:00
Richard R
2f99b3987e ci: update Playwright workflow to improve test execution and reporting 2026-06-05 10:20:49 -06:00
Richard R
5f28be5d58 refactor(env): remove legacy no-auth mode and enforce required auth env vars
Eliminate all code paths, configuration, and documentation related to running
without authentication. Require AUTH_SECRET and BASE_URL at startup, updating
middleware, server logic, and runtime checks to assume auth is always enabled.
Simplify onboarding, settings, and test helpers to reflect mandatory auth.
Update environment examples, Docker and deployment docs, and CI/test configs.
Remove no-auth-specific UI flows, test cases, and feature toggles.
2026-05-31 12:09:37 -06:00
Richard R
aa7700e844 refactor(core): improve streaming, validation, and rate limit logic
- Stream request body in blob upload fallback route to enforce size limits
  without buffering entire payload
- Enhance admin provider panel error handling for multi-status responses
- Adjust positive integer validation to require value >= 1
- Use full rate limit config for job event recording and prune by largest window
- Fix PDF layout job event userId usage in docx-to-pdf upload route
- Add missing windows array fallback in documents register route
- Minor CI workflow and env example corrections
- Update audiobooks blobstore test to use beforeAll directly

These changes improve efficiency, correctness, and maintainability across API, admin, rate limiting, and test modules.
2026-05-30 16:41:17 -06:00
Richard R
36fe5f66a3 ci(workflow): add migration step to vitest CI pipeline
include drizzle migration before running Vitest to ensure database schema
is up to date during test execution
2026-05-30 15:45:42 -06:00
Richard R
b16328abe9 fix(vitest): update test command from 'test:vitest' to 'test:unit' 2026-05-30 11:01:44 -06:00
Richard R
e1483ab80a refactor(worker): modularize worker entrypoint and add vitest infra
move worker startup logic from server.ts to runtime.ts for better modularity
add worker-loop-policy.ts for loop control abstraction
delete legacy unit tests under tests/unit/ related to compute worker and control-plane
add vitest config and test directories for compute-core and compute-worker
add vitest workflow for CI
update package.json scripts for new test commands and add vitest as dev dependency

BREAKING CHANGE: worker entrypoint is now compute/worker/src/runtime.ts instead of server.ts; legacy Playwright-based unit tests for compute worker are removed in favor of Vitest
2026-05-30 11:00:01 -06:00
Richard R
5428a21b67 refactor(onboarding): modularize onboarding flow logic and improve testability
Extract onboarding flow step resolution and async runner logic into a
dedicated module for better separation of concerns and maintainability.
Refactor context/provider to use new helpers and simplify settings modal
control. Add unit tests for onboarding step resolution and async runner
coalescing. Update e2e helpers and selectors for onboarding modals.
Adjust environment file handling for CI and Docker contexts.

- Add .dockerignore for build hygiene
- Refactor onboarding context and modal logic
- Add src/lib/client/onboarding-flow.ts with core onboarding helpers
- Add onboarding-flow unit tests
- Update Playwright config and scripts for CI env loading
- Improve test selectors for claim modal
- Update .gitignore to exclude all .env* except .env.example

This change increases onboarding logic modularity and reliability, and improves CI/test environment handling.
2026-05-29 18:03:17 -06:00
Richard R
ebc1dbcda0 ci(playwright): install nats-server from linux tarball asset 2026-05-26 19:48:55 -06:00
Richard R
b2cf4aa7aa ci(playwright): install nats for embedded worker startup 2026-05-26 19:00:07 -06:00
Richard R
fb0c95dfb1 hard-cut batch D: enforce worker bundle isolation and CI guard 2026-05-26 16:00:26 -06:00
Richard R
f1aa1c3e3b feat(compute): add external compute worker backend and integration
Introduce support for external compute worker mode (`COMPUTE_MODE=worker`) using a new `WorkerComputeBackend`. This enables offloading heavy ONNX Whisper alignment and PDF layout parsing to a standalone worker service (Redis + BullMQ), improving scalability and compatibility with serverless/limited environments.

- Add `@openreader/compute-core` as a shared package for ONNX inference and PDF parsing logic.
- Implement `WorkerComputeBackend` and worker contract/types for remote job execution.
- Update compute backend selection logic and remove previous worker mode guards.
- Extend `WhisperAlignInput` and `PdfLayoutInput` types to support object keys for remote data access.
- Refactor local compute backend to use `@openreader/compute-core` and support both buffer and object key inputs.
- Update job runner, TTS segment alignment, and PDF layout parsing flows to use new compute backend APIs.
- Add scripts, Docker workflow, and documentation for deploying and running the compute worker.
- Update environment variable docs and examples for worker mode, including storage requirements and configuration.
- Document published images and stack changes to reflect the new compute worker architecture.

BREAKING CHANGE: `COMPUTE_MODE=worker` now requires an external compute worker service and S3-compatible object storage. Embedded SeaweedFS (`weed mini`) is not supported in worker mode. See the new documentation for deployment and configuration details.
2026-05-19 15:21:25 -06:00
Richard R
92762fe858 ci(docs): separate release-triggered deploy and enhance changelog feed sync
Extract release-based deploy triggers to new workflow for clarity and
reliability. Update docs-deploy.yml to streamline dispatch inputs and
remove release/workflow_run triggers. Extend build-changelog-feed.mjs to
support incremental changelog sync via workflow_dispatch with release tag
and action, enabling precise upserts and deletions of changelog entries.
2026-05-14 17:14:43 -06:00
Richard R
64182e6c8d chore(workspace): streamline monorepo and CI with pnpm workspace, dependency, and workflow updates
Update GitHub Actions workflows to latest action versions and improve pnpm usage by introducing per-package workspace configs. Add docs-site/pnpm-workspace.yaml and update Dockerfile/package.json for better monorepo support. Upgrade dependencies across main and docs-site packages for compatibility and security. Refine TypeScript types and hooks for improved type safety and maintainability.
2026-05-14 15:44:22 -06:00
Richard R
288af0f370 build(deps): update pnpm to v11.1.2 and enforce workspace settings
Upgrade pnpm version to 11.1.2 across Dockerfile, workflows, and package.json.
Add pnpm-workspace.yaml to enable workspace features. Update all docs-related
scripts and CI jobs to use --ignore-workspace for isolated docs-site installs.
Set packageManager field in package.json for explicit version control.

This ensures consistent dependency management and prepares for multi-package
workspace support.
2026-05-14 14:25:29 -06:00
Richard R
9e09f80067 feat(changelog): add changelog feed support and settings integration
Introduce changelog feed manifest support, including client and shared
utilities for fetching and parsing changelog data. Add a new
Settings modal panel for viewing changelog entries, with version
detection based on the current app version. Expose a configurable
changelog feed URL in both environment variables and admin panel
runtime settings. Update documentation and deployment workflow to
support changelog feed generation and consumption. Include unit tests
for changelog utilities.
2026-05-14 09:45:04 -06:00
Richard R
8349ef7ae8 build(deps): pin SeaweedFS Docker image to 4.18 for stability
Update CI workflow and Dockerfile to use chrislusf/seaweedfs:4.18 instead of
latest. This avoids upload regressions observed with version 4.19 and ensures
consistent builds.
2026-04-16 16:32:48 -06:00
Richard R
230b614b90 feat: enhance docs deployment workflow with conditional execution for workflow runs 2026-02-19 13:02:18 -07:00
Richard R
092fc3ae0d feat: enhance workflow for versioning documentation with optional inputs and improved commit handling 2026-02-19 12:53:01 -07:00
Richard R
5ffacb251f fix: remove legacy image name from Docker outputs in workflow 2026-02-19 11:05:42 -07:00
Richard R
7f9659fd99 chore: Remove unnecessary Pages configuration step from docs deployment workflow 2026-02-19 10:32:28 -07:00
Richard R
7482a4265e feat: Enhance authentication handling by adding error logging and improving session checks in the API and ClaimDataModal component 2026-02-19 02:13:43 -07:00
Richard R
0fbd9d5c38 feat: Update Docker image name to openreader while maintaining openreader-webui as a legacy alias and updating documentation. 2026-02-18 02:54:01 -07:00
Richard R
3ede092adc feat: Make word highlighting enabled by default by inverting its environment variable logic, updating documentation, and removing NEXT_PUBLIC_NODE_ENV from Playwright tests. 2026-02-17 22:12:11 -07:00
Richard R
fa406a8cb2 refactor(deps): remove ffprobe-static and fix playwright flakiness
Replace ffprobe-static with ffmpeg-static for metadata extraction in
migration scripts. This removes the requirement for ffprobe binaries
across Docker, CI, and Vercel environments, simplifying the dependency
graph.
2026-02-16 01:41:54 -07:00
Richard R
f9366dbfa2 refactor(audio): replace ffprobe with ffmpeg for metadata extraction
- Remove ffprobe-static dependency and associated configuration
- Reimplement ffprobeAudio to parse duration and title tags from ffmpeg output
- Update CI workflow to install SeaweedFS binary and set required environment variables
- Enhance E2E test setup to handle auth-protected endpoints during document cleanup
2026-02-15 12:13:29 -07:00
Richard R
260513ebaa ci(config): configure CI environment and optimize function memory
- Set FFMPEG_BIN and FFPROBE_BIN environment variables in Playwright workflow
- Reduce memory allocation for audiobook and whisper API routes to 1024MB
2026-02-15 11:33:46 -07:00
Richard R
91bcf232e1 fix(docs): simplify docs versioning workflow 2026-02-10 15:15:44 -07:00
Richard R
6d5fb65a72 feat(docs): add configurable rate limiting and external docs
Add environment variables for fine-grained control over TTS rate limiting
and Better Auth behavior. Move documentation to external Docusaurus site
with automated deployment workflows.

- TTS rate limiting can now be enabled/disabled via TTS_ENABLE_RATE_LIMIT
- Customizable daily limits for anonymous/authenticated users and IP backstops
- Better Auth rate limiting can be disabled via DISABLE_AUTH_RATE_LIMIT
- Rename library import env vars to IMPORT_LIBRARY_DIRS/DIR
- Add docs-site with Docusaurus and GitHub Actions workflows
- Update README to reference external documentation
2026-02-10 15:11:37 -07:00
Richard R
6016481e0a ci: enhance docker publish workflow for multi-arch support
Add support for building and publishing multi-architecture Docker images (amd64 and arm64) using a matrix strategy. Introduce a new 'prepare' job for metadata computation and a 'merge' job for creating manifest lists. Enable manual workflow dispatch with optional latest tagging. Improve caching and provenance handling for better efficiency and reliability.
2026-01-19 20:58:47 -07:00
Richard R
33d49d6966 fix: Enhance audiobook API validation and prevent critical meta file deletion 2026-01-19 16:40:04 -07:00
Richard Roberson
50538bfaba feat(epub): Add EPUB text highlighting for the current TTS sentence with a new configuration option 2025-11-18 12:31:07 -07:00
Richard Roberson
d1dd3bd351 ci(playwright): add ffmpeg installation and verification 2025-11-15 15:12:06 -07:00
Richard Roberson
21870ed576 fix(api): stabilize DOCX to PDF conversion and cleanup
- Isolate concurrent LibreOffice runs with per-job profile directories
  (soffice -env:UserInstallation) and per-job temp folders
- Poll for generated PDF and verify stable file size before reading
- Consolidate temp artifacts under docstore/tmp and clean up via rm -r
- Add headless/nologo flags and improve error handling

ui(accessibility):
- ConfirmDialog exposes proper dialog roles
- DocumentFolder toggle adds type, aria-expanded, aria-controls, and title;
  associate content panel with an id
- HTMLViewer wraps content in .html-container for HTML/TXT views

test: add comprehensive Playwright specs and helpers
- Accessibility, API health, deletion flows, folders, navigation, playback,
  and upload scenarios
- Add sample.md and unsupported.xyz; update sample.pdf; extend helpers

ci: run Playwright workflow on version1.0.0 branch
2025-11-12 16:21:11 -07:00
Richard Roberson
dad1dfd26f chore(ci): update Playwright workflow to use pnpm and remove .npmrc 2025-11-09 15:56:09 -07:00
Richard Roberson
b21bdb3a21 feat(tts): add multi-provider TTS support with Deepinfra and custom OpenAI-compatible endpoints
Add comprehensive multi-provider TTS support enabling users to choose between OpenAI, Deepinfra, and custom OpenAI-compatible endpoints. Implement provider-specific voice management with automatic voice restoration per provider-model combination, and migrate package manager to pnpm for improved dependency handling.

Key changes:
- Add TTS provider selection (OpenAI, Deepinfra, custom-openai) in settings UI
- Implement provider-specific model and voice lists with dynamic fetching
- Add voice persistence per provider-model combination in savedVoices
- Support Deepinfra models: Kokoro-82M, Orpheus-3B, Sesame-1B with their voice libraries
- Migrate to pnpm with frozen lockfile for reproducible builds
- Update Docker configuration to use pnpm and Deepinfra API defaults
- Add migration logic for existing users to infer provider from stored baseUrl
- Update test helpers and Playwright configuration for Deepinfra API
- Add example docker-compose.yml with Kokoro-FastAPI integration

BREAKING CHANGE: Voice selection is now provider-model specific. Previously saved voices will be migrated to the new savedVoices structure, but users may need to reselect voices if switching providers.
2025-11-09 15:47:17 -07:00
Richard Roberson
b9ce1d6a8f Change tts url for tests 2025-06-22 12:58:11 -06:00
Richard Roberson
43a44abcb2 Fix tests 2025-03-30 21:26:56 -06:00
Richard Roberson
12b46cc8e3 Use smaller libreoffice-writer 2025-03-04 23:47:18 -07:00
Richard Roberson
520236b95c add DOCX support (requires libreoffice) + upload test 2025-03-04 22:47:37 -07:00
Richard Roberson
4e07e737f0 Stop using audio blobs 2025-03-02 21:16:24 -07:00
Richard Roberson
4aaa74414f Update tests 2025-03-02 20:19:52 -07:00
Richard Roberson
136fdb9652 Fix Apple WebKit PDF Upload 2025-03-01 23:36:25 -07:00
Richard Roberson
28a3804843 Setup playwright e2e initial test 2025-03-01 22:44:30 -07:00
Richard Roberson
3568f3c8f9 Update README.md 2025-03-01 19:52:06 -07:00
Richard Roberson
991304d134 Use github packages
Some checks failed
Build and Publish Docker Image / build (push) Has been cancelled
2025-02-26 05:42:57 -07:00
Richard Roberson
868bb88988 Update docker publish flow
Some checks failed
Build and Publish Docker Image / build (push) Has been cancelled
2025-02-21 11:30:06 -07:00
Richard Roberson
c9d799994e Add docker image push action
Some checks failed
Build and Publish Docker Image / build (push) Has been cancelled
2025-02-10 20:50:14 -07:00