Refactor EPUB and HTML word highlighting to use a shared, position-preserving
text normalization pipeline for mapping TTS alignment offsets to DOM positions.
Replace the previous fuzzy token-matching logic with a direct char→DOM map
based on the canonical "audio" text form, ensuring highlights stay in sync
across languages and with Unicode-aware hyphenation handling.
- Add `highlight-char-map.ts` for shared normalization logic
- Update normalization to handle Unicode hyphenation consistently across
compute, client, and shared layers
- Remove unused fuzzy token mapping and cache logic from EPUB highlighting
- Update tests for new highlight mapping and normalization behavior
BREAKING CHANGE: Removes legacy fuzzy token-based word highlight mapping in favor of direct char offset mapping; highlight cache and tokenization APIs are removed.
* refactor(user): overhaul user data cleanup and export for cascading deletes and TTS segment support
Revise user data cleanup logic to ensure proper cascading deletion of user-related
database rows and S3 objects, including shared document and preview artifacts.
Introduce explicit checks for last ownership before removing shared resources.
Add TTS segment cache and audio cleanup to document and user deletion flows.
Expand user data export to include TTS segment entries and audio, job events,
document settings, and linked auth sessions. Update schema with ON DELETE CASCADE
for userTtsChars and userJobEvents. Add new migration scripts and comprehensive
unit tests for cleanup and export scenarios.
* test(user): expand cleanup and export coverage for shared docs and TTS segment files
Add tests for user data cleanup with shared document ownership and for TTS segment
variant export scenarios, including cases with duplicate audio keys and storage
disabled. Refactor cleanup logic to use document row types and transactional
ownership removal with last-owner checks. Update TTS segment cache clearing to
report deleted segment count by unique entry. Adjust export logic to always
include all TTS segment variants and conditionally export file buckets based on
storage availability. Update migration to use NOT VALID/VALIDATE for new
cascading constraints.
* refactor(documents): centralize owned document deletion and add mutation locking
Introduce `deleteOwnedDocument` utility to encapsulate all logic for removing a user's ownership of a document, including TTS segment cache cleanup, preview artifact removal, and S3 blob deletion for last-owner cases. Add `withDocumentMutationLock` to serialize concurrent mutations on the same document, using advisory locks in Postgres and a local queue fallback. Refactor all API routes and user data flows to use these utilities, ensuring transactional safety and preventing race conditions during document deletion or transfer. Update tests for new flows and add coverage for document cleanup sequencing and locking behavior.
* feat(data): enhance anonymous claim to support document settings and TTS segment copy
Expand the anonymous data claim process to include document settings transfer and TTS segment S3 prefix copying. Remove foreign key constraints from user_tts_chars to allow non-user buckets. Update claim modal and onboarding flow to display claimed document settings. Refactor TTS char count transfer to merge all dates and fix upsert logic. Add S3 copy utility for TTS segments and corresponding tests. Update migrations and schema to reflect relaxed constraints.
* refactor(data): improve error handling and rollback for document and TTS segment operations
Enhance robustness of document deletion and TTS segment transfer by improving
error logging, partial rollback, and degraded state reporting. Add best-effort
cleanup and recovery mechanisms to prevent orphaned data and ensure diagnostic
information is captured for unexpected failures. Update user data cleanup to
log and swallow restoration errors without masking primary failures. Refine
claim logic to handle unmapped TTS audio keys safely.
* refactor(data): unify document mutation locking and transaction handling
Replace mutation-lock with document-lock to centralize document mutation
serialization and database transaction management. Introduce runInDbTransaction
utility to abstract SQLite/Postgres transaction differences. Update document
deletion, user data cleanup, and rate limiter logic to delegate transaction
handling and locking to shared helpers. Remove dialect-specific branching and
inline transaction logic for improved maintainability and testability.
BREAKING CHANGE: withDocumentMutationLock is removed in favor of withDocumentLock and runInDbTransaction
* fix(documents): improve rollback error handling in deleteOwnedDocument
Enhance error handling during document deletion by ensuring that failures
in restoring document ownership do not obscure the original error. Log
degraded events when ownership restoration fails after a deletion error,
providing additional context for debugging and monitoring. This change
improves reliability and traceability of document deletion operations.
* feat(tasks): introduce scheduled task engine and admin UI for background jobs
Add a general-purpose scheduled task system with a persistent registry and status tracking, supporting background maintenance jobs such as orphaned blob reaping, expired upload cleanup, job event pruning, and TTS usage retention. Implement a new `scheduled_tasks` table, task engine, and handlers for each maintenance operation. Integrate an admin UI panel for monitoring, manual runs, and configuration of tasks. Update document and user data cleanup flows to delegate shared blob and preview deletion to the scheduled reaper. Add Vercel cron integration for serverless environments.
BREAKING CHANGE: Document and user storage cleanup now relies on background scheduled tasks for shared blob and preview deletion; immediate inline deletion is no longer performed.
* refactor(user): streamline document and TTS segment transfer logic
Simplify user document transfer by consolidating storage and metadata handling. Remove inline deletion of shared blobs; only metadata is moved, and TTS segment transfer is controlled via options. Add `skipStorage` option for test scenarios to bypass storage operations. Update tests to reflect new transfer behavior.
* chore(instrumentation): delegate node-specific setup to separate module
Move Node.js-specific instrumentation logic to a dedicated file. Update
registration to dynamically import the node module only when running in a
Node.js environment. This separation clarifies environment-specific behavior
and improves maintainability.
* ci(config): update scheduled task cron to run daily at midnight
Change cron schedule for /api/admin/tasks/tick from hourly to once daily at
midnight to reduce task frequency and align with updated operational
requirements.
* refactor(tasks): enforce positive interval and improve scheduled task updates
Add database-level check constraints to ensure scheduled task intervals are always
positive for both Postgres and SQLite. Update admin API and UI to support sub-minute
intervals and stricter validation. Refactor scheduled task update logic to upsert
rows, ensuring tasks can be updated even if not yet present in the database.
Improve temporary upload cleanup to delete in paginated batches. Enhance tests to
cover new constraints and update behaviors.
* chore(docker): remove ffmpeg-static from runtime dependencies in image build
Eliminate ffmpeg-static from the production node_modules during Docker image
assembly to streamline the deployment artifact and avoid bundling unused binaries.
* refactor(admin): redesign task panel UI with Card layout and running indicator
Replace the bordered div layout in AdminTasksPanel with the Card component for
improved visual hierarchy. Add a dynamic running indicator using a custom
RunningDot component to clearly show active tasks. Update control alignment and
button states for better usability. Remove Badge-based status display in favor
of a more streamlined appearance.
* feat(tasks): add document blob lease for safe orphan reaping and improve scheduled task robustness
Introduce a document blob lease mechanism to prevent race conditions between document registration and orphaned blob cleanup. The new `document_blob_leases` table ensures that only one process can claim a document blob for mutation or deletion at a time. Update the orphan reaper to acquire a lease before deleting blobs and to re-check for ownership after acquiring the lease, avoiding accidental deletion of in-flight uploads.
Enhance scheduled task infrastructure with per-task fencing tokens to prevent stale runners from overwriting newer results, enforce runtime limits with abort signals, and expose scheduler mode and minimum interval to the admin panel and API. Adjust task handlers to accept a context with abort support, and update documentation and environment variable references for the new cron secret and scheduling behavior.
BREAKING CHANGE: Scheduled tasks now require a `document_blob_leases` table and updated handler signatures. Vercel deployments must set `CRON_SECRET` for scheduled maintenance.
* refactor(admin): add loading skeleton to tasks panel for improved UX
Introduce a TasksSkeleton component to display animated placeholders while scheduled tasks are loading. Replace direct rendering of empty task rows with the skeleton when data is pending, enhancing perceived responsiveness and user experience in the admin tasks panel.
* test: clean up playwright anonymous users
* test: make blob lease expiry deterministic
* fix(documents): improve blob lease release error handling and test stale lease scenario
Handle errors during blob lease release by logging warnings instead of allowing them to mask original results. Update unit tests to verify that releasing a stale lease does not affect a replacement lease.
* refactor(documents): implement exponential backoff with jitter for blob lease retries
Replace fixed retry delay with capped exponential backoff and jitter to reduce
contention and thundering herd effect when acquiring document blob leases. This
improves lease acquisition fairness and efficiency under high load.
Move pdf.js worker and standard font resolution logic into dedicated
utility functions for improved maintainability and clarity. Replace
inline workerSrc setup with a reusable configurePdfjsNodeRuntime helper.
Update Next.js config to use a shared asset trace list for pdf.js runtime
files, reducing duplication and simplifying future updates.
Move encodeParserVersion to a dedicated module for unified access and remove local duplicates. Update all consumers to import from the new entrypoint. Tighten artifact readiness checks in both client and API route to ensure 'ready' is only reported when the artifact is accessible, with retries and stricter validation. Add tests for edge cases and operation state validation.
Standardize import of PDF_PARSER_VERSION across server modules to reference
@openreader/compute-core/api-contracts. This clarifies versioning boundaries
and improves maintainability by consolidating contract exports.
This change removes legacy server-side PDF parse state management and transitions
to a fully worker-owned model for PDF parsing operations. Key updates include:
- Deletes all code related to server-managed parse state, including:
- parse-state.ts, parse-state-backfill.ts, parse-state-healing.ts,
parsed-pdf-reuse.ts, pdf-parse-operation.ts, and related job logic
- Removes the user-pdf-layout-job queue and associated job logic
- Refactors API routes for parsed PDF documents and events to use the new
worker-owned PDF parse operation flow under src/lib/server/pdf-parse/
- Updates S3 parsed PDF artifact keying to include parser version for
deduplication and compatibility
- Refactors client API and hooks to handle new error and progress reporting
- Removes all tests for the legacy parse state and job system, updating remaining
tests to mock the new worker-owned flow
BREAKING CHANGE: Server no longer manages per-user PDF parse state or jobs.
All PDF parsing is now managed by the compute worker and new artifact keying.
Legacy parse state and jobs are no longer supported. Existing parsed PDFs may
need to be reprocessed for compatibility with the new model.
Add PDF_PARSER_VERSION constant and propagate parser versioning throughout
the PDF parsing, job, and API layers. Implement normalization of parse state
to ensure compatibility with the current parser version, and enable reuse of
parsed PDF results when possible. Add isPlaybackReady state to document hooks
and TTS player, improving playback UX by disabling controls until content is
ready. Update tests to reflect new playback readiness logic.
BREAKING CHANGE: PDF parse state and job logic now require explicit parserVersion;
older parse states may be treated as pending until reprocessed.
Replace dynamic require.resolve lookup with direct path construction to
standard_fonts directory under node_modules. Removes use of createRequire
and streamlines runtime font path resolution logic for PDF parsing.
Refactor Dockerfile to optimize layering, embed entrypoint migration tools and compute worker as deployable bundles, and merge only required dependencies for runtime scripts. Add docker/entrypoint-migration-tools package and update workspace configuration. Refactor PDF parsing to resolve standard font data path robustly at runtime. Update dependencies and output config for Next.js standalone mode. Enhance entrypoint script to launch embedded compute worker from deployed bundle.
- Dockerfile now deploys migration tools and compute worker as separate bundles
- compute/core/pdf uses new pdfjs-runtime utility for robust font path resolution
- docker/entrypoint-migration-tools added for migration script dependencies
- scripts/openreader-entrypoint.mjs launches embedded worker from deployed bundle
- next.config.ts enables standalone output and includes standard_fonts in tracing
- Dependency updates across package.json and compute/worker/package.json
BREAKING CHANGE: Docker image structure and runtime entrypoint logic have changed; custom deployment scripts or Docker overrides may require updates.
* refactor(ui): replace anchor tags with next/link in SidebarNavLink and UserMenu
Update SidebarNavLink to use next/link for navigation instead of anchor tags,
ensuring proper routing and improved accessibility in Next.js. Refactor UserMenu
to remove legacy Link wrappers and directly use SidebarNavLink for signin and
signup links. This streamlines navigation components and aligns with Next.js
best practices.
* fix(pdf): handle non-zero viewport origins and improve layout model ordering
Update normalizeTextItemsForLayout to correctly apply viewport transforms,
including non-zero page origins, ensuring accurate mapping of PDF text items
to top-left coordinates. Refactor runLayoutModel to implement a custom order
sequence builder for layout regions using model order logits, improving
region ordering consistency with model semantics. Update related tests to
cover viewport transform edge cases.
* refactor(pdf): improve text normalization with font ascent and vertical overlap logic
Enhance text normalization by incorporating font ascent and descent data to more accurately position glyphs, especially for decorative initials. Update merge logic to better detect line membership using vertical overlap, ensuring drop caps and overlapping glyphs are merged correctly. Extend tests to cover these layout scenarios.
* style(range): redesign slider with precision gauge and ruler ticks
Revamp the range input to feature a minimalist "precision gauge" style.
Introduce a hairline rail, ruler notches for discrete steps, and a slim
needle thumb. Add CSS variables and logic for per-instance tick sizing
and coloring. Remove bulky inline class-based styling in favor of
centralized CSS for improved maintainability and visual clarity.
* refactor(ui): modularize PDF loader and range slider visuals
Move PDF layout scan visualization and range slider styles into dedicated CSS
modules, isolating their styles from the global scope. Integrate PdfLayoutScan
component into the PDF viewer loader UI for animated parse progress. Refactor
progress bars to use a reusable progress-fill class with animated sheen effect.
Update range input to use CSS module for precision gauge styling.
* style(reader): remove grid overlay from PdfLayoutScan visualization
* refactor(api): add staleness detection for inflight worker operation states
Integrate isWorkerOperationStateStale checks into document parse API endpoints to ensure inflight worker operation states are not reused if stale. Introduce helper for staleness detection and corresponding unit tests. Enhance SSE event streaming with keepalive intervals and improve progress acknowledgment error handling.
* feat(worker): recover and fail stale in-flight pdf ops on startup
Add orphaned operation recovery logic to detect and mark stale in-flight pdf_layout jobs as failed during worker startup. Extend OperationStateStore with listOpStates for state enumeration. Update tests and documentation to cover recovery behavior and new environment variable COMPUTE_PDF_JOB_ATTEMPTS.
* refactor(worker): distinguish staleness thresholds for running and queued pdf ops
Update orphan recovery logic to apply separate timeouts for 'running' and
'queued' pdf_layout operations. Adjust tests to verify that only stale
'running' operations are failed, while stale 'queued' operations remain
untouched.
* feat(worker): extend orphan recovery to handle whisper_align ops and improve logging
Update orphan recovery to detect and fail stale 'running' whisper_align operations
in addition to pdf_layout. Refactor recovery logic to generalize staleness checks
across operation kinds and enhance log output with detailed operation info.
Expand tests to verify correct handling of both whisper_align and pdf_layout
operations in running and queued states.
* refactor(control-plane): introduce revision-based CAS for operation state updates
Add revision tracking and compare-and-set (CAS) semantics to operation state
stores, enabling atomic state transitions and preventing lost updates. Extend
the OperationStateStore interface with getOpStateRecord and compareAndSetOpState
methods. Update orchestrator and worker runtime to utilize CAS for marking
operations as failed only if the state is unchanged. Enhance in-memory,
JetStream, and test control plane implementations to support revision logic.
This change improves concurrency safety and correctness of operation state
management across distributed components.
* feat(ui): add parse failure state to PDF layout scan animation
Display a distinct "parse halted" visual state in the PDF layout scan
component and PDF viewer page when parsing fails. The loader animation
is replaced by a static, dimmed page with an alert glyph and updated
styling, ensuring users are not misled by an active animation after a
failure. CSS and component logic updated to support the new state.
* feat(worker): extract orphaned operation recovery to module with periodic sweep
Move orphaned operation recovery logic into a dedicated orphan-recovery module,
introducing a periodic sweep timer that triggers recovery every 15 seconds while
the worker is connected. Refactor runtime to delegate orphan detection and
handling to the new module, improving modularity and maintainability. Add
unit tests for orphan-recovery to ensure correctness.
* fix(ui): adjust PDF viewer layout and update parse loader description
* refactor(pdf): streamline layout model region extraction and update test coverage
- Replace custom order sequence logic with softmax-based class selection in runLayoutModel
- Remove unused sigmoid and buildOrderSequence functions
- Simplify detection loop to filter and map regions directly
- Add targeted tests for layout model extraction logic
- Update CSS animation naming for consistency
- Clarify test description and add inline comments for orphan recovery scenario
* fix(pdf): add strict validation for layout model output shapes and extend test coverage
Add explicit error handling for invalid or inconsistent pred_boxes and logits array lengths in runLayoutModel to prevent silent failures. Expand test suite to verify correct region filtering and error scenarios, ensuring only labeled regions are returned and malformed outputs are handled robustly.
* docker fixes
* fix(worker): default COMPUTE_PREWARM_MODELS to false and update docs, env, and compose
Set COMPUTE_PREWARM_MODELS to false by default in runtime, docker-compose,
and .env.example to avoid pre-downloading ONNX models during worker startup.
Update Dockerfile and deployment documentation to reflect this change and
clarify configuration for model prewarming.
* fix(docker): downgrade pnpm version to 10.33.4 for compatibility
---------
Co-authored-by: Richard R <me@richardr.dev>
move worker startup logic from server.ts to runtime.ts for better modularity
add worker-loop-policy.ts for loop control abstraction
delete legacy unit tests under tests/unit/ related to compute worker and control-plane
add vitest config and test directories for compute-core and compute-worker
add vitest workflow for CI
update package.json scripts for new test commands and add vitest as dev dependency
BREAKING CHANGE: worker entrypoint is now compute/worker/src/runtime.ts instead of server.ts; legacy Playwright-based unit tests for compute worker are removed in favor of Vitest
Introduce per-page progress events for PDF parsing by adding onPageStarted
hooks throughout the compute worker and core PDF pipeline. Add
compute/worker/src/pdf-progress.ts to encapsulate progress event builders.
Update server logic to emit page start and page parsed events, improving
client feedback during long-running PDF jobs.
Add src/lib/server/compute/abort-like-error.ts and integrate isAbortLikeError
to robustly detect and suppress expected aborts in API event streaming.
Update API route to ignore abort-like errors on worker proxy crash.
Add unit tests for abort-like error detection and PDF progress event
generation. Refactor PDF parse logic to allow empty merged regions without
throwing, ensuring downstream flows remain resilient to edge cases.
These changes improve observability, error handling, and reliability for
document parsing and event streaming.
Introduce `serverLogger` utility based on pino for consistent, structured logging across all server and API modules. Replace direct console logging with `serverLogger` and add request-scoped logging helpers. Update environment variable handling, documentation, and deployment guides to reflect new logging configuration (`LOG_FORMAT`, `LOG_LEVEL`, `COMPUTE_LOG_LEVEL`). Enforce no-console in server code via ESLint and add pino/pino-pretty dependencies.
This change standardizes log output, improves observability, and prepares the codebase for ingestion by log platforms.
Expand documentation and example env files to explicitly distinguish which
environment variables belong in the app/root `.env` versus the `compute/worker/.env*`
files when deploying in external worker mode. This clarifies routing/auth
settings versus worker runtime configuration, reducing misconfiguration risk.
Update all configuration, manifest, and documentation references to use the q4
Whisper ONNX model files instead of int8. Adjust expected file names, hashes,
and environment variable descriptions to reflect this new default. This aligns
runtime, deployment, and developer documentation with the updated model artifact
expectations for improved consistency.
Refactored docstore directory logic to dynamically locate the monorepo
root by searching for a pnpm-workspace.yaml marker. If found, the
docstore directory is anchored at the monorepo root; otherwise, it
defaults to the current working directory. This enhances consistency
when running in different environments.
Also updated process shutdown and SeaweedFS launch arguments in the
entrypoint script for improved reliability and log filtering.
Eliminates the START_EMBEDDED_COMPUTE_WORKER environment variable from configuration,
documentation, and entrypoint logic. Embedded compute worker startup is now strictly
determined by the absence of COMPUTE_WORKER_URL. If embedded mode is triggered,
the presence of the nats-server binary is mandatory; otherwise, an error is thrown.
Documentation and example env files have been updated to reflect this streamlined
behavior and clarify requirements for both embedded and external compute worker setups.
Move shared type contracts from `contracts` to new `api-contracts` module in `compute/core` for clearer API boundaries. Remove legacy `contracts`, `pdf`, and `whisper` index files. Update all imports to use `api-contracts`. Refactor server job logic by replacing `parsePdfJob.ts` with modular `user-pdf-layout-job.ts` and `user-whisper-align-job.ts`, updating all relevant API routes and compute integration. This improves maintainability and separation of concerns across compute and server layers.
Enhance worker operation handling by adding detailed logging for operation
acceptance, reuse, replacement, and stale detection, including opKey hashes and
reasons for replacement. Remove the alignMutex from Whisper alignment logic to
simplify concurrency control and reduce unnecessary serialization of alignment
requests. These changes improve observability and throughput for alignment and
layout parsing jobs.
Introduce COMPUTE_NATS_REPLICAS environment variable to control the number of
replicas for JetStream streams and key-value buckets. Only values 1, 3, or 5
are accepted, defaulting to 1 for other inputs. Update worker server logic and
deployment documentation to describe and support this configuration.
Add lazy NATS connection lifecycle and idle disconnect logic to the worker,
enabling automatic disconnection after 120s of inactivity to support Railway
container sleep. Update deployment docs to describe this behavior, including
caveats and reconnection flow.
Move and re-export all PDF parsing and TTS-related types from
@openreader/compute-core/types, eliminating local type duplication.
Update imports throughout the codebase to use the new centralized type
module. Remove obsolete worker-contract file and update tsconfig paths
and compute-core exports for the new types entry point. Refactor API
routes and job logic to use the new type imports and shared parse-state
healing utility. This streamlines type management and improves
consistency between compute and app layers.
Introduce automatic healing for stale PDF parse states in API routes. Add
`isDocumentParseStateStale` utility and use a shared `COMPUTE_OP_STALE_MS`
environment variable (with config getter) to control the stale window for both
worker op replacement and app-side parse-state healing. Update documentation and
environment variable references to reflect new config.
Enables automatic marking of stuck parses as failed, improving reliability and
retry behavior in distributed compute environments.
- Replace static JSON imports with runtime manifest loading in pdf and whisper model modules for compatibility with ESM and bundlers
- Refactor Next.js config to clarify compute mode logic and improve worker bundling conditions
- Update test imports to use package entrypoints instead of relative paths
- Remove redundant whisper alignment/model tests now covered elsewhere or by integration
- No breaking changes to public API or model handling logic
Replace static timeout constants with functions that derive alignment and ffmpeg
decode timeouts from compute timeout config. This enables adaptive timeout
behavior based on runtime configuration, improving robustness under varying
resource constraints. Update error handling and internal APIs to propagate
timeout values accordingly.
Remove unused Node.js engine constraint from package.json for broader compatibility.
