Commit graph

557 commits

Author SHA1 Message Date
Richard R
e70b3619e0 tests: harden pdf readiness waits and extend tts/pdf timeouts 2026-05-24 20:58:57 -06:00
Richard R
3f17874fe5 tts ensure: skip whisper alignment in test namespaces 2026-05-24 20:58:48 -06:00
Richard R
b2517bc91d pdf parse: dedupe shared jobs and stop enqueue-on-read 2026-05-24 20:58:34 -06:00
Richard R
12c586d7be refactor(core,server): migrate contracts to api-contracts and modularize job handlers
Move shared type contracts from `contracts` to new `api-contracts` module in `compute/core` for clearer API boundaries. Remove legacy `contracts`, `pdf`, and `whisper` index files. Update all imports to use `api-contracts`. Refactor server job logic by replacing `parsePdfJob.ts` with modular `user-pdf-layout-job.ts` and `user-whisper-align-job.ts`, updating all relevant API routes and compute integration. This improves maintainability and separation of concerns across compute and server layers.
2026-05-24 11:39:25 -06:00
Richard R
e75114a943 refactor(onboarding): introduce onboarding state coordinator and central registry
Add useOnboardingCoordinator hook and onboarding-state registry to manage
onboarding-related state such as privacy acceptance and first-visit tracking.
Refactor SettingsModal to utilize the new onboarding state abstractions,
removing legacy privacy gating logic and improving maintainability.

Includes unit tests for onboarding-state registry to ensure correctness.
2026-05-23 17:07:50 -06:00
Richard R
df321b8832 fix(jobs): make TTS segment cache invalidation non-blocking in parsePdfJob
Move TTS segment cache invalidation to a fire-and-forget promise after
document update to prevent cache errors from blocking parse readiness.
Log warnings for both cache invalidation failures and warnings, but do not
interrupt the main job flow. This improves job robustness and avoids
unnecessary failures due to cache issues.
2026-05-23 05:55:22 -06:00
Richard R
65d25e3ce5 refactor(ui): centralize privacy gating logic for settings modal
Consolidate privacy acceptance checks into reusable hooks and callbacks within
SettingsModal. Replace scattered gating with a single entry point for opening
settings or changelog views, ensuring consistent enforcement. Update test
helpers to robustly dismiss onboarding and settings overlays by avoiding
state race conditions.
2026-05-23 05:50:54 -06:00
Richard R
3dcda2b4b7 refactor(logging): enhance logger configuration and streamline event logging 2026-05-22 13:11:51 -06:00
Richard R
ad82aede73 feat(worker,whisper): improve logging for operation lifecycle and remove align mutex
Enhance worker operation handling by adding detailed logging for operation
acceptance, reuse, replacement, and stale detection, including opKey hashes and
reasons for replacement. Remove the alignMutex from Whisper alignment logic to
simplify concurrency control and reduce unnecessary serialization of alignment
requests. These changes improve observability and throughput for alignment and
layout parsing jobs.
2026-05-22 12:49:33 -06:00
Richard R
653dbedaee refactor(worker): add configurable NATS JetStream replica count
Introduce COMPUTE_NATS_REPLICAS environment variable to control the number of
replicas for JetStream streams and key-value buckets. Only values 1, 3, or 5
are accepted, defaulting to 1 for other inputs. Update worker server logic and
deployment documentation to describe and support this configuration.
2026-05-22 09:32:25 -06:00
Richard R
64a68b1011 docs(worker): document and implement idle-based NATS disconnect for Railway sleep
Add lazy NATS connection lifecycle and idle disconnect logic to the worker,
enabling automatic disconnection after 120s of inactivity to support Railway
container sleep. Update deployment docs to describe this behavior, including
caveats and reconnection flow.
2026-05-22 09:01:14 -06:00
Richard R
0fcf4e2dab style(config): update eslint import rules for compute-core/types subpath
Allow '@openreader/compute-core/types' as a permitted subpath in ESLint
import rules. Update the lint message to reflect the new allowed subpath,
ensuring consistency with recent type centralization efforts.
2026-05-22 02:01:52 -06:00
Richard R
b6be71e3b1 refactor(types,api): centralize PDF and TTS type exports to compute-core/types
Move and re-export all PDF parsing and TTS-related types from
@openreader/compute-core/types, eliminating local type duplication.
Update imports throughout the codebase to use the new centralized type
module. Remove obsolete worker-contract file and update tsconfig paths
and compute-core exports for the new types entry point. Refactor API
routes and job logic to use the new type imports and shared parse-state
healing utility. This streamlines type management and improves
consistency between compute and app layers.
2026-05-22 01:51:51 -06:00
Richard R
09e8898683 feat(pdf,config): implement stale parse-state healing and unify op stale window config
Introduce automatic healing for stale PDF parse states in API routes. Add
`isDocumentParseStateStale` utility and use a shared `COMPUTE_OP_STALE_MS`
environment variable (with config getter) to control the stale window for both
worker op replacement and app-side parse-state healing. Update documentation and
environment variable references to reflect new config.

Enables automatic marking of stuck parses as failed, improving reliability and
retry behavior in distributed compute environments.
2026-05-22 01:21:18 -06:00
Richard R
1d4a301a5d fix(pdf): correct parse status handling and loader display logic
Update PDF viewer and document hook to improve parse status management:
- Set parseStatus to null on clear instead of 'pending' for accurate state
- Explicitly set initial parseStatus and reset parseProgress when loading a new document
- Adjust loader display logic to check for resolved parse status before showing expanded loader

Prevents UI inconsistencies and ensures parse progress is reflected correctly during document transitions.
2026-05-22 00:17:15 -06:00
Richard R
50c4330ab1 refactor(core,config,tests): update ONNX model manifest loading and test imports
- Replace static JSON imports with runtime manifest loading in pdf and whisper model modules for compatibility with ESM and bundlers
- Refactor Next.js config to clarify compute mode logic and improve worker bundling conditions
- Update test imports to use package entrypoints instead of relative paths
- Remove redundant whisper alignment/model tests now covered elsewhere or by integration
- No breaking changes to public API or model handling logic
2026-05-21 23:49:56 -06:00
Richard R
b2bb36911a test(whisper): fix playwright ESM loading via dynamic imports 2026-05-21 21:53:55 -06:00
Richard R
2fd9f05652 fix(config): load webpack DefinePlugin via createRequire 2026-05-21 21:49:03 -06:00
Richard R
7a49ff9896 chore(lint): fix eslint violations in worker config and e2e tests 2026-05-21 21:44:58 -06:00
Richard R
057d2a4e32 chore(eslint): restrict compute-core subpath imports to local-runtime 2026-05-21 21:42:35 -06:00
Richard R
55acf05ad6 refactor(compute-core): rename pdf and whisper modules to clean capability layout 2026-05-21 21:41:37 -06:00
Richard R
3a2c6ffa42 refactor(compute-core): split runtime into config/platform and move contracts barrel 2026-05-21 21:40:16 -06:00
Richard R
37b6999c2c refactor(compute-core): collapse to root/local-runtime entrypoints 2026-05-21 21:30:44 -06:00
Richard R
37d11bf9b8 refactor(compute): remove legacy app whisper duplicate and unify core whisper boundary 2026-05-21 21:15:10 -06:00
Richard R
1612e6694b refactor(compute-core): capability-boundary exports for runtime and pdf-layout 2026-05-21 21:11:28 -06:00
Richard R
d8c6d00de6 docs(env): clarify local vs worker compute env ownership 2026-05-21 20:58:32 -06:00
Richard R
a29e6fc250 docs(compute): align worker env docs and model override vars 2026-05-21 20:57:10 -06:00
Richard R
754fbee227 refactor(compute-core): centralize runtime cpu config exports 2026-05-21 20:54:22 -06:00
Richard R
51fbfa117d worker: avoid idle consumer blocking shared compute gate 2026-05-21 20:06:24 -06:00
Richard R
4664806ba0 worker: enrich single-line request logs with status and opId 2026-05-21 19:48:46 -06:00
Richard R
bcdff35200 worker: simplify request logs to single-line entries 2026-05-21 19:44:25 -06:00
Richard R
661af71fa4 refactor(whisper): implement dynamic timeout configuration for alignment and decoding
Replace static timeout constants with functions that derive alignment and ffmpeg
decode timeouts from compute timeout config. This enables adaptive timeout
behavior based on runtime configuration, improving robustness under varying
resource constraints. Update error handling and internal APIs to propagate
timeout values accordingly.

Remove unused Node.js engine constraint from package.json for broader compatibility.
2026-05-21 19:29:03 -06:00
Richard R
7cdd89b26d compute-core: resolve pdf layout license asset by module path 2026-05-21 19:09:01 -06:00
Richard R
1f13ecaed9 worker: use default JetStream long-poll for consumer next 2026-05-21 19:04:16 -06:00
Richard R
4d12c6913a test(export): streamline export cleanup and improve test reliability
Remove unnecessary UI reload and modal interactions during audiobook export
cleanup to reduce flakiness under load. Set explicit timeout for long-running
MP3 export test to ensure consistent execution. These changes enhance test
stability and maintainability.
2026-05-21 18:50:11 -06:00
Richard R
384369f48b refactor: fix tests + remove deprecated PDF layout processing files and update imports
- Deleted mergeTextWithRegions.ts, LICENSE.txt, manifest.json, parsePdf.ts, renderPage.ts, runLayoutModel.ts, stitchCrossPageBlocks.ts, and types.ts as they are no longer needed.
- Updated unit tests to reflect new import paths from @openreader/compute-core.
- Adjusted tsconfig.json to include new paths for pdf-layout module.
2026-05-21 18:48:17 -06:00
Richard R
3ebcb3e6fd tts: make ensure route abort-aware and reduce abort log noise 2026-05-21 16:46:47 -06:00
Richard R
3c25c93e8b tts: remove redundant route-level alignment timeout wrappers 2026-05-21 16:33:55 -06:00
Richard R
c9dad9c23c compute: reuse shared timeout helpers across worker and local 2026-05-21 16:32:24 -06:00
Richard R
d70c6aa5c6 docs: document shared compute timeout env vars across app and worker 2026-05-21 16:27:03 -06:00
Richard R
e384b66b7d compute: align local and worker-client timeout behavior 2026-05-21 16:17:47 -06:00
Richard R
fbb6f484ce compute: centralize timeout config in compute-core and use in worker 2026-05-21 16:15:34 -06:00
Richard R
0d494cf5e5 docs(config): update PDF job attempts default to 1 for layout parsing
Align COMPUTE_PDF_JOB_ATTEMPTS default value with new ONNX-based PDF layout
parsing approach by setting it to 1 in environment examples, worker config,
and deployment documentation. This reflects improved reliability and reduces
unnecessary retries for PDF processing jobs.
2026-05-21 16:04:17 -06:00
Richard R
f8672b073e refactor(compute): unify job concurrency and CPU thread allocation logic
Standardize compute job concurrency configuration by introducing a shared
COMPUTE_JOB_CONCURRENCY setting, replacing separate PDF and Whisper concurrency
controls. Add cross-platform CPU core detection and thread budgeting utilities,
and update both ONNX model execution and concurrency limiters to use dynamic
thread allocation per job. Refactor environment variable docs and examples to
reflect unified concurrency management. Streamline PDF.js font path resolution
for improved reliability across environments.
2026-05-21 15:58:29 -06:00
Richard R
671d5c6adb feat(pdf,worker): enhance PDF layout parsing UX and add robust timeout controls
Introduce idle and hard cap timeouts for PDF layout parsing in the compute worker,
ensuring long-running jobs are capped and progress is tracked with explicit
timeouts. Refactor NATS JetStream client initialization to use a configurable
API timeout. Update PDF viewer UI to treat parse state as 'pending' by default,
improving clarity during document preparation and parsing. Refine parse loader
labels and progress logic for more accurate user feedback.
2026-05-21 13:50:40 -06:00
Richard R
a2c714c2a5 refactor(worker,tts): unify PDF and Whisper retry config, add alignment timeouts
Update compute worker to use separate environment variable for PDF job attempts
(`COMPUTE_PDF_JOB_ATTEMPTS`) and set Whisper job max deliveries to 1, clarifying
and separating retry logic between job types. Adjust `.env.example` and deployment
docs to comment out defaults and document new/renamed variables for advanced tuning.

In TTS segment ensure route, introduce explicit timeouts for Whisper alignment
operations, with configurable durations for local and worker modes, improving
robustness and error handling for long-running alignments.
2026-05-21 12:37:21 -06:00
Richard R
fe9104e8fc docs(pdf,spinner,config): update parse state UI, spinner theming, and worker env docs
Revise PDF viewer to introduce an explicit 'unknown' parse state and improve
loader and status messaging for better user clarity. Refactor Spinner to use
'currentColor' for border styling and simplify conditional rendering based on
className. Expand compute worker .env.example with explicit concurrency,
timeout, and job attempt variables for enhanced configuration transparency.
2026-05-21 11:59:12 -06:00
Richard R
08df437e61 chore: add .npmrc for npm configuration 2026-05-21 11:18:41 -06:00
Richard R
bde8d74a97 refactor(pdf): improve parse loader UX and PDF viewer readiness signaling
Enhance PDF parse loader transitions by introducing a delayed expansion for detailed progress, reducing UI flicker during parse-to-render handoff. Refactor PDFViewer to ensure document readiness is signaled only once per load, preventing redundant callbacks and improving synchronization between parsing and rendering states.
2026-05-21 10:54:07 -06:00
Richard R
dd494d50aa Add force-reparse confirmation flow in PDF UI 2026-05-21 10:34:29 -06:00