Commit graph

569 commits

Author SHA1 Message Date
Richard R
c0f0438fff hard-cut batch E: embedded nats+worker startup in single-container entrypoint 2026-05-26 16:25:39 -06:00
Richard R
fb0c95dfb1 hard-cut batch D: enforce worker bundle isolation and CI guard 2026-05-26 16:00:26 -06:00
Richard R
f77ee8cfd4 hard-cut batch B: worker-proxy parsed SSE and remove app progress bus 2026-05-26 15:58:14 -06:00
Richard R
f5eb593554 hard-cut batch A: worker-only app compute backend 2026-05-26 15:56:39 -06:00
Richard R
821d0fdc6b build(tsconfig): add path alias for compute-core control-plane module 2026-05-26 15:45:19 -06:00
Richard R
df05a7d7a3 refactor(control-plane): hard-cut worker events and shared SSE helpers 2026-05-26 11:41:33 -06:00
Richard R
084cbdfac1 refactor(worker-control-plane): route ops through core orchestrator 2026-05-26 11:16:54 -06:00
Richard R
00fc2d5e36 feat(control-plane): add shared orchestrator and in-memory adapters 2026-05-26 11:09:43 -06:00
Richard R
cd06201f5c refactor(parse-progress): rely on sse reconnect and document mode behavior 2026-05-25 21:37:43 -06:00
Richard R
be7ec2c254 feat(worker-events): stream op events via jetstream and proxy in app 2026-05-25 21:37:27 -06:00
Richard R
2df49b54de feat(parse-progress): add local in-memory realtime bus 2026-05-25 21:06:18 -06:00
Richard R
52083c6a09 build(config): add path alias for api-contracts in compute-core
Update tsconfig.json to include a path mapping for
@openreader/compute-core/api-contracts, enabling cleaner imports and
improving modularity for API contract usage across the codebase.
2026-05-25 12:00:31 -06:00
Richard R
e70b3619e0 tests: harden pdf readiness waits and extend tts/pdf timeouts 2026-05-24 20:58:57 -06:00
Richard R
3f17874fe5 tts ensure: skip whisper alignment in test namespaces 2026-05-24 20:58:48 -06:00
Richard R
b2517bc91d pdf parse: dedupe shared jobs and stop enqueue-on-read 2026-05-24 20:58:34 -06:00
Richard R
12c586d7be refactor(core,server): migrate contracts to api-contracts and modularize job handlers
Move shared type contracts from `contracts` to new `api-contracts` module in `compute/core` for clearer API boundaries. Remove legacy `contracts`, `pdf`, and `whisper` index files. Update all imports to use `api-contracts`. Refactor server job logic by replacing `parsePdfJob.ts` with modular `user-pdf-layout-job.ts` and `user-whisper-align-job.ts`, updating all relevant API routes and compute integration. This improves maintainability and separation of concerns across compute and server layers.
2026-05-24 11:39:25 -06:00
Richard R
e75114a943 refactor(onboarding): introduce onboarding state coordinator and central registry
Add useOnboardingCoordinator hook and onboarding-state registry to manage
onboarding-related state such as privacy acceptance and first-visit tracking.
Refactor SettingsModal to utilize the new onboarding state abstractions,
removing legacy privacy gating logic and improving maintainability.

Includes unit tests for onboarding-state registry to ensure correctness.
2026-05-23 17:07:50 -06:00
Richard R
df321b8832 fix(jobs): make TTS segment cache invalidation non-blocking in parsePdfJob
Move TTS segment cache invalidation to a fire-and-forget promise after
document update to prevent cache errors from blocking parse readiness.
Log warnings for both cache invalidation failures and warnings, but do not
interrupt the main job flow. This improves job robustness and avoids
unnecessary failures due to cache issues.
2026-05-23 05:55:22 -06:00
Richard R
65d25e3ce5 refactor(ui): centralize privacy gating logic for settings modal
Consolidate privacy acceptance checks into reusable hooks and callbacks within
SettingsModal. Replace scattered gating with a single entry point for opening
settings or changelog views, ensuring consistent enforcement. Update test
helpers to robustly dismiss onboarding and settings overlays by avoiding
state race conditions.
2026-05-23 05:50:54 -06:00
Richard R
3dcda2b4b7 refactor(logging): enhance logger configuration and streamline event logging 2026-05-22 13:11:51 -06:00
Richard R
ad82aede73 feat(worker,whisper): improve logging for operation lifecycle and remove align mutex
Enhance worker operation handling by adding detailed logging for operation
acceptance, reuse, replacement, and stale detection, including opKey hashes and
reasons for replacement. Remove the alignMutex from Whisper alignment logic to
simplify concurrency control and reduce unnecessary serialization of alignment
requests. These changes improve observability and throughput for alignment and
layout parsing jobs.
2026-05-22 12:49:33 -06:00
Richard R
653dbedaee refactor(worker): add configurable NATS JetStream replica count
Introduce COMPUTE_NATS_REPLICAS environment variable to control the number of
replicas for JetStream streams and key-value buckets. Only values 1, 3, or 5
are accepted, defaulting to 1 for other inputs. Update worker server logic and
deployment documentation to describe and support this configuration.
2026-05-22 09:32:25 -06:00
Richard R
64a68b1011 docs(worker): document and implement idle-based NATS disconnect for Railway sleep
Add lazy NATS connection lifecycle and idle disconnect logic to the worker,
enabling automatic disconnection after 120s of inactivity to support Railway
container sleep. Update deployment docs to describe this behavior, including
caveats and reconnection flow.
2026-05-22 09:01:14 -06:00
Richard R
0fcf4e2dab style(config): update eslint import rules for compute-core/types subpath
Allow '@openreader/compute-core/types' as a permitted subpath in ESLint
import rules. Update the lint message to reflect the new allowed subpath,
ensuring consistency with recent type centralization efforts.
2026-05-22 02:01:52 -06:00
Richard R
b6be71e3b1 refactor(types,api): centralize PDF and TTS type exports to compute-core/types
Move and re-export all PDF parsing and TTS-related types from
@openreader/compute-core/types, eliminating local type duplication.
Update imports throughout the codebase to use the new centralized type
module. Remove obsolete worker-contract file and update tsconfig paths
and compute-core exports for the new types entry point. Refactor API
routes and job logic to use the new type imports and shared parse-state
healing utility. This streamlines type management and improves
consistency between compute and app layers.
2026-05-22 01:51:51 -06:00
Richard R
09e8898683 feat(pdf,config): implement stale parse-state healing and unify op stale window config
Introduce automatic healing for stale PDF parse states in API routes. Add
`isDocumentParseStateStale` utility and use a shared `COMPUTE_OP_STALE_MS`
environment variable (with config getter) to control the stale window for both
worker op replacement and app-side parse-state healing. Update documentation and
environment variable references to reflect new config.

Enables automatic marking of stuck parses as failed, improving reliability and
retry behavior in distributed compute environments.
2026-05-22 01:21:18 -06:00
Richard R
1d4a301a5d fix(pdf): correct parse status handling and loader display logic
Update PDF viewer and document hook to improve parse status management:
- Set parseStatus to null on clear instead of 'pending' for accurate state
- Explicitly set initial parseStatus and reset parseProgress when loading a new document
- Adjust loader display logic to check for resolved parse status before showing expanded loader

Prevents UI inconsistencies and ensures parse progress is reflected correctly during document transitions.
2026-05-22 00:17:15 -06:00
Richard R
50c4330ab1 refactor(core,config,tests): update ONNX model manifest loading and test imports
- Replace static JSON imports with runtime manifest loading in pdf and whisper model modules for compatibility with ESM and bundlers
- Refactor Next.js config to clarify compute mode logic and improve worker bundling conditions
- Update test imports to use package entrypoints instead of relative paths
- Remove redundant whisper alignment/model tests now covered elsewhere or by integration
- No breaking changes to public API or model handling logic
2026-05-21 23:49:56 -06:00
Richard R
b2bb36911a test(whisper): fix playwright ESM loading via dynamic imports 2026-05-21 21:53:55 -06:00
Richard R
2fd9f05652 fix(config): load webpack DefinePlugin via createRequire 2026-05-21 21:49:03 -06:00
Richard R
7a49ff9896 chore(lint): fix eslint violations in worker config and e2e tests 2026-05-21 21:44:58 -06:00
Richard R
057d2a4e32 chore(eslint): restrict compute-core subpath imports to local-runtime 2026-05-21 21:42:35 -06:00
Richard R
55acf05ad6 refactor(compute-core): rename pdf and whisper modules to clean capability layout 2026-05-21 21:41:37 -06:00
Richard R
3a2c6ffa42 refactor(compute-core): split runtime into config/platform and move contracts barrel 2026-05-21 21:40:16 -06:00
Richard R
37b6999c2c refactor(compute-core): collapse to root/local-runtime entrypoints 2026-05-21 21:30:44 -06:00
Richard R
37d11bf9b8 refactor(compute): remove legacy app whisper duplicate and unify core whisper boundary 2026-05-21 21:15:10 -06:00
Richard R
1612e6694b refactor(compute-core): capability-boundary exports for runtime and pdf-layout 2026-05-21 21:11:28 -06:00
Richard R
d8c6d00de6 docs(env): clarify local vs worker compute env ownership 2026-05-21 20:58:32 -06:00
Richard R
a29e6fc250 docs(compute): align worker env docs and model override vars 2026-05-21 20:57:10 -06:00
Richard R
754fbee227 refactor(compute-core): centralize runtime cpu config exports 2026-05-21 20:54:22 -06:00
Richard R
51fbfa117d worker: avoid idle consumer blocking shared compute gate 2026-05-21 20:06:24 -06:00
Richard R
4664806ba0 worker: enrich single-line request logs with status and opId 2026-05-21 19:48:46 -06:00
Richard R
bcdff35200 worker: simplify request logs to single-line entries 2026-05-21 19:44:25 -06:00
Richard R
661af71fa4 refactor(whisper): implement dynamic timeout configuration for alignment and decoding
Replace static timeout constants with functions that derive alignment and ffmpeg
decode timeouts from compute timeout config. This enables adaptive timeout
behavior based on runtime configuration, improving robustness under varying
resource constraints. Update error handling and internal APIs to propagate
timeout values accordingly.

Remove unused Node.js engine constraint from package.json for broader compatibility.
2026-05-21 19:29:03 -06:00
Richard R
7cdd89b26d compute-core: resolve pdf layout license asset by module path 2026-05-21 19:09:01 -06:00
Richard R
1f13ecaed9 worker: use default JetStream long-poll for consumer next 2026-05-21 19:04:16 -06:00
Richard R
4d12c6913a test(export): streamline export cleanup and improve test reliability
Remove unnecessary UI reload and modal interactions during audiobook export
cleanup to reduce flakiness under load. Set explicit timeout for long-running
MP3 export test to ensure consistent execution. These changes enhance test
stability and maintainability.
2026-05-21 18:50:11 -06:00
Richard R
384369f48b refactor: fix tests + remove deprecated PDF layout processing files and update imports
- Deleted mergeTextWithRegions.ts, LICENSE.txt, manifest.json, parsePdf.ts, renderPage.ts, runLayoutModel.ts, stitchCrossPageBlocks.ts, and types.ts as they are no longer needed.
- Updated unit tests to reflect new import paths from @openreader/compute-core.
- Adjusted tsconfig.json to include new paths for pdf-layout module.
2026-05-21 18:48:17 -06:00
Richard R
3ebcb3e6fd tts: make ensure route abort-aware and reduce abort log noise 2026-05-21 16:46:47 -06:00
Richard R
3c25c93e8b tts: remove redundant route-level alignment timeout wrappers 2026-05-21 16:33:55 -06:00