Commit graph

3 commits

Author SHA1 Message Date
Richard R
e1483ab80a refactor(worker): modularize worker entrypoint and add vitest infra
move worker startup logic from server.ts to runtime.ts for better modularity
add worker-loop-policy.ts for loop control abstraction
delete legacy unit tests under tests/unit/ related to compute worker and control-plane
add vitest config and test directories for compute-core and compute-worker
add vitest workflow for CI
update package.json scripts for new test commands and add vitest as dev dependency

BREAKING CHANGE: worker entrypoint is now compute/worker/src/runtime.ts instead of server.ts; legacy Playwright-based unit tests for compute worker are removed in favor of Vitest
2026-05-30 11:00:01 -06:00
Richard R
bab3416a36 refactor(compute): add timing metrics to worker job results and status
Introduce WorkerJobTiming interface to capture queue wait, S3 fetch, and compute
durations for compute worker jobs. Update job result and status response types
to include timing data. Enhance worker implementation to record and attach
timing metrics for alignment jobs, improving observability of job processing
latency. Update exports and type usage to support new timing fields.
2026-05-19 15:55:35 -06:00
Richard R
f1aa1c3e3b feat(compute): add external compute worker backend and integration
Introduce support for external compute worker mode (`COMPUTE_MODE=worker`) using a new `WorkerComputeBackend`. This enables offloading heavy ONNX Whisper alignment and PDF layout parsing to a standalone worker service (Redis + BullMQ), improving scalability and compatibility with serverless/limited environments.

- Add `@openreader/compute-core` as a shared package for ONNX inference and PDF parsing logic.
- Implement `WorkerComputeBackend` and worker contract/types for remote job execution.
- Update compute backend selection logic and remove previous worker mode guards.
- Extend `WhisperAlignInput` and `PdfLayoutInput` types to support object keys for remote data access.
- Refactor local compute backend to use `@openreader/compute-core` and support both buffer and object key inputs.
- Update job runner, TTS segment alignment, and PDF layout parsing flows to use new compute backend APIs.
- Add scripts, Docker workflow, and documentation for deploying and running the compute worker.
- Update environment variable docs and examples for worker mode, including storage requirements and configuration.
- Document published images and stack changes to reflect the new compute worker architecture.

BREAKING CHANGE: `COMPUTE_MODE=worker` now requires an external compute worker service and S3-compatible object storage. Embedded SeaweedFS (`weed mini`) is not supported in worker mode. See the new documentation for deployment and configuration details.
2026-05-19 15:21:25 -06:00