Adopt a consistent findWorkspaceRoot helper across bootstrap, storage, and database modules to reliably locate the monorepo root. Update environment file loading and docstore path construction to use the workspace root, ensuring correct file resolution in multi-package setups. This addresses issues with relative paths and improves compatibility in monorepo and containerized environments.
Update resolveDocstoreDir to handle multiple environments:
- Prefer monorepo root for local development
- Check process.env.PWD for containerized bootstrap
- Fallback to standard /app/docstore path
- Default to process cwd
Enhances robustness of docstore path resolution across deployment scenarios.
Introduce compose.local-slim.yml and compose.local-full.yml for local builds of
OpenReader and compute-worker. Update deployment documentation to describe new
compose variants and usage. Adjust package.json scripts to support local-slim
and local-full workflows. Rename compose files and update stack names for
clarity.
Fix/refactor reconciliation logic to wrap getOpStateRecord and listOpStates methods
from the state store, ensuring they are invoked as functions rather than accessed
as properties. This improves type safety and prevents potential runtime errors
from improper method usage.
- Extract database schema and migrations into @openreader/database
- Extract startup and orchestration scripts into @openreader/bootstrap
- Move compute-worker into packages/compute-worker
- Remove runtime dependency on drizzle-kit
- Update Dockerfile to deploy isolated packages without merging node_modules
Introduce a comprehensive Docker Compose documentation page covering slim, full, and local-build stack examples. Add example Compose files under docker/examples/ for streamlined deployment. Update local development docs to reference the new Compose guide. Revise package.json scripts to support the new Compose workflows and remove legacy compute-worker compose commands. Update sidebar to include the Docker Compose documentation for improved discoverability.
- Move operation contracts from api/contracts to operations/contracts for clearer separation
- Delete legacy API modules: contracts.ts, operation-keys.ts, public-operation.ts
- Add new modular files: compute-operation.ts, http-hooks.ts, routes.ts, operations/keys.ts, operations/reconciliation.ts
- Refactor all imports to use new operations/contracts and keys modules
- Rename protocol types from PublicOperation to ComputeOperation for clarity
- Update OpenAPI schema and all references to use ComputeOperation/Event naming
- Extract whisper decoder helpers to inference/whisper/decoder.ts
- Add new infra: nats-session.ts for NATS session management
- Update tests and storage logic to use new contracts and keys modules
- Refactor worker loop and orchestrator to use modular job definitions
- Update client and protocol in src/lib/server/compute-worker to use new ComputeOperation types
This refactor modularizes compute-worker operation logic, clarifies type naming, and improves maintainability by separating contracts, keys, and reconciliation logic. No breaking changes to external API, but all internal references and tests are updated for the new structure.
- Remove legacy inference and platform files from `src/inference/` and migrate responsibilities to new modular locations under `src/infrastructure/`, `src/inference/pdf/`, and `src/inference/whisper/`
- Consolidate environment/config logic into `src/infrastructure/config.ts`
- Move docstore and ffmpeg platform utilities to `src/infrastructure/platform.ts`
- Refactor PDF and Whisper inference code to use new document layout, layout model, and timestamp utilities
- Update API contracts to define and export types and constants previously scattered in inference/types
- Adjust all imports in jobs, storage, and tests to reference new module structure and type locations
- Inline parser version and encoding logic into API contracts
- Remove obsolete files and update test fixtures for new type locations
This update improves codebase modularity, maintainability, and separation of concerns. No changes to inference or orchestration functionality.
BREAKING CHANGE: inference module structure, type imports, and config utilities have changed; downstream code must update imports and integration points.
- Remove legacy compute-worker monolith under `src/compute/`, including PDF and Whisper inference, control-plane, and runtime orchestration
- Move PDF and Whisper inference logic to new `src/inference/` module
- Move control-plane, orchestrator, and state machine logic to `src/operations/`
- Move NATS/JetStream adapters to `src/infrastructure/`
- Move job orchestration, progress, and artifact persistence to `src/jobs/`
- Update imports throughout tests and main app to reference new module structure
- Update Dockerfile and scripts to use new asset paths
- Add new entrypoints: `src/api/app.ts` and `src/api/contracts.ts`
- Remove obsolete files and update `.gitignore` for new dev artifacts
This refactor modularizes the compute-worker, improving maintainability and separation of concerns. No functional changes to inference or orchestration logic.
BREAKING CHANGE: compute-worker internal APIs, directory structure, and imports have changed; downstream code must update imports and integration points.
- Removed deprecated functions related to document parsing and blob storage in blobstore.ts.
- Introduced new PDF rendering logic in pdf-preview-renderer.ts and pdf-preview-pdfjs-runtime.ts.
- Updated previews-render.ts to utilize the new PDF rendering functions.
- Refactored user-whisper-align-job.ts to use the compute-worker client for alignment requests.
- Enhanced artifact.ts and operation.ts to validate parsed PDF artifacts and resolve current PDF parses.
- Updated snapshot.ts to align with new worker operation types.
- Adjusted runtime-config.ts to check for compute-worker availability.
- Modified types in parsed-pdf.ts and tts.ts to reflect changes in the compute-worker protocol.
- Added unit tests for PDF artifact validation and compute-worker client contract.
- Removed obsolete pdf-op-key.vitest.spec.ts test file.
Track previous session existence to avoid minting a new anonymous user when a
session ends and the user is redirected to the sign-in page. This prevents
clobbering account-scoped preferences by ensuring bootstrapping does not occur
during the logout redirect flow.
Add periodic idle status logs with activity context to improve monitoring and
debugging of worker idle/disconnect behavior. Track last activity reason and
include it in status logs and idle disconnect eligibility checks. Update
activity marking calls to specify reason for more granular observability.
Review-driven: thread ttsUpstreamTimeoutMs into the speech-sdk branch as an
overall budget across SDK retries, and fail fast on model ids missing the
provider/model form instead of falling through to provider defaults.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds 'speech-sdk' as a fifth built-in TTS provider backed by @speech-sdk/core
(Apache 2.0). Models use the provider/model format; requests go from the
server directly to the provider's API with the user's own key. Existing
providers, defaults, and config are untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Configure webpack to treat @napi-rs/canvas as an external dependency under
the 'canvas' module name when building the server bundle. This adjustment
ensures proper module resolution and avoids bundling native dependencies
directly, improving deployment reliability for environments requiring
@napi-rs/canvas.
Add explicit host validation to only allow audio output URLs from
replicate.delivery and its subdomains, blocking potential SSRF attacks
from malicious model outputs. Update tests to cover allowed and
disallowed host scenarios.
Update DocumentPreview text preview to use a CSS color-mix radial pattern
with increased opacity and a masked vignette effect. Improves visual
clarity and consistency with design tokens.
Switch SSRF protection in web-loader to use @microsoft/antissrf, removing
custom IP range logic. This centralizes and future-proofs SSRF mitigation by
leveraging maintained policy sets and DNS-aware enforcement. Error handling in
the import-url API route is updated to recognize new policy error messages.
Dependency added to package.json.
Add detection for CGNAT (100.64.0.0/10) and benchmarking (198.18.0.0/15) ranges
in IPv4 private IP checks. Update IPv6 site-local detection to include full
range. Refactor fetchWithLimit to return both HTML and the resolved final URL,
ensuring relative URLs are resolved against the actual fetched location.
Enhance SSRF mitigation in web-loader by expanding private IP detection for both
IPv4 and IPv6, including checks for multicast and broadcast addresses. Update
filename sanitization logic in UploadMenuDialog to collapse consecutive
disallowed characters and trim underscores, ensuring cleaner filenames. Remove
redundant userId check in import-url API route for streamlined authentication.
Add input field for specifying a custom document title when importing from a web URL.
Use the provided title for the generated markdown file name and success message,
falling back to the scraped title if left empty. Reset the title field after import
completion for improved UX.
Replace ModalFrame and related modal patterns in settings and upload dialogs with
the new SidebarDialog component to establish a consistent sidebar-based dialog
layout. Refactor SettingsModal and UploadMenuDialog to use SidebarDialog, updating
header, navigation, and content handling accordingly. Update DocumentList to use
DocumentUploader in compact mode for sidebar upload action. Export SidebarDialog
from the UI module. Minor fixes in web-loader for IP parsing and option naming.
This change standardizes dialog UI structure and improves maintainability.
Introduce a unified upload menu dialog enabling users to upload files,
create markdown/text documents, or import content directly from web URLs.
Implement a secure server-side loader for fetching and parsing web pages
with SSRF protection, content-type checks, and size limits. Add new API
route for importing documents from URLs, and update dependencies to
support HTML parsing and markdown conversion.
- Add UploadMenuDialog component with file, create, and URL import tabs
- Implement SSRF-mitigated web-loader using linkedom, readability, turndown
- Add /api/documents/import-url endpoint for web imports
- Update DocumentList to use new upload dialog
- Add BrowserIcon and supporting UI elements
- Update dependencies: @mozilla/readability, linkedom, turndown
- Expose importUrl client API
This improves document onboarding flexibility and enhances security for
web-based imports.
Add unit test verifying that U+00B7 middle dot is not stripped from Catalan
words during audio text preprocessing, reflecting multilingual requirements.
Move canonical audio text preprocessing logic to new shared module `audio-text.ts`
to ensure consistent normalization across worker, client, and NLP code. Update
all consumers to import and use the shared rules, eliminating duplication and
preventing drift. Expand glyph stripping to handle additional decorative
characters. Add unit tests for new cases and update documentation accordingly.
Add SSE `retry:` directive to suggest a 2-minute reconnect delay when the worker
enters idle sleep and drops silent EventSource streams. This prevents clients
from immediately reconnecting and re-waking the container, reducing unnecessary
resource usage. Update SSE encoding to support the `retry` field and add tests
for correct behavior. Improve idle disconnect logging for better observability.
Update the EPUB highlighting cache key to include aligned word texts, ensuring
that changes in alignment or word content invalidate stale cached spans even if
word counts remain unchanged. Clear the word-region cache when rendered text
maps are replaced or reset, preventing incorrect highlights after remapping.
Also add locale tracking to HTML sentence highlighting state to support
language-aware word segmentation during alignment.
Replace EPUB-specific word highlight mapping with shared token-sequence alignment
logic used by HTML and PDF viewers. Remove bespoke alignment code and tests in
favor of a single primitive (`locateAlignmentWordSpans`) that robustly maps
spoken words to rendered text regions, tolerant of transcription and formatting
differences. Update EPUB highlighting to cache per-segment word-region spans for
efficient re-use. Add unit tests for the new alignment logic.
Switch mp3 transcoding to output a seekable temp file instead of piping to stdout.
This allows ffmpeg to backpatch the Xing/Info header with correct total-duration
metadata, which is required for HTML5 audio elements to handle VBR mp3 playback
accurately. Prevents playback issues caused by missing or incorrect duration,
such as premature or stalled `ended` events.
Update documentation to specify that OpenReader uses the Kokoro voice set
for Kokoro models and the standard OpenAI voices for other models when no
voice listing endpoint is available. Improves clarity on provider fallback
behavior.
Align non-kokoro custom-openai models to use the canonical OpenAI voice set
instead of a separate custom set. Updates logic and tests to reflect this
standardization for improved consistency across OpenAI-compatible providers.
Avoid bootstrapping anonymous sessions when users are on sign-in or sign-up
pages to prevent overwriting account-scoped preferences after login. Adjust
loading logic to account for this scenario.
Update documentation to specify that only the speech endpoint is required for
OpenAI-compatible TTS providers, with voice listing now optional and auto-discovered.
Document support for multiple audio formats (mp3, wav, ogg, flac) and note that
API keys are optional for local or unauthenticated servers. Add Supertonic to
the list of compatible implementations and improve troubleshooting guidance.
feat(tts): support OpenAI-compatible servers that emit wav instead of mp3
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
* **New Features**
* Normalize all incoming TTS audio to MP3 so stored/served audio is consistently MP3.
* Broader discovery of voices from OpenAI‑compatible endpoints by probing multiple voice routes and respecting provider defaults.
* **Reliability**
* Retry flow when providers reject explicit response format to improve compatibility.
* Improved audio format detection, safer transcoding with cancellation and cleanup, and logging when normalization occurs.
* **Tests**
* Added unit tests covering audio format detection across multiple codecs.
* **Documentation**
* Added a Supertonic TTS provider setup guide.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Update OpenAI SDK instantiation to provide a placeholder apiKey when none is
supplied, enabling support for local or unauthenticated OpenAI-compatible
TTS servers. Ensure Authorization header is cleared when not needed.
Include documentation for configuring the Supertonic TTS provider. Update
the main TTS providers list and sidebar to reference the new guide, ensuring
users can easily find setup instructions for Supertonic.
Update audio format sniffing to require a full 10-byte ID3 header for mp3 detection and add stricter validation of MPEG frame headers to avoid false positives. Refactor ffmpeg process spawning to register abort listeners before module import, ensuring abort signals are handled promptly and preventing orphaned processes. Expand tests to cover edge cases in mp3 header detection.
The wav->mp3 normalization path re-encoded pristine high-fidelity TTS (e.g.
Supertonic's 44.1 kHz wav) at 64k CBR, which lowpasses hard and introduces
high-frequency artifacts that cause listening fatigue. Switch the live/segment
transcode to libmp3lame VBR -q:a 2 (~170-210 kbps for speech), effectively
transparent. The audiobook export still re-encodes to 64k for file size.
