* refactor(ui): replace anchor tags with next/link in SidebarNavLink and UserMenu Update SidebarNavLink to use next/link for navigation instead of anchor tags, ensuring proper routing and improved accessibility in Next.js. Refactor UserMenu to remove legacy Link wrappers and directly use SidebarNavLink for signin and signup links. This streamlines navigation components and aligns with Next.js best practices. * fix(pdf): handle non-zero viewport origins and improve layout model ordering Update normalizeTextItemsForLayout to correctly apply viewport transforms, including non-zero page origins, ensuring accurate mapping of PDF text items to top-left coordinates. Refactor runLayoutModel to implement a custom order sequence builder for layout regions using model order logits, improving region ordering consistency with model semantics. Update related tests to cover viewport transform edge cases. * refactor(pdf): improve text normalization with font ascent and vertical overlap logic Enhance text normalization by incorporating font ascent and descent data to more accurately position glyphs, especially for decorative initials. Update merge logic to better detect line membership using vertical overlap, ensuring drop caps and overlapping glyphs are merged correctly. Extend tests to cover these layout scenarios. * style(range): redesign slider with precision gauge and ruler ticks Revamp the range input to feature a minimalist "precision gauge" style. Introduce a hairline rail, ruler notches for discrete steps, and a slim needle thumb. Add CSS variables and logic for per-instance tick sizing and coloring. Remove bulky inline class-based styling in favor of centralized CSS for improved maintainability and visual clarity. * refactor(ui): modularize PDF loader and range slider visuals Move PDF layout scan visualization and range slider styles into dedicated CSS modules, isolating their styles from the global scope. Integrate PdfLayoutScan component into the PDF viewer loader UI for animated parse progress. Refactor progress bars to use a reusable progress-fill class with animated sheen effect. Update range input to use CSS module for precision gauge styling. * style(reader): remove grid overlay from PdfLayoutScan visualization * refactor(api): add staleness detection for inflight worker operation states Integrate isWorkerOperationStateStale checks into document parse API endpoints to ensure inflight worker operation states are not reused if stale. Introduce helper for staleness detection and corresponding unit tests. Enhance SSE event streaming with keepalive intervals and improve progress acknowledgment error handling. * feat(worker): recover and fail stale in-flight pdf ops on startup Add orphaned operation recovery logic to detect and mark stale in-flight pdf_layout jobs as failed during worker startup. Extend OperationStateStore with listOpStates for state enumeration. Update tests and documentation to cover recovery behavior and new environment variable COMPUTE_PDF_JOB_ATTEMPTS. * refactor(worker): distinguish staleness thresholds for running and queued pdf ops Update orphan recovery logic to apply separate timeouts for 'running' and 'queued' pdf_layout operations. Adjust tests to verify that only stale 'running' operations are failed, while stale 'queued' operations remain untouched. * feat(worker): extend orphan recovery to handle whisper_align ops and improve logging Update orphan recovery to detect and fail stale 'running' whisper_align operations in addition to pdf_layout. Refactor recovery logic to generalize staleness checks across operation kinds and enhance log output with detailed operation info. Expand tests to verify correct handling of both whisper_align and pdf_layout operations in running and queued states. * refactor(control-plane): introduce revision-based CAS for operation state updates Add revision tracking and compare-and-set (CAS) semantics to operation state stores, enabling atomic state transitions and preventing lost updates. Extend the OperationStateStore interface with getOpStateRecord and compareAndSetOpState methods. Update orchestrator and worker runtime to utilize CAS for marking operations as failed only if the state is unchanged. Enhance in-memory, JetStream, and test control plane implementations to support revision logic. This change improves concurrency safety and correctness of operation state management across distributed components. * feat(ui): add parse failure state to PDF layout scan animation Display a distinct "parse halted" visual state in the PDF layout scan component and PDF viewer page when parsing fails. The loader animation is replaced by a static, dimmed page with an alert glyph and updated styling, ensuring users are not misled by an active animation after a failure. CSS and component logic updated to support the new state. * feat(worker): extract orphaned operation recovery to module with periodic sweep Move orphaned operation recovery logic into a dedicated orphan-recovery module, introducing a periodic sweep timer that triggers recovery every 15 seconds while the worker is connected. Refactor runtime to delegate orphan detection and handling to the new module, improving modularity and maintainability. Add unit tests for orphan-recovery to ensure correctness. * fix(ui): adjust PDF viewer layout and update parse loader description * refactor(pdf): streamline layout model region extraction and update test coverage - Replace custom order sequence logic with softmax-based class selection in runLayoutModel - Remove unused sigmoid and buildOrderSequence functions - Simplify detection loop to filter and map regions directly - Add targeted tests for layout model extraction logic - Update CSS animation naming for consistency - Clarify test description and add inline comments for orphan recovery scenario * fix(pdf): add strict validation for layout model output shapes and extend test coverage Add explicit error handling for invalid or inconsistent pred_boxes and logits array lengths in runLayoutModel to prevent silent failures. Expand test suite to verify correct region filtering and error scenarios, ensuring only labeled regions are returned and malformed outputs are handled robustly.
10 KiB
| title |
|---|
| Local Development |
import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';
Prerequisites
Node.js + pnpm (required)
brew install nvm pnpm
mkdir -p ~/.nvm
echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.zshrc
echo '[ -s "$(brew --prefix nvm)/nvm.sh" ] && . "$(brew --prefix nvm)/nvm.sh"' >> ~/.zshrc
source ~/.zshrc
nvm install --lts
nvm use --lts
node -v
pnpm -v
# Debian/Ubuntu example
sudo apt update
sudo apt install -y curl
curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
nvm install --lts
nvm use --lts
corepack enable
corepack prepare pnpm@latest --activate
node -v
pnpm -v
SeaweedFS weed binary (required unless using external S3)
brew install seaweedfs
weed version
:::warning SeaweedFS Compatibility Note (April 16, 2026)
If you see intermittent S3 InternalError upload failures with embedded storage, use SeaweedFS 4.18.
OpenReader currently pins 4.18 in CI and Docker builds while 4.19 compatibility is investigated.
:::
# Linux amd64 example (pin 4.18)
mkdir -p "$HOME/.local/bin"
curl -fsSL -o /tmp/seaweedfs.tar.gz \
https://github.com/seaweedfs/seaweedfs/releases/download/4.18/linux_amd64.tar.gz
tar -xzf /tmp/seaweedfs.tar.gz -C /tmp weed
install -m 0755 /tmp/weed "$HOME/.local/bin/weed"
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"
weed version
:::warning SeaweedFS Compatibility Note (April 16, 2026)
If you see intermittent S3 InternalError upload failures with embedded storage, use SeaweedFS 4.18.
OpenReader currently pins 4.18 in CI and Docker builds while 4.19 compatibility is investigated.
:::
NATS Server nats-server (required for embedded compute mode)
If COMPUTE_WORKER_URL is unset, startup launches embedded compute worker + NATS, so nats-server must be available on host PATH.
If you always use an external worker (COMPUTE_WORKER_URL set), this is not required.
brew install nats-server
nats-server -v
# Linux amd64 example
mkdir -p "$HOME/.local/bin"
curl -fsSL -o /tmp/nats-server.zip \
https://github.com/nats-io/nats-server/releases/latest/download/nats-server-v2.12.1-linux-amd64.zip
unzip -j /tmp/nats-server.zip '*/nats-server' -d /tmp
install -m 0755 /tmp/nats-server "$HOME/.local/bin/nats-server"
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"
nats-server -v
LibreOffice (optional, for DOCX conversion)
brew install libreoffice
# Debian/Ubuntu example
sudo apt update
sudo apt install -y libreoffice
Word-by-word highlighting (optional)
No extra native Whisper CLI build step is required.
Word-by-word highlighting and PDF layout parsing are worker-backed in current releases.
If you need mirrors or pinned artifact locations, set WHISPER_MODEL_BASE_URL in .env (current defaults expect q4 Whisper files at that base URL).
External compute worker dev stack (optional)
Use this only when you intentionally run compute-worker as a separate service.
Default local flow does not need compute/worker/.env; embedded worker startup reads root .env.
Full worker deployment details are in Compute Worker (NATS JetStream).
Start only NATS + compute-worker via compose watch:
docker compose --env-file compute/worker/.env -f compute/worker/docker-compose.yml up --watch
# or: pnpm compute:dev:watch
compute/worker/.env.example contains a starter config for standalone worker service deployments.
Run the main app separately on the host:
pnpm dev
For app -> external worker routing, set in root .env:
COMPUTE_WORKER_URL=http://localhost:8081
COMPUTE_WORKER_TOKEN=<same-token-used-by-worker>
Ownership in external worker mode:
- root
.env: app routing/auth (COMPUTE_WORKER_URL,COMPUTE_WORKER_TOKEN) plus optional shared timeout/stale/retry overrides such asCOMPUTE_PDF_JOB_ATTEMPTS compute/worker/.env*(or worker platform env): worker runtime variables (NATS_*,S3_*, model base URLs, worker tuning)
For embedded worker startup (COMPUTE_WORKER_URL unset), worker tuning values such as COMPUTE_PDF_JOB_ATTEMPTS must be set in the root .env because compute/worker/.env* is ignored in that mode.
Worker mode requires worker-reachable shared object storage (S3-compatible endpoint). For external worker mode, object storage must be shared/reachable by both app and worker services.
Steps
Required flow
- Clone the repository.
git clone https://github.com/richardr1126/openreader.git
cd openreader
- Install dependencies.
pnpm i
- Configure the environment.
cp .env.example .env
Then edit .env.
Default embedded worker flow (no external worker URL):
# Leave COMPUTE_WORKER_URL unset.
# Entry point auto-starts embedded worker+NATS when available.
External worker flow:
COMPUTE_WORKER_URL=http://localhost:8081
COMPUTE_WORKER_TOKEN=<same-token-used-by-worker>
Use the same ownership split:
- root
.env: app routing/auth (COMPUTE_WORKER_URL,COMPUTE_WORKER_TOKEN) plus optional shared timeout/stale overrides compute/worker/.env*(or worker platform env): worker runtime variables (NATS_*,S3_*, model base URLs, worker tuning)
Use one of these .env mode templates:
API_BASE=http://host.docker.internal:8880/v1
API_KEY=none
BASE_URL=http://localhost:3003
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
# Optional when you need multiple local origins:
# AUTH_TRUSTED_ORIGINS=http://localhost:3003,http://127.0.0.1:3003
# API_BASE / API_KEY are seeded into the admin "default-openai" shared provider
# on first boot, then no longer read. Manage them in Settings → Admin afterwards.
API_BASE=http://host.docker.internal:8880/v1
API_KEY=none
BASE_URL=http://localhost:3003
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
# Comma-separated emails to auto-promote to admin on signin.
ADMIN_EMAILS=you@example.com
API_BASE=http://host.docker.internal:8880/v1
API_KEY=none
USE_EMBEDDED_WEED_MINI=false
BASE_URL=http://localhost:3003
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
S3_BUCKET=your-bucket
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=your-access-key
S3_SECRET_ACCESS_KEY=your-secret-key
# Optional for non-AWS providers:
# S3_ENDPOINT=https://your-s3-compatible-endpoint
# S3_FORCE_PATH_STYLE=true
API_BASE=http://host.docker.internal:8880/v1
API_KEY=none
BASE_URL=http://localhost:3003
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
COMPUTE_WORKER_URL=http://localhost:8081
COMPUTE_WORKER_TOKEN=<same-token-used-by-worker>
USE_EMBEDDED_WEED_MINI=false
S3_BUCKET=your-bucket
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=your-access-key
S3_SECRET_ACCESS_KEY=your-secret-key
# Optional for non-AWS providers:
# S3_ENDPOINT=https://your-s3-compatible-endpoint
# S3_FORCE_PATH_STYLE=true
:::note Env vars vs. admin panel
On first boot, API_KEY / API_BASE can bootstrap default-openai, and RUNTIME_SEED_JSON / RUNTIME_SEED_JSON_PATH can seed runtime config + providers. After that, the admin UI is authoritative and editing bootstrap env vars no longer changes app behavior. See Admin Panel.
:::
:::note User BYOK restriction default
If you want each user to enter personal provider credentials, set restrictUserApiKeys=false (from Settings → Admin, or by seeding runtimeConfig.restrictUserApiKeys=false in runtime seed JSON).
:::
:::info For all environment variables, see Environment Variables. :::
See Auth for app/auth behavior. See Admin Panel for the shared-provider and feature-flag management UI. Storage configuration details are in Object / Blob Storage. Refer to Database for database modes. Learn about migration behavior and commands in Migrations.
- Start the app.
pnpm dev
If you use embedded worker startup (no COMPUTE_WORKER_URL) and the host is missing nats-server,
install nats-server locally or switch to external worker mode.
pnpm build
pnpm start
:::warning API Base Reachability
API_BASE must be reachable from the Next.js server process, not just your browser.
:::
Visit http://localhost:3003.
Optional workflows
Run manual DB migrations only for troubleshooting or explicit migration workflows:
- Migrations run automatically on startup through the shared entrypoint for both
pnpm devandpnpm start.
pnpm migrate
:::info
If POSTGRES_URL is set, migrations target Postgres; otherwise local SQLite is used. To disable automatic startup migrations, set RUN_DRIZZLE_MIGRATIONS=false and/or RUN_FS_MIGRATIONS=false. You can run storage migration manually with pnpm migrate-fs.
:::