danvics/openreader
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
openreader/docs-site/docs/deploy/local-development.md
Richard R 848ee4faad feat(tasks): add document blob lease for safe orphan reaping and improve scheduled task robustness 
Introduce a document blob lease mechanism to prevent race conditions between document registration and orphaned blob cleanup. The new `document_blob_leases` table ensures that only one process can claim a document blob for mutation or deletion at a time. Update the orphan reaper to acquire a lease before deleting blobs and to re-check for ownership after acquiring the lease, avoiding accidental deletion of in-flight uploads.

Enhance scheduled task infrastructure with per-task fencing tokens to prevent stale runners from overwriting newer results, enforce runtime limits with abort signals, and expose scheduler mode and minimum interval to the admin panel and API. Adjust task handlers to accept a context with abort support, and update documentation and environment variable references for the new cron secret and scheduling behavior.

BREAKING CHANGE: Scheduled tasks now require a `document_blob_leases` table and updated handler signatures. Vercel deployments must set `CRON_SECRET` for scheduled maintenance.
2026-06-07 12:08:57 -06:00

11 KiB
Raw Blame History

title
Local Development

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';

Prerequisites

Node.js + pnpm (required) 
brew install nvm pnpm
mkdir -p ~/.nvm
echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.zshrc
echo '[ -s "$(brew --prefix nvm)/nvm.sh" ] && . "$(brew --prefix nvm)/nvm.sh"' >> ~/.zshrc
source ~/.zshrc
nvm install --lts
nvm use --lts
node -v
pnpm -v

# Debian/Ubuntu example
sudo apt update
sudo apt install -y curl
curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
nvm install --lts
nvm use --lts
corepack enable
corepack prepare pnpm@latest --activate
node -v
pnpm -v
SeaweedFS weed binary (required unless using external S3) 
brew install seaweedfs
weed version

:::warning SeaweedFS Compatibility Note (April 16, 2026) If you see intermittent S3 InternalError upload failures with embedded storage, use SeaweedFS 4.18. OpenReader currently pins 4.18 in CI and Docker builds while 4.19 compatibility is investigated. :::

# Linux amd64 example (pin 4.18)
mkdir -p "$HOME/.local/bin"
curl -fsSL -o /tmp/seaweedfs.tar.gz \
  https://github.com/seaweedfs/seaweedfs/releases/download/4.18/linux_amd64.tar.gz
tar -xzf /tmp/seaweedfs.tar.gz -C /tmp weed
install -m 0755 /tmp/weed "$HOME/.local/bin/weed"
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"
weed version

:::warning SeaweedFS Compatibility Note (April 16, 2026) If you see intermittent S3 InternalError upload failures with embedded storage, use SeaweedFS 4.18. OpenReader currently pins 4.18 in CI and Docker builds while 4.19 compatibility is investigated. :::

NATS Server nats-server (required for embedded compute mode)

If COMPUTE_WORKER_URL is unset, startup launches embedded compute worker + NATS, so nats-server must be available on host PATH.

If you always use an external worker (COMPUTE_WORKER_URL set), this is not required.

brew install nats-server
nats-server -v

# Linux amd64 example
mkdir -p "$HOME/.local/bin"
curl -fsSL -o /tmp/nats-server.zip \
  https://github.com/nats-io/nats-server/releases/latest/download/nats-server-v2.12.1-linux-amd64.zip
unzip -j /tmp/nats-server.zip '*/nats-server' -d /tmp
install -m 0755 /tmp/nats-server "$HOME/.local/bin/nats-server"
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"
nats-server -v
LibreOffice (optional, for DOCX conversion) 
brew install libreoffice

# Debian/Ubuntu example
sudo apt update
sudo apt install -y libreoffice
Word-by-word highlighting (optional)

No extra native Whisper CLI build step is required.

Word-by-word highlighting and PDF layout parsing are worker-backed in current releases.

If you need mirrors or pinned artifact locations, set WHISPER_MODEL_BASE_URL in .env (current defaults expect q4 Whisper files at that base URL).

External compute worker dev stack (optional)

Use this only when you intentionally run compute-worker as a separate service. Default local flow does not need compute/worker/.env; embedded worker startup reads root .env. Full worker deployment details are in Compute Worker (NATS JetStream).

Start only NATS + compute-worker via compose watch:

docker compose --env-file compute/worker/.env -f compute/worker/docker-compose.yml up --watch
# or: pnpm compute:dev:watch

compute/worker/.env.example contains a starter config for standalone worker service deployments.

Run the main app separately on the host:

pnpm dev

For app -> external worker routing, set in root .env:

COMPUTE_WORKER_URL=http://localhost:8081
COMPUTE_WORKER_TOKEN=<same-token-used-by-worker>

Ownership in external worker mode:

  • root .env: app routing/auth (COMPUTE_WORKER_URL, COMPUTE_WORKER_TOKEN) plus optional shared timeout/stale/retry overrides such as COMPUTE_PDF_JOB_ATTEMPTS
  • compute/worker/.env* (or worker platform env): worker runtime variables (NATS_*, S3_*, model base URLs, worker tuning)

For embedded worker startup (COMPUTE_WORKER_URL unset), worker tuning values such as COMPUTE_PDF_JOB_ATTEMPTS must be set in the root .env because compute/worker/.env* is ignored in that mode.

Worker mode requires worker-reachable shared object storage (S3-compatible endpoint). For external worker mode, object storage must be shared/reachable by both app and worker services.

Steps

Required flow

  1. Clone the repository.
git clone https://github.com/richardr1126/openreader.git
cd openreader
  1. Install dependencies.
pnpm i
  1. Configure the environment.
cp .env.example .env

Then edit .env.

Default embedded worker flow (no external worker URL):

# Leave COMPUTE_WORKER_URL unset.
# Entry point auto-starts embedded worker+NATS when available.

External worker flow:

COMPUTE_WORKER_URL=http://localhost:8081
COMPUTE_WORKER_TOKEN=<same-token-used-by-worker>

Use the same ownership split:

  • root .env: app routing/auth (COMPUTE_WORKER_URL, COMPUTE_WORKER_TOKEN) plus optional shared timeout/stale overrides
  • compute/worker/.env* (or worker platform env): worker runtime variables (NATS_*, S3_*, model base URLs, worker tuning)

Use one of these .env mode templates:

API_BASE=http://host.docker.internal:8880/v1
BASE_URL=http://localhost:3003
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
# Optional when you need multiple local origins:
# AUTH_TRUSTED_ORIGINS=http://localhost:3003,http://127.0.0.1:3003

# API_BASE and optional API_KEY are seeded into the admin "default-openai" shared provider
# on first boot, then no longer read. Manage them in Settings → Admin afterwards.
API_BASE=http://host.docker.internal:8880/v1
BASE_URL=http://localhost:3003
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
# Comma-separated emails to auto-promote to admin on signin.
ADMIN_EMAILS=you@example.com

API_BASE=http://host.docker.internal:8880/v1
USE_EMBEDDED_WEED_MINI=false
BASE_URL=http://localhost:3003
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
S3_BUCKET=your-bucket
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=your-access-key
S3_SECRET_ACCESS_KEY=your-secret-key
# Optional for non-AWS providers:
# S3_ENDPOINT=https://your-s3-compatible-endpoint
# S3_FORCE_PATH_STYLE=true

API_BASE=http://host.docker.internal:8880/v1
BASE_URL=http://localhost:3003
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
COMPUTE_WORKER_URL=http://localhost:8081
COMPUTE_WORKER_TOKEN=<same-token-used-by-worker>
USE_EMBEDDED_WEED_MINI=false
S3_BUCKET=your-bucket
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=your-access-key
S3_SECRET_ACCESS_KEY=your-secret-key
# Optional for non-AWS providers:
# S3_ENDPOINT=https://your-s3-compatible-endpoint
# S3_FORCE_PATH_STYLE=true

:::note Env vars vs. admin panel On first boot, API_KEY / API_BASE can bootstrap default-openai, and RUNTIME_SEED_JSON / RUNTIME_SEED_JSON_PATH can seed runtime config + providers. After that, the admin UI is authoritative and editing bootstrap env vars no longer changes app behavior. See Admin Panel. :::

:::note User BYOK restriction default If you want each user to enter personal provider credentials, set restrictUserApiKeys=false (from Settings → Admin, or by seeding runtimeConfig.restrictUserApiKeys=false in runtime seed JSON). :::

:::info For all environment variables, see Environment Variables. :::

See Auth for app/auth behavior. See Admin Panel for the shared-provider and feature-flag management UI. Storage configuration details are in Object / Blob Storage. Refer to Database for database modes. Learn about migration behavior and commands in Migrations.

:::info Scheduled maintenance tasks Local and self-hosted Node.js deployments start the scheduled-task loop in-process and check for due work once per minute. No CRON_SECRET is required unless you intentionally invoke the cron HTTP route yourself. Manage task intervals and inspect failures from Settings → Admin → Scheduled tasks. :::

  1. Start the app.
pnpm dev

If you use embedded worker startup (no COMPUTE_WORKER_URL) and the host is missing nats-server, install nats-server locally or switch to external worker mode.

pnpm build
pnpm start

:::warning API Base Reachability API_BASE must be reachable from the Next.js server process, not just your browser. :::

Visit http://localhost:3003.

Optional workflows

Run manual DB migrations only for troubleshooting or explicit migration workflows:

  • Migrations run automatically on startup through the shared entrypoint for both pnpm dev and pnpm start.
pnpm migrate

:::info If POSTGRES_URL is set, migrations target Postgres; otherwise local SQLite is used. To disable automatic startup migrations, set RUN_DRIZZLE_MIGRATIONS=false and/or RUN_FS_MIGRATIONS=false. You can run storage migration manually with pnpm migrate-fs. :::