openreader/docs-site/docs/reference/environment-variables.md
Richard R e670c38523
Refactor user data cleanup and TTS storage flows (#105)
* refactor(user): overhaul user data cleanup and export for cascading deletes and TTS segment support

Revise user data cleanup logic to ensure proper cascading deletion of user-related
database rows and S3 objects, including shared document and preview artifacts.
Introduce explicit checks for last ownership before removing shared resources.
Add TTS segment cache and audio cleanup to document and user deletion flows.
Expand user data export to include TTS segment entries and audio, job events,
document settings, and linked auth sessions. Update schema with ON DELETE CASCADE
for userTtsChars and userJobEvents. Add new migration scripts and comprehensive
unit tests for cleanup and export scenarios.

* test(user): expand cleanup and export coverage for shared docs and TTS segment files

Add tests for user data cleanup with shared document ownership and for TTS segment
variant export scenarios, including cases with duplicate audio keys and storage
disabled. Refactor cleanup logic to use document row types and transactional
ownership removal with last-owner checks. Update TTS segment cache clearing to
report deleted segment count by unique entry. Adjust export logic to always
include all TTS segment variants and conditionally export file buckets based on
storage availability. Update migration to use NOT VALID/VALIDATE for new
cascading constraints.

* refactor(documents): centralize owned document deletion and add mutation locking

Introduce `deleteOwnedDocument` utility to encapsulate all logic for removing a user's ownership of a document, including TTS segment cache cleanup, preview artifact removal, and S3 blob deletion for last-owner cases. Add `withDocumentMutationLock` to serialize concurrent mutations on the same document, using advisory locks in Postgres and a local queue fallback. Refactor all API routes and user data flows to use these utilities, ensuring transactional safety and preventing race conditions during document deletion or transfer. Update tests for new flows and add coverage for document cleanup sequencing and locking behavior.

* feat(data): enhance anonymous claim to support document settings and TTS segment copy

Expand the anonymous data claim process to include document settings transfer and TTS segment S3 prefix copying. Remove foreign key constraints from user_tts_chars to allow non-user buckets. Update claim modal and onboarding flow to display claimed document settings. Refactor TTS char count transfer to merge all dates and fix upsert logic. Add S3 copy utility for TTS segments and corresponding tests. Update migrations and schema to reflect relaxed constraints.

* refactor(data): improve error handling and rollback for document and TTS segment operations

Enhance robustness of document deletion and TTS segment transfer by improving
error logging, partial rollback, and degraded state reporting. Add best-effort
cleanup and recovery mechanisms to prevent orphaned data and ensure diagnostic
information is captured for unexpected failures. Update user data cleanup to
log and swallow restoration errors without masking primary failures. Refine
claim logic to handle unmapped TTS audio keys safely.

* refactor(data): unify document mutation locking and transaction handling

Replace mutation-lock with document-lock to centralize document mutation
serialization and database transaction management. Introduce runInDbTransaction
utility to abstract SQLite/Postgres transaction differences. Update document
deletion, user data cleanup, and rate limiter logic to delegate transaction
handling and locking to shared helpers. Remove dialect-specific branching and
inline transaction logic for improved maintainability and testability.

BREAKING CHANGE: withDocumentMutationLock is removed in favor of withDocumentLock and runInDbTransaction

* fix(documents): improve rollback error handling in deleteOwnedDocument

Enhance error handling during document deletion by ensuring that failures
in restoring document ownership do not obscure the original error. Log
degraded events when ownership restoration fails after a deletion error,
providing additional context for debugging and monitoring. This change
improves reliability and traceability of document deletion operations.

* feat(tasks): introduce scheduled task engine and admin UI for background jobs

Add a general-purpose scheduled task system with a persistent registry and status tracking, supporting background maintenance jobs such as orphaned blob reaping, expired upload cleanup, job event pruning, and TTS usage retention. Implement a new `scheduled_tasks` table, task engine, and handlers for each maintenance operation. Integrate an admin UI panel for monitoring, manual runs, and configuration of tasks. Update document and user data cleanup flows to delegate shared blob and preview deletion to the scheduled reaper. Add Vercel cron integration for serverless environments.

BREAKING CHANGE: Document and user storage cleanup now relies on background scheduled tasks for shared blob and preview deletion; immediate inline deletion is no longer performed.

* refactor(user): streamline document and TTS segment transfer logic

Simplify user document transfer by consolidating storage and metadata handling. Remove inline deletion of shared blobs; only metadata is moved, and TTS segment transfer is controlled via options. Add `skipStorage` option for test scenarios to bypass storage operations. Update tests to reflect new transfer behavior.

* chore(instrumentation): delegate node-specific setup to separate module

Move Node.js-specific instrumentation logic to a dedicated file. Update
registration to dynamically import the node module only when running in a
Node.js environment. This separation clarifies environment-specific behavior
and improves maintainability.

* ci(config): update scheduled task cron to run daily at midnight

Change cron schedule for /api/admin/tasks/tick from hourly to once daily at
midnight to reduce task frequency and align with updated operational
requirements.

* refactor(tasks): enforce positive interval and improve scheduled task updates

Add database-level check constraints to ensure scheduled task intervals are always
positive for both Postgres and SQLite. Update admin API and UI to support sub-minute
intervals and stricter validation. Refactor scheduled task update logic to upsert
rows, ensuring tasks can be updated even if not yet present in the database.
Improve temporary upload cleanup to delete in paginated batches. Enhance tests to
cover new constraints and update behaviors.

* chore(docker): remove ffmpeg-static from runtime dependencies in image build

Eliminate ffmpeg-static from the production node_modules during Docker image
assembly to streamline the deployment artifact and avoid bundling unused binaries.

* refactor(admin): redesign task panel UI with Card layout and running indicator

Replace the bordered div layout in AdminTasksPanel with the Card component for
improved visual hierarchy. Add a dynamic running indicator using a custom
RunningDot component to clearly show active tasks. Update control alignment and
button states for better usability. Remove Badge-based status display in favor
of a more streamlined appearance.

* feat(tasks): add document blob lease for safe orphan reaping and improve scheduled task robustness

Introduce a document blob lease mechanism to prevent race conditions between document registration and orphaned blob cleanup. The new `document_blob_leases` table ensures that only one process can claim a document blob for mutation or deletion at a time. Update the orphan reaper to acquire a lease before deleting blobs and to re-check for ownership after acquiring the lease, avoiding accidental deletion of in-flight uploads.

Enhance scheduled task infrastructure with per-task fencing tokens to prevent stale runners from overwriting newer results, enforce runtime limits with abort signals, and expose scheduler mode and minimum interval to the admin panel and API. Adjust task handlers to accept a context with abort support, and update documentation and environment variable references for the new cron secret and scheduling behavior.

BREAKING CHANGE: Scheduled tasks now require a `document_blob_leases` table and updated handler signatures. Vercel deployments must set `CRON_SECRET` for scheduled maintenance.

* refactor(admin): add loading skeleton to tasks panel for improved UX

Introduce a TasksSkeleton component to display animated placeholders while scheduled tasks are loading. Replace direct rendering of empty task rows with the skeleton when data is pending, enhancing perceived responsiveness and user experience in the admin tasks panel.

* test: clean up playwright anonymous users

* test: make blob lease expiry deterministic

* fix(documents): improve blob lease release error handling and test stale lease scenario

Handle errors during blob lease release by logging warnings instead of allowing them to mask original results. Update unit tests to verify that releasing a stale lease does not affect a replacement lease.

* refactor(documents): implement exponential backoff with jitter for blob lease retries

Replace fixed retry delay with capped exponential backoff and jitter to reduce
contention and thundering herd effect when acquiring document blob leases. This
improves lease acquisition fairness and efficiency under high load.
2026-06-07 13:33:20 -06:00

14 KiB

title toc_max_heading_level
Environment Variables 3

This page is the source-of-truth reference for OpenReader environment variables.

:::note Recommended configuration path Use Settings → Admin as the primary source of truth for shared providers and runtime site features. API_BASE / API_KEY are optional one-time provider bootstrap seeds. Runtime site features are seeded with RUNTIME_SEED_JSON / RUNTIME_SEED_JSON_PATH. :::

Quick Reference Table

Variable Area Default When to set
LOG_FORMAT Runtime logging pretty Set json for structured logs
LOG_LEVEL Runtime logging info Set app server log level
API_BASE TTS provider bootstrap seed unset Optional first-boot base URL for default-openai
API_KEY TTS provider bootstrap seed unset Optional first-boot API key for default-openai
BASE_URL Auth unset Required at startup
AUTH_SECRET Auth unset Required at startup
AUTH_TRUSTED_ORIGINS Auth empty Add extra allowed origins
USE_ANONYMOUS_AUTH_SESSIONS Auth false Set true to allow anonymous auth sessions
GITHUB_CLIENT_ID Auth/OAuth unset Set with GITHUB_CLIENT_SECRET to enable GitHub sign-in
GITHUB_CLIENT_SECRET Auth/OAuth unset Set with GITHUB_CLIENT_ID to enable GitHub sign-in
ADMIN_EMAILS Admin empty Comma-separated emails auto-promoted to admin
CRON_SECRET Scheduled tasks unset Required for Vercel cron invocations
POSTGRES_URL Database unset (SQLite mode) Set to switch metadata/auth DB to Postgres
USE_EMBEDDED_WEED_MINI Storage true when unset Set false to use external S3-compatible storage only
WEED_MINI_DIR Storage docstore/seaweedfs Override embedded SeaweedFS data directory
WEED_MINI_WAIT_SEC Storage 20 Tune SeaweedFS startup wait timeout
S3_ACCESS_KEY_ID Storage auto-generated in embedded mode Set explicitly for stable/external credentials
S3_SECRET_ACCESS_KEY Storage auto-generated in embedded mode Set explicitly for stable/external credentials
S3_BUCKET Storage openreader-documents in embedded mode Required for external S3-compatible storage
S3_REGION Storage us-east-1 in embedded mode Required for external S3-compatible storage
S3_ENDPOINT Storage derived in embedded mode Set for S3-compatible providers (MinIO/SeaweedFS/R2/etc.)
S3_FORCE_PATH_STYLE Storage true in embedded mode Set per provider requirement
S3_PREFIX Storage openreader Customize object key prefix
IMPORT_LIBRARY_DIR Library import docstore/library fallback Set a single server library root
IMPORT_LIBRARY_DIRS Library import unset Set multiple roots (comma/colon/semicolon separated)
EMBEDDED_COMPUTE_WORKER_PORT Compute 8081 Override embedded worker bind port
EMBEDDED_NATS_PORT Compute 4222 Override embedded NATS client port
EMBEDDED_NATS_MONITOR_PORT Compute 8222 Override embedded NATS monitor port
EMBEDDED_NATS_STORE_DIR Compute docstore/nats/jetstream Override embedded JetStream storage directory
NATS_URL Compute nats://127.0.0.1:4222 in embedded startup Override embedded startup or set standalone worker URL
COMPUTE_LOG_LEVEL Compute info Compute worker log level
COMPUTE_JOB_CONCURRENCY Compute 1 Shared compute concurrency cap
COMPUTE_WHISPER_TIMEOUT_MS Compute 30000 Whisper alignment timeout budget
COMPUTE_PDF_TIMEOUT_MS Compute 300000 PDF parse timeout budget
COMPUTE_PDF_JOB_ATTEMPTS Compute 1 Max JetStream deliveries for PDF layout jobs
COMPUTE_OP_STALE_MS Compute max(30m, 4x max compute timeout) Shared stale window for compute op replacement
WHISPER_MODEL_BASE_URL Compute model source onnx-community default Override Whisper ONNX model base URL
PDF_LAYOUT_MODEL_BASE_URL Compute model source PP-DocLayoutV3 default Override PDF layout ONNX model base URL
COMPUTE_WORKER_URL External compute mode unset Set only for standalone external worker mode
COMPUTE_WORKER_TOKEN External compute mode unset Required for standalone external worker auth
FFMPEG_BIN Audio runtime auto-detected (ffmpeg-static) Override ffmpeg binary path
DISABLE_AUTH_RATE_LIMIT Auth request throttling false Set true to disable Better Auth request rate limiting
ENABLE_TEST_NAMESPACE Testing/CI unset Honor x-openreader-test-namespace header in production builds
RUN_DRIZZLE_MIGRATIONS DB migrations true Set false to skip startup Drizzle migrations
RUN_FS_MIGRATIONS Storage migrations true Set false to skip startup filesystem -> S3/DB migration pass
RUNTIME_SEED_JSON_PATH Runtime JSON seed unset Absolute path to first-boot JSON seed document
RUNTIME_SEED_JSON Runtime JSON seed unset Inline first-boot JSON seed document

Runtime Logging

LOG_FORMAT

Controls log output format for server-side Pino loggers.

  • Default: pretty
  • Allowed values: pretty, json
  • Applies to app server and compute worker

LOG_LEVEL

App server log level.

  • Default: info

TTS Provider and Request Behavior

API_BASE

Optional first-boot bootstrap base URL for the auto-created default-openai shared provider.

  • Example: http://host.docker.internal:8880/v1
  • Read only for provider bootstrap when shared providers are empty. Setting API_BASE is sufficient; API_KEY may be blank.
  • After bootstrap, provider configuration is DB-backed and managed in Settings → Admin → Shared providers.

API_KEY

Optional first-boot bootstrap API key for the auto-created default-openai shared provider.

  • Read only for provider bootstrap when shared providers are empty.
  • Stored encrypted at rest after bootstrap.
  • After bootstrap, provider configuration is DB-backed and managed in Settings → Admin → Shared providers.

Auth and Identity

BASE_URL

Required external base URL for this OpenReader instance.

  • Required at startup
  • Example: http://localhost:3003 or https://reader.example.com

AUTH_SECRET

Required secret key used by auth/session handling.

  • Required at startup
  • Generate with openssl rand -hex 32

AUTH_TRUSTED_ORIGINS

Additional allowed origins for auth requests.

  • Comma-separated list
  • BASE_URL origin is trusted automatically

USE_ANONYMOUS_AUTH_SESSIONS

Controls whether auth-enabled deployments can create/use anonymous sessions.

  • Default: false

GITHUB_CLIENT_ID

GitHub OAuth client ID.

  • Set with GITHUB_CLIENT_SECRET

GITHUB_CLIENT_SECRET

GitHub OAuth client secret.

  • Set with GITHUB_CLIENT_ID

ADMIN_EMAILS

Comma-separated list of email addresses auto-promoted to admin.

  • Requires auth to be enabled
  • Admins can manage shared providers and runtime site features in-app

CRON_SECRET

Bearer-token secret for GET /api/admin/tasks/tick.

  • Required on Vercel so scheduled maintenance tasks can run from the configured Vercel Cron.
  • Vercel automatically sends Authorization: Bearer <CRON_SECRET> on cron invocations.
  • Generate a strong random value, for example with openssl rand -hex 32.
  • Self-hosted Node.js deployments run the scheduler in-process and do not require this variable.

Database and Object Blob Storage

POSTGRES_URL

Switches metadata/auth storage from SQLite to Postgres.

  • Unset: SQLite at docstore/sqlite3.db
  • Set: Postgres mode

USE_EMBEDDED_WEED_MINI

Controls embedded SeaweedFS startup.

  • Default behavior: treated as enabled when unset
  • Set false to rely on external S3-compatible storage

WEED_MINI_DIR

Data directory for embedded SeaweedFS (weed mini).

  • Default: docstore/seaweedfs

WEED_MINI_WAIT_SEC

Max wait time for embedded SeaweedFS startup.

  • Default: 20

S3_ACCESS_KEY_ID

S3 access key.

  • Optional in embedded mode (auto-generated when unset)
  • Required for external S3 mode

S3_SECRET_ACCESS_KEY

S3 secret key.

  • Optional in embedded mode (auto-generated when unset)
  • Required for external S3 mode

S3_BUCKET

S3 bucket name.

  • Embedded default: openreader-documents
  • Required for external S3 mode

S3_REGION

S3 region.

  • Embedded default: us-east-1
  • Required for external S3 mode

S3_ENDPOINT

Custom endpoint for S3-compatible providers.

  • Optional for AWS
  • Typical for MinIO/SeaweedFS/R2

S3_FORCE_PATH_STYLE

Force path-style S3 URLs.

  • Embedded default: true

S3_PREFIX

Object key prefix.

  • Default: openreader

Library Import

IMPORT_LIBRARY_DIR

Single library source directory.

IMPORT_LIBRARY_DIRS

Multiple library roots.

  • Supports comma, colon, or semicolon-separated values

Compute Worker and Model Configuration

EMBEDDED_COMPUTE_WORKER_PORT

Embedded compute worker port.

  • Default: 8081

EMBEDDED_NATS_PORT

Embedded NATS client port.

  • Default: 4222

EMBEDDED_NATS_MONITOR_PORT

Embedded NATS monitor port.

  • Default: 8222

EMBEDDED_NATS_STORE_DIR

Embedded NATS JetStream data directory.

  • Default: docstore/nats/jetstream

NATS_URL

NATS URL used by compute services.

  • Embedded startup default: nats://127.0.0.1:4222

COMPUTE_LOG_LEVEL

Compute worker log level.

  • Default: info

COMPUTE_JOB_CONCURRENCY

Max concurrent compute jobs per worker.

  • Default: 1

COMPUTE_WHISPER_TIMEOUT_MS

Whisper alignment timeout budget.

  • Default: 30000

COMPUTE_PDF_TIMEOUT_MS

PDF parse timeout budget.

  • Default: 300000

COMPUTE_PDF_JOB_ATTEMPTS

Max JetStream deliveries for PDF layout jobs.

  • Default: 1
  • In embedded worker mode, set this in the root .env

COMPUTE_OP_STALE_MS

Stale operation window before worker/app cleanup logic can replace an op.

  • Default: max(30m, 4x max compute timeout)

WHISPER_MODEL_BASE_URL

Base URL for Whisper ONNX model downloads.

PDF_LAYOUT_MODEL_BASE_URL

Base URL for PDF layout model downloads.

COMPUTE_WORKER_URL

External compute worker URL.

  • Leave unset for embedded worker mode

COMPUTE_WORKER_TOKEN

Shared token for app-to-external-worker requests.

Audio Runtime

FFMPEG_BIN

Override ffmpeg binary path used for audio processing.

  • Used by audiobook processing routes and compute worker Whisper audio decode.

Testing and CI

DISABLE_AUTH_RATE_LIMIT

Disables Better Auth request rate limiting.

  • Default: false

ENABLE_TEST_NAMESPACE

Enables the x-openreader-test-namespace header path in production builds.

Migration Controls

RUN_DRIZZLE_MIGRATIONS

Controls startup Drizzle schema migrations.

  • Default: true
  • Set false to skip startup migration run

RUN_FS_MIGRATIONS

Controls startup filesystem-to-S3/DB migration pass.

  • Default: true
  • Set false to skip startup storage migration run

Runtime JSON Seed (v4)

RUNTIME_SEED_JSON_PATH

Path-based first-boot seed document.

  • If both RUNTIME_SEED_JSON_PATH and RUNTIME_SEED_JSON are set, path wins.
  • Value must point to a JSON file readable by the app process.

RUNTIME_SEED_JSON

Inline first-boot seed document.

  • Used only when RUNTIME_SEED_JSON_PATH is unset.
  • Must be a JSON object with version: 1.

Supported top-level keys:

  • version (required, must be 1)
  • runtimeConfig (optional object, strict-validated against runtime schema)
  • providers (optional array of shared provider seed entries)

Example:

{
  "version": 1,
  "runtimeConfig": {
    "enableUserSignups": true,
    "restrictUserApiKeys": true,
    "defaultTtsProvider": "custom-openai",
    "enableTtsProvidersTab": true,
    "enableAudiobookExport": true,
    "enableDocxConversion": true,
    "showAllProviderModels": true,
    "disableTtsRateLimit": true,
    "ttsDailyLimitAnonymous": 50000,
    "ttsDailyLimitAuthenticated": 500000,
    "ttsIpDailyLimitAnonymous": 100000,
    "ttsIpDailyLimitAuthenticated": 1000000,
    "ttsCacheMaxSizeBytes": 268435456,
    "ttsCacheTtlMs": 1800000,
    "ttsUpstreamMaxRetries": 2,
    "ttsUpstreamTimeoutMs": 285000,
    "disableComputeRateLimit": true,
    "computeParseBurstMax": 8,
    "computeParseBurstWindowSec": 60,
    "computeParseSustainedMax": 24,
    "computeParseSustainedWindowSec": 600,
    "maxUploadMb": 200,
    "changelogFeedUrl": "https://docs.openreader.richardr.dev/changelog/manifest.json"
  },
  "providers": [
    {
      "slug": "default-openai",
      "displayName": "Default (seeded)",
      "providerType": "custom-openai",
      "baseUrl": "http://localhost:8880/v1",
      "defaultModel": "kokoro",
      "enabled": true
    }
  ]
}

Provider fallback behavior:

  • If the JSON seed includes providers (including an empty array), API_BASE / API_KEY fallback is skipped.
  • If the JSON seed does not include a providers key, the legacy API_BASE / API_KEY bootstrap fallback can still create default-openai when provider rows are empty. API_BASE alone is sufficient for an upstream that does not require authentication.

Precedence summary:

  • Runtime reads: admin DB runtime rows override built-in defaults.
  • Seed input (RUNTIME_SEED_JSON*) only populates missing runtime rows on first boot; it does not overwrite existing/admin-edited rows.
  • Provider bootstrap order: JSON providers section > API_BASE/API_KEY fallback > no provider bootstrap.