e670c38523
* refactor(user): overhaul user data cleanup and export for cascading deletes and TTS segment support Revise user data cleanup logic to ensure proper cascading deletion of user-related database rows and S3 objects, including shared document and preview artifacts. Introduce explicit checks for last ownership before removing shared resources. Add TTS segment cache and audio cleanup to document and user deletion flows. Expand user data export to include TTS segment entries and audio, job events, document settings, and linked auth sessions. Update schema with ON DELETE CASCADE for userTtsChars and userJobEvents. Add new migration scripts and comprehensive unit tests for cleanup and export scenarios. * test(user): expand cleanup and export coverage for shared docs and TTS segment files Add tests for user data cleanup with shared document ownership and for TTS segment variant export scenarios, including cases with duplicate audio keys and storage disabled. Refactor cleanup logic to use document row types and transactional ownership removal with last-owner checks. Update TTS segment cache clearing to report deleted segment count by unique entry. Adjust export logic to always include all TTS segment variants and conditionally export file buckets based on storage availability. Update migration to use NOT VALID/VALIDATE for new cascading constraints. * refactor(documents): centralize owned document deletion and add mutation locking Introduce `deleteOwnedDocument` utility to encapsulate all logic for removing a user's ownership of a document, including TTS segment cache cleanup, preview artifact removal, and S3 blob deletion for last-owner cases. Add `withDocumentMutationLock` to serialize concurrent mutations on the same document, using advisory locks in Postgres and a local queue fallback. Refactor all API routes and user data flows to use these utilities, ensuring transactional safety and preventing race conditions during document deletion or transfer. Update tests for new flows and add coverage for document cleanup sequencing and locking behavior. * feat(data): enhance anonymous claim to support document settings and TTS segment copy Expand the anonymous data claim process to include document settings transfer and TTS segment S3 prefix copying. Remove foreign key constraints from user_tts_chars to allow non-user buckets. Update claim modal and onboarding flow to display claimed document settings. Refactor TTS char count transfer to merge all dates and fix upsert logic. Add S3 copy utility for TTS segments and corresponding tests. Update migrations and schema to reflect relaxed constraints. * refactor(data): improve error handling and rollback for document and TTS segment operations Enhance robustness of document deletion and TTS segment transfer by improving error logging, partial rollback, and degraded state reporting. Add best-effort cleanup and recovery mechanisms to prevent orphaned data and ensure diagnostic information is captured for unexpected failures. Update user data cleanup to log and swallow restoration errors without masking primary failures. Refine claim logic to handle unmapped TTS audio keys safely. * refactor(data): unify document mutation locking and transaction handling Replace mutation-lock with document-lock to centralize document mutation serialization and database transaction management. Introduce runInDbTransaction utility to abstract SQLite/Postgres transaction differences. Update document deletion, user data cleanup, and rate limiter logic to delegate transaction handling and locking to shared helpers. Remove dialect-specific branching and inline transaction logic for improved maintainability and testability. BREAKING CHANGE: withDocumentMutationLock is removed in favor of withDocumentLock and runInDbTransaction * fix(documents): improve rollback error handling in deleteOwnedDocument Enhance error handling during document deletion by ensuring that failures in restoring document ownership do not obscure the original error. Log degraded events when ownership restoration fails after a deletion error, providing additional context for debugging and monitoring. This change improves reliability and traceability of document deletion operations. * feat(tasks): introduce scheduled task engine and admin UI for background jobs Add a general-purpose scheduled task system with a persistent registry and status tracking, supporting background maintenance jobs such as orphaned blob reaping, expired upload cleanup, job event pruning, and TTS usage retention. Implement a new `scheduled_tasks` table, task engine, and handlers for each maintenance operation. Integrate an admin UI panel for monitoring, manual runs, and configuration of tasks. Update document and user data cleanup flows to delegate shared blob and preview deletion to the scheduled reaper. Add Vercel cron integration for serverless environments. BREAKING CHANGE: Document and user storage cleanup now relies on background scheduled tasks for shared blob and preview deletion; immediate inline deletion is no longer performed. * refactor(user): streamline document and TTS segment transfer logic Simplify user document transfer by consolidating storage and metadata handling. Remove inline deletion of shared blobs; only metadata is moved, and TTS segment transfer is controlled via options. Add `skipStorage` option for test scenarios to bypass storage operations. Update tests to reflect new transfer behavior. * chore(instrumentation): delegate node-specific setup to separate module Move Node.js-specific instrumentation logic to a dedicated file. Update registration to dynamically import the node module only when running in a Node.js environment. This separation clarifies environment-specific behavior and improves maintainability. * ci(config): update scheduled task cron to run daily at midnight Change cron schedule for /api/admin/tasks/tick from hourly to once daily at midnight to reduce task frequency and align with updated operational requirements. * refactor(tasks): enforce positive interval and improve scheduled task updates Add database-level check constraints to ensure scheduled task intervals are always positive for both Postgres and SQLite. Update admin API and UI to support sub-minute intervals and stricter validation. Refactor scheduled task update logic to upsert rows, ensuring tasks can be updated even if not yet present in the database. Improve temporary upload cleanup to delete in paginated batches. Enhance tests to cover new constraints and update behaviors. * chore(docker): remove ffmpeg-static from runtime dependencies in image build Eliminate ffmpeg-static from the production node_modules during Docker image assembly to streamline the deployment artifact and avoid bundling unused binaries. * refactor(admin): redesign task panel UI with Card layout and running indicator Replace the bordered div layout in AdminTasksPanel with the Card component for improved visual hierarchy. Add a dynamic running indicator using a custom RunningDot component to clearly show active tasks. Update control alignment and button states for better usability. Remove Badge-based status display in favor of a more streamlined appearance. * feat(tasks): add document blob lease for safe orphan reaping and improve scheduled task robustness Introduce a document blob lease mechanism to prevent race conditions between document registration and orphaned blob cleanup. The new `document_blob_leases` table ensures that only one process can claim a document blob for mutation or deletion at a time. Update the orphan reaper to acquire a lease before deleting blobs and to re-check for ownership after acquiring the lease, avoiding accidental deletion of in-flight uploads. Enhance scheduled task infrastructure with per-task fencing tokens to prevent stale runners from overwriting newer results, enforce runtime limits with abort signals, and expose scheduler mode and minimum interval to the admin panel and API. Adjust task handlers to accept a context with abort support, and update documentation and environment variable references for the new cron secret and scheduling behavior. BREAKING CHANGE: Scheduled tasks now require a `document_blob_leases` table and updated handler signatures. Vercel deployments must set `CRON_SECRET` for scheduled maintenance. * refactor(admin): add loading skeleton to tasks panel for improved UX Introduce a TasksSkeleton component to display animated placeholders while scheduled tasks are loading. Replace direct rendering of empty task rows with the skeleton when data is pending, enhancing perceived responsiveness and user experience in the admin tasks panel. * test: clean up playwright anonymous users * test: make blob lease expiry deterministic * fix(documents): improve blob lease release error handling and test stale lease scenario Handle errors during blob lease release by logging warnings instead of allowing them to mask original results. Update unit tests to verify that releasing a stale lease does not affect a replacement lease. * refactor(documents): implement exponential backoff with jitter for blob lease retries Replace fixed retry delay with capped exponential backoff and jitter to reduce contention and thundering herd effect when acquiring document blob leases. This improves lease acquisition fairness and efficiency under high load.
373 lines
11 KiB
Markdown
373 lines
11 KiB
Markdown
---
|
|
title: Local Development
|
|
---
|
|
|
|
import Tabs from '@theme/Tabs';
|
|
import TabItem from '@theme/TabItem';
|
|
|
|
## Prerequisites
|
|
|
|
<details>
|
|
<summary><strong>Node.js + pnpm (required)</strong></summary>
|
|
|
|
<Tabs groupId="local-dev-node-pnpm-os">
|
|
<TabItem value="macos" label="macOS" default>
|
|
|
|
```bash
|
|
brew install nvm pnpm
|
|
mkdir -p ~/.nvm
|
|
echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.zshrc
|
|
echo '[ -s "$(brew --prefix nvm)/nvm.sh" ] && . "$(brew --prefix nvm)/nvm.sh"' >> ~/.zshrc
|
|
source ~/.zshrc
|
|
nvm install --lts
|
|
nvm use --lts
|
|
node -v
|
|
pnpm -v
|
|
```
|
|
|
|
</TabItem>
|
|
<TabItem value="linux" label="Linux">
|
|
|
|
```bash
|
|
# Debian/Ubuntu example
|
|
sudo apt update
|
|
sudo apt install -y curl
|
|
curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
|
|
export NVM_DIR="$HOME/.nvm"
|
|
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
|
|
nvm install --lts
|
|
nvm use --lts
|
|
corepack enable
|
|
corepack prepare pnpm@latest --activate
|
|
node -v
|
|
pnpm -v
|
|
```
|
|
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary><strong>SeaweedFS <code>weed</code> binary (required unless using external S3)</strong></summary>
|
|
|
|
<Tabs groupId="local-dev-seaweed-os">
|
|
<TabItem value="macos" label="macOS" default>
|
|
|
|
```bash
|
|
brew install seaweedfs
|
|
weed version
|
|
```
|
|
|
|
:::warning SeaweedFS Compatibility Note (April 16, 2026)
|
|
If you see intermittent S3 `InternalError` upload failures with embedded storage, use SeaweedFS `4.18`.
|
|
OpenReader currently pins `4.18` in CI and Docker builds while `4.19` compatibility is investigated.
|
|
:::
|
|
|
|
</TabItem>
|
|
<TabItem value="linux" label="Linux">
|
|
|
|
```bash
|
|
# Linux amd64 example (pin 4.18)
|
|
mkdir -p "$HOME/.local/bin"
|
|
curl -fsSL -o /tmp/seaweedfs.tar.gz \
|
|
https://github.com/seaweedfs/seaweedfs/releases/download/4.18/linux_amd64.tar.gz
|
|
tar -xzf /tmp/seaweedfs.tar.gz -C /tmp weed
|
|
install -m 0755 /tmp/weed "$HOME/.local/bin/weed"
|
|
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
|
|
export PATH="$HOME/.local/bin:$PATH"
|
|
weed version
|
|
```
|
|
|
|
:::warning SeaweedFS Compatibility Note (April 16, 2026)
|
|
If you see intermittent S3 `InternalError` upload failures with embedded storage, use SeaweedFS `4.18`.
|
|
OpenReader currently pins `4.18` in CI and Docker builds while `4.19` compatibility is investigated.
|
|
:::
|
|
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary><strong>NATS Server <code>nats-server</code> (required for embedded compute mode)</strong></summary>
|
|
|
|
If `COMPUTE_WORKER_URL` is unset, startup launches embedded compute worker + NATS, so `nats-server` must be available on host PATH.
|
|
|
|
If you always use an external worker (`COMPUTE_WORKER_URL` set), this is not required.
|
|
|
|
<Tabs groupId="local-dev-nats-os">
|
|
<TabItem value="macos" label="macOS" default>
|
|
|
|
```bash
|
|
brew install nats-server
|
|
nats-server -v
|
|
```
|
|
|
|
</TabItem>
|
|
<TabItem value="linux" label="Linux">
|
|
|
|
```bash
|
|
# Linux amd64 example
|
|
mkdir -p "$HOME/.local/bin"
|
|
curl -fsSL -o /tmp/nats-server.zip \
|
|
https://github.com/nats-io/nats-server/releases/latest/download/nats-server-v2.12.1-linux-amd64.zip
|
|
unzip -j /tmp/nats-server.zip '*/nats-server' -d /tmp
|
|
install -m 0755 /tmp/nats-server "$HOME/.local/bin/nats-server"
|
|
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
|
|
export PATH="$HOME/.local/bin:$PATH"
|
|
nats-server -v
|
|
```
|
|
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary><strong>LibreOffice (optional, for DOCX conversion)</strong></summary>
|
|
|
|
<Tabs groupId="local-dev-libreoffice-os">
|
|
<TabItem value="macos" label="macOS" default>
|
|
|
|
```bash
|
|
brew install libreoffice
|
|
```
|
|
|
|
</TabItem>
|
|
<TabItem value="linux" label="Linux">
|
|
|
|
```bash
|
|
# Debian/Ubuntu example
|
|
sudo apt update
|
|
sudo apt install -y libreoffice
|
|
```
|
|
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary><strong>Word-by-word highlighting (optional)</strong></summary>
|
|
|
|
No extra native Whisper CLI build step is required.
|
|
|
|
Word-by-word highlighting and PDF layout parsing are worker-backed in current releases.
|
|
|
|
If you need mirrors or pinned artifact locations, set `WHISPER_MODEL_BASE_URL` in `.env` (current defaults expect q4 Whisper files at that base URL).
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary><strong>External compute worker dev stack (optional)</strong></summary>
|
|
|
|
Use this only when you intentionally run compute-worker as a separate service.
|
|
Default local flow does not need `compute/worker/.env`; embedded worker startup reads root `.env`.
|
|
Full worker deployment details are in [Compute Worker (NATS JetStream)](./compute-worker).
|
|
|
|
Start only NATS + compute-worker via compose watch:
|
|
|
|
```bash
|
|
docker compose --env-file compute/worker/.env -f compute/worker/docker-compose.yml up --watch
|
|
# or: pnpm compute:dev:watch
|
|
```
|
|
|
|
`compute/worker/.env.example` contains a starter config for standalone worker service deployments.
|
|
|
|
Run the main app separately on the host:
|
|
|
|
```bash
|
|
pnpm dev
|
|
```
|
|
|
|
For app -> external worker routing, set in root `.env`:
|
|
|
|
```env
|
|
COMPUTE_WORKER_URL=http://localhost:8081
|
|
COMPUTE_WORKER_TOKEN=<same-token-used-by-worker>
|
|
```
|
|
|
|
Ownership in external worker mode:
|
|
- root `.env`: app routing/auth (`COMPUTE_WORKER_URL`, `COMPUTE_WORKER_TOKEN`) plus optional shared timeout/stale/retry overrides such as `COMPUTE_PDF_JOB_ATTEMPTS`
|
|
- `compute/worker/.env*` (or worker platform env): worker runtime variables (`NATS_*`, `S3_*`, model base URLs, worker tuning)
|
|
|
|
For embedded worker startup (`COMPUTE_WORKER_URL` unset), worker tuning values such as `COMPUTE_PDF_JOB_ATTEMPTS` must be set in the root `.env` because `compute/worker/.env*` is ignored in that mode.
|
|
|
|
Worker mode requires worker-reachable shared object storage (S3-compatible endpoint).
|
|
For external worker mode, object storage must be shared/reachable by both app and worker services.
|
|
|
|
</details>
|
|
|
|
## Steps
|
|
|
|
### Required flow
|
|
|
|
1. Clone the repository.
|
|
|
|
```bash
|
|
git clone https://github.com/richardr1126/openreader.git
|
|
cd openreader
|
|
```
|
|
|
|
2. Install dependencies.
|
|
|
|
```bash
|
|
pnpm i
|
|
```
|
|
|
|
3. Configure the environment.
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
```
|
|
|
|
Then edit `.env`.
|
|
|
|
Default embedded worker flow (no external worker URL):
|
|
|
|
```env
|
|
# Leave COMPUTE_WORKER_URL unset.
|
|
# Entry point auto-starts embedded worker+NATS when available.
|
|
```
|
|
|
|
External worker flow:
|
|
|
|
```env
|
|
COMPUTE_WORKER_URL=http://localhost:8081
|
|
COMPUTE_WORKER_TOKEN=<same-token-used-by-worker>
|
|
```
|
|
|
|
Use the same ownership split:
|
|
- root `.env`: app routing/auth (`COMPUTE_WORKER_URL`, `COMPUTE_WORKER_TOKEN`) plus optional shared timeout/stale overrides
|
|
- `compute/worker/.env*` (or worker platform env): worker runtime variables (`NATS_*`, `S3_*`, model base URLs, worker tuning)
|
|
|
|
Use one of these `.env` mode templates:
|
|
|
|
<Tabs groupId="local-env-modes">
|
|
<TabItem value="auth-enabled" label="Auth Enabled" default>
|
|
|
|
```env
|
|
API_BASE=http://host.docker.internal:8880/v1
|
|
BASE_URL=http://localhost:3003
|
|
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
|
|
# Optional when you need multiple local origins:
|
|
# AUTH_TRUSTED_ORIGINS=http://localhost:3003,http://127.0.0.1:3003
|
|
```
|
|
|
|
</TabItem>
|
|
<TabItem value="auth-with-admin" label="Auth + Admin Panel">
|
|
|
|
```env
|
|
# API_BASE and optional API_KEY are seeded into the admin "default-openai" shared provider
|
|
# on first boot, then no longer read. Manage them in Settings → Admin afterwards.
|
|
API_BASE=http://host.docker.internal:8880/v1
|
|
BASE_URL=http://localhost:3003
|
|
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
|
|
# Comma-separated emails to auto-promote to admin on signin.
|
|
ADMIN_EMAILS=you@example.com
|
|
```
|
|
|
|
</TabItem>
|
|
<TabItem value="external-s3" label="External S3">
|
|
|
|
```env
|
|
API_BASE=http://host.docker.internal:8880/v1
|
|
USE_EMBEDDED_WEED_MINI=false
|
|
BASE_URL=http://localhost:3003
|
|
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
|
|
S3_BUCKET=your-bucket
|
|
S3_REGION=us-east-1
|
|
S3_ACCESS_KEY_ID=your-access-key
|
|
S3_SECRET_ACCESS_KEY=your-secret-key
|
|
# Optional for non-AWS providers:
|
|
# S3_ENDPOINT=https://your-s3-compatible-endpoint
|
|
# S3_FORCE_PATH_STYLE=true
|
|
```
|
|
|
|
</TabItem>
|
|
<TabItem value="worker-mode" label="External Worker Service">
|
|
|
|
```env
|
|
API_BASE=http://host.docker.internal:8880/v1
|
|
BASE_URL=http://localhost:3003
|
|
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
|
|
COMPUTE_WORKER_URL=http://localhost:8081
|
|
COMPUTE_WORKER_TOKEN=<same-token-used-by-worker>
|
|
USE_EMBEDDED_WEED_MINI=false
|
|
S3_BUCKET=your-bucket
|
|
S3_REGION=us-east-1
|
|
S3_ACCESS_KEY_ID=your-access-key
|
|
S3_SECRET_ACCESS_KEY=your-secret-key
|
|
# Optional for non-AWS providers:
|
|
# S3_ENDPOINT=https://your-s3-compatible-endpoint
|
|
# S3_FORCE_PATH_STYLE=true
|
|
```
|
|
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
:::note Env vars vs. admin panel
|
|
On first boot, `API_KEY` / `API_BASE` can bootstrap `default-openai`, and `RUNTIME_SEED_JSON` / `RUNTIME_SEED_JSON_PATH` can seed runtime config + providers. After that, the admin UI is authoritative and editing bootstrap env vars no longer changes app behavior. See [Admin Panel](../configure/admin-panel).
|
|
:::
|
|
|
|
:::note User BYOK restriction default
|
|
If you want each user to enter personal provider credentials, set `restrictUserApiKeys=false` (from **Settings → Admin**, or by seeding `runtimeConfig.restrictUserApiKeys=false` in runtime seed JSON).
|
|
:::
|
|
|
|
:::info
|
|
For all environment variables, see [Environment Variables](../reference/environment-variables).
|
|
:::
|
|
|
|
See [Auth](../configure/auth) for app/auth behavior.
|
|
See [Admin Panel](../configure/admin-panel) for the shared-provider and feature-flag management UI.
|
|
Storage configuration details are in [Object / Blob Storage](../configure/object-blob-storage).
|
|
Refer to [Database](../configure/database) for database modes.
|
|
Learn about migration behavior and commands in [Migrations](../configure/migrations).
|
|
|
|
:::info Scheduled maintenance tasks
|
|
Local and self-hosted Node.js deployments start the scheduled-task loop in-process and check for due work once per minute. No `CRON_SECRET` is required unless you intentionally invoke the cron HTTP route yourself. Manage task intervals and inspect failures from **Settings → Admin → Scheduled tasks**.
|
|
:::
|
|
|
|
4. Start the app.
|
|
|
|
<Tabs groupId="local-run-mode">
|
|
<TabItem value="dev" label="Dev (recommended)" default>
|
|
|
|
```bash
|
|
pnpm dev
|
|
```
|
|
|
|
If you use embedded worker startup (no `COMPUTE_WORKER_URL`) and the host is missing `nats-server`,
|
|
install `nats-server` locally or switch to external worker mode.
|
|
|
|
</TabItem>
|
|
<TabItem value="prod" label="Build + Start">
|
|
|
|
```bash
|
|
pnpm build
|
|
pnpm start
|
|
```
|
|
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
:::warning API Base Reachability
|
|
`API_BASE` must be reachable from the Next.js server process, not just your browser.
|
|
:::
|
|
|
|
Visit [http://localhost:3003](http://localhost:3003).
|
|
|
|
### Optional workflows
|
|
|
|
Run manual DB migrations only for troubleshooting or explicit migration workflows:
|
|
|
|
- Migrations run automatically on startup through the shared entrypoint for both `pnpm dev` and `pnpm start`.
|
|
|
|
```bash
|
|
pnpm migrate
|
|
```
|
|
|
|
:::info
|
|
If `POSTGRES_URL` is set, migrations target Postgres; otherwise local SQLite is used. To disable automatic startup migrations, set `RUN_DRIZZLE_MIGRATIONS=false` and/or `RUN_FS_MIGRATIONS=false`. You can run storage migration manually with `pnpm migrate-fs`.
|
|
:::
|