openreader/docs-site/docs/configure/admin-panel.md at main

Refactor user data cleanup and TTS storage flows (#105 )

* refactor(user): overhaul user data cleanup and export for cascading deletes and TTS segment support

Revise user data cleanup logic to ensure proper cascading deletion of user-related
database rows and S3 objects, including shared document and preview artifacts.
Introduce explicit checks for last ownership before removing shared resources.
Add TTS segment cache and audio cleanup to document and user deletion flows.
Expand user data export to include TTS segment entries and audio, job events,
document settings, and linked auth sessions. Update schema with ON DELETE CASCADE
for userTtsChars and userJobEvents. Add new migration scripts and comprehensive
unit tests for cleanup and export scenarios.

* test(user): expand cleanup and export coverage for shared docs and TTS segment files

Add tests for user data cleanup with shared document ownership and for TTS segment
variant export scenarios, including cases with duplicate audio keys and storage
disabled. Refactor cleanup logic to use document row types and transactional
ownership removal with last-owner checks. Update TTS segment cache clearing to
report deleted segment count by unique entry. Adjust export logic to always
include all TTS segment variants and conditionally export file buckets based on
storage availability. Update migration to use NOT VALID/VALIDATE for new
cascading constraints.

* refactor(documents): centralize owned document deletion and add mutation locking

Introduce `deleteOwnedDocument` utility to encapsulate all logic for removing a user's ownership of a document, including TTS segment cache cleanup, preview artifact removal, and S3 blob deletion for last-owner cases. Add `withDocumentMutationLock` to serialize concurrent mutations on the same document, using advisory locks in Postgres and a local queue fallback. Refactor all API routes and user data flows to use these utilities, ensuring transactional safety and preventing race conditions during document deletion or transfer. Update tests for new flows and add coverage for document cleanup sequencing and locking behavior.

* feat(data): enhance anonymous claim to support document settings and TTS segment copy

Expand the anonymous data claim process to include document settings transfer and TTS segment S3 prefix copying. Remove foreign key constraints from user_tts_chars to allow non-user buckets. Update claim modal and onboarding flow to display claimed document settings. Refactor TTS char count transfer to merge all dates and fix upsert logic. Add S3 copy utility for TTS segments and corresponding tests. Update migrations and schema to reflect relaxed constraints.

* refactor(data): improve error handling and rollback for document and TTS segment operations

Enhance robustness of document deletion and TTS segment transfer by improving
error logging, partial rollback, and degraded state reporting. Add best-effort
cleanup and recovery mechanisms to prevent orphaned data and ensure diagnostic
information is captured for unexpected failures. Update user data cleanup to
log and swallow restoration errors without masking primary failures. Refine
claim logic to handle unmapped TTS audio keys safely.

* refactor(data): unify document mutation locking and transaction handling

Replace mutation-lock with document-lock to centralize document mutation
serialization and database transaction management. Introduce runInDbTransaction
utility to abstract SQLite/Postgres transaction differences. Update document
deletion, user data cleanup, and rate limiter logic to delegate transaction
handling and locking to shared helpers. Remove dialect-specific branching and
inline transaction logic for improved maintainability and testability.

BREAKING CHANGE: withDocumentMutationLock is removed in favor of withDocumentLock and runInDbTransaction

* fix(documents): improve rollback error handling in deleteOwnedDocument

Enhance error handling during document deletion by ensuring that failures
in restoring document ownership do not obscure the original error. Log
degraded events when ownership restoration fails after a deletion error,
providing additional context for debugging and monitoring. This change
improves reliability and traceability of document deletion operations.

* feat(tasks): introduce scheduled task engine and admin UI for background jobs

Add a general-purpose scheduled task system with a persistent registry and status tracking, supporting background maintenance jobs such as orphaned blob reaping, expired upload cleanup, job event pruning, and TTS usage retention. Implement a new `scheduled_tasks` table, task engine, and handlers for each maintenance operation. Integrate an admin UI panel for monitoring, manual runs, and configuration of tasks. Update document and user data cleanup flows to delegate shared blob and preview deletion to the scheduled reaper. Add Vercel cron integration for serverless environments.

BREAKING CHANGE: Document and user storage cleanup now relies on background scheduled tasks for shared blob and preview deletion; immediate inline deletion is no longer performed.

* refactor(user): streamline document and TTS segment transfer logic

Simplify user document transfer by consolidating storage and metadata handling. Remove inline deletion of shared blobs; only metadata is moved, and TTS segment transfer is controlled via options. Add `skipStorage` option for test scenarios to bypass storage operations. Update tests to reflect new transfer behavior.

* chore(instrumentation): delegate node-specific setup to separate module

Move Node.js-specific instrumentation logic to a dedicated file. Update
registration to dynamically import the node module only when running in a
Node.js environment. This separation clarifies environment-specific behavior
and improves maintainability.

* ci(config): update scheduled task cron to run daily at midnight

Change cron schedule for /api/admin/tasks/tick from hourly to once daily at
midnight to reduce task frequency and align with updated operational
requirements.

* refactor(tasks): enforce positive interval and improve scheduled task updates

Add database-level check constraints to ensure scheduled task intervals are always
positive for both Postgres and SQLite. Update admin API and UI to support sub-minute
intervals and stricter validation. Refactor scheduled task update logic to upsert
rows, ensuring tasks can be updated even if not yet present in the database.
Improve temporary upload cleanup to delete in paginated batches. Enhance tests to
cover new constraints and update behaviors.

* chore(docker): remove ffmpeg-static from runtime dependencies in image build

Eliminate ffmpeg-static from the production node_modules during Docker image
assembly to streamline the deployment artifact and avoid bundling unused binaries.

* refactor(admin): redesign task panel UI with Card layout and running indicator

Replace the bordered div layout in AdminTasksPanel with the Card component for
improved visual hierarchy. Add a dynamic running indicator using a custom
RunningDot component to clearly show active tasks. Update control alignment and
button states for better usability. Remove Badge-based status display in favor
of a more streamlined appearance.

* feat(tasks): add document blob lease for safe orphan reaping and improve scheduled task robustness

Introduce a document blob lease mechanism to prevent race conditions between document registration and orphaned blob cleanup. The new `document_blob_leases` table ensures that only one process can claim a document blob for mutation or deletion at a time. Update the orphan reaper to acquire a lease before deleting blobs and to re-check for ownership after acquiring the lease, avoiding accidental deletion of in-flight uploads.

Enhance scheduled task infrastructure with per-task fencing tokens to prevent stale runners from overwriting newer results, enforce runtime limits with abort signals, and expose scheduler mode and minimum interval to the admin panel and API. Adjust task handlers to accept a context with abort support, and update documentation and environment variable references for the new cron secret and scheduling behavior.

BREAKING CHANGE: Scheduled tasks now require a `document_blob_leases` table and updated handler signatures. Vercel deployments must set `CRON_SECRET` for scheduled maintenance.

* refactor(admin): add loading skeleton to tasks panel for improved UX

Introduce a TasksSkeleton component to display animated placeholders while scheduled tasks are loading. Replace direct rendering of empty task rows with the skeleton when data is pending, enhancing perceived responsiveness and user experience in the admin tasks panel.

* test: clean up playwright anonymous users

* test: make blob lease expiry deterministic

* fix(documents): improve blob lease release error handling and test stale lease scenario

Handle errors during blob lease release by logging warnings instead of allowing them to mask original results. Update unit tests to verify that releasing a stale lease does not affect a replacement lease.

* refactor(documents): implement exponential backoff with jitter for blob lease retries

Replace fixed retry delay with capped exponential backoff and jitter to reduce
contention and thundering herd effect when acquiring document blob leases. This
improves lease acquisition fairness and efficiency under high load.

2026-06-07 13:33:20 -06:00

10 KiB

Raw Permalink Blame History

title
Admin Panel

The admin panel lets a designated set of users manage shared TTS providers and site-wide feature flags directly from the Settings modal — without touching env vars or redeploying.

It is gated behind authentication, so you must have auth enabled to use it (Auth).

Designating admins

Set ADMIN_EMAILS to a comma-separated list of emails:

AUTH_SECRET=...        # required for auth
BASE_URL=...           # required for auth
ADMIN_EMAILS=alice@example.com,bob@example.com

On every session resolution the server compares the user's email against this list and writes user.is_admin = true (or false for emails removed from the list). No restart is required to demote — the next page load picks it up.

When the logged-in user is an admin, an Admin tab appears in Settings → sidebar with two sub-tabs:

Shared providers — server-side TTS provider instances visible to all users.
Site features — runtime-editable replacements for what were previously build-time public env flags.

Shared TTS providers

Each shared provider is one named instance bound to one of the four built-in provider types (custom-openai, openai, replicate, deepinfra). The admin form has:

Field	Notes
Slug	URL-safe identifier exposed to users (e.g. `kokoro-prod`). Must not collide with a built-in id. Lowercase alphanumeric + hyphens.
Display name	Shown in the user's provider dropdown, suffixed with "(shared)".
Provider type	One of the four built-ins. Determines voice/model resolution.
Base URL	Optional. Falls through to the provider type's default when blank.
API key	Encrypted at rest with AES-256-GCM (key derived from `AUTH_SECRET` via scrypt). On edit, leave blank to keep the existing key.
Default model	Optional. Used as the initial model when a user selects this provider.
Enabled	Toggle to hide the provider from non-admin users without deleting it.

When a non-admin user picks a shared provider in Settings → TTS Provider:

The API key / base URL fields are hidden — those credentials never leave the server.
The TTS request still goes through the user's browser, but the server replaces the slug with the matching admin row's decrypted key and base URL before calling the upstream provider.
The user's per-request x-openai-key / x-openai-base-url headers are ignored for shared slugs.

Whether users can supply their own personal built-in provider keys is controlled by the site feature restrictUserApiKeys:

true: users are restricted to shared providers only.
false: users may also use per-user BYOK credentials for built-in providers.

Auto-seeded "default-openai"

On first boot, if admin_providers is empty and API_BASE or API_KEY is set, OpenReader creates a single shared provider with:

slug default-openai, displayName Default (from env), providerType custom-openai
baseUrl from API_BASE, apiKey from API_KEY when provided (blank keys are supported)
defaultModel set to kokoro (you can edit it in Admin → Shared providers)

After this seed runs, the legacy API_KEY / API_BASE env vars are no longer read by the TTS routes — the DB row is authoritative. You can rename, edit, disable, or delete this row like any other from the admin UI, and remove the env vars from your .env when convenient.

:::warning Upgrading from v2.2.0 In v2.2.0 and earlier, API_KEY / API_BASE were read live by the TTS routes on every request. As of v3.0.0 they are one-shot seeds consumed only on the first boot where admin_providers is empty. After upgrading, boot the app once and confirm a default-openai row exists in Settings → Admin → Shared providers with the correct base URL. If it is missing or wrong (e.g. the env vars were not set on first boot, or the table was already non-empty from a pre-release), create or edit the shared provider manually — TTS will not fall back to the env vars. :::

Site features

Runtime-editable settings, one row per key:

Key	What it controls
`defaultTtsProvider`	Default provider id new users start with (built-in id or shared slug).
`changelogFeedUrl`	Public changelog manifest URL used by the Settings modal changelog panel.
`enableUserSignups`	Controls whether new accounts can be created. Existing accounts can still sign in when this is `false`.
`restrictUserApiKeys`	Restrict user-supplied API keys/base URLs; when `true`, only admin shared providers are allowed.
`enableTtsProvidersTab`	Whether the user-facing TTS Provider tab in Settings is shown.
`showAllProviderModels`	When `false`, users are restricted to each provider's default model (shared provider `defaultModel` or built-in provider default).
`enableAudiobookExport`	Show the audiobook export entry points on PDF/EPUB pages.
`enableDocxConversion`	Accept .docx uploads (converted to PDF server-side).

Word-by-word highlighting and PDF layout parsing capability are controlled by compute-worker server env configuration, not an admin runtime flag.

Each row shows a source badge:

from seed — the value was seeded on first boot (from RUNTIME_SEED_JSON / RUNTIME_SEED_JSON_PATH).
admin — explicit admin override. Use Reset on the row to clear it back to built-in default behavior.
default — no seed/admin row exists; built-in default is active.

:::warning Security note for restrictUserApiKeys Turning restrictUserApiKeys off allows user-supplied API keys to flow through this server. Use this only for trusted/self-hosted deployments where that tradeoff is acceptable. :::

Rate limiting

A dedicated Rate limiting group (within the same admin panel) collects the daily quotas, the PDF parsing throttle, and the upload size cap:

Key	What it controls
`disableTtsRateLimit`	Disable the per-user/IP daily TTS character limits. When `false`, the daily-limit fields below it apply.
`disableComputeRateLimit`	Disable per-user PDF parsing rate limiting. When `false`, the burst/sustained limit fields below it apply.
`maxUploadMb`	Maximum size (MB) accepted for a single document upload. Enforced server-side and signed into the presigned S3 PUT.

The Disable TTS daily rate limiting and Disable PDF parsing rate limiting toggles each reveal a collapsible group of numeric inputs when set to false:

TTS: anonymous/authenticated per-user daily limits and anonymous/authenticated IP daily backstops.
PDF parsing: burst limit + window (seconds) and sustained limit + window (seconds). The sustained window doubles as a concurrency cap.

TTS upstream

At the end of the Site features tab, a dedicated TTS upstream group controls server-side request and cache tuning (DB-backed runtime settings, not env vars):

Key	What it controls
`ttsUpstreamMaxRetries`	Maximum retry attempts for upstream TTS 429/5xx responses.
`ttsUpstreamTimeoutMs`	Upstream request timeout for OpenAI-compatible TTS calls.
`ttsCacheMaxSizeBytes`	Maximum size of the in-memory TTS audio cache.
`ttsCacheTtlMs`	Time-to-live for cached TTS audio buffers.

In v4 these settings are admin-only and are no longer configurable through environment variables.

Scheduled tasks

The Scheduled tasks section controls background maintenance jobs such as expired-upload cleanup, orphaned-blob reaping, and rate-limit ledger pruning.

Enable or disable each task, adjust its interval, or run it immediately.
Runs use database-backed leases so multiple app instances do not normally execute the same task concurrently.
A task that exceeds four minutes is aborted and recorded as failed. A crashed run can be reclaimed after its stale lease expires.
Failures and the latest successful summary appear on the task card and in server logs.

Self-hosted Node.js deployments tick the scheduler in-process once per minute. Vercel uses the authenticated /api/admin/tasks/tick cron route; the checked-in Vercel Hobby schedule runs once daily, so intervals shorter than one day are unavailable there. See Vercel Deployment.

Migrating off env vars

In v4, runtime site features are managed by admin settings and optional JSON seed. To minimize env surface area:

Deploy this version with your existing env values in place.
Boot the app once. Open Settings → Admin and verify:
- Seeded settings appear as from seed (if you supplied a runtime JSON seed).
- A default-openai row exists in Shared providers (if you had API_BASE or API_KEY set).
Remove any bootstrap env vars you no longer need from .env.
Redeploy. Behavior is unchanged — the DB is now the source of truth.

You can keep API_BASE / API_KEY if you intentionally want bootstrap fallback behavior on empty provider tables.

How keys are protected

API keys are encrypted in the admin_providers table with AES-256-GCM. The encryption key is derived from AUTH_SECRET via scrypt.
The masked-list view (GET /api/admin/providers, used by the admin UI itself) returns •••• + last-4 only — never plaintext or ciphertext.
The public list endpoint (GET /api/tts/shared-providers, called by every user's browser) returns only { slug, displayName, providerType, defaultModel }. Keys and base URLs are never exposed to the client.
Non-admin users cannot enumerate admin providers' credentials or base URLs through any API.

:::danger Rotating AUTH_SECRET invalidates all stored admin provider keys Because the encryption key for admin_providers is derived from AUTH_SECRET, changing AUTH_SECRET makes every stored API key undecryptable. After rotating it, shared providers will fail to authenticate upstream until you re-enter each provider's API key in Settings → Admin → Shared providers (edit the row and paste the key again). There is no automated re-encryption path. If you must rotate AUTH_SECRET, plan to re-enter admin provider keys immediately afterward. :::

Auth — required to use the admin panel.
TTS Providers — built-in provider catalog and per-user behavior.
Environment Variables — ADMIN_EMAILS, provider bootstrap vars, and runtime JSON seed.

10 KiB Raw Permalink Blame History