diff --git a/docs-site/docs/configure/database.md b/docs-site/docs/configure/database.md index 45586af..d878e9e 100644 --- a/docs-site/docs/configure/database.md +++ b/docs-site/docs/configure/database.md @@ -4,6 +4,11 @@ title: Database This page covers database mode selection for OpenReader. +## Scope of this page + +- Focus: SQL metadata, state, and relational tables. +- Not covered here: object key layout and blob transport details (see [Object / Blob Storage](./object-blob-storage)). + ## Database mode - SQLite (default): embedded DB at `docstore/sqlite3.db`; good for local/self-host single-instance setups. @@ -17,9 +22,24 @@ This page covers database mode selection for OpenReader. - User settings preferences (`user_preferences`) when auth is enabled. - User reading progress (`user_document_progress`) when auth is enabled. - Document preview job/asset metadata (`document_previews`) for server-side PDF/EPUB thumbnails. +- TTS segment metadata (`tts_segments`) for server-side playback caching: + - Segment identity + settings hash + - Audio object key and duration + - Optional alignment payload for word highlighting + - Status/error state + - Text fingerprint/hash (plaintext segment text is not stored) App-specific tables are manually maintained in Drizzle schema files, while auth tables are generated by the Better Auth CLI. Both are migrated together via Drizzle. See [Migrations](./migrations) for details. +## What the database does not store + +- Raw document file bytes +- Audiobook audio bytes +- TTS segment audio bytes +- Generated preview image bytes + +Those payloads live in object storage. SQL stores the metadata, references, and status. + ## Related variables - `POSTGRES_URL` diff --git a/docs-site/docs/configure/migrations.md b/docs-site/docs/configure/migrations.md index 66c8f4b..91c50f8 100644 --- a/docs-site/docs/configure/migrations.md +++ b/docs-site/docs/configure/migrations.md @@ -20,6 +20,10 @@ Startup migration phases: - DB schema migrations (`pnpm migrate`) - Storage/data migration (`pnpm migrate-fs`) for legacy filesystem content into S3 + DB rows +Recent schema addition: + +- `0001_tts_segments` (SQLite + Postgres) creates the `tts_segments` table used by server-side TTS segment caching and alignment metadata. + :::info In most setups, you do not need to run migration commands manually because startup handles this automatically. ::: @@ -99,6 +103,13 @@ App-specific tables are manually maintained in the standard Drizzle schema files Both sets of schema files are included in the Drizzle configs, so `drizzle-kit generate` and `drizzle-kit migrate` handle all tables together. +When app schema changes (for example `tts_segments`), keep these in sync: + +- `src/db/schema_sqlite.ts` +- `src/db/schema_postgres.ts` +- `drizzle/sqlite/*.sql` + `drizzle/sqlite/meta/_journal.json` +- `drizzle/postgres/*.sql` + `drizzle/postgres/meta/_journal.json` + diff --git a/docs-site/docs/configure/object-blob-storage.md b/docs-site/docs/configure/object-blob-storage.md index 99e1698..802a96e 100644 --- a/docs-site/docs/configure/object-blob-storage.md +++ b/docs-site/docs/configure/object-blob-storage.md @@ -7,10 +7,17 @@ import TabItem from '@theme/TabItem'; This page documents storage backends, blob upload routing, and core Docker mount behavior. +## Scope of this page + +- Focus: object/blob backends, keyspaces, upload/read paths, and storage debugging. +- Not covered here: relational metadata tables and SQL state modeling (see [Database](./database)). + ## Storage backends -- Embedded (default): SQLite metadata + embedded SeaweedFS (`weed mini`) blobs. -- External: Postgres + external S3-compatible object storage. +- Embedded (default): embedded SeaweedFS (`weed mini`) blob storage. +- External: external S3-compatible object storage. + +Metadata database mode (SQLite vs Postgres) is configured separately in [Database](./database). :::warning SeaweedFS Compatibility Note (April 16, 2026) OpenReader currently pins embedded SeaweedFS to `4.18` in CI and Docker builds. @@ -106,3 +113,55 @@ Embedded default example: `S3_ENDPOINT=http://127.0.0.1:8333` (or your mapped ho + +## TTS Segment Storage + +Server-side TTS segment audio is stored in object storage under the `tts_segments_v1` keyspace. + +Typical key layout: + +- `${S3_PREFIX}/tts_segments_v1/users//docs////.mp3` +- `${S3_PREFIX}/tts_segments_v1/ns//users//docs/...` (test namespace mode) + +Notes: + +- For the corresponding SQL metadata model (`tts_segments`), see [Database](./database). + +## Account Deletion Cleanup + +Account deletion performs best-effort object cleanup: + +- Document blobs + preview artifacts +- Audiobook blobs +- TTS segment blobs under `tts_segments_v1` + +If object deletion fails, account deletion still proceeds and orphaned objects may require manual cleanup. + +## TTS Segment Storage Debug Commands + +Use these commands to inspect segment objects. + + + + +```bash +# List all TTS segment objects +aws s3 ls "s3://$S3_BUCKET/$S3_PREFIX/tts_segments_v1/" --recursive + +# Filter to one document id (replace ) +aws s3 ls "s3://$S3_BUCKET/$S3_PREFIX/tts_segments_v1/" --recursive | grep "/docs//" +``` + + + + +```bash +# List all TTS segment objects +aws s3 ls "s3://$S3_BUCKET/$S3_PREFIX/tts_segments_v1/" --recursive --endpoint-url "$S3_ENDPOINT" + +# Filter to one document id (replace ) +aws s3 ls "s3://$S3_BUCKET/$S3_PREFIX/tts_segments_v1/" --recursive --endpoint-url "$S3_ENDPOINT" | grep "/docs//" +``` + + + diff --git a/docs-site/docs/deploy/local-development.md b/docs-site/docs/deploy/local-development.md index b771b49..e8806fa 100644 --- a/docs-site/docs/deploy/local-development.md +++ b/docs-site/docs/deploy/local-development.md @@ -7,45 +7,137 @@ import TabItem from '@theme/TabItem'; ## Prerequisites -- Node.js (recommended with [nvm](https://github.com/nvm-sh/nvm)) -- `pnpm` (recommended) or `npm` +
+Node.js + pnpm (required) + + + ```bash -npm install -g pnpm +brew install nvm pnpm +mkdir -p ~/.nvm +echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.zshrc +echo '[ -s "$(brew --prefix nvm)/nvm.sh" ] && . "$(brew --prefix nvm)/nvm.sh"' >> ~/.zshrc +source ~/.zshrc +nvm install --lts +nvm use --lts +node -v +pnpm -v ``` -- A reachable TTS API server -- [SeaweedFS](https://github.com/seaweedfs/seaweedfs) `weed` binary (required unless using external S3 storage) + + + +```bash +# Debian/Ubuntu example +sudo apt update +sudo apt install -y curl +curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash +export NVM_DIR="$HOME/.nvm" +[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" +nvm install --lts +nvm use --lts +corepack enable +corepack prepare pnpm@latest --activate +node -v +pnpm -v +``` + + + + +
+ +
+SeaweedFS weed binary (required unless using external S3) + + + + +```bash +brew install seaweedfs +weed version +``` :::warning SeaweedFS Compatibility Note (April 16, 2026) If you see intermittent S3 `InternalError` upload failures with embedded storage, use SeaweedFS `4.18`. OpenReader currently pins `4.18` in CI and Docker builds while `4.19` compatibility is investigated. ::: - - + + ```bash -brew install seaweedfs +# Linux amd64 example (pin 4.18) +mkdir -p "$HOME/.local/bin" +curl -fsSL -o /tmp/seaweedfs.tar.gz \ + https://github.com/seaweedfs/seaweedfs/releases/download/4.18/linux_amd64.tar.gz +tar -xzf /tmp/seaweedfs.tar.gz -C /tmp weed +install -m 0755 /tmp/weed "$HOME/.local/bin/weed" +echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc +export PATH="$HOME/.local/bin:$PATH" +weed version ``` - - +:::warning SeaweedFS Compatibility Note (April 16, 2026) +If you see intermittent S3 `InternalError` upload failures with embedded storage, use SeaweedFS `4.18`. +OpenReader currently pins `4.18` in CI and Docker builds while `4.19` compatibility is investigated. +::: -Install the `weed` binary from the [SeaweedFS releases](https://github.com/seaweedfs/seaweedfs/releases) and ensure it is available on `PATH`. - - + -Optional, depending on features: +
-- [libreoffice](https://www.libreoffice.org) (required for DOCX conversion) +
+LibreOffice (optional, for DOCX conversion) + + + ```bash brew install libreoffice ``` -- [whisper.cpp](https://github.com/ggml-org/whisper.cpp) (optional, for word-by-word highlighting) + + + +```bash +# Debian/Ubuntu example +sudo apt update +sudo apt install -y libreoffice +``` + + + + +
+ +
+whisper.cpp (optional, for word-by-word highlighting) + +Install build dependencies: + + + + +```bash +brew install cmake +``` + + + + +```bash +# Debian/Ubuntu example +sudo apt update +sudo apt install -y git build-essential cmake +``` + + + + +Build whisper.cpp: ```bash # clone and build whisper.cpp (no model download needed – OpenReader handles that) @@ -58,12 +150,21 @@ cmake --build build -j --config Release echo WHISPER_CPP_BIN="$(pwd)/build/bin/whisper-cli" ``` +If you are not on Debian/Ubuntu, install equivalent packages with your distro package manager: + +- Fedora/RHEL: use `dnf` (`gcc gcc-c++ make cmake curl git tar xz`) +- Arch: use `pacman` (`base-devel cmake curl git tar xz`) + :::tip Set `WHISPER_CPP_BIN` in your `.env` to enable word-by-word highlighting. ::: +
+ ## Steps +### Required flow + 1. Clone the repository. ```bash @@ -85,14 +186,47 @@ cp .env.example .env Then edit `.env`. -- No auth mode: leave `BASE_URL` or `AUTH_SECRET` unset. -- Auth enabled mode: set both `BASE_URL` (typically `http://localhost:3003`) and `AUTH_SECRET` (generate with `openssl rand -hex 32`). +Use one of these `.env` mode templates: -Optional: + + -- `AUTH_TRUSTED_ORIGINS=http://localhost:3003,http://192.168.0.116:3003` -- Stable S3 credentials via `S3_ACCESS_KEY_ID` and `S3_SECRET_ACCESS_KEY` -- External S3 storage by setting `USE_EMBEDDED_WEED_MINI=false` and related S3 vars +```env +API_BASE=http://host.docker.internal:8880/v1 +API_KEY=none +# Leave BASE_URL and AUTH_SECRET unset to keep auth disabled +``` + + + + +```env +API_BASE=http://host.docker.internal:8880/v1 +API_KEY=none +BASE_URL=http://localhost:3003 +AUTH_SECRET= +# Optional when you need multiple local origins: +# AUTH_TRUSTED_ORIGINS=http://localhost:3003,http://127.0.0.1:3003 +``` + + + + +```env +API_BASE=http://host.docker.internal:8880/v1 +API_KEY=none +USE_EMBEDDED_WEED_MINI=false +S3_BUCKET=your-bucket +S3_REGION=us-east-1 +S3_ACCESS_KEY_ID=your-access-key +S3_SECRET_ACCESS_KEY=your-secret-key +# Optional for non-AWS providers: +# S3_ENDPOINT=https://your-s3-compatible-endpoint +# S3_FORCE_PATH_STYLE=true +``` + + + :::info For all environment variables, see [Environment Variables](../reference/environment-variables). @@ -103,23 +237,10 @@ Storage configuration details are in [Object / Blob Storage](../configure/object Refer to [Database](../configure/database) for database modes. Learn about migration behavior and commands in [Migrations](../configure/migrations). -4. Run DB migrations. - -- Migrations run automatically on startup through the shared entrypoint for both `pnpm dev` and `pnpm start`. -- You only need manual migration commands for one-off troubleshooting or explicit migration workflows: - -```bash -pnpm migrate -``` - -:::info -If `POSTGRES_URL` is set, migrations target Postgres; otherwise local SQLite is used. To disable automatic startup migrations, set `RUN_DRIZZLE_MIGRATIONS=false` and/or `RUN_FS_MIGRATIONS=false`. You can run storage migration manually with `pnpm migrate-fs`. -::: - -5. Start the app. +4. Start the app. - + ```bash pnpm dev @@ -141,3 +262,17 @@ pnpm start ::: Visit [http://localhost:3003](http://localhost:3003). + +### Optional workflows + +Run manual DB migrations only for troubleshooting or explicit migration workflows: + +- Migrations run automatically on startup through the shared entrypoint for both `pnpm dev` and `pnpm start`. + +```bash +pnpm migrate +``` + +:::info +If `POSTGRES_URL` is set, migrations target Postgres; otherwise local SQLite is used. To disable automatic startup migrations, set `RUN_DRIZZLE_MIGRATIONS=false` and/or `RUN_FS_MIGRATIONS=false`. You can run storage migration manually with `pnpm migrate-fs`. +::: diff --git a/docs-site/docs/docker-quick-start.md b/docs-site/docs/docker-quick-start.md index 6c6467a..c885897 100644 --- a/docs-site/docs/docker-quick-start.md +++ b/docs-site/docs/docker-quick-start.md @@ -8,34 +8,24 @@ import TabItem from '@theme/TabItem'; ## Prerequisites - A recent Docker version installed -- A TTS API server that OpenReader can reach (Kokoro-FastAPI, KittenTTS-FastAPI, Orpheus-FastAPI, Replicate, DeepInfra, OpenAI, or equivalent) +- A TTS API server that OpenReader can reach: + - [Kokoro-FastAPI](./configure/tts-provider-guides/kokoro-fastapi) + - [KittenTTS-FastAPI](./configure/tts-provider-guides/kitten-tts-fastapi) + - [Orpheus-FastAPI](./configure/tts-provider-guides/orpheus-fastapi) + - [Replicate](./configure/tts-provider-guides/replicate) + - [DeepInfra](./configure/tts-provider-guides/deepinfra) + - [OpenAI](./configure/tts-provider-guides/openai) + - [Other OpenAI-compatible providers](./configure/tts-provider-guides/other) :::warning SeaweedFS Compatibility Note (April 16, 2026) OpenReader currently pins embedded SeaweedFS to `4.18` in CI and Docker builds. `4.19` introduced intermittent `InternalError` responses on S3 `PutObject` in our upload flow. ::: -:::note -If you have suitable hardware, you can run Kokoro locally with Docker. See [Kokoro-FastAPI](./configure/tts-provider-guides/kokoro-fastapi). -::: - ## 1. Start the Docker container - - -Auth disabled, embedded storage ephemeral, no library import: - -```bash -docker run --name openreader \ - --restart unless-stopped \ - -p 3003:3003 \ - -p 8333:8333 \ - ghcr.io/richardr1126/openreader:latest -``` - - - + Persistent storage, embedded SeaweedFS `weed mini`, optional auth, optional library mount: @@ -53,8 +43,18 @@ docker run --name openreader \ ghcr.io/richardr1126/openreader:latest ``` - - +What this command enables: + +- `-p 3003:3003`: exposes the OpenReader web app/API. +- `-p 8333:8333`: exposes embedded SeaweedFS S3 endpoint for direct browser presigned upload/download. +- `-v openreader_docstore:/app/docstore`: persists SQLite metadata, SeaweedFS blob data, and migration/runtime state. +- `-v /path/to/your/library:/app/docstore/library:ro`: mounts a read-only importable library source. +- `-e API_BASE=...`: sets the server-side default TTS endpoint OpenReader calls. +- `-e API_KEY=...`: sets the server-side default TTS API key (`none` is fine for local backends that do not require auth). +- `-e BASE_URL=...` and `-e AUTH_SECRET=...`: together they turn on auth/session mode for local sign-in flows. + + + Use this when the app should be reachable from other devices on your LAN: @@ -72,15 +72,45 @@ docker run --name openreader \ ghcr.io/richardr1126/openreader:latest ``` -Replace `` with the Docker host IP address on your local network to allow access from other devices. +Replace `YOUR_LAN_IP` with the Docker host IP address on your local network to allow access from other devices. - +What this command enables: + +- LAN access from phones/tablets/other computers via `http://:3003`. +- `BASE_URL` points auth/session cookies and callbacks at your LAN URL. +- `AUTH_TRUSTED_ORIGINS` allows localhost loopback origins in addition to your primary LAN origin. +- `USE_ANONYMOUS_AUTH_SESSIONS=true` allows guest sessions while auth is enabled. +- `API_BASE` still sets the default server-side TTS endpoint. +- `openreader_docstore` volume keeps data persistent across restarts. + + + + +Auth disabled, embedded storage ephemeral, no library import: + +```bash +docker run --name openreader \ + --restart unless-stopped \ + -p 3003:3003 \ + -p 8333:8333 \ + ghcr.io/richardr1126/openreader:latest +``` + +What this command enables: + +- Fastest startup with no extra env vars. +- No persistent volume (`/app/docstore` stays container-local), so data is ephemeral unless you add a mount. +- Auth remains disabled because `BASE_URL` and `AUTH_SECRET` are not set. +- TTS endpoint/key are not preset server-side (`API_BASE`/`API_KEY` not set), so configure provider settings in the app UI. + + :::tip Quick Tips -- Remove `/app/docstore/library` if you do not need server library import. -- Remove either `BASE_URL` or `AUTH_SECRET` to keep auth disabled. -- Set `API_BASE` to your reachable TTS server base URL. +- Set `API_BASE` to a TTS endpoint the container can reach (`host.docker.internal` works for host-local services). +- Auth is enabled only when both `BASE_URL` and `AUTH_SECRET` are set. +- Use a `/app/docstore` mount if you want data to survive container/image replacement. +- Startup automatically runs DB/storage migrations via the shared entrypoint. ::: :::warning Port `8333` Exposure @@ -89,21 +119,10 @@ Expose `8333` for direct browser presigned upload/download with embedded Seaweed If `8333` is not reachable from the browser, direct presigned access is unavailable. Uploads can still fall back to `/api/documents/blob/upload/fallback`, and document reads/downloads continue through `/api/documents/blob`. ::: -:::info Auth and Migrations -- Auth is enabled only when both `BASE_URL` and `AUTH_SECRET` are set. -- DB/storage migrations run automatically at container startup via the shared entrypoint. -::: - -:::info Related Docs -- [Environment Variables](./reference/environment-variables) -- [Auth](./configure/auth) -- [Database](./configure/database) -- [Object / Blob Storage](./configure/object-blob-storage) -- [Migrations](./configure/migrations) -::: - ## 2. Configure settings in the app UI +Visit [http://localhost:3003](http://localhost:3003) after startup. + - Set TTS provider and model in Settings - Set TTS API base URL and API key if needed - Select the model voice from the voice dropdown @@ -123,4 +142,10 @@ docker pull ghcr.io/richardr1126/openreader:latest If you use a mounted volume for `/app/docstore`, your persisted data remains after image updates. ::: -Visit [http://localhost:3003](http://localhost:3003) after startup. +:::info Related Docs +- [Environment Variables](./reference/environment-variables) +- [Auth](./configure/auth) +- [Database](./configure/database) +- [Object / Blob Storage](./configure/object-blob-storage) +- [Migrations](./configure/migrations) +::: diff --git a/docs-site/docs/introduction.md b/docs-site/docs/introduction.md index 79a617d..c4d09e0 100644 --- a/docs-site/docs/introduction.md +++ b/docs-site/docs/introduction.md @@ -21,13 +21,13 @@ It supports multiple TTS providers including OpenAI, Replicate, DeepInfra, and c - [**Replicate**](https://replicate.com/explore): includes a built-in catalog and supports any Replicate model ID via `Other` - [**DeepInfra**](https://deepinfra.com/models/text-to-speech): Kokoro-82M and other hosted models - [**OpenAI API**](https://platform.openai.com/docs/pricing#transcription-and-speech): `tts-1`, `tts-1-hd`, and `gpt-4o-mini-tts` -- 🛜 **Server-side Document Storage** - - Documents are persisted in server blob/object storage for consistent access -- 📚 **External Library Import** - - Import documents from server-mounted folders -- 🎧 **Server-side Audiobook Export** in `m4b`/`mp3` with resumable chapter generation - 📖 **Read Along Experience** - Real-time highlighting for PDF/EPUB, with optional word-level [whisper.cpp](https://github.com/ggml-org/whisper.cpp) timestamps +- 🛜 **Document Storage** + - Documents are persisted in server blob/object storage for consistent access +- ⚡ **Segment-based TTS Playback** for reusable generation + preloading + - Stores segment audio in object storage for fast replay/resume +- 🎧 **Audiobook Export** in `m4b`/`mp3` with resumable chapter generation - 🔐 **Auth Optional by Design** - Run no-auth for local use, or enable auth with user isolation and claim flow - 🗂️ **Flexible Storage and Database Modes** with embedded defaults or external S3/Postgres diff --git a/docs-site/docs/reference/environment-variables.md b/docs-site/docs/reference/environment-variables.md index 2f1ff0b..682a357 100644 --- a/docs-site/docs/reference/environment-variables.md +++ b/docs-site/docs/reference/environment-variables.md @@ -170,6 +170,7 @@ Secret key used by auth/session handling. - Required with `BASE_URL` to enable auth - Generate with `openssl rand -hex 32` +- Also used to HMAC-hash server-side TTS segment text fingerprints - Related docs: [Auth](../configure/auth) ### AUTH_TRUSTED_ORIGINS