danvics/openreader
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(tts,storage,dev): update documentation for TTS segment storage, schema, and local setup

Browse source 
Expand and clarify documentation for TTS segment storage, including key layout, cleanup, and CLI access in object storage. Add details on the `tts_segments` schema and migration process. Revise local development and Docker quick start guides with improved prerequisites, platform-specific setup instructions, and updated TTS provider references. Document new usage of `AUTH_SECRET` for TTS segment text fingerprinting.
This commit is contained in:
Richard R 2026-05-04 11:29:02 -06:00
parent 9999cafec1
commit 0a7d392d77
7 changed files with 335 additions and 84 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

20
docs-site/docs/configure/database.md
View file

@ -4,6 +4,11 @@ title: Database
This page covers database mode selection for OpenReader.
## Scope of this page
- Focus: SQL metadata, state, and relational tables.
- Not covered here: object key layout and blob transport details (see [Object / Blob Storage](./object-blob-storage)).
## Database mode
- SQLite (default): embedded DB at `docstore/sqlite3.db`; good for local/self-host single-instance setups.
@ -17,9 +22,24 @@ This page covers database mode selection for OpenReader.
- User settings preferences (`user_preferences`) when auth is enabled.
- User reading progress (`user_document_progress`) when auth is enabled.
- Document preview job/asset metadata (`document_previews`) for server-side PDF/EPUB thumbnails.
- TTS segment metadata (`tts_segments`) for server-side playback caching:
- Segment identity + settings hash
- Audio object key and duration
- Optional alignment payload for word highlighting
- Status/error state
- Text fingerprint/hash (plaintext segment text is not stored)
App-specific tables are manually maintained in Drizzle schema files, while auth tables are generated by the Better Auth CLI. Both are migrated together via Drizzle. See [Migrations](./migrations) for details.
## What the database does not store
- Raw document file bytes
- Audiobook audio bytes
- TTS segment audio bytes
- Generated preview image bytes
Those payloads live in object storage. SQL stores the metadata, references, and status.
## Related variables
- `POSTGRES_URL`

11
docs-site/docs/configure/migrations.md
View file

@ -20,6 +20,10 @@ Startup migration phases:
- DB schema migrations (`pnpm migrate`)
- Storage/data migration (`pnpm migrate-fs`) for legacy filesystem content into S3 + DB rows
Recent schema addition:
- `0001_tts_segments` (SQLite + Postgres) creates the `tts_segments` table used by server-side TTS segment caching and alignment metadata.
:::info
In most setups, you do not need to run migration commands manually because startup handles this automatically.
:::
@ -99,6 +103,13 @@ App-specific tables are manually maintained in the standard Drizzle schema files
Both sets of schema files are included in the Drizzle configs, so `drizzle-kit generate` and `drizzle-kit migrate` handle all tables together.
When app schema changes (for example `tts_segments`), keep these in sync:
- `src/db/schema_sqlite.ts`
- `src/db/schema_postgres.ts`
- `drizzle/sqlite/*.sql` + `drizzle/sqlite/meta/_journal.json`
- `drizzle/postgres/*.sql` + `drizzle/postgres/meta/_journal.json`
<Tabs groupId="generate-migration-commands">
<TabItem value="project-script" label="Project Script" default>

63
docs-site/docs/configure/object-blob-storage.md
View file

@ -7,10 +7,17 @@ import TabItem from '@theme/TabItem';
This page documents storage backends, blob upload routing, and core Docker mount behavior.
## Scope of this page
- Focus: object/blob backends, keyspaces, upload/read paths, and storage debugging.
- Not covered here: relational metadata tables and SQL state modeling (see [Database](./database)).
## Storage backends
- Embedded (default): SQLite metadata + embedded SeaweedFS (`weed mini`) blobs.
- External: Postgres + external S3-compatible object storage.
- Embedded (default): embedded SeaweedFS (`weed mini`) blob storage.
- External: external S3-compatible object storage.
Metadata database mode (SQLite vs Postgres) is configured separately in [Database](./database).
:::warning SeaweedFS Compatibility Note (April 16, 2026)
OpenReader currently pins embedded SeaweedFS to `4.18` in CI and Docker builds.
@ -106,3 +113,55 @@ Embedded default example: `S3_ENDPOINT=http://127.0.0.1:8333` (or your mapped ho
</TabItem>
</Tabs>
## TTS Segment Storage
Server-side TTS segment audio is stored in object storage under the `tts_segments_v1` keyspace.
Typical key layout:
- `${S3_PREFIX}/tts_segments_v1/users/<url-encoded-user-id>/docs/<document-id>/<document-version>/<settings-hash>/<segment-id>.mp3`
- `${S3_PREFIX}/tts_segments_v1/ns/<test-namespace>/users/<url-encoded-user-id>/docs/...` (test namespace mode)
Notes:
- For the corresponding SQL metadata model (`tts_segments`), see [Database](./database).
## Account Deletion Cleanup
Account deletion performs best-effort object cleanup:
- Document blobs + preview artifacts
- Audiobook blobs
- TTS segment blobs under `tts_segments_v1`
If object deletion fails, account deletion still proceeds and orphaned objects may require manual cleanup.
## TTS Segment Storage Debug Commands
Use these commands to inspect segment objects.
<Tabs groupId="tts-segment-storage-access-cli">
<TabItem value="aws-s3" label="AWS S3" default>
```bash
# List all TTS segment objects
aws s3 ls "s3://$S3_BUCKET/$S3_PREFIX/tts_segments_v1/" --recursive
# Filter to one document id (replace <document-id>)
aws s3 ls "s3://$S3_BUCKET/$S3_PREFIX/tts_segments_v1/" --recursive | grep "/docs/<document-id>/"
```
</TabItem>
<TabItem value="s3-compatible" label="Embedded / MinIO / R2 / etc">
```bash
# List all TTS segment objects
aws s3 ls "s3://$S3_BUCKET/$S3_PREFIX/tts_segments_v1/" --recursive --endpoint-url "$S3_ENDPOINT"
# Filter to one document id (replace <document-id>)
aws s3 ls "s3://$S3_BUCKET/$S3_PREFIX/tts_segments_v1/" --recursive --endpoint-url "$S3_ENDPOINT" | grep "/docs/<document-id>/"
```
</TabItem>
</Tabs>

209
docs-site/docs/deploy/local-development.md
View file

@ -7,45 +7,137 @@ import TabItem from '@theme/TabItem';
## Prerequisites
- Node.js (recommended with [nvm](https://github.com/nvm-sh/nvm))
- `pnpm` (recommended) or `npm`
<details>
<summary><strong>Node.js + pnpm (required)</strong></summary>
<Tabs groupId="local-dev-node-pnpm-os">
<TabItem value="macos" label="macOS" default>
```bash
npm install -g pnpm
brew install nvm pnpm
mkdir -p ~/.nvm
echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.zshrc
echo '[ -s "$(brew --prefix nvm)/nvm.sh" ] && . "$(brew --prefix nvm)/nvm.sh"' >> ~/.zshrc
source ~/.zshrc
nvm install --lts
nvm use --lts
node -v
pnpm -v
```
- A reachable TTS API server
- [SeaweedFS](https://github.com/seaweedfs/seaweedfs) `weed` binary (required unless using external S3 storage)
</TabItem>
<TabItem value="linux" label="Linux">
```bash
# Debian/Ubuntu example
sudo apt update
sudo apt install -y curl
curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
nvm install --lts
nvm use --lts
corepack enable
corepack prepare pnpm@latest --activate
node -v
pnpm -v
```
</TabItem>
</Tabs>
</details>
<details>
<summary><strong>SeaweedFS <code>weed</code> binary (required unless using external S3)</strong></summary>
<Tabs groupId="local-dev-seaweed-os">
<TabItem value="macos" label="macOS" default>
```bash
brew install seaweedfs
weed version
```
:::warning SeaweedFS Compatibility Note (April 16, 2026)
If you see intermittent S3 `InternalError` upload failures with embedded storage, use SeaweedFS `4.18`.
OpenReader currently pins `4.18` in CI and Docker builds while `4.19` compatibility is investigated.
:::
<Tabs groupId="seaweedfs-install">
<TabItem value="macos" label="macOS" default>
</TabItem>
<TabItem value="linux" label="Linux">
```bash
brew install seaweedfs
# Linux amd64 example (pin 4.18)
mkdir -p "$HOME/.local/bin"
curl -fsSL -o /tmp/seaweedfs.tar.gz \
https://github.com/seaweedfs/seaweedfs/releases/download/4.18/linux_amd64.tar.gz
tar -xzf /tmp/seaweedfs.tar.gz -C /tmp weed
install -m 0755 /tmp/weed "$HOME/.local/bin/weed"
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/bin:$PATH"
weed version
```
</TabItem>
<TabItem value="linux" label="Linux">
:::warning SeaweedFS Compatibility Note (April 16, 2026)
If you see intermittent S3 `InternalError` upload failures with embedded storage, use SeaweedFS `4.18`.
OpenReader currently pins `4.18` in CI and Docker builds while `4.19` compatibility is investigated.
:::
Install the `weed` binary from the [SeaweedFS releases](https://github.com/seaweedfs/seaweedfs/releases) and ensure it is available on `PATH`.
</TabItem>
</TabItem>
</Tabs>
Optional, depending on features:
</details>
- [libreoffice](https://www.libreoffice.org) (required for DOCX conversion)
<details>
<summary><strong>LibreOffice (optional, for DOCX conversion)</strong></summary>
<Tabs groupId="local-dev-libreoffice-os">
<TabItem value="macos" label="macOS" default>
```bash
brew install libreoffice
```
- [whisper.cpp](https://github.com/ggml-org/whisper.cpp) (optional, for word-by-word highlighting)
</TabItem>
<TabItem value="linux" label="Linux">
```bash
# Debian/Ubuntu example
sudo apt update
sudo apt install -y libreoffice
```
</TabItem>
</Tabs>
</details>
<details>
<summary><strong>whisper.cpp (optional, for word-by-word highlighting)</strong></summary>
Install build dependencies:
<Tabs groupId="local-dev-whisper-deps-os">
<TabItem value="macos" label="macOS" default>
```bash
brew install cmake
```
</TabItem>
<TabItem value="linux" label="Linux">
```bash
# Debian/Ubuntu example
sudo apt update
sudo apt install -y git build-essential cmake
```
</TabItem>
</Tabs>
Build whisper.cpp:
```bash
# clone and build whisper.cpp (no model download needed – OpenReader handles that)
@ -58,12 +150,21 @@ cmake --build build -j --config Release
echo WHISPER_CPP_BIN="$(pwd)/build/bin/whisper-cli"
```
If you are not on Debian/Ubuntu, install equivalent packages with your distro package manager:
- Fedora/RHEL: use `dnf` (`gcc gcc-c++ make cmake curl git tar xz`)
- Arch: use `pacman` (`base-devel cmake curl git tar xz`)
:::tip
Set `WHISPER_CPP_BIN` in your `.env` to enable word-by-word highlighting.
:::
</details>
## Steps
### Required flow
1. Clone the repository.
```bash
@ -85,14 +186,47 @@ cp .env.example .env
Then edit `.env`.
- No auth mode: leave `BASE_URL` or `AUTH_SECRET` unset.
- Auth enabled mode: set both `BASE_URL` (typically `http://localhost:3003`) and `AUTH_SECRET` (generate with `openssl rand -hex 32`).
Use one of these `.env` mode templates:
Optional:
<Tabs groupId="local-env-modes">
<TabItem value="no-auth" label="No Auth (simple)" default>
- `AUTH_TRUSTED_ORIGINS=http://localhost:3003,http://192.168.0.116:3003`
- Stable S3 credentials via `S3_ACCESS_KEY_ID` and `S3_SECRET_ACCESS_KEY`
- External S3 storage by setting `USE_EMBEDDED_WEED_MINI=false` and related S3 vars
```env
API_BASE=http://host.docker.internal:8880/v1
API_KEY=none
# Leave BASE_URL and AUTH_SECRET unset to keep auth disabled
```
</TabItem>
<TabItem value="auth-enabled" label="Auth Enabled">
```env
API_BASE=http://host.docker.internal:8880/v1
API_KEY=none
BASE_URL=http://localhost:3003
AUTH_SECRET=<generate-with-openssl-rand-hex-32>
# Optional when you need multiple local origins:
# AUTH_TRUSTED_ORIGINS=http://localhost:3003,http://127.0.0.1:3003
```
</TabItem>
<TabItem value="external-s3" label="External S3">
```env
API_BASE=http://host.docker.internal:8880/v1
API_KEY=none
USE_EMBEDDED_WEED_MINI=false
S3_BUCKET=your-bucket
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=your-access-key
S3_SECRET_ACCESS_KEY=your-secret-key
# Optional for non-AWS providers:
# S3_ENDPOINT=https://your-s3-compatible-endpoint
# S3_FORCE_PATH_STYLE=true
```
</TabItem>
</Tabs>
:::info
For all environment variables, see [Environment Variables](../reference/environment-variables).
@ -103,23 +237,10 @@ Storage configuration details are in [Object / Blob Storage](../configure/object
Refer to [Database](../configure/database) for database modes.
Learn about migration behavior and commands in [Migrations](../configure/migrations).
4. Run DB migrations.
- Migrations run automatically on startup through the shared entrypoint for both `pnpm dev` and `pnpm start`.
- You only need manual migration commands for one-off troubleshooting or explicit migration workflows:
```bash
pnpm migrate
```
:::info
If `POSTGRES_URL` is set, migrations target Postgres; otherwise local SQLite is used. To disable automatic startup migrations, set `RUN_DRIZZLE_MIGRATIONS=false` and/or `RUN_FS_MIGRATIONS=false`. You can run storage migration manually with `pnpm migrate-fs`.
:::
5. Start the app.
4. Start the app.
<Tabs groupId="local-run-mode">
<TabItem value="dev" label="Dev" default>
<TabItem value="dev" label="Dev (recommended)" default>
```bash
pnpm dev
@ -141,3 +262,17 @@ pnpm start
:::
Visit [http://localhost:3003](http://localhost:3003).
### Optional workflows
Run manual DB migrations only for troubleshooting or explicit migration workflows:
- Migrations run automatically on startup through the shared entrypoint for both `pnpm dev` and `pnpm start`.
```bash
pnpm migrate
```
:::info
If `POSTGRES_URL` is set, migrations target Postgres; otherwise local SQLite is used. To disable automatic startup migrations, set `RUN_DRIZZLE_MIGRATIONS=false` and/or `RUN_FS_MIGRATIONS=false`. You can run storage migration manually with `pnpm migrate-fs`.
:::

105
docs-site/docs/docker-quick-start.md
View file

@ -8,34 +8,24 @@ import TabItem from '@theme/TabItem';
## Prerequisites
- A recent Docker version installed
- A TTS API server that OpenReader can reach (Kokoro-FastAPI, KittenTTS-FastAPI, Orpheus-FastAPI, Replicate, DeepInfra, OpenAI, or equivalent)
- A TTS API server that OpenReader can reach:
- [Kokoro-FastAPI](./configure/tts-provider-guides/kokoro-fastapi)
- [KittenTTS-FastAPI](./configure/tts-provider-guides/kitten-tts-fastapi)
- [Orpheus-FastAPI](./configure/tts-provider-guides/orpheus-fastapi)
- [Replicate](./configure/tts-provider-guides/replicate)
- [DeepInfra](./configure/tts-provider-guides/deepinfra)
- [OpenAI](./configure/tts-provider-guides/openai)
- [Other OpenAI-compatible providers](./configure/tts-provider-guides/other)
:::warning SeaweedFS Compatibility Note (April 16, 2026)
OpenReader currently pins embedded SeaweedFS to `4.18` in CI and Docker builds.
`4.19` introduced intermittent `InternalError` responses on S3 `PutObject` in our upload flow.
:::
:::note
If you have suitable hardware, you can run Kokoro locally with Docker. See [Kokoro-FastAPI](./configure/tts-provider-guides/kokoro-fastapi).
:::
## 1. Start the Docker container
<Tabs groupId="docker-start-mode">
<TabItem value="minimal" label="Minimal" default>
Auth disabled, embedded storage ephemeral, no library import:
```bash
docker run --name openreader \
--restart unless-stopped \
-p 3003:3003 \
-p 8333:8333 \
ghcr.io/richardr1126/openreader:latest
```
</TabItem>
<TabItem value="localhost" label="Localhost">
<TabItem value="localhost" label="Localhost" default>
Persistent storage, embedded SeaweedFS `weed mini`, optional auth, optional library mount:
@ -53,8 +43,18 @@ docker run --name openreader \
ghcr.io/richardr1126/openreader:latest
```
</TabItem>
<TabItem value="local-network" label="LAN Host">
What this command enables:
- `-p 3003:3003`: exposes the OpenReader web app/API.
- `-p 8333:8333`: exposes embedded SeaweedFS S3 endpoint for direct browser presigned upload/download.
- `-v openreader_docstore:/app/docstore`: persists SQLite metadata, SeaweedFS blob data, and migration/runtime state.
- `-v /path/to/your/library:/app/docstore/library:ro`: mounts a read-only importable library source.
- `-e API_BASE=...`: sets the server-side default TTS endpoint OpenReader calls.
- `-e API_KEY=...`: sets the server-side default TTS API key (`none` is fine for local backends that do not require auth).
- `-e BASE_URL=...` and `-e AUTH_SECRET=...`: together they turn on auth/session mode for local sign-in flows.
</TabItem>
<TabItem value="local-network" label="LAN Host">
Use this when the app should be reachable from other devices on your LAN:
@ -72,15 +72,45 @@ docker run --name openreader \
ghcr.io/richardr1126/openreader:latest
```
Replace `<YOUR_LAN_IP>` with the Docker host IP address on your local network to allow access from other devices.
Replace `YOUR_LAN_IP` with the Docker host IP address on your local network to allow access from other devices.
</TabItem>
What this command enables:
- LAN access from phones/tablets/other computers via `http://<YOUR_LAN_IP>:3003`.
- `BASE_URL` points auth/session cookies and callbacks at your LAN URL.
- `AUTH_TRUSTED_ORIGINS` allows localhost loopback origins in addition to your primary LAN origin.
- `USE_ANONYMOUS_AUTH_SESSIONS=true` allows guest sessions while auth is enabled.
- `API_BASE` still sets the default server-side TTS endpoint.
- `openreader_docstore` volume keeps data persistent across restarts.
</TabItem>
<TabItem value="minimal" label="Minimal">
Auth disabled, embedded storage ephemeral, no library import:
```bash
docker run --name openreader \
--restart unless-stopped \
-p 3003:3003 \
-p 8333:8333 \
ghcr.io/richardr1126/openreader:latest
```
What this command enables:
- Fastest startup with no extra env vars.
- No persistent volume (`/app/docstore` stays container-local), so data is ephemeral unless you add a mount.
- Auth remains disabled because `BASE_URL` and `AUTH_SECRET` are not set.
- TTS endpoint/key are not preset server-side (`API_BASE`/`API_KEY` not set), so configure provider settings in the app UI.
</TabItem>
</Tabs>
:::tip Quick Tips
- Remove `/app/docstore/library` if you do not need server library import.
- Remove either `BASE_URL` or `AUTH_SECRET` to keep auth disabled.
- Set `API_BASE` to your reachable TTS server base URL.
- Set `API_BASE` to a TTS endpoint the container can reach (`host.docker.internal` works for host-local services).
- Auth is enabled only when both `BASE_URL` and `AUTH_SECRET` are set.
- Use a `/app/docstore` mount if you want data to survive container/image replacement.
- Startup automatically runs DB/storage migrations via the shared entrypoint.
:::
:::warning Port `8333` Exposure
@ -89,21 +119,10 @@ Expose `8333` for direct browser presigned upload/download with embedded Seaweed
If `8333` is not reachable from the browser, direct presigned access is unavailable. Uploads can still fall back to `/api/documents/blob/upload/fallback`, and document reads/downloads continue through `/api/documents/blob`.
:::
:::info Auth and Migrations
- Auth is enabled only when both `BASE_URL` and `AUTH_SECRET` are set.
- DB/storage migrations run automatically at container startup via the shared entrypoint.
:::
:::info Related Docs
- [Environment Variables](./reference/environment-variables)
- [Auth](./configure/auth)
- [Database](./configure/database)
- [Object / Blob Storage](./configure/object-blob-storage)
- [Migrations](./configure/migrations)
:::
## 2. Configure settings in the app UI
Visit [http://localhost:3003](http://localhost:3003) after startup.
- Set TTS provider and model in Settings
- Set TTS API base URL and API key if needed
- Select the model voice from the voice dropdown
@ -123,4 +142,10 @@ docker pull ghcr.io/richardr1126/openreader:latest
If you use a mounted volume for `/app/docstore`, your persisted data remains after image updates.
:::
Visit [http://localhost:3003](http://localhost:3003) after startup.
:::info Related Docs
- [Environment Variables](./reference/environment-variables)
- [Auth](./configure/auth)
- [Database](./configure/database)
- [Object / Blob Storage](./configure/object-blob-storage)
- [Migrations](./configure/migrations)
:::

10
docs-site/docs/introduction.md
View file

@ -21,13 +21,13 @@ It supports multiple TTS providers including OpenAI, Replicate, DeepInfra, and c
- [**Replicate**](https://replicate.com/explore): includes a built-in catalog and supports any Replicate model ID via `Other`
- [**DeepInfra**](https://deepinfra.com/models/text-to-speech): Kokoro-82M and other hosted models
- [**OpenAI API**](https://platform.openai.com/docs/pricing#transcription-and-speech): `tts-1`, `tts-1-hd`, and `gpt-4o-mini-tts`
- 🛜 **Server-side Document Storage**
- Documents are persisted in server blob/object storage for consistent access
- 📚 **External Library Import**
- Import documents from server-mounted folders
- 🎧 **Server-side Audiobook Export** in `m4b`/`mp3` with resumable chapter generation
- 📖 **Read Along Experience**
- Real-time highlighting for PDF/EPUB, with optional word-level [whisper.cpp](https://github.com/ggml-org/whisper.cpp) timestamps
- 🛜 **Document Storage**
- Documents are persisted in server blob/object storage for consistent access
- ⚡ **Segment-based TTS Playback** for reusable generation + preloading
- Stores segment audio in object storage for fast replay/resume
- 🎧 **Audiobook Export** in `m4b`/`mp3` with resumable chapter generation
- 🔐 **Auth Optional by Design**
- Run no-auth for local use, or enable auth with user isolation and claim flow
- 🗂️ **Flexible Storage and Database Modes** with embedded defaults or external S3/Postgres

1
docs-site/docs/reference/environment-variables.md
View file

@ -170,6 +170,7 @@ Secret key used by auth/session handling.
- Required with `BASE_URL` to enable auth
- Generate with `openssl rand -hex 32`
- Also used to HMAC-hash server-side TTS segment text fingerprints
- Related docs: [Auth](../configure/auth)
### AUTH_TRUSTED_ORIGINS