Commit graph

222 commits

Author SHA1 Message Date
Richard R
81d249ed52 feat(storage): implement S3/SeaweedFS blob storage for documents
Replaces local filesystem document storage with S3-compatible object storage.
Adds embedded SeaweedFS 'weed mini' for local development and Docker deployments.
Updates document upload flow to use presigned URLs with a server fallback proxy.
Refactors auth configuration to use BASE_URL and AUTH_SECRET.

BREAKING CHANGE: Renamed BETTER_AUTH_URL to BASE_URL and BETTER_AUTH_SECRET to AUTH_SECRET. Removed legacy document upload/content endpoints. Requires S3 environment variables (auto-configured for embedded SeaweedFS).
2026-02-10 12:29:15 -07:00
Richard R
4a5f3060f2 refactor(documents): migrate from IndexedDB to server-backed storage with caching
- Replace client-side IndexedDB with server-first document storage
- Add document caching layer for offline capability and performance
- Implement document transfer during anonymous-to-authenticated account linking
- Add database indexes for improved query performance
- Update document contexts to use new caching system
- Refactor document upload and deletion APIs
- Add audiobook pruning for missing files
- Improve test isolation with namespace support
- Add unit tests for document cache and transfer functions
2026-02-08 11:15:57 -07:00
Richard R
2c23dc9043 refactor(db): remove conditional database usage from API layer
Ensure all metadata operations consistently use the database layer,
removing filesystem-only fallback paths and conditional DB checks.

- Simplified migration scripts to always run on startup
- Updated document/audiobook APIs to always query DB
- Added ensureDbIndexed() calls across all routes
- Extracted test namespace utilities to dedicated module
- Removed migration-manager.ts (functionality consolidated)
- Updated rate limiter to assume DB is always available

BREAKING CHANGE: Database is now required in all configurations.
When auth is disabled, SQLite is used by default at /app/docstore/sqlite3.db.
2026-02-03 12:17:30 -07:00
Richard R
c24710b2ca refactor(audiobooks): implement user-specific storage with composite primary keys
- Change audiobooks and audiobookChapters tables to use composite PK (id, userId)
- Migrate audiobooks from flat storage to user-specific directories under audiobooks_users
- Add support for claiming unclaimed audiobooks on account creation
- Improve auth rate limiting with retry logic and DISABLE_AUTH_RATE_LIMIT for tests
- Fix iOS/Safari audio playback with unlock mechanism and playback rate watchdog
- Update API routes to handle user-scoped audiobook access with fallback to unclaimed
- Transfer audiobooks when linking anonymous accounts to real accounts
- Remove foreign key constraint from audiobookChapters to support composite PK
- Add cascade delete to account and session foreign keys
2026-02-03 12:17:06 -07:00
Richard R
30ce65e747 refactor(auth): bootstrap anonymous sessions via AuthLoader
Replace privacy popup with PrivacyModal and reorganize auth UI components
Remove signed-out persistence and centralize session handling in useAuthSession
Move document viewers and Kokoro voice helpers into dedicated modules
2026-01-28 02:39:44 -07:00
Richard R
20111b1e36 fix(tests): try to get tests working again 2026-01-28 01:04:01 -07:00
Richard R
297752e83e fix(db): select drizzle config for migrations
Add dedicated Drizzle config files for sqlite and postgres and generate
new migration outputs for each dialect.

Update migrate scripts to load env files and auto-pick the correct
config unless explicitly provided, and tighten API auth handling by
using shared auth context helpers and enforcing userId checks when auth
is enabled.
2026-01-27 16:53:58 -07:00
Richard R
8189087ad9 fix: reduce pdf flicker and harden tts
- Smooth react-pdf page turns with stable canvas staging and fade-in
- Retry transient empty page text extraction during fast page turns
- Avoid noisy TTS preload toasts, guard requests when quota is exhausted,
  and improve Howler retry/unload behavior to prevent pool exhaustion
- Enforce auth FK cascade deletes and tighten rate-limiter bucket updates
  using affected-row checks
2026-01-26 17:01:51 -07:00
Richard R
c57b913cb5 feat(db/auth): migrate auth storage to Drizzle and add claimable data flow
- replace custom DB adapter with Drizzle setup (SQLite/Postgres schemas) and add `drizzle.config.ts` plus migrations in `drizzle/` and `drizzle_pg/`
- switch better-auth to drizzleAdapter, add auth helpers (`getAuthContext`, `requireAuthContext`, `requireAudiobookOwned`), and make `useAuth`/`useAuthSession` safe no-ops when auth is disabled
- persist documents/audiobooks in DB with ownership checks, unclaimed fallback, and ref-counted deletes; add FS scan helper for no-auth mode
- gate docx-to-pdf, library, voices, whisper, and migration endpoints behind auth when enabled
- add unclaimed data scan/claim flow with `/api/user/claim`, `ClaimDataModal`, and server-side scan/claim helpers
- refactor rate limiting and account deletion to use Drizzle tables (`user_tts_chars`, `user`)
- run migrations via `scripts/migrate-if-auth.mjs` (auto on `pnpm start` + `pnpm migrate`), remove Docker entrypoint and old better-auth migration file
- update README and lockfile for the new migration workflow and dependencies
2026-01-26 17:01:36 -07:00
Richard R
9561d5a816 fix(ui): adjust modal positioning for responsive design
Update modal components to start from top on mobile devices and center on larger screens, improving usability across screen sizes. Also refine transition handling in privacy popup for cleaner unmounting.
2026-01-25 15:13:02 -07:00
Richard R
fb8fc41994 fix(ui): adjust dropdown positioning and scrolling in voices control and improve the ui label for speed controls 2026-01-25 14:55:45 -07:00
Richard R
c345628200 fix: remove unused imports and fix privacy popup text
- Remove unused SettingsIcon and UserMenu imports from epub and pdf pages
- Remove unused UserMenu import from home page
- Remove unused isDev const from home page
- Update SettingsModal to only import clearSignedOut
- Fix apostrophe encoding in privacy popup description
2026-01-25 14:41:49 -07:00
Richard R
50c3829f98 update(auth): enhance rate limiting with device/IP backstops and improve UI
- Add device ID and IP-based rate limiting to prevent abuse
- Refactor UI components for better header menus and settings
- Update privacy popup with detailed data usage info
- Improve PDF handling and caching to prevent react-pdf warnings
- Update README with new Docker instructions and environment variables
2026-01-25 14:38:31 -07:00
Richard R
4689f3676f refactor(auth): consolidate authentication contexts and enhance rate limit UI integration
Streamline auth client factory and remove redundant session manager
Introduce unified auto rate limit context for seamless status updates
Implement problem details in TTS responses for improved error clarity
Add rate limit pause controls and banners across document viewers
Refine PDF scaling logic with height-aware resize handling
Update privacy popup triggers and first-visit modal behavior

BREAKING CHANGE: Daily character limits adjusted (anonymous: 250K → 50K, authenticated: 1M → 500K)
2026-01-25 11:12:19 -07:00
Richard R
9ba20c8a9e feat(auth): add user authentication and rate limiting
- Implement user sign-in, sign-up, and account management using better-auth
- Add rate limiting for TTS API with daily character limits for authenticated and anonymous users
- Integrate SQLite and PostgreSQL database support for user sessions and data persistence
- Update UI components to include authentication flows, user menu, and privacy popup
- Modify Dockerfile and package.json for new dependencies and entrypoint script
- Bump version to v1.3.0
2026-01-24 17:36:11 -07:00
Richard R
64825b626a chore: Update various dependencies and introduce pnpm overrides for specific packages. 2026-01-24 12:33:48 -07:00
Richard R
54b9691524 refactor(nlp): enhance EPUB text splitting for oversized sentences and remove unused functions
- Add logic to split oversized sentences in `splitTextToTtsBlocksEPUB` using `splitOversizedText` to ensure blocks stay within `MAX_BLOCK_LENGTH`
- Remove `extractRawSentences` and `processTextWithMapping` functions as they are no longer needed
- Update tests to reflect the changes, including a new test for oversized sentence splitting and removal of related test suites
2026-01-21 14:27:20 -07:00
Richard R
efe6ffec86 fix(nlp): add EPUB-specific text splitting for TTS blocks
Introduce `splitTextToTtsBlocksEPUB` function to handle EPUB text splitting,
treating single newlines as paragraph boundaries for precise highlighting.
Update TTS context to conditionally use EPUB splitting based on document type.
Enhance PDFViewer to clear highlights when current sentence is null.
Add comprehensive tests for the new functionality and refactor existing ones.
2026-01-21 14:04:19 -07:00
Richard R
63c316fc4b fix(pdfviewer): improve sentence highlight cleanup when current sentence is null
Handle case where currentSentence is null by canceling retry loops and clearing stale highlights to prevent lingering highlights. Remove redundant check inside the highlight function.
2026-01-21 13:32:04 -07:00
Richard R
5a40e4b00a fix: improve word highlight cleanup in PDFViewer and update test
- Add cleanup function to prevent memory leaks in word highlighting
- Update test to allow blocks ending with ., !, or ? for better accuracy
2026-01-21 13:17:57 -07:00
Richard R
c28c074e43 refactor(nlp): improve text splitting and normalization for TTS blocks
- Rename and enhance text processing functions in nlp.ts for better handling of oversized texts, sentence boundaries, and PDF artifacts
- Update PDFViewer to add layout-aware highlighting with retry logic for sentence and word highlights
- Adjust PDFContext and TTSContext to use new normalized text functions
- Expand unit tests for new splitting behaviors, including long texts and punctuation preferences
2026-01-21 12:09:59 -07:00
Richard R
409b25977c tests: Add NLP text processing, audiobook chapter encoding/decoding, and SHA256 hashing testing. 2026-01-19 17:45:55 -07:00
Richard R
8dbf904a21 feat: Implement FFmpeg metadata escaping and robust chapter index validation for audiobook generation, alongside unit test reorganization. 2026-01-19 17:33:29 -07:00
Richard R
9c941b57ed fix: Enhance path sanitization in getAudiobooksDir to prevent directory traversal and update getMigratedDocumentFileName tests to assert truncated- prefix and hash suffix. 2026-01-19 17:17:51 -07:00
Richard R
47d838039a feat: Add bookId validation to API routes and implement robust, length-limited filename generation with unit tests. 2026-01-19 17:04:02 -07:00
Richard R
33d49d6966 fix: Enhance audiobook API validation and prevent critical meta file deletion 2026-01-19 16:40:04 -07:00
Richard R
f947ece01a feat: Implement document selection modal to allow users to select specific documents for sync, load, and import operations. 2026-01-19 15:53:29 -07:00
Richard R
741126c0b4 feat: Display a toast notification after library migration, powered by new migration status in the API response. 2026-01-19 15:27:07 -07:00
Richard R
57ac17fb34 refactor(api): force dynamic rendering for API routes
Add 'force-dynamic' export to audiobook and documents API routes to ensure
dynamic rendering. Remove 'nodejs' runtime export from TTS route for consistency.
2026-01-19 14:33:49 -07:00
Richard R
199b937a25 refactor: overhaul document storage and audiobook architecture
- refactor(documents): implement stable SHA256-based document IDs
- feat(library): add support for external document libraries
- refactor(audiobook): consolidate server logic and migrate to V1 storage layout
- feat(audiobook): add reset functionality and improve chapter management
- fix(ui): enhance audiobook export modal with download/regenerate controls
- test: add coverage for upload hashing and export reset flow
2026-01-19 14:22:21 -07:00
Richard Roberson
d5ec96d395 feat(ui): implement better document grid with file previews
Some checks failed
Build and Publish Docker Image / build (push) Has been cancelled
- Introduces `DocumentPreview` component to display rich previews for documents.
- PDFs generate a thumbnail of their first page.
- EPUBs extract and display their cover image.
- HTML, TXT, and Markdown files show a text snippet.
- Refactors document list and folder views to a responsive grid layout.
- Expands main content areas to accommodate the new grid.
- Updates Dockerfile to use Node.js LTS image.
- Updates various development and runtime dependencies.
2025-12-12 13:13:49 -07:00
Richard Roberson
30ade1d79a fix(ui): fix chapter list overflow and menu positioning
Some checks failed
Build and Publish Docker Image / build (push) Has been cancelled
- Add scroll functionality to the chapter list to prevent content
  overflow.
- Conditionally adjust the positioning of chapter action menus
  to ensure they remain visible within the modal, preventing
  them from being cut off for chapters at the top of the list.
2025-11-22 18:37:38 -07:00
Richard Roberson
d2189541e8 fix(tailwind): fix footer layout and tailwind config
Some checks failed
Build and Publish Docker Image / build (push) Has been cancelled
- Remove unused `isDev` variable from `Footer.tsx`.
- Add `inline` and `inline-flex` classes to footer elements for improved layout.
- Move custom `xs` breakpoint definition to `theme.extend` in `tailwind.config.ts`.
2025-11-22 16:32:03 -07:00
Richard Roberson
a755a0b3f9 refactor(ui): make UI labels and descriptions more concise 2025-11-22 15:27:04 -07:00
Richard Roberson
e86782bf38 feat: improve document list and self-hosting clarity
- Introduce a toggleable grid/list view for the document list, enhancing organization and display flexibility. This involved updates across `DocumentList`, `DocumentFolder`, `DocumentListItem`, and `SortControls`.
- Add a new `CodeBlock` component and integrate detailed Docker-based self-hosting instructions into the footer.
- Enhance privacy policy popover with clearer details on Deepinfra usage and strongly recommend self-hosting for secure experience.
- Implement conditional rendering and feature gating based on the `isDev` environment variable, differentiating features between the production demo and self-hosted instances. Affected components include `page.tsx`, `DocumentSettings.tsx`, `SettingsModal.tsx`, and `config.ts`.
- Clarify that advanced features like audiobook export and word-by-word highlighting via `whisper.cpp` are exclusive to self-hosted setups and are disabled in the demo.
- Expand the list of supported document types on the homepage to include MD and TXT.
- Integrate new `ListIcon`, `GridIcon`, and `CopyIcon` to support UI enhancements.
- Add a custom `xs` breakpoint in `tailwind.config.ts` for improved responsive design.
2025-11-22 15:18:46 -07:00
Richard Roberson
e39a5b8bcf refactor(player): simplify popover input focus logic
Removed `isPopoverOpen` state and associated `useEffect` hook.
Auto-focus and select logic for the input are now directly handled
within `handlePopoverOpen`, reducing state management complexity.
2025-11-22 02:06:28 -07:00
Richard Roberson
7046666b3f fix(tts): ensure robust caching for audio and alignments
- New buildCacheKey function creates unique identifiers for TTS cache entries.
- Cache keys now include sentence, voice, speed, provider, and model parameters.
- Prevents serving cached audio or alignment data that mismatches current TTS parameters.
- Removes redundant audioCache.clear() calls when voice or speed change, as entries are now distinct.
2025-11-22 01:57:27 -07:00
Richard Roberson
773778ee90 feat(epub): add custom reader navigation and TOC
- Implement custom previous/next page buttons for EPUB viewer.
- Display current page number out of total pages.
- Introduce an in-viewer, toggleable table of contents (TOC) for quick chapter navigation.
- Hide default `react-reader` navigation arrows and title bar to prevent redundancy.
- Adjust `react-reader` styles to optimize content area and ensure custom controls are visible.
- Update description for EPUB theme setting in document settings.
- Add `ChevronLeftIcon` and `ChevronRightIcon` to icon library.
- Update Playwright tests to reflect new navigation button labels.
2025-11-22 01:33:53 -07:00
Richard Roberson
b576910523 refactor(client): centralize client-side API calls and refine types
Abstracted direct fetch calls across components and contexts into new functions within `src/lib/client.ts`. This provides a consistent and centralized interface for interacting with backend APIs.

- Introduced `src/lib/client.ts` to encapsulate API request logic.
- Standardized audio buffer types (`TTSAudioBuffer`, `TTSAudioBytes`) in `src/types/tts.ts`.
- Moved client-specific request types (`TTSRequestPayload`, `TTSRequestHeaders`, `TTSRetryOptions`) to `src/types/client.ts`.
- Updated API routes and consumer components/contexts to leverage the new client library functions and type definitions.
- Removed `src/utils/audio.ts` as its utility functions are now part of `src/lib/client.ts`.
2025-11-21 23:33:41 -07:00
Richard Roberson
7a29f73d07 refactor(api): move audiobook endpoints to new path
- Relocated all audiobook-related API routes from `/api/audio/convert/*` to `/api/audiobook/*`.
- This change affects endpoints for chapter conversion, retrieval, deletion, and overall audiobook status.
- Updated client-side calls in `AudiobookExportModal.tsx`, `EPUBContext.tsx`, and `PDFContext.tsx` to reflect the new API paths.
- Modified API tests (`api.spec.ts`, `export.spec.ts`) to target the restructured endpoints.
- The new API structure provides better organization and a clearer, more consistent interface for audiobook functionality.
2025-11-21 19:29:49 -07:00
Richard Roberson
733b4180eb feat(tts): add word-by-word highlighting with whisper.cpp
- Introduce `/api/whisper` endpoint which uses `whisper.cpp` (via a `WHISPER_CPP_BIN` executable) and `ffmpeg` to generate word-level audio alignments from provided audio and text.
- Integrate word-level alignments into `TTSContext`, tracking the currently spoken word based on audio seek position and provided timestamps. Alignments are cached in-memory and fetched asynchronously.
- Add new configuration options (`pdfWordHighlightEnabled`, `epubWordHighlightEnabled`) to `ConfigContext` and `Dexie` for enabling/disabling the feature.
- Implement visual word highlighting in both `PDFViewer` and `EPUBViewer` by mapping TTS-aligned words to rendered text elements.
- Enhance `EPUBContext` and `PDFContext` with new `highlightWordIndex` and `clearWordHighlights` functions, utilizing fuzzy string matching (`cmpstr`) to robustly align spoken words with displayed text for accurate highlighting.
- Update `DocumentSettings` to include user-facing toggles for the new highlighting modes.
2025-11-21 19:22:54 -07:00
Richard Roberson
ac194ed749 refactor(audiobook): update audiobook export modal generation message to include TTS options. 2025-11-18 12:39:23 -07:00
Richard Roberson
50538bfaba feat(epub): Add EPUB text highlighting for the current TTS sentence with a new configuration option 2025-11-18 12:31:07 -07:00
Richard Roberson
cc0372405f chore(env): hide audiobook export button in production 2025-11-16 21:41:25 -07:00
Richard Roberson
ff150eec01 refactor(audiobook): standardize chapter regeneration APIs and UX
- Removed `onProgress` callback from `regenerateChapter` and related functions across `TTS`, `EPUB`, and `PDF` contexts to simplify the chapter regeneration API.
- Updated `AudiobookExportModal` to align with the refined regeneration API, including removing granular chapter progress display and adding a hint about TTS caching behavior.
- Introduced `TTSAudiobookChapter` interface and renamed `ContinuationMergeResult` to `TTSSmartMergeResult` and `PageTurnEstimate` to `TTSPageTurnEstimate` for better type consistency and clarity.
- Applied minor styling adjustments to buttons and listbox components in modals for visual consistency.
- Added `'use client'` directive to several client-side components for Next.js 13+ compatibility.
- Updated Dockerfile build command from `pnpm run build` to `pnpm build`.
- Added Playwright tests to verify backend chapter state after regeneration.
2025-11-16 21:10:32 -07:00
Richard Roberson
04def62ff5 refactor(core): standardize module structure and enhance TTS types
- Reorganized utility modules from `src/utils` to `src/lib` for clearer separation of concerns.
- Introduced new, dedicated type definitions in `src/types` for improved type safety in configuration and TTS API interactions.
  - Replaced `src/types/appConfig.ts` with `src/types/config.ts`.
  - Added `src/types/tts.ts` for TTS request payloads, error structures, and retry options.
- Updated module imports across several contexts (`Config`, `EPUB`, `HTML`, `PDF`, `TTS`) and components to reflect the new `lib` and `types` locations.
- Enhanced TTS API request and error handling in `src/app/api/tts/route.ts` and TTS-consuming contexts with explicit types.
- Simplified `ProgressCard`, `ProgressPopup`, and `AudiobookExportModal` components by removing the `isProcessing` prop, centralizing processing state management.
- Streamlined `HTMLContext` by removing `createFullAudioBook` and `isAudioCombining` properties, focusing its scope.
2025-11-16 15:12:35 -07:00
Richard Roberson
7271afa2f5 refactor(config): centralize app config in Dexie
Migrates scattered configuration items and last document locations
from the legacy `config` table to new, dedicated Dexie stores.

- Introduces a singleton `app-config` table (`AppConfigRow`) to standardize
  and structure all application settings in a single object.
- Creates a `last-locations` table for efficient storage of document read positions.
- Updates `ConfigContext` to utilize `dexie-react-hooks`' `useLiveQuery`
  for reactive and simplified state management.
- Implements a database upgrade path (from DB_VERSION 4 to 5) to migrate
  all existing user settings seamlessly.
- Simplifies config access and updates with new `getAppConfig` and `updateAppConfig`
  utility functions.
- Removes the deprecated `config` table after successful migration.
- Updates consumers like `SettingsModal` and test helpers to align with
  the new config structure.
2025-11-15 17:29:56 -07:00
Richard Roberson
071d967f8f feat(pdf): modernize text highlighting and parsing for fluidity
- Offload computationally intensive text matching for real-time highlighting to a dedicated Web Worker, ensuring the main thread remains responsive during playback.
- Implement a new overlay-based highlighting system that renders independently of the PDF's text layer, providing smoother and more reliable visual feedback without interfering with document rendering.
- Introduce a new setting allowing users to enable or disable real-time text highlighting in PDFs, offering personalized control over the reading interface.
- Upgrade the underlying text comparison algorithm to Dice similarity for more accurate and context-aware matching of spoken words to on-screen text, improving synchronization precision.
- Improve sentence boundary detection, especially for quoted dialogue and complex structures, by enhancing the NLP processing logic, leading to a more natural audio-text flow.
2025-11-15 14:06:44 -07:00
Richard Roberson
2f39e8f014 perf(test): optimize test execution and error handling
- Reduce Playwright worker allocation from 75% to 50% for better resource management
- Disable parallel test execution temporarily to isolate flaky test behavior
- Enhance audio utility to fail fast on user cancellation, preventing unnecessary retries
- Consolidate test-specific utilities within individual test suites for better isolation
- Introduce export functionality test coverage with new export.spec.ts suite
2025-11-14 18:52:49 -07:00
Richard Roberson
5316596a1b refactor(db): migrate to Dexie with reactive queries and simplify data layer
Replaces custom IndexedDB implementation with Dexie ORM, eliminating 850+ lines of
boilerplate code and introducing reactive live queries across all document types.
Transforms document management from imperative refresh patterns to automatic
reactive updates using dexie-react-hooks.

Simplifies TTS backend by removing concurrency semaphore while maintaining
request de-duplication through in-flight tracking. Streamlines document hooks
by removing manual state management and refresh methods.

Updates package dependencies and type definitions to support new database
architecture while maintaining full backward compatibility for existing
documents and settings.

BREAKING CHANGE: Document hooks no longer expose refresh() methods as updates
are now reactive through live queries.
2025-11-14 16:14:00 -07:00