diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 00000000..11d4b9cf --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1 @@ +- This project uses the AGENTS.md file to give detailed information about the repository structure and development commands. Make sure to read this file before starting development. diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..136692d0 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,267 @@ +# AGENTS.md + +## Important instructions + +- Never create migration files manually. Always use the provided command to generate migrations +- If you realize an automated migration is incorrect, make sure to remove all the associated entries from the `_journal.json` and the newly created files located in `app/drizzle/` before re-generating the migration + +## Project Overview + +Zerobyte is a backup automation tool built on top of Restic that provides a web interface for scheduling, managing, and monitoring encrypted backups. It supports multiple volume backends (NFS, SMB, WebDAV, local directories) and repository backends (S3, Azure, GCS, local, and rclone-based storage). + +## Technology Stack + +- **Runtime**: Bun 1.3.1 +- **Server**: Hono (web framework) with Bun runtime +- **Client**: React Router v7 (SSR) with React 19 +- **Database**: SQLite with Drizzle ORM +- **Validation**: ArkType for runtime schema validation +- **Styling**: Tailwind CSS v4 + Radix UI components +- **Architecture**: Unified application structure (not a monorepo) +- **Code Quality**: Biome (formatter & linter) +- **Containerization**: Docker with multi-stage builds + +## Repository Structure + +This is a unified application with the following structure: + +- `app/server` - Bun-based API server with Hono +- `app/client` - React Router SSR frontend components and modules +- `app/schemas` - Shared ArkType schemas for validation +- `app/drizzle` - Database migrations + +### Type Checking + +```bash +# Run type checking and generate React Router types +bun run tsc +``` + +### Building + +```bash +# Build for production +bun run build +``` + +### Database Migrations + +```bash +# Generate new migration from schema changes +bun gen:migrations + +# Generate a custom empty migration +bunx drizzle-kit generate --custom --name=fix-timestamps-to-ms + +``` + +### API Client Generation + +```bash +# Generate TypeScript API client from OpenAPI spec +# Note: Server is always running don't need to start it separately +bun run gen:api-client +``` + +### Code Quality + +```bash +# Format and lint (Biome) +bunx biome check --write . + +# Format only +bunx biome format --write . + +# Lint only +bunx biome lint . +``` + +## Architecture + +### Server Architecture + +The server follows a modular service-oriented architecture: + +**Entry Point**: `app/server/index.ts` + +- Initializes servers using `react-router-hono-server`: + 1. Main API server on port 4096 (REST API + serves static frontend) + 2. Docker volume plugin server on Unix socket `/run/docker/plugins/zerobyte.sock` (optional, if Docker is available) + +**Modules** (`app/server/modules/`): +Each module follows a controller � service � database pattern: + +- `auth/` - User authentication and session management +- `volumes/` - Volume mounting/unmounting (NFS, SMB, WebDAV, directories) +- `repositories/` - Restic repository management (S3, Azure, GCS, local, rclone) +- `backups/` - Backup schedule management and execution +- `notifications/` - Notification system with multiple providers (Discord, email, Gotify, Ntfy, Slack, Pushover) +- `driver/` - Docker volume plugin implementation +- `events/` - Server-Sent Events for real-time updates +- `system/` - System information and capabilities +- `lifecycle/` - Application startup/shutdown hooks + +**Backends** (`app/server/modules/backends/`): +Each volume backend (NFS, SMB, WebDAV, directory) implements mounting logic using system tools (mount.nfs, mount.cifs, davfs2). + +**Jobs** (`app/server/jobs/`): +Cron-based background jobs managed by the Scheduler: + +- `backup-execution.ts` - Runs scheduled backups (every minute) +- `cleanup-dangling.ts` - Removes stale mounts (hourly) +- `healthchecks.ts` - Checks volume health (every 5 minutes) +- `repository-healthchecks.ts` - Validates repositories (every 10 minutes) +- `cleanup-sessions.ts` - Expires old sessions (daily) + +**Core** (`app/server/core/`): + +- `scheduler.ts` - Job scheduling system using node-cron +- `capabilities.ts` - Detects available system features (Docker support, etc.) +- `constants.ts` - Application-wide constants + +**Utils** (`app/server/utils/`): + +- `restic.ts` - Restic CLI wrapper with type-safe output parsing +- `spawn.ts` - Safe subprocess execution helpers +- `logger.ts` - Winston-based logging +- `crypto.ts` - Encryption utilities +- `errors.ts` - Error handling middleware + +**Database** (`app/server/db/`): + +- Uses Drizzle ORM with SQLite +- Schema in `schema.ts` defines: volumes, repositories, backup schedules, notifications, users, sessions +- Migrations: `app/drizzle/` + +### Client Architecture + +**Framework**: React Router v7 with SSR +**Entry Point**: `app/root.tsx` + +The client uses: + +- TanStack Query for server state management +- Auto-generated API client from OpenAPI spec (in `app/client/api-client/`) +- Radix UI primitives with custom Tailwind styling +- Server-Sent Events hook (`use-server-events.ts`) for real-time updates + +Routes are organized in feature modules at `app/client/modules/*/routes/`. + +### Shared Schemas + +`app/schemas/` contains ArkType schemas used by both client and server: + +- Volume configurations (NFS, SMB, WebDAV, directory) +- Repository configurations (S3, Azure, GCS, local, rclone) +- Restic command output parsing types +- Backend status types + +These schemas provide runtime validation and TypeScript types. + +## Restic Integration + +Zerobyte is a wrapper around Restic for backup operations. Key integration points: + +**Repository Management**: + +- Creates/initializes Restic repositories via `restic init` +- Supports multiple backends: local, S3, Azure Blob Storage, Google Cloud Storage, or any rclone-supported backend +- Stores single encryption password in `/var/lib/zerobyte/restic/password` (auto-generated on first run) + +**Backup Operations**: + +- Executes `restic backup` with user-defined schedules (cron expressions) +- Supports include/exclude patterns for selective backups +- Parses JSON output for progress tracking and statistics +- Implements retention policies via `restic forget --prune` + +**Repository Utilities** (`utils/restic.ts`): + +- `buildRepoUrl()` - Constructs repository URLs for different backends +- `buildEnv()` - Sets environment variables (credentials, cache dir) +- `ensurePassfile()` - Manages encryption password file +- Type-safe parsing of Restic JSON output using ArkType schemas + +**Rclone Integration** (`app/server/modules/repositories/`): + +- Allows using any rclone backend as a Restic repository +- Dynamically generates rclone config and passes via environment variables +- Supports backends like Dropbox, Google Drive, OneDrive, Backblaze B2, etc. + +## Docker Volume Plugin + +When Docker socket is available (`/var/run/docker.sock`), Zerobyte registers as a Docker volume plugin: + +**Plugin Location**: `/run/docker/plugins/zerobyte.sock` +**Implementation**: `app/server/modules/driver/driver.controller.ts` + +This allows other containers to mount Zerobyte volumes using Docker. + +The plugin implements the Docker Volume Plugin API v1. + +## Environment & Configuration + +**Runtime Environment Variables**: + +- Database path: `./data/zerobyte.db` (configurable via `drizzle.config.ts`) +- Restic cache: `/var/lib/zerobyte/restic/cache` +- Restic password: `/var/lib/zerobyte/restic/password` +- Volume mounts: `/var/lib/zerobyte/mounts/` +- Local repositories: `/var/lib/zerobyte/repositories/` + +**Capabilities Detection**: +On startup, the server detects available capabilities (see `core/capabilities.ts`): + +- **Docker**: Requires `/var/run/docker.sock` access +- System will gracefully degrade if capabilities are unavailable + +## Common Workflows + +### Adding a New Volume Backend + +1. Create backend implementation in `app/server/modules/backends//` +2. Implement `mount()` and `unmount()` methods +3. Add schema to `app/schemas/volumes.ts` +4. Update `volumeConfigSchema` discriminated union +5. Update backend factory in `app/server/modules/backends/backend.ts` + +### Adding a New Repository Backend + +1. Add backend type to `app/schemas/restic.ts` +2. Update `buildRepoUrl()` in `app/server/utils/restic.ts` +3. Update `buildEnv()` to handle credentials/configuration +4. Add DTO schemas in `app/server/modules/repositories/repositories.dto.ts` +5. Update repository service to handle new backend + +### Adding a New Scheduled Job + +1. Create job class in `app/server/jobs/.ts` extending `Job` +2. Implement `run()` method +3. Register in `app/server/modules/lifecycle/startup.ts` with cron expression: + ```typescript + Scheduler.build(YourJob).schedule("* * * * *"); + ``` + +## Important Notes + +- **Code Style**: Uses Biome with tabs (not spaces), 120 char line width, double quotes +- **Imports**: Organize imports is disabled in Biome - do not auto-organize +- **TypeScript**: Uses `"type": "module"` - all imports must include extensions when targeting Node/Bun +- **Validation**: Prefer ArkType over Zod - it's used throughout the codebase +- **Database**: Timestamps are stored as Unix epoch integers, not ISO strings +- **Security**: Restic password file has 0600 permissions - never expose it +- **Mounting**: Requires privileged container or CAP_SYS_ADMIN for FUSE mounts +- **API Documentation**: OpenAPI spec auto-generated at `/api/v1/openapi.json`, docs at `/api/v1/docs` + +## Docker Development Setup + +The `docker-compose.yml` defines two services: + +- `zerobyte-dev` - Development with hot reload (uses `development` stage) +- `zerobyte-prod` - Production build (uses `production` stage) + +Both mount: + +- `/var/lib/zerobyte` for persistent data +- `/dev/fuse` device for FUSE mounting +- Optionally `/var/run/docker.sock` for Docker plugin functionality