feat: file-based provisionning of volumes and repos (#656)

* feat: file-based provisionning of volumes and repos

docs: provisioning example

chore: ui improvements

* chore: ci issues
This commit is contained in:
Nico 2026-03-12 18:31:42 +01:00 committed by GitHub
parent f7f56c6c83
commit d74f516336
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
37 changed files with 3546 additions and 348 deletions

View file

@ -100,16 +100,19 @@ Zerobyte can be customized using environment variables. Below are the available
| `LOG_LEVEL` | Logging verbosity. Options: `debug`, `info`, `warn`, `error`. | `info` |
| `SERVER_IDLE_TIMEOUT` | Idle timeout for the server in seconds. | `60` |
| `RCLONE_CONFIG_DIR` | Path to the rclone config directory inside the container. Change this if running as a non-root user. | `/root/.config/rclone` |
| `PROVISIONING_PATH` | Path to a JSON file with operator-managed repositories and volumes to sync at startup. | (none) |
### Secret References
### Provisioned Resources
For enhanced security, Zerobyte supports dynamic secret resolution for sensitive fields (like passwords, access keys, etc.) in volume and repository configurations. Instead of storing the encrypted secret in the database, you can use one of the following prefixes:
Zerobyte can sync operator-managed repositories and volumes from a JSON file at startup. This is useful when you want credentials or connection details to live in deployment-time configuration instead of being entered through the UI.
- `env://VAR_NAME`: Reads the secret from the environment variable `VAR_NAME`.
- `file://SECRET_NAME`: Reads the secret from `/run/secrets/SECRET_NAME` (standard Docker Secrets path).
Provisioned resources:
**Example:**
When configuring an S3 repository, you can set the Secret Access Key to `env://S3_SECRET_KEY` and then provide that variable in your `docker-compose.yml`.
- appear in the normal repositories and volumes screens
- stay read-only in the UI
- can resolve credential fields from environment variables or `/run/secrets/*` during startup sync
See `examples/provisioned-resources/README.md` for a full example.
### Simplified setup (No remote mounts)

View file

@ -337,6 +337,7 @@ export type ListVolumesResponses = {
200: Array<{
id: number;
shortId: string;
provisioningId: string | null;
name: string;
type: 'nfs' | 'smb' | 'directory' | 'webdav' | 'rclone' | 'sftp';
status: 'mounted' | 'unmounted' | 'error';
@ -462,6 +463,7 @@ export type CreateVolumeResponses = {
201: {
id: number;
shortId: string;
provisioningId: string | null;
name: string;
type: 'nfs' | 'smb' | 'directory' | 'webdav' | 'rclone' | 'sftp';
status: 'mounted' | 'unmounted' | 'error';
@ -635,6 +637,7 @@ export type GetVolumeResponses = {
volume: {
id: number;
shortId: string;
provisioningId: string | null;
name: string;
type: 'nfs' | 'smb' | 'directory' | 'webdav' | 'rclone' | 'sftp';
status: 'mounted' | 'unmounted' | 'error';
@ -776,6 +779,7 @@ export type UpdateVolumeResponses = {
200: {
id: number;
shortId: string;
provisioningId: string | null;
name: string;
type: 'nfs' | 'smb' | 'directory' | 'webdav' | 'rclone' | 'sftp';
status: 'mounted' | 'unmounted' | 'error';
@ -986,6 +990,7 @@ export type ListRepositoriesResponses = {
200: Array<{
id: string;
shortId: string;
provisioningId: string | null;
name: string;
type: 'local' | 's3' | 'r2' | 'gcs' | 'azure' | 'rclone' | 'rest' | 'sftp';
config: {
@ -1406,6 +1411,7 @@ export type GetRepositoryResponses = {
200: {
id: string;
shortId: string;
provisioningId: string | null;
name: string;
type: 'local' | 's3' | 'r2' | 'gcs' | 'azure' | 'rclone' | 'rest' | 'sftp';
config: {
@ -1779,6 +1785,7 @@ export type UpdateRepositoryResponses = {
200: {
id: string;
shortId: string;
provisioningId: string | null;
name: string;
type: 'local' | 's3' | 'r2' | 'gcs' | 'azure' | 'rclone' | 'rest' | 'sftp';
config: {
@ -2435,6 +2442,7 @@ export type ListBackupSchedulesResponses = {
volume: {
id: number;
shortId: string;
provisioningId: string | null;
name: string;
type: 'nfs' | 'smb' | 'directory' | 'webdav' | 'rclone' | 'sftp';
status: 'mounted' | 'unmounted' | 'error';
@ -2495,6 +2503,7 @@ export type ListBackupSchedulesResponses = {
repository: {
id: string;
shortId: string;
provisioningId: string | null;
name: string;
type: 'local' | 's3' | 'r2' | 'gcs' | 'azure' | 'rclone' | 'rest' | 'sftp';
config: {
@ -2806,6 +2815,7 @@ export type GetBackupScheduleResponses = {
volume: {
id: number;
shortId: string;
provisioningId: string | null;
name: string;
type: 'nfs' | 'smb' | 'directory' | 'webdav' | 'rclone' | 'sftp';
status: 'mounted' | 'unmounted' | 'error';
@ -2866,6 +2876,7 @@ export type GetBackupScheduleResponses = {
repository: {
id: string;
shortId: string;
provisioningId: string | null;
name: string;
type: 'local' | 's3' | 'r2' | 'gcs' | 'azure' | 'rclone' | 'rest' | 'sftp';
config: {
@ -3158,6 +3169,7 @@ export type GetBackupScheduleForVolumeResponses = {
volume: {
id: number;
shortId: string;
provisioningId: string | null;
name: string;
type: 'nfs' | 'smb' | 'directory' | 'webdav' | 'rclone' | 'sftp';
status: 'mounted' | 'unmounted' | 'error';
@ -3218,6 +3230,7 @@ export type GetBackupScheduleForVolumeResponses = {
repository: {
id: string;
shortId: string;
provisioningId: string | null;
name: string;
type: 'local' | 's3' | 'r2' | 'gcs' | 'azure' | 'rclone' | 'rest' | 'sftp';
config: {
@ -3687,6 +3700,7 @@ export type GetScheduleMirrorsResponses = {
repository: {
id: string;
shortId: string;
provisioningId: string | null;
name: string;
type: 'local' | 's3' | 'r2' | 'gcs' | 'azure' | 'rclone' | 'rest' | 'sftp';
config: {
@ -3898,6 +3912,7 @@ export type UpdateScheduleMirrorsResponses = {
repository: {
id: string;
shortId: string;
provisioningId: string | null;
name: string;
type: 'local' | 's3' | 'r2' | 'gcs' | 'azure' | 'rclone' | 'rest' | 'sftp';
config: {

View file

@ -0,0 +1,23 @@
import { Badge } from "~/client/components/ui/badge";
import { Tooltip, TooltipContent, TooltipTrigger } from "~/client/components/ui/tooltip";
type ManagedBadgeProps = {
label?: string;
message?: string;
};
const defaultMessage =
"This resource is provisioned at startup. Changes are useful for testing, but the next provisioning sync can overwrite or recreate it.";
export const ManagedBadge = ({ label = "Managed", message = defaultMessage }: ManagedBadgeProps) => {
return (
<Tooltip>
<TooltipTrigger asChild>
<Badge variant="secondary">{label}</Badge>
</TooltipTrigger>
<TooltipContent>
<p className="max-w-80 text-sm">{message}</p>
</TooltipContent>
</Tooltip>
);
};

View file

@ -24,6 +24,7 @@ import { parseError } from "~/client/lib/errors";
import { Alert, AlertDescription } from "~/client/components/ui/alert";
import type { RepositoryConfig } from "@zerobyte/core/restic";
import { useNavigate } from "@tanstack/react-router";
import { ManagedBadge } from "~/client/components/managed-badge";
const riskyLocationFieldsByBackend = {
local: ["path"],
@ -124,7 +125,10 @@ export function EditRepositoryPage({ repositoryId }: { repositoryId: string }) {
<div className="flex items-center justify-center w-10 h-10 rounded-lg bg-primary/10">
<Database className="w-5 h-5 text-primary" />
</div>
<CardTitle>Edit Repository</CardTitle>
<div className="flex items-center gap-2">
<CardTitle>Edit Repository</CardTitle>
{repository.provisioningId && <ManagedBadge />}
</div>
</div>
</CardHeader>
<CardContent className="space-y-6">

View file

@ -126,7 +126,11 @@ export function RepositoriesPage() {
className="hover:bg-accent/50 hover:cursor-pointer h-12"
onClick={() => navigate({ to: `/repositories/${repository.shortId}` })}
>
<TableCell className="font-medium text-strong-accent">{repository.name}</TableCell>
<TableCell className="font-medium text-strong-accent">
<div className="flex items-center gap-2">
<span>{repository.name}</span>
</div>
</TableCell>
<TableCell>
<span className="flex items-center gap-2 text-muted-foreground">
<RepositoryIcon backend={repository.type} />

View file

@ -28,6 +28,7 @@ import { parseError } from "~/client/lib/errors";
import { useNavigate } from "@tanstack/react-router";
import { CompressionStatsChart } from "../components/compression-stats-chart";
import { cn } from "~/client/lib/utils";
import { ManagedBadge } from "~/client/components/managed-badge";
type Props = {
repository: Repository;
@ -109,7 +110,10 @@ export const RepositoryInfoTabContent = ({ repository, initialStats }: Props) =>
<div className="flex flex-col gap-6 @container">
<div className="flex flex-col @medium:flex-row items-start @medium:items-center justify-between gap-4">
<div>
<h2 className="text-lg font-semibold tracking-tight">Repository Settings</h2>
<div className="flex items-center gap-2">
<h2 className="text-lg font-semibold tracking-tight">Repository Settings</h2>
{repository.provisioningId && <ManagedBadge />}
</div>
</div>
<div className="flex flex-wrap items-center gap-2">
<Button
@ -174,6 +178,10 @@ export const RepositoryInfoTabContent = ({ repository, initialStats }: Props) =>
<div className="text-sm font-medium text-muted-foreground">Backend</div>
<p className="text-sm">{repository.type}</p>
</div>
<div className="flex flex-col gap-1">
<div className="text-sm font-medium text-muted-foreground">Management</div>
<p className="text-sm">{repository.provisioningId ? "Provisioned" : "Manual"}</p>
</div>
<div className="flex flex-col gap-1">
<div className="text-sm font-medium text-muted-foreground">Compression Mode</div>
<p className="text-sm">{repository.compressionMode || "off"}</p>

View file

@ -1,5 +1,5 @@
import { useMutation } from "@tanstack/react-query";
import { Download, KeyRound, User, X, Settings as SettingsIcon, Building2 } from "lucide-react";
import { Download, Fingerprint, KeyRound, User, X, Settings as SettingsIcon, Building2 } from "lucide-react";
import { useState } from "react";
import { toast } from "sonner";
import { downloadResticPasswordMutation } from "~/client/api-client/@tanstack/react-query.gen";
@ -45,7 +45,7 @@ export function SettingsPage({ appContext, initialMembers, initialSsoSettings, i
const activeTab = tab || "account";
const navigate = useNavigate();
const { activeMember } = useOrganizationContext();
const { activeMember, activeOrganization } = useOrganizationContext();
const isOrgAdmin = activeMember?.role === "owner" || activeMember?.role === "admin";
const handleLogout = async () => {
@ -295,6 +295,25 @@ export function SettingsPage({ appContext, initialMembers, initialSsoSettings, i
{isOrgAdmin && (
<TabsContent value="organization" className="mt-0 space-y-4">
<Card className="p-0 gap-0">
<div className="border-b border-border/50 bg-card-header p-6">
<CardTitle className="flex items-center gap-2">
<Fingerprint className="size-5" />
Organization Details
</CardTitle>
<CardDescription className="mt-1.5">Reference details for the active organization</CardDescription>
</div>
<CardContent className="p-6 space-y-2">
<Label htmlFor="organization-id">Organization ID</Label>
<Input
id="organization-id"
value={activeOrganization.id}
readOnly
className="max-w-xl font-mono text-sm"
/>
</CardContent>
</Card>
<Card className="p-0 gap-0">
<div className="border-b border-border/50 bg-card-header p-6">
<CardTitle className="flex items-center gap-2">

View file

@ -137,7 +137,11 @@ export function VolumesPage() {
className="hover:bg-muted/50 hover:cursor-pointer transition-colors h-12"
onClick={() => navigate({ to: `/volumes/${volume.shortId}` })}
>
<TableCell className="font-medium font-mono text-strong-accent">{volume.name}</TableCell>
<TableCell className="font-medium font-mono text-strong-accent">
<div className="flex items-center gap-2">
<span>{volume.name}</span>
</div>
</TableCell>
<TableCell className="font-mono text-muted-foreground">
<VolumeIcon backend={volume.type} />
</TableCell>

View file

@ -27,6 +27,7 @@ import {
import type { UpdateVolumeResponse } from "~/client/api-client/types.gen";
import { useNavigate } from "@tanstack/react-router";
import { parseError } from "~/client/lib/errors";
import { ManagedBadge } from "~/client/components/managed-badge";
type Props = {
volume: Volume;
@ -122,7 +123,10 @@ export const VolumeInfoTabContent = ({ volume, statfs }: Props) => {
<Card className="p-6 @container">
<div className="flex flex-col @xl:flex-row items-start @xl:items-center justify-between gap-4 mb-6">
<div>
<span className="text-lg font-semibold">Volume Configuration</span>
<div className="flex items-center gap-2">
<span className="text-lg font-semibold">Volume Configuration</span>
{volume.provisioningId && <ManagedBadge />}
</div>
</div>
<div className="flex flex-col @xl:flex-row w-full @xl:w-auto gap-2">
{volume.status !== "mounted" ? (

View file

@ -0,0 +1,4 @@
ALTER TABLE `repositories_table` ADD `provisioning_id` text;--> statement-breakpoint
ALTER TABLE `volumes_table` ADD `provisioning_id` text;--> statement-breakpoint
CREATE UNIQUE INDEX `repositories_table_org_provisioning_id_uidx` ON `repositories_table` (`organization_id`,`provisioning_id`);--> statement-breakpoint
CREATE UNIQUE INDEX `volumes_table_org_provisioning_id_uidx` ON `volumes_table` (`organization_id`,`provisioning_id`);

File diff suppressed because it is too large Load diff

View file

@ -38,6 +38,7 @@ const envSchema = z
APP_SECRET: z.string().min(32).max(256),
BASE_URL: z.string(),
ENABLE_DEV_PANEL: z.string().default("false"),
PROVISIONING_PATH: z.string().optional(),
})
.transform((s) => ({
__prod__: s.NODE_ENV === "production",
@ -57,6 +58,7 @@ const envSchema = z
baseUrl: s.BASE_URL,
isSecure: s.BASE_URL?.startsWith("https://") ?? false,
enableDevPanel: s.ENABLE_DEV_PANEL === "true",
provisioningPath: s.PROVISIONING_PATH,
}));
const parseConfig = (env: unknown) => {

View file

@ -203,6 +203,7 @@ export const volumesTable = sqliteTable(
{
id: int().primaryKey({ autoIncrement: true }),
shortId: text("short_id").$type<ShortId>().notNull().unique(),
provisioningId: text("provisioning_id"),
name: text().notNull(),
type: text().$type<BackendType>().notNull(),
status: text().$type<BackendStatus>().notNull().default("unmounted"),
@ -223,7 +224,10 @@ export const volumesTable = sqliteTable(
.notNull()
.references(() => organization.id, { onDelete: "cascade" }),
},
(table) => [unique().on(table.name, table.organizationId)],
(table) => [
unique().on(table.name, table.organizationId),
uniqueIndex("volumes_table_org_provisioning_id_uidx").on(table.organizationId, table.provisioningId),
],
);
export type Volume = typeof volumesTable.$inferSelect;
export type VolumeInsert = typeof volumesTable.$inferInsert;
@ -231,35 +235,42 @@ export type VolumeInsert = typeof volumesTable.$inferInsert;
/**
* Repositories Table
*/
export const repositoriesTable = sqliteTable("repositories_table", {
id: text().primaryKey(),
shortId: text("short_id").$type<ShortId>().notNull().unique(),
name: text().notNull(),
type: text().$type<RepositoryBackend>().notNull(),
config: text("config", { mode: "json" }).$type<RepositoryConfig>().notNull(),
compressionMode: text("compression_mode").$type<CompressionMode>().default("auto"),
status: text().$type<RepositoryStatus>().default("unknown"),
lastChecked: int("last_checked", { mode: "number" }),
lastError: text("last_error"),
doctorResult: text("doctor_result", { mode: "json" }).$type<DoctorResult>(),
stats: text("stats", { mode: "json" }).$type<ResticStatsDto | null>(),
statsUpdatedAt: int("stats_updated_at", { mode: "number" }),
uploadLimitEnabled: int("upload_limit_enabled", { mode: "boolean" }).notNull().default(false),
uploadLimitValue: real("upload_limit_value").notNull().default(1),
uploadLimitUnit: text("upload_limit_unit").$type<BandwidthUnit>().notNull().default("Mbps"),
downloadLimitEnabled: int("download_limit_enabled", { mode: "boolean" }).notNull().default(false),
downloadLimitValue: real("download_limit_value").notNull().default(1),
downloadLimitUnit: text("download_limit_unit").$type<BandwidthUnit>().notNull().default("Mbps"),
createdAt: int("created_at", { mode: "number" })
.notNull()
.default(sql`(unixepoch() * 1000)`),
updatedAt: int("updated_at", { mode: "number" })
.notNull()
.default(sql`(unixepoch() * 1000)`),
organizationId: text("organization_id")
.notNull()
.references(() => organization.id, { onDelete: "cascade" }),
});
export const repositoriesTable = sqliteTable(
"repositories_table",
{
id: text().primaryKey(),
shortId: text("short_id").$type<ShortId>().notNull().unique(),
provisioningId: text("provisioning_id"),
name: text().notNull(),
type: text().$type<RepositoryBackend>().notNull(),
config: text("config", { mode: "json" }).$type<RepositoryConfig>().notNull(),
compressionMode: text("compression_mode").$type<CompressionMode>().default("auto"),
status: text().$type<RepositoryStatus>().default("unknown"),
lastChecked: int("last_checked", { mode: "number" }),
lastError: text("last_error"),
doctorResult: text("doctor_result", { mode: "json" }).$type<DoctorResult>(),
stats: text("stats", { mode: "json" }).$type<ResticStatsDto | null>(),
statsUpdatedAt: int("stats_updated_at", { mode: "number" }),
uploadLimitEnabled: int("upload_limit_enabled", { mode: "boolean" }).notNull().default(false),
uploadLimitValue: real("upload_limit_value").notNull().default(1),
uploadLimitUnit: text("upload_limit_unit").$type<BandwidthUnit>().notNull().default("Mbps"),
downloadLimitEnabled: int("download_limit_enabled", { mode: "boolean" }).notNull().default(false),
downloadLimitValue: real("download_limit_value").notNull().default(1),
downloadLimitUnit: text("download_limit_unit").$type<BandwidthUnit>().notNull().default("Mbps"),
createdAt: int("created_at", { mode: "number" })
.notNull()
.default(sql`(unixepoch() * 1000)`),
updatedAt: int("updated_at", { mode: "number" })
.notNull()
.default(sql`(unixepoch() * 1000)`),
organizationId: text("organization_id")
.notNull()
.references(() => organization.id, { onDelete: "cascade" }),
},
(table) => [
uniqueIndex("repositories_table_org_provisioning_id_uidx").on(table.organizationId, table.provisioningId),
],
);
export type Repository = typeof repositoriesTable.$inferSelect;
export type RepositoryInsert = typeof repositoriesTable.$inferInsert;

View file

@ -14,6 +14,9 @@ import { VolumeAutoRemountJob } from "~/server/jobs/auto-remount";
import { cache } from "~/server/utils/cache";
import { withContext } from "~/server/core/request-context";
import { backupsService } from "../backups/backups.service";
import { config } from "~/server/core/config";
import { syncProvisionedResources } from "../provisioning/provisioning";
import { toMessage } from "~/server/utils/errors";
const ensureLatestConfigurationSchema = async () => {
const volumes = await db.query.volumesTable.findMany({});
@ -53,6 +56,10 @@ export const startup = async () => {
await Scheduler.start();
await Scheduler.clear();
await syncProvisionedResources(config.provisioningPath).catch((error) => {
logger.error(`Provisioning sync failed: ${toMessage(error)}`);
});
await ensureLatestConfigurationSchema();
const { deletedSchedules } = await backupsService.cleanupOrphanedSchedules().catch((err) => {

View file

@ -0,0 +1,391 @@
import { afterEach, describe, expect, mock, spyOn, test } from "bun:test";
import * as fs from "node:fs/promises";
import * as os from "node:os";
import * as path from "node:path";
import { db } from "~/server/db/db";
import { backupSchedulesTable, repositoriesTable } from "~/server/db/schema";
import { restic } from "~/server/core/restic";
import { createTestSession } from "~/test/helpers/auth";
import { generateShortId } from "~/server/utils/id";
import { provisionedResourcesSchema, readProvisionedResourcesFile, syncProvisionedResources } from "./provisioning";
describe("provisioning", () => {
afterEach(() => {
mock.restore();
});
test("rejects duplicate ids for the same organization", () => {
expect(() =>
provisionedResourcesSchema.parse({
version: 1,
repositories: [
{
id: "shared-id",
organizationId: "acme",
name: "Repository one",
backend: "local",
config: {
backend: "local",
path: "/tmp/one",
},
},
{
id: "shared-id",
organizationId: "acme",
name: "Repository two",
backend: "local",
config: {
backend: "local",
path: "/tmp/two",
},
},
],
volumes: [],
}),
).toThrow("Duplicate provisioned repository id for organization acme: shared-id");
});
test("syncs provisioned repositories and volumes into the database", async () => {
const { organizationId } = await createTestSession();
process.env.ZEROBYTE_PROVISIONED_ACCESS_KEY = "access-key-from-env";
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "zerobyte-provisioning-"));
const provisioningPath = path.join(tempDir, "provisioning.json");
await fs.writeFile(
provisioningPath,
JSON.stringify({
version: 1,
repositories: [
{
id: "aws-prod",
organizationId,
name: "AWS Production",
backend: "s3",
compressionMode: "auto",
config: {
backend: "s3",
endpoint: "https://s3.amazonaws.com",
bucket: "company-backups",
accessKeyId: "env://ZEROBYTE_PROVISIONED_ACCESS_KEY",
secretAccessKey: "plain-secret-key",
},
},
],
volumes: [
{
id: "shared-directory",
organizationId,
name: "Shared Directory",
backend: "directory",
autoRemount: true,
config: {
backend: "directory",
path: tempDir,
},
},
],
}),
);
const parsed = await readProvisionedResourcesFile(provisioningPath);
expect(parsed.repositories).toHaveLength(1);
await syncProvisionedResources(provisioningPath);
const repositories = await db.query.repositoriesTable.findMany({ where: { organizationId } });
const volumes = await db.query.volumesTable.findMany({ where: { organizationId } });
const repository = repositories.find((item) => item.name === "AWS Production");
const volume = volumes.find((item) => item.name === "Shared Directory");
expect(repository).toBeTruthy();
if (!repository || repository.config.backend !== "s3") {
throw new Error("Expected provisioned repository to be stored as an s3 repository");
}
expect(repository.config.accessKeyId).toBe("access-key-from-env");
expect(repository.provisioningId).toBeDefined();
expect(volume).toBeTruthy();
expect(volume?.status).toBe("mounted");
expect(volume?.provisioningId).toBeDefined();
});
test("removes managed resources when delete is set", async () => {
const { organizationId } = await createTestSession();
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "zerobyte-provisioning-"));
const provisioningPath = path.join(tempDir, "provisioning.json");
await fs.writeFile(
provisioningPath,
JSON.stringify({
version: 1,
repositories: [
{
id: "repo-to-remove",
organizationId,
name: "Repo to remove",
backend: "local",
config: {
backend: "local",
path: tempDir,
isExistingRepository: true,
},
},
],
volumes: [
{
id: "volume-to-remove",
organizationId,
name: "Volume to remove",
backend: "directory",
config: {
backend: "directory",
path: tempDir,
},
},
],
}),
);
await syncProvisionedResources(provisioningPath);
await fs.writeFile(
provisioningPath,
JSON.stringify({
version: 1,
repositories: [
{
id: "repo-to-remove",
organizationId,
name: "Repo to remove",
backend: "local",
delete: true,
config: {
backend: "local",
path: tempDir,
isExistingRepository: true,
},
},
],
volumes: [
{
id: "volume-to-remove",
organizationId,
name: "Volume to remove",
backend: "directory",
delete: true,
config: {
backend: "directory",
path: tempDir,
},
},
],
}),
);
await syncProvisionedResources(provisioningPath);
const repositories = await db.query.repositoriesTable.findMany({ where: { organizationId } });
const volumes = await db.query.volumesTable.findMany({ where: { organizationId } });
expect(repositories.filter((item) => item.provisioningId)).toHaveLength(0);
expect(volumes.filter((item) => item.provisioningId)).toHaveLength(0);
});
test("renaming a provisioned volume updates it in place", async () => {
const { organizationId } = await createTestSession();
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "zerobyte-provisioning-"));
const provisioningPath = path.join(tempDir, "provisioning.json");
const provisionedVolumeId = "shared-directory";
await db.insert(repositoriesTable).values({
id: crypto.randomUUID(),
shortId: generateShortId(),
name: "Schedule Repository",
type: "local",
config: {
backend: "local",
path: tempDir,
isExistingRepository: true,
},
organizationId,
});
await fs.writeFile(
provisioningPath,
JSON.stringify({
version: 1,
repositories: [],
volumes: [
{
id: provisionedVolumeId,
organizationId,
name: "Shared Directory",
backend: "directory",
config: {
backend: "directory",
path: tempDir,
},
},
],
}),
);
await syncProvisionedResources(provisioningPath);
const initialVolume = await db.query.volumesTable.findFirst({
where: { organizationId, provisioningId: `provisioned:${organizationId}:${provisionedVolumeId}` },
});
expect(initialVolume).toBeTruthy();
if (!initialVolume) {
throw new Error("Expected initial provisioned volume to exist");
}
const repository = await db.query.repositoriesTable.findFirst({ where: { organizationId } });
expect(repository).toBeTruthy();
if (!repository) {
throw new Error("Expected repository to exist");
}
const [schedule] = await db
.insert(backupSchedulesTable)
.values({
shortId: generateShortId(),
name: "Daily Backup",
volumeId: initialVolume.id,
repositoryId: repository.id,
cronExpression: "0 0 * * *",
organizationId,
})
.returning();
await fs.writeFile(
provisioningPath,
JSON.stringify({
version: 1,
repositories: [],
volumes: [
{
id: provisionedVolumeId,
organizationId,
name: "Shared Directory Renamed",
backend: "directory",
config: {
backend: "directory",
path: tempDir,
},
},
],
}),
);
await syncProvisionedResources(provisioningPath);
const volumes = await db.query.volumesTable.findMany({ where: { organizationId } });
expect(volumes).toHaveLength(1);
expect(volumes[0]?.id).toBe(initialVolume.id);
expect(volumes[0]?.name).toBe("Shared Directory Renamed");
expect(volumes[0]?.provisioningId).toBe(`provisioned:${organizationId}:${provisionedVolumeId}`);
const persistedSchedule = await db.query.backupSchedulesTable.findFirst({ where: { id: schedule.id } });
expect(persistedSchedule).toBeTruthy();
expect(persistedSchedule?.volumeId).toBe(initialVolume.id);
});
test("does not partially sync resources when resolving a provisioned secret fails", async () => {
const { organizationId } = await createTestSession();
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "zerobyte-provisioning-"));
const provisioningPath = path.join(tempDir, "provisioning.json");
await fs.writeFile(
provisioningPath,
JSON.stringify({
version: 1,
repositories: [
{
id: "broken-secret-repo",
organizationId,
name: "Broken Secret Repo",
backend: "rest",
config: {
backend: "rest",
url: "https://rest.example.test",
password: "file://ZEROBYTE_WEBDAV_PASSWORD",
},
},
],
volumes: [
{
id: "shared-directory",
organizationId,
name: "Shared Directory",
backend: "directory",
config: {
backend: "directory",
path: tempDir,
},
},
],
}),
);
await expect(syncProvisionedResources(provisioningPath)).rejects.toThrow(
"Provisioned secret file not found: /run/secrets/ZEROBYTE_WEBDAV_PASSWORD",
);
const repositories = await db.query.repositoriesTable.findMany({ where: { organizationId } });
const volumes = await db.query.volumesTable.findMany({ where: { organizationId } });
expect(repositories.filter((repository) => repository.provisioningId)).toHaveLength(0);
expect(volumes.filter((volume) => volume.provisioningId)).toHaveLength(0);
});
test("initializes a non-existing provisioned repository on first sync", async () => {
const { organizationId } = await createTestSession();
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "zerobyte-provisioning-"));
const provisioningPath = path.join(tempDir, "provisioning.json");
const initMock = mock(() => Promise.resolve({ success: true, error: null }));
spyOn(restic, "init").mockImplementation(initMock);
await fs.writeFile(
provisioningPath,
JSON.stringify({
version: 1,
repositories: [
{
id: "new-local-repo",
organizationId,
name: "New Local Repo",
backend: "local",
config: {
backend: "local",
path: tempDir,
isExistingRepository: false,
},
},
],
volumes: [],
}),
);
await syncProvisionedResources(provisioningPath);
expect(initMock).toHaveBeenCalledTimes(1);
const repository = await db.query.repositoriesTable.findFirst({
where: {
organizationId,
provisioningId: `provisioned:${organizationId}:new-local-repo`,
},
});
expect(repository).toBeTruthy();
expect(repository?.status).toBe("healthy");
expect(repository?.lastChecked).toEqual(expect.any(Number));
expect(repository?.lastError).toBeNull();
});
});

View file

@ -0,0 +1,262 @@
import fs from "node:fs/promises";
import path from "node:path";
import { eq } from "drizzle-orm";
import {
COMPRESSION_MODES,
REPOSITORY_BACKENDS,
repositoryConfigSchema,
type RepositoryConfig,
} from "@zerobyte/core/restic";
import { logger } from "@zerobyte/core/node";
import { z } from "zod";
import { config as appConfig } from "~/server/core/config";
import { restic } from "~/server/core/restic";
import { db } from "~/server/db/db";
import { repositoriesTable, volumesTable } from "~/server/db/schema";
import { mapRepositoryConfigSecrets } from "~/server/modules/repositories/repository-config-secrets";
import { mapVolumeConfigSecrets } from "~/server/modules/volumes/volume-config-secrets";
import { BACKEND_TYPES, volumeConfigSchema, type BackendConfig } from "~/schemas/volumes";
import { cryptoUtils } from "~/server/utils/crypto";
import { toMessage } from "~/server/utils/errors";
import { generateShortId } from "~/server/utils/id";
const envSecretPrefix = "env://";
const fileSecretPrefix = "file://";
const provisionedRepositorySchema = z.object({
id: z.string().min(1),
organizationId: z.string().min(1),
name: z.string().min(1),
compressionMode: z.enum(COMPRESSION_MODES).optional(),
config: repositoryConfigSchema,
backend: z.enum(REPOSITORY_BACKENDS),
delete: z.boolean().default(false),
});
type ProvisionedRepository = z.infer<typeof provisionedRepositorySchema>;
const provisionedVolumeSchema = z.object({
id: z.string().min(1),
organizationId: z.string().min(1),
name: z.string().min(1),
autoRemount: z.boolean().default(true),
config: volumeConfigSchema,
delete: z.boolean().default(false),
backend: z.enum(BACKEND_TYPES),
});
type ProvisionedVolume = z.infer<typeof provisionedVolumeSchema>;
export const provisionedResourcesSchema = z
.object({
version: z.literal(1).default(1),
repositories: z.array(provisionedRepositorySchema).default([]),
volumes: z.array(provisionedVolumeSchema).default([]),
})
.superRefine((value, ctx) => {
const repositoryIds = new Set<string>();
for (const repository of value.repositories) {
const key = `${repository.organizationId}:${repository.id}`;
if (repositoryIds.has(key)) {
ctx.addIssue({
code: "custom",
message: `Duplicate provisioned repository id for organization ${repository.organizationId}: ${repository.id}`,
path: ["repositories"],
});
}
repositoryIds.add(key);
}
const volumeIds = new Set<string>();
for (const volume of value.volumes) {
const key = `${volume.organizationId}:${volume.id}`;
if (volumeIds.has(key)) {
ctx.addIssue({
code: "custom",
message: `Duplicate provisioned volume id for organization ${volume.organizationId}: ${volume.id}`,
path: ["volumes"],
});
}
volumeIds.add(key);
}
});
type ProvisionedResources = z.infer<typeof provisionedResourcesSchema>;
export const readProvisionedResourcesFile = async (filePath: string): Promise<ProvisionedResources> => {
const content = await fs.readFile(filePath, "utf-8");
const parsed = JSON.parse(content) as unknown;
return provisionedResourcesSchema.parse(parsed);
};
const resolveProvisioningSecret = async (value: string): Promise<string> => {
if (!value) {
return value;
}
if (value.startsWith(envSecretPrefix)) {
const name = value.slice(envSecretPrefix.length);
if (!name) {
throw new Error("Provisioned env secret reference is missing a variable name");
}
const resolved = process.env[name];
if (resolved === undefined) {
throw new Error(`Environment variable not set: ${name}`);
}
return resolved;
}
if (value.startsWith(fileSecretPrefix)) {
const secretName = value.slice(fileSecretPrefix.length).replace(/^\/+/, "");
if (!secretName) {
throw new Error("Provisioned file secret reference is missing a secret name");
}
if (secretName.includes("/") || secretName.includes("\\") || secretName.includes("\0")) {
throw new Error("Provisioned file secret reference must be a single path segment");
}
const secretPath = path.join("/run/secrets", secretName);
const content = await fs.readFile(secretPath, "utf-8").catch(() => {
throw new Error(`Provisioned secret file not found: ${secretPath}`);
});
return content.trimEnd();
}
return value;
};
const sealProvisionedSecret = async (value: string): Promise<string> => {
const resolved = await resolveProvisioningSecret(value);
return cryptoUtils.sealSecret(resolved);
};
const encryptProvisionedRepositoryConfig = async (config: RepositoryConfig): Promise<RepositoryConfig> => {
return await mapRepositoryConfigSecrets(config, sealProvisionedSecret);
};
const encryptProvisionedVolumeConfig = async (config: BackendConfig): Promise<BackendConfig> => {
return await mapVolumeConfigSecrets(config, sealProvisionedSecret);
};
const syncProvisionedRepositories = async (repositories: ProvisionedRepository[]) => {
const existingRepositories = await db.query.repositoriesTable.findMany({
where: { AND: [{ provisioningId: { isNotNull: true } }] },
});
for (const repository of repositories) {
const provisioningId = `provisioned:${repository.organizationId}:${repository.id}`;
if (repository.delete) {
await db.delete(repositoriesTable).where(eq(repositoriesTable.provisioningId, provisioningId));
continue;
}
const existing = existingRepositories.find((r) => r.provisioningId === provisioningId);
const encryptedConfig = await encryptProvisionedRepositoryConfig(repository.config);
if (!existing) {
const id = Bun.randomUUIDv7();
await db.insert(repositoriesTable).values({
id,
provisioningId: provisioningId,
shortId: generateShortId(),
name: repository.name,
type: repository.backend,
config: encryptedConfig,
compressionMode: repository.compressionMode,
status: "unknown",
organizationId: repository.organizationId,
});
if (!repository.config.isExistingRepository) {
const result = await restic
.init(encryptedConfig, repository.organizationId, { timeoutMs: appConfig.serverIdleTimeout * 1000 })
.catch((error) => ({ success: false, error }));
await db
.update(repositoriesTable)
.set({
status: result.error ? "error" : "healthy",
lastChecked: Date.now(),
lastError: result.error ? toMessage(result.error) : null,
updatedAt: Date.now(),
})
.where(eq(repositoriesTable.id, id));
if (result.error) {
logger.error(`Provisioned repository ${repository.name} failed to initialize: ${toMessage(result.error)}`);
}
}
continue;
}
const updatePayload = {
name: repository.name,
type: repository.backend,
config: encryptedConfig,
compressionMode: repository.compressionMode,
organizationId: repository.organizationId,
updatedAt: Date.now(),
};
await db.update(repositoriesTable).set(updatePayload).where(eq(repositoriesTable.id, existing.id));
}
};
const syncProvisionedVolumes = async (volumes: ProvisionedVolume[]) => {
const existingVolumes = await db.query.volumesTable.findMany({
where: { AND: [{ provisioningId: { isNotNull: true } }] },
});
for (const volume of volumes) {
const provisioningId = `provisioned:${volume.organizationId}:${volume.id}`;
if (volume.delete) {
await db.delete(volumesTable).where(eq(volumesTable.provisioningId, provisioningId));
continue;
}
const existing = existingVolumes.find((v) => v.provisioningId === provisioningId);
if (!existing) {
await db.insert(volumesTable).values({
shortId: generateShortId(),
provisioningId: provisioningId,
name: volume.name,
type: volume.backend,
config: await encryptProvisionedVolumeConfig(volume.config),
autoRemount: volume.autoRemount,
status: volume.autoRemount ? "mounted" : "unmounted",
organizationId: volume.organizationId,
});
continue;
}
const updatePayload = {
name: volume.name,
type: volume.backend,
config: await encryptProvisionedVolumeConfig(volume.config),
autoRemount: volume.autoRemount,
organizationId: volume.organizationId,
updatedAt: Date.now(),
};
await db.update(volumesTable).set(updatePayload).where(eq(volumesTable.id, existing.id));
}
};
export const syncProvisionedResources = async (filePath?: string) => {
if (!filePath) {
return;
}
const resources = await readProvisionedResourcesFile(filePath);
await syncProvisionedRepositories(resources.repositories);
await syncProvisionedVolumes(resources.volumes);
logger.info(
`Synchronized ${resources.repositories.length} provisioned repositor${resources.repositories.length === 1 ? "y" : "ies"} and ${resources.volumes.length} provisioned volume${resources.volumes.length === 1 ? "" : "s"}`,
);
};

View file

@ -40,6 +40,29 @@ const createRepositoryRecord = async (organizationId: string) => {
return repository;
};
const createManagedRepositoryRecord = async (organizationId: string) => {
const [repository] = await db
.insert(repositoriesTable)
.values({
id: Bun.randomUUIDv7(),
provisioningId: `provisioned:${crypto.randomUUID()}`,
shortId: generateShortId(),
name: `Managed-${crypto.randomUUID()}`,
type: "local",
config: {
backend: "local",
path: `/tmp/repository-${crypto.randomUUID()}`,
isExistingRepository: true,
},
compressionMode: "off",
status: "healthy",
organizationId,
})
.returning();
return repository;
};
describe("repositories security", () => {
test("should return 401 if no session cookie is provided", async () => {
const res = await app.request("/api/v1/repositories");
@ -175,6 +198,30 @@ describe("repositories security", () => {
expect(res.status).toBe(400);
});
test("should accept env:// values as plain secrets on create", async () => {
const { headers } = await createTestSession();
const res = await app.request("/api/v1/repositories", {
method: "POST",
headers: {
...headers,
"Content-Type": "application/json",
},
body: JSON.stringify({
name: "S3 repo",
compressionMode: "auto",
config: {
backend: "s3",
endpoint: "https://s3.amazonaws.com",
bucket: "bucket-name",
accessKeyId: "access-key",
secretAccessKey: "env://ZEROBYTE_REPOSITORY_SECRET",
},
}),
});
expect(res.status).not.toBe(400);
});
});
});
@ -338,4 +385,47 @@ describe("repositories updates", () => {
}
});
});
test("GET marks provisioned repositories as managed", async () => {
const { headers, organizationId } = await createTestSession();
const repository = await createManagedRepositoryRecord(organizationId);
const res = await app.request(`/api/v1/repositories/${repository.shortId}`, { headers });
expect(res.status).toBe(200);
const body = await res.json();
expect(body.provisioningId).toBeDefined();
});
test("PATCH allows updates for managed repositories", async () => {
const { headers, organizationId } = await createTestSession();
const repository = await createManagedRepositoryRecord(organizationId);
const res = await app.request(`/api/v1/repositories/${repository.shortId}`, {
method: "PATCH",
headers: {
...headers,
"Content-Type": "application/json",
},
body: JSON.stringify({
name: "Updated repository",
}),
});
expect(res.status).toBe(200);
const body = await res.json();
expect(body.name).toBe("Updated repository");
});
test("DELETE allows managed repositories", async () => {
const { headers, organizationId } = await createTestSession();
const repository = await createManagedRepositoryRecord(organizationId);
const res = await app.request(`/api/v1/repositories/${repository.shortId}`, {
method: "DELETE",
headers,
});
expect(res.status).toBe(200);
});
});

View file

@ -14,6 +14,7 @@ import {
export const repositorySchema = z.object({
id: z.string(),
shortId: z.string(),
provisioningId: z.string().nullable(),
name: z.string(),
type: z.enum(REPOSITORY_BACKENDS),
config: repositoryConfigSchema,

View file

@ -1,4 +1,3 @@
import crypto from "node:crypto";
import nodePath from "node:path";
import { and, eq } from "drizzle-orm";
import { BadRequestError, ConflictError, InternalServerError, NotFoundError } from "http-errors-enhanced";
@ -18,7 +17,6 @@ import { repoMutex } from "../../core/repository-mutex";
import { db } from "../../db/db";
import { repositoriesTable, type Repository } from "../../db/schema";
import { cache, cacheKeys } from "../../utils/cache";
import { cryptoUtils } from "../../utils/crypto";
import { toMessage } from "../../utils/errors";
import { generateShortId } from "../../utils/id";
import { addCommonArgs, buildEnv, buildRepoUrl, cleanupTemporaryKeys } from "@zerobyte/core/restic/server";
@ -30,6 +28,7 @@ import { findCommonAncestor } from "@zerobyte/core/utils";
import { prepareSnapshotDump } from "./helpers/dump";
import { executeDoctor } from "./helpers/doctor";
import type { ShortId } from "~/server/utils/branded";
import { decryptRepositoryConfig, encryptRepositoryConfig } from "./repository-config-secrets";
const runningDoctors = new Map<string, AbortController>();
@ -57,94 +56,16 @@ const listRepositories = async () => {
return repositories;
};
const encryptConfig = async (config: RepositoryConfig): Promise<RepositoryConfig> => {
const encryptedConfig: Record<string, unknown> = { ...config };
if (config.customPassword) {
encryptedConfig.customPassword = await cryptoUtils.sealSecret(config.customPassword);
}
if (config.cacert) {
encryptedConfig.cacert = await cryptoUtils.sealSecret(config.cacert);
}
switch (config.backend) {
case "s3":
case "r2":
encryptedConfig.accessKeyId = await cryptoUtils.sealSecret(config.accessKeyId);
encryptedConfig.secretAccessKey = await cryptoUtils.sealSecret(config.secretAccessKey);
break;
case "gcs":
encryptedConfig.credentialsJson = await cryptoUtils.sealSecret(config.credentialsJson);
break;
case "azure":
encryptedConfig.accountKey = await cryptoUtils.sealSecret(config.accountKey);
break;
case "rest":
if (config.username) {
encryptedConfig.username = await cryptoUtils.sealSecret(config.username);
}
if (config.password) {
encryptedConfig.password = await cryptoUtils.sealSecret(config.password);
}
break;
case "sftp":
encryptedConfig.privateKey = await cryptoUtils.sealSecret(config.privateKey);
break;
}
return encryptedConfig as RepositoryConfig;
};
const decryptConfig = async (config: RepositoryConfig): Promise<RepositoryConfig> => {
const decryptedConfig: Record<string, unknown> = { ...config };
if (config.customPassword) {
decryptedConfig.customPassword = await cryptoUtils.resolveSecret(config.customPassword);
}
if (config.cacert) {
decryptedConfig.cacert = await cryptoUtils.resolveSecret(config.cacert);
}
switch (config.backend) {
case "s3":
case "r2":
decryptedConfig.accessKeyId = await cryptoUtils.resolveSecret(config.accessKeyId);
decryptedConfig.secretAccessKey = await cryptoUtils.resolveSecret(config.secretAccessKey);
break;
case "gcs":
decryptedConfig.credentialsJson = await cryptoUtils.resolveSecret(config.credentialsJson);
break;
case "azure":
decryptedConfig.accountKey = await cryptoUtils.resolveSecret(config.accountKey);
break;
case "rest":
if (config.username) {
decryptedConfig.username = await cryptoUtils.resolveSecret(config.username);
}
if (config.password) {
decryptedConfig.password = await cryptoUtils.resolveSecret(config.password);
}
break;
case "sftp":
decryptedConfig.privateKey = await cryptoUtils.resolveSecret(config.privateKey);
break;
}
return decryptedConfig as RepositoryConfig;
};
const createRepository = async (name: string, config: RepositoryConfig, compressionMode?: CompressionMode) => {
const organizationId = getOrganizationId();
const id = crypto.randomUUID();
const id = Bun.randomUUIDv7();
const shortId = generateShortId();
if (config.backend === "local" && !config.isExistingRepository) {
config.path = `${config.path}/${shortId}`;
}
const encryptedConfig = await encryptConfig(config);
const encryptedConfig = await encryptRepositoryConfig(config);
const [created] = await db
.insert(repositoriesTable)
@ -761,9 +682,9 @@ const updateRepository = async (shortId: ShortId, updates: UpdateRepositoryBody)
parsedConfig = nextConfig;
}
const decryptedExisting = await decryptConfig(existingConfig);
const decryptedExisting = await decryptRepositoryConfig(existingConfig);
const configChanged = updates.config && JSON.stringify(decryptedExisting) !== JSON.stringify(parsedConfig);
const encryptedConfig = updates.config ? await encryptConfig(parsedConfig) : existingConfig;
const encryptedConfig = updates.config ? await encryptRepositoryConfig(parsedConfig) : existingConfig;
const updatedAt = Date.now();
const updatePayload = {

View file

@ -0,0 +1,79 @@
import type { RepositoryConfig } from "@zerobyte/core/restic";
import { cryptoUtils } from "~/server/utils/crypto";
type SecretTransformer = (value: string) => Promise<string>;
const transformOptionalSecret = async (
value: string | undefined,
transformSecret: SecretTransformer,
): Promise<string | undefined> => {
if (!value) {
return value;
}
return await transformSecret(value);
};
export const mapRepositoryConfigSecrets = async (
config: RepositoryConfig,
transformSecret: SecretTransformer,
): Promise<RepositoryConfig> => {
const customPassword = await transformOptionalSecret(config.customPassword, transformSecret);
const cacert = await transformOptionalSecret(config.cacert, transformSecret);
switch (config.backend) {
case "s3":
case "r2":
return {
...config,
customPassword,
cacert,
accessKeyId: await transformSecret(config.accessKeyId),
secretAccessKey: await transformSecret(config.secretAccessKey),
};
case "gcs":
return {
...config,
customPassword,
cacert,
credentialsJson: await transformSecret(config.credentialsJson),
};
case "azure":
return {
...config,
customPassword,
cacert,
accountKey: await transformSecret(config.accountKey),
};
case "rest":
return {
...config,
customPassword,
cacert,
username: await transformOptionalSecret(config.username, transformSecret),
password: await transformOptionalSecret(config.password, transformSecret),
};
case "sftp":
return {
...config,
customPassword,
cacert,
privateKey: await transformSecret(config.privateKey),
};
case "local":
case "rclone":
return {
...config,
customPassword,
cacert,
};
}
};
export const encryptRepositoryConfig = async (config: RepositoryConfig): Promise<RepositoryConfig> => {
return await mapRepositoryConfigSecrets(config, cryptoUtils.sealSecret);
};
export const decryptRepositoryConfig = async (config: RepositoryConfig): Promise<RepositoryConfig> => {
return await mapRepositoryConfigSecrets(config, cryptoUtils.resolveSecret);
};

View file

@ -1,9 +1,33 @@
import { test, describe, expect } from "bun:test";
import { db } from "~/server/db/db";
import { volumesTable } from "~/server/db/schema";
import { createApp } from "~/server/app";
import { createTestSession, getAuthHeaders } from "~/test/helpers/auth";
import { generateShortId } from "~/server/utils/id";
const app = createApp();
const createManagedVolumeRecord = async (organizationId: string) => {
const [volume] = await db
.insert(volumesTable)
.values({
shortId: generateShortId(),
provisioningId: `provisioned:${organizationId}:${generateShortId()}`,
name: `Managed-${Date.now()}`,
type: "directory",
status: "mounted",
config: {
backend: "directory",
path: "/tmp",
},
autoRemount: true,
organizationId,
})
.returning();
return volume;
};
describe("volumes security", () => {
test("should return 401 if no session cookie is provided", async () => {
const res = await app.request("/api/v1/volumes");
@ -92,5 +116,48 @@ describe("volumes security", () => {
expect(res.status).toBe(400);
});
test("should mark provisioned volumes as managed", async () => {
const { headers, organizationId } = await createTestSession();
const volume = await createManagedVolumeRecord(organizationId);
const res = await app.request(`/api/v1/volumes/${volume.shortId}`, { headers });
expect(res.status).toBe(200);
const body = await res.json();
expect(body.volume.provisioningId).toBeDefined();
});
test("should allow updates for managed volumes", async () => {
const { headers, organizationId } = await createTestSession();
const volume = await createManagedVolumeRecord(organizationId);
const res = await app.request(`/api/v1/volumes/${volume.shortId}`, {
method: "PUT",
headers: {
...headers,
"Content-Type": "application/json",
},
body: JSON.stringify({
name: "Updated volume",
}),
});
expect(res.status).toBe(200);
const body = await res.json();
expect(body.name).toBe("Updated volume");
});
test("should allow deletion for managed volumes", async () => {
const { headers, organizationId } = await createTestSession();
const volume = await createManagedVolumeRecord(organizationId);
const res = await app.request(`/api/v1/volumes/${volume.shortId}`, {
method: "DELETE",
headers,
});
expect(res.status).toBe(200);
});
});
});

View file

@ -0,0 +1,47 @@
import type { BackendConfig } from "~/schemas/volumes";
import { cryptoUtils } from "~/server/utils/crypto";
type SecretTransformer = (value: string) => Promise<string>;
const transformOptionalSecret = async (
value: string | undefined,
transformSecret: SecretTransformer,
): Promise<string | undefined> => {
if (!value) {
return value;
}
return await transformSecret(value);
};
export const mapVolumeConfigSecrets = async (
config: BackendConfig,
transformSecret: SecretTransformer,
): Promise<BackendConfig> => {
switch (config.backend) {
case "smb":
return {
...config,
password: await transformOptionalSecret(config.password, transformSecret),
};
case "webdav":
return {
...config,
password: await transformOptionalSecret(config.password, transformSecret),
};
case "sftp":
return {
...config,
password: await transformOptionalSecret(config.password, transformSecret),
privateKey: await transformOptionalSecret(config.privateKey, transformSecret),
};
case "nfs":
case "directory":
case "rclone":
return config;
}
};
export const encryptVolumeConfig = async (config: BackendConfig): Promise<BackendConfig> => {
return await mapVolumeConfigSecrets(config, cryptoUtils.sealSecret);
};

View file

@ -5,6 +5,7 @@ import { BACKEND_STATUS, BACKEND_TYPES, volumeConfigSchema } from "~/schemas/vol
export const volumeSchema = z.object({
id: z.number(),
shortId: z.string(),
provisioningId: z.string().nullable(),
name: z.string(),
type: z.enum(BACKEND_TYPES),
status: z.enum(BACKEND_STATUS),

View file

@ -5,7 +5,6 @@ import { and, eq } from "drizzle-orm";
import { BadRequestError, InternalServerError, NotFoundError } from "http-errors-enhanced";
import { db } from "../../db/db";
import { volumesTable } from "../../db/schema";
import { cryptoUtils } from "../../utils/crypto";
import { toMessage } from "../../utils/errors";
import { generateShortId } from "../../utils/id";
import { getStatFs, type StatFs } from "../../utils/mountinfo";
@ -19,29 +18,7 @@ import { volumeConfigSchema, type BackendConfig } from "~/schemas/volumes";
import { getOrganizationId } from "~/server/core/request-context";
import { isNodeJSErrnoException } from "~/server/utils/fs";
import { asShortId, type ShortId } from "~/server/utils/branded";
async function encryptSensitiveFields(config: BackendConfig): Promise<BackendConfig> {
switch (config.backend) {
case "smb":
return {
...config,
password: config.password ? await cryptoUtils.sealSecret(config.password) : undefined,
};
case "webdav":
return {
...config,
password: config.password ? await cryptoUtils.sealSecret(config.password) : undefined,
};
case "sftp":
return {
...config,
password: config.password ? await cryptoUtils.sealSecret(config.password) : undefined,
privateKey: config.privateKey ? await cryptoUtils.sealSecret(config.privateKey) : undefined,
};
default:
return config;
}
}
import { encryptVolumeConfig } from "./volume-config-secrets";
const listVolumes = async () => {
const organizationId = getOrganizationId();
@ -70,7 +47,7 @@ const createVolume = async (name: string, backendConfig: BackendConfig) => {
}
const shortId = generateShortId();
const encryptedConfig = await encryptSensitiveFields(backendConfig);
const encryptedConfig = await encryptVolumeConfig(backendConfig);
const [created] = await db
.insert(volumesTable)
@ -206,7 +183,7 @@ const updateVolume = async (shortId: ShortId, volumeData: UpdateVolumeBody) => {
}
const newConfig = newConfigResult.data;
const encryptedConfig = await encryptSensitiveFields(newConfig);
const encryptedConfig = await encryptVolumeConfig(newConfig);
const [updated] = await db
.update(volumesTable)
@ -240,19 +217,21 @@ const updateVolume = async (shortId: ShortId, volumeData: UpdateVolumeBody) => {
const testConnection = async (backendConfig: BackendConfig) => {
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "zerobyte-test-"));
const encryptedConfig = await encryptVolumeConfig(backendConfig);
const mockVolume = {
id: 0,
shortId: asShortId("test"),
name: "test-connection",
path: tempDir,
config: backendConfig,
config: encryptedConfig,
createdAt: Date.now(),
updatedAt: Date.now(),
lastHealthCheck: Date.now(),
type: backendConfig.backend,
type: encryptedConfig.backend,
status: "unmounted" as const,
lastError: null,
provisioningId: null,
autoRemount: true,
organizationId: "test-org",
};

View file

@ -1,8 +1,5 @@
import crypto from "node:crypto";
import fs from "node:fs/promises";
import path from "node:path";
import { config } from "../core/config";
import { isNodeJSErrnoException } from "./fs";
import { promisify } from "node:util";
const hkdf = promisify(crypto.hkdf);
@ -11,9 +8,6 @@ const algorithm = "aes-256-gcm" as const;
const keyLength = 32;
const encryptionPrefix = "encv1:";
const envSecretPrefix = "env://";
const fileSecretPrefix = "file://";
/**
* Checks if a given string is encrypted by looking for the encryption prefix.
*/
@ -21,68 +15,6 @@ const isEncrypted = (val?: string): boolean => {
return typeof val === "string" && val.startsWith(encryptionPrefix);
};
/**
* Checks if a string looks like a supported secret reference.
* - env://VAR_NAME -> reads process.env.VAR_NAME
* - file://name -> reads a file /run/secrets/name
*/
const isSecretReference = (val?: string): boolean => {
return typeof val === "string" && (val.startsWith(envSecretPrefix) || val.startsWith(fileSecretPrefix));
};
/**
* Resolves an environment variable secret reference.
*/
const resolveEnvSecret = (ref: string): string => {
const name = ref.slice(envSecretPrefix.length);
if (!name) {
throw new Error("env:// reference is missing variable name");
}
const value = process.env[name];
if (value === undefined) {
throw new Error(`Environment variable not set: ${name}`);
}
return value;
};
/**
* Resolves a file-based secret reference.
* Reads the secret from /run/secrets/{name}
*/
const resolveFileSecret = async (ref: string): Promise<string> => {
const secretName = ref.slice(fileSecretPrefix.length);
if (!secretName) {
throw new Error("file:// reference is missing secret name");
}
const normalizedName = secretName.replace(/^\/+/, "");
if (!normalizedName) {
throw new Error("file:// reference is missing secret name");
}
if (normalizedName.includes("\0") || normalizedName.includes("/") || normalizedName.includes("\\")) {
throw new Error("Invalid secret reference: secret name must be a single path segment");
}
const resolvedPath = path.join("/run/secrets", normalizedName);
try {
const content = await fs.readFile(resolvedPath, "utf-8");
return content.trimEnd();
} catch (error) {
if (isNodeJSErrnoException(error)) {
if (error.code === "ENOENT") {
throw new Error(`Secret file not found: ${resolvedPath}`);
}
if (error.code === "EACCES") {
throw new Error(`Permission denied reading secret file: ${resolvedPath}`);
}
}
throw new Error(`Failed to read secret file ${resolvedPath}: ${(error as Error).message}`);
}
};
/**
* Given a string, encrypts it using a randomly generated salt and the APP_SECRET.
* Returns the input unchanged if it's empty or already encrypted.
@ -137,51 +69,34 @@ const decrypt = async (encryptedData: string) => {
/**
* Resolves secret references and encrypted database values.
*
* - encv1:... -> decrypt
* - env://VAR -> read process.env.VAR
* - file://name -> read /run/secrets/name
* - otherwise returns value unchanged
*/
const resolveSecret = async (value: string): Promise<string> => {
if (!value) {
return value;
}
if (isEncrypted(value)) {
return decrypt(value);
}
if (value.startsWith(envSecretPrefix)) {
return resolveEnvSecret(value);
}
if (value.startsWith(fileSecretPrefix)) {
return resolveFileSecret(value);
}
return value;
};
/**
* Prepares a secret value for storage.
*
* - env://... and file://... are stored as-is (references)
* - encv1:... is stored as-is (already encrypted)
* - otherwise encrypt before storing
*/
const sealSecret = async (value: string): Promise<string> => {
if (!value) {
return value;
}
if (isEncrypted(value) || isSecretReference(value)) {
if (isEncrypted(value)) {
return value;
}
return encrypt(value);
};
const sealOptionalSecret = async (value?: string): Promise<string | undefined> => {
if (typeof value === "undefined") {
return value;
}
return sealSecret(value);
};
async function deriveSecret(label: string) {
const derivedKey = await hkdf("sha256", config.appSecret, "", label, 32);
@ -195,6 +110,7 @@ function generateResticPassword(): string {
export const cryptoUtils = {
resolveSecret,
sealSecret,
sealOptionalSecret,
deriveSecret,
generateResticPassword,
isEncrypted,

View file

@ -34,6 +34,7 @@
"@tanstack/react-router": "^1.166.7",
"@tanstack/react-router-ssr-query": "^1.166.7",
"@tanstack/react-start": "^1.166.7",
"@zerobyte/core": "workspace:*",
"better-auth": "^1.5.4",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",

View file

@ -10,7 +10,7 @@ This folder contains runnable, copy/paste-friendly examples for running Zerobyte
- [Simplified Docker Compose (no remote mounts)](simplified-docker-compose/README.md) — reduced-privilege deployment (no `SYS_ADMIN`, no `/dev/fuse`).
- [Bind-mount a local directory](directory-bind-mount/README.md) — back up a host folder by mounting it into the container.
- [Mount an rclone config](rclone-config-mount/README.md) — use rclone-based repository backends by mounting your rclone config.
- [Secret placeholders + Docker secrets](secrets-placeholders/README.md) — keep secrets out of the DB using `env://...` and `file://...` references.
- [Provisioned repositories and volumes](provisioned-resources/README.md) — operator-managed provisioning with env/file references kept in a mounted JSON file.
### Advanced setups

View file

@ -0,0 +1,5 @@
TZ=UTC
BASE_URL=http://localhost:4096
APP_SECRET=replace-with-openssl-rand-hex-32
ZEROBYTE_AWS_ACCESS_KEY_ID=replace-with-your-s3-access-key-id
ZEROBYTE_WEBDAV_PASSWORD=replace-with-your-webdav-password

View file

@ -0,0 +1,3 @@
.env
provisioning.json
secrets/aws_secret_access_key

View file

@ -0,0 +1,89 @@
# Provisioned repositories and volumes
This example shows how to keep operator-managed repositories and volumes in a mounted JSON file instead of creating them through the UI.
At startup, Zerobyte reads the provisioning file, resolves any `env://...` or `file://...` references, encrypts the resolved secrets into the database, and syncs the resources into the normal repositories and volumes lists as managed entries.
## Why this model
- Secret references stay in deployment-time config instead of the UI.
- Provisioned repositories and volumes show up in the normal UI and API.
- Secret rotation is just an env/secret update plus a restart.
## What this example includes
- `docker-compose.yml` mounts a provisioning file and a Docker secret.
- `.env.example` provides the environment variables used by `env://...` references.
- `provisioning.example.json` provisions one S3 repository and one WebDAV volume.
- `secrets/aws_secret_access_key.example` shows the file consumed by `file://aws_secret_access_key`.
## Prerequisites
- Docker + Docker Compose
- An existing Zerobyte organization ID (found in the UI under Settings > Organization)
- An S3-compatible repository target and a WebDAV share, or your own equivalent values
If this is a brand-new Zerobyte instance, finish first-run setup first so you have a real organization ID, then enable provisioning and restart the container.
## Setup
1. Copy the example files:
```bash
cp .env.example .env
cp provisioning.example.json provisioning.json
cp secrets/aws_secret_access_key.example secrets/aws_secret_access_key
```
2. Edit `.env`:
- Set `APP_SECRET` to a real secret, for example `openssl rand -hex 32`
- Set `ZEROBYTE_AWS_ACCESS_KEY_ID`
- Set `ZEROBYTE_WEBDAV_PASSWORD`
- Adjust `BASE_URL` and `TZ` if needed
3. Edit `provisioning.json`:
- Replace `organizationId` with your existing Zerobyte organization ID
- Update the S3 endpoint/bucket values
- Update the WebDAV server, path, and username
4. Edit `secrets/aws_secret_access_key` and replace the placeholder value with the real secret access key.
5. Start the stack:
```bash
docker compose up -d
```
## How secret references work
- `env://ZEROBYTE_AWS_ACCESS_KEY_ID` reads from a container environment variable.
- `env://ZEROBYTE_WEBDAV_PASSWORD` reads from a container environment variable.
- `file://aws_secret_access_key` reads `/run/secrets/aws_secret_access_key` inside the container.
- The resolved values are encrypted before Zerobyte stores them in the database.
`file://...` references are always resolved from `/run/secrets` and must be a single filename, not a nested path.
## Access
- UI/API: `http://<host>:4096`
## What you'll see in Zerobyte
- `AWS Production Backups` appears in the repositories list as a managed repository.
- `Team A WebDAV` appears in the volumes list as a managed volume.
- Both resources are read-only in the UI, but can still be used by backup schedules.
- Changes to `provisioning.json`, `.env`, or mounted secret files apply on the next container restart.
## Rotating or removing provisioned resources
- Rotate an env-based secret: update `.env`, then restart Zerobyte.
- Rotate a file-based secret: update `secrets/aws_secret_access_key`, then restart Zerobyte.
- Remove a resource: add `delete: true`, then restart Zerobyte.
## Notes
- This example keeps `SYS_ADMIN` and `/dev/fuse` enabled because the sample volume uses WebDAV.
- Each provisioned entry must reference an existing `organizationId`.
- Each entry includes both a top-level `backend` and the matching `config.backend`.

View file

@ -0,0 +1,28 @@
services:
zerobyte:
image: ghcr.io/nicotsx/zerobyte:latest
container_name: zerobyte
restart: unless-stopped
cap_add:
- SYS_ADMIN
devices:
- /dev/fuse:/dev/fuse
ports:
- "4096:4096"
environment:
- TZ=${TZ:-UTC}
- BASE_URL=${BASE_URL:?BASE_URL must be set}
- APP_SECRET=${APP_SECRET:?APP_SECRET must be set}
- PROVISIONING_PATH=/config/provisioning.json
- ZEROBYTE_AWS_ACCESS_KEY_ID=${ZEROBYTE_AWS_ACCESS_KEY_ID:?ZEROBYTE_AWS_ACCESS_KEY_ID must be set}
- ZEROBYTE_WEBDAV_PASSWORD=${ZEROBYTE_WEBDAV_PASSWORD:?ZEROBYTE_WEBDAV_PASSWORD must be set}
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/lib/zerobyte:/var/lib/zerobyte
- ./provisioning.json:/config/provisioning.json:ro
secrets:
- aws_secret_access_key
secrets:
aws_secret_access_key:
file: ./secrets/aws_secret_access_key

View file

@ -0,0 +1,36 @@
{
"version": 1,
"repositories": [
{
"id": "aws-prod",
"organizationId": "replace-with-existing-organization-id",
"name": "AWS Production Backups",
"backend": "s3",
"compressionMode": "auto",
"config": {
"backend": "s3",
"endpoint": "https://s3.amazonaws.com",
"bucket": "company-backups",
"accessKeyId": "env://ZEROBYTE_AWS_ACCESS_KEY_ID",
"secretAccessKey": "file://aws_secret_access_key"
}
}
],
"volumes": [
{
"id": "webdav-team-a",
"organizationId": "replace-with-existing-organization-id",
"name": "Team A WebDAV",
"backend": "webdav",
"config": {
"backend": "webdav",
"server": "backup.internal.example.com",
"path": "/team-a",
"username": "team-a",
"password": "env://ZEROBYTE_WEBDAV_PASSWORD",
"port": 443,
"ssl": true
}
}
]
}

View file

@ -0,0 +1 @@
replace-with-your-s3-secret-access-key

View file

@ -1,6 +0,0 @@
# Timezone used by the container
TZ=UTC
# Optional example secret injected via environment variable.
# Use it in Zerobyte config fields as: env://ZEROBYTE_SMB_PASSWORD
ZEROBYTE_SMB_PASSWORD=

View file

@ -1,5 +0,0 @@
# Secret files - do not commit
smb-password.txt
*-password.txt
*.secret
*.key

View file

@ -1,61 +0,0 @@
# Secret placeholders (env:// and file://) + Docker secrets
Zerobyte supports **secret placeholders** in many configuration fields (repositories, volumes, notifications).
Instead of storing raw secrets in the database, you can store a reference that gets resolved at runtime.
Supported formats:
- `env://VAR_NAME` → reads `process.env.VAR_NAME`
- `file://name` → reads `/run/secrets/name` (Docker Compose / Docker secrets)
This example shows how to run Zerobyte with:
- an environment variable you can reference via `env://...`
- a Docker secret you can reference via `file://...`
## Prerequisites
- Docker + Docker Compose
This example includes `SYS_ADMIN` and `/dev/fuse` because its intended for SMB volumes (remote mounts).
## Setup
1. Copy the env file:
```bash
cp .env.example .env
```
2. Create a Docker secret file.
⚠️ **Important**: never commit real credentials. This folder includes a `.gitignore` to help prevent accidentally committing secret files.
```bash
printf "your-smb-password" > smb-password.txt
```
3. Start Zerobyte:
```bash
docker compose up -d
```
## Using placeholders in Zerobyte
You can now use the placeholders for example in these Zerobyte configuration fields:
| UI section | Type | Field | Example value |
| --- | --- | --- | --- |
| Volumes → Create volume | SMB | Password | `file://smb_password` or `env://ZEROBYTE_SMB_PASSWORD` |
| Volumes → Create volume | WebDAV | Password | `file://webdav_password` or `env://ZEROBYTE_WEBDAV_PASSWORD` |
| Repositories → Create repository | S3 | Secret access key | `file://aws_secret_access_key` or `env://AWS_SECRET_ACCESS_KEY` |
| Repositories → Create repository | SFTP | SSH Private key | `file://sftp_private_key` or `env://SFTP_PRIVATE_KEY` |
| Notifications → Create notification | Telegram | Bot token | `file://telegram_bot_token` or `env://TELEGRAM_BOT_TOKEN` |
Notes:
- Placeholder names used in these examples are arbitrary; you can choose any valid name.
- Placeholder names are case-sensitive.
- With `file://...`, the secret name must be a single path segment (no `/` or `\\`).
- You can still paste a raw secret, but placeholders can be considered safer and easier to rotate.

View file

@ -1,26 +0,0 @@
services:
zerobyte:
image: ghcr.io/nicotsx/zerobyte:latest
container_name: zerobyte
restart: unless-stopped
# Required for mounting remote volumes (SMB/NFS/WebDAV) from inside the container
cap_add:
- SYS_ADMIN
devices:
- /dev/fuse:/dev/fuse
ports:
- "4096:4096"
environment:
- TZ=${TZ:-UTC}
# Example env-backed secret (refer to it in Zerobyte as env://ZEROBYTE_SMB_PASSWORD)
- ZEROBYTE_SMB_PASSWORD=${ZEROBYTE_SMB_PASSWORD:-}
secrets:
# Example file-backed secret (refer to it in Zerobyte as file://smb_password)
- smb_password
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/lib/zerobyte:/var/lib/zerobyte
secrets:
smb_password:
file: ./smb-password.txt