open-webui: more consistent example

Open WebUI 0.9.6 still initializes its embedding function during startup, even when OFFLINE_MODE=true and BYPASS_EMBEDDING_AND_RETRIEVAL=true. With OFFLINE_MODE=true and no cached SentenceTransformer model, startup failed with:

ValueError: No embedding model is loaded. Set RAG_EMBEDDING_MODEL to a valid SentenceTransformer model name, or configure an external RAG_EMBEDDING_ENGINE (ollama, openai, azure_openai).

Configure the tests to use a dummy Ollama embedding endpoint so startup does not try to load or download a local embedding model. The endpoint is not meant to be contacted by these tests; it only keeps startup on the external embedding configuration path.

.github/dependabot.yml

@@ -0,0 +1,9 @@
+# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    reviewers:
+      - ibizaman

.github/workflows/auto-merge.yaml

@@ -23,15 +23,28 @@ on:
         required: false
 
 jobs:
-  automerge:
+  automerge-rebase:
     runs-on: ubuntu-latest
     steps:
       - uses: reitermarkus/automerge@v2
         with:
           token: ${{ secrets.GH_TOKEN_FOR_UPDATES }}
-          merge-method: squash
+          merge-method: rebase
           do-not-merge-labels: never-merge
           required-labels: automerge
           pull-request: ${{ github.event.inputs.pull-request }}
           review: ${{ github.event.inputs.review }}
           dry-run: false
+
+  automerge-merge:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: reitermarkus/automerge@v2
+        with:
+          token: ${{ secrets.GH_TOKEN_FOR_UPDATES }}
+          merge-method: merge
+          do-not-merge-labels: never-merge
+          required-labels: automerge-merge
+          pull-request: ${{ github.event.inputs.pull-request }}
+          review: ${{ github.event.inputs.review }}
+          dry-run: false

.github/workflows/build.yaml

@@ -0,0 +1,159 @@
+# name: build
+# on: push
+# jobs:
+#   checks:
+#     uses: nixbuild/nixbuild-action/.github/workflows/ci-workflow.yml@v19
+#     with:
+#       nix_conf: |
+#         allow-import-from-derivation = true
+#     secrets:
+#       nixbuild_token: ${{ secrets.nixbuild_token }}
+
+
+name: "build"
+on:
+  pull_request:
+  push:
+    branches: [ "main" ]
+
+jobs:
+  path-filter:
+    runs-on: ubuntu-latest
+    outputs:
+      changed: ${{ steps.filter.outputs.any_changed }}
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - uses: tj-actions/changed-files@v47
+        id: filter
+        with:
+          files: |
+            lib/**
+            modules/**
+            !modules/**/docs/**
+            test/**
+            flake.lock
+            flake.nix
+            .github/workflows/build.yaml
+          separator: "\n"
+
+      - env:
+          ALL_CHANGED_FILES: ${{ steps.filter.outputs.all_changed_files }}
+        run: |
+          echo $ALL_CHANGED_FILES
+
+
+  build-matrix:
+    needs: [ "path-filter" ]
+    if: needs.path-filter.outputs.changed == 'true'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+      - name: Install Nix
+        uses: cachix/install-nix-action@v31
+        with:
+          github_access_token: ${{ secrets.GITHUB_TOKEN }}
+          enable_kvm: true
+          extra_nix_config: |
+            keep-outputs = true
+            keep-failed = true
+      - name: Setup Caching
+        uses: cachix/cachix-action@v17
+        with:
+          name: selfhostblocks
+          authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
+      - name: Generate Matrix
+        id: generate-matrix
+        run: |
+          set -euox pipefail
+
+          nix flake show --allow-import-from-derivation --json \
+              | jq -c '.["checks"]["x86_64-linux"] | keys' > .output
+
+          cat .output
+
+          echo dynamic_list="$(cat .output)" >> "$GITHUB_OUTPUT"
+    outputs:
+      check: ${{ steps.generate-matrix.outputs.dynamic_list }}
+
+  manual:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+      - name: Install Nix
+        uses: cachix/install-nix-action@v31
+        with:
+          github_access_token: ${{ secrets.GITHUB_TOKEN }}
+          enable_kvm: true
+          extra_nix_config: |
+            keep-outputs = true
+            keep-failed = true
+      - name: Setup Caching
+        uses: cachix/cachix-action@v17
+        with:
+          name: selfhostblocks
+          authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
+      - name: Build
+        run: |
+          nix \
+            --print-build-logs \
+            --option keep-going true \
+            --show-trace \
+            build .#manualHtml
+
+  tests:
+    runs-on: ubuntu-latest
+    needs: [ "build-matrix" ]
+    strategy:
+      fail-fast: false
+      matrix:
+        check: ${{ fromJson(needs.build-matrix.outputs.check) }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+      - name: Install Nix
+        uses: cachix/install-nix-action@v31
+        with:
+          github_access_token: ${{ secrets.GITHUB_TOKEN }}
+          enable_kvm: true
+          extra_nix_config: |
+            keep-outputs = true
+            keep-failed = true
+      - name: Setup Caching
+        uses: cachix/cachix-action@v17
+        with:
+          name: selfhostblocks
+          authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
+      - name: Build
+        run: |
+          echo "resultPath=$(nix eval .#checks.x86_64-linux.${{ matrix.check }} --raw)" >> $GITHUB_ENV
+          nix build --print-build-logs --show-trace .#checks.x86_64-linux.${{ matrix.check }}
+      - name: Upload Build Result
+        uses: actions/upload-artifact@v7
+        if: always() && startsWith(matrix.check, 'vm_')
+        with:
+          name: ${{ matrix.check }}
+          path: ${{ env.resultPath }}/trace/*
+          overwrite: true
+          if-no-files-found: ignore
+
+  results:
+    name: Final Results
+    runs-on: ubuntu-latest
+    needs: [ manual, tests ]
+    if: '!cancelled()'
+    steps:
+      - run: |
+          result="${{ needs.manual.result }}"
+          if ! [[ $result == "success" || $result == "skipped" ]]; then
+            exit 1
+          fi
+          result="${{ needs.tests.result }}"
+          if ! [[ $result == "success" || $result == "skipped" ]]; then
+            exit 1
+          fi
+          exit 0

.github/workflows/demo.yml

@@ -0,0 +1,111 @@
+name: Demo
+
+on:
+  workflow_dispatch:
+  pull_request:
+  push:
+    branches:
+      - main
+
+jobs:
+  path-filter:
+    runs-on: ubuntu-latest
+    outputs:
+      changed: ${{ steps.filter.outputs.any_changed }}
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - uses: tj-actions/changed-files@v47
+        id: filter
+        with:
+          files: |
+            demo/**
+            lib/**
+            modules/**
+            !modules/**/docs/**
+            test/**
+            flake.lock
+            flake.nix
+            .github/workflows/demo.yml
+          separator: "\n"
+
+      - env:
+          ALL_CHANGED_FILES: ${{ steps.filter.outputs.all_changed_files }}
+        run: |
+          echo $ALL_CHANGED_FILES
+
+  build:
+    needs: [ "path-filter" ]
+    if: needs.path-filter.outputs.changed == 'true'
+    strategy:
+      fail-fast: false
+      matrix:
+        demo:
+          - name:  homeassistant
+            flake: basic
+          - name:  homeassistant
+            flake: ldap
+
+          - name:  nextcloud
+            flake: basic
+          - name:  nextcloud
+            flake: ldap
+          - name:  nextcloud
+            flake: sso
+
+          - name:  minimal
+            flake: minimal
+          - name:  minimal
+            flake: lowlevel
+          - name:  minimal
+            flake: sops
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Install Nix
+        uses: cachix/install-nix-action@v31
+        with:
+          github_access_token: ${{ secrets.GITHUB_TOKEN }}
+          enable_kvm: true
+          extra_nix_config: |
+            keep-outputs = true
+            keep-failed = true
+
+      - uses: cachix/cachix-action@v17
+        with:
+          name: selfhostblocks
+          authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
+
+      - name: Build ${{ matrix.demo.name }} .#${{ matrix.demo.flake }}
+        run: |
+          cd demo/${{ matrix.demo.name }}
+          nix flake update --override-input selfhostblocks ../.. selfhostblocks
+          nix \
+            --print-build-logs \
+            --option keep-going true \
+            --show-trace \
+            build .#nixosConfigurations.${{ matrix.demo.flake }}.config.system.build.toplevel
+          nix \
+            --print-build-logs \
+            --option keep-going true \
+            --show-trace \
+            build .#nixosConfigurations.${{ matrix.demo.flake }}.config.system.build.vm
+
+  result:
+    runs-on: ubuntu-latest
+    needs: [ "build" ]
+    if: '!cancelled()'
+    steps:
+      - run: |
+          result="${{ needs.build.result }}"
+          if [[ $result == "success" || $result == "skipped" ]]; then
+            exit 0
+          else
+            exit 1
+          fi

.github/workflows/format.yaml

@@ -0,0 +1,24 @@
+name: "format"
+on:
+  pull_request:
+  push:
+    branches: [ "main" ]
+
+jobs:
+  format:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+      - name: Install Nix
+        uses: cachix/install-nix-action@v31
+        with:
+          github_access_token: ${{ secrets.GITHUB_TOKEN }}
+      - name: Setup Caching
+        uses: cachix/cachix-action@v17
+        with:
+          name: selfhostblocks
+          authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
+      - name: Check Formatting
+        run: |
+          find . -name '*.nix' | nix fmt -- --ci

.github/workflows/garnix.yaml

@@ -1,20 +0,0 @@
-name: "Final Results"
-
-on:
-  check_suite:
-    types: [completed]
-
-jobs:
-  results:
-    name: Final Results
-    runs-on: ubuntu-latest
-    steps:
-      - run: echo
-      - run: exit 1
-        # see https://stackoverflow.com/a/67532120/4907315
-        if: >-
-          ${{
-               contains(needs.*.result, 'failure')
-            || contains(needs.*.result, 'cancelled')
-            || contains(needs.*.result, 'skipped')
-          }}

.github/workflows/lock-update.yaml

@@ -1,24 +1,43 @@
+# See shb.skarabox.com/misc.html
 name: Update Flake Lock
 
 on:
-  workflow_dispatch:
   schedule:
-    - cron: '0 0 * * *' # runs daily at 00:00
+    - cron: '0 */3 * * *' # runs every 3 hours at :00
+
+  workflow_dispatch:
+    inputs:
+      bisect_future:
+        description: "Bisect in the future instead of the past"
+        required: false
+        default: "false"
 
 jobs:
   lockfile:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v6
       - name: Install Nix
-        uses: DeterminateSystems/nix-installer-action@main
+        uses: cachix/install-nix-action@v31
         with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          extra-conf: "system-features = nixos-test benchmark big-parallel kvm"
+          github_access_token: ${{ secrets.GITHUB_TOKEN }}
+      - name: Cache nixpkgs mirror
+        uses: actions/cache@v5
+        with:
+          path: .cache/nixpkgs.git
+          key: nixpkgs-mirror-v1
       - name: Update flake.lock
-        uses: DeterminateSystems/update-flake-lock@main
-        with:
-          token: ${{ secrets.GH_TOKEN_FOR_UPDATES }}
-          pr-labels: |
-            automerge
+        env:
+          GH_TOKEN: ${{ secrets.GH_TOKEN_FOR_UPDATES }}
+          BISECT_FUTURE: ${{ inputs.bisect_future }}
+        run: |
+          git config user.email "update-flake-lock-pr@selfhostblocks.com"
+          git config user.name "Update Flake Lock PR"
+          if [ "$BISECT_FUTURE" = false ]; then
+            BISECT_FUTURE=
+          elif [ -n "$BISECT_FUTURE" ]; then
+            BISECT_FUTURE=future
+          fi
+          nix run .#update-flake-lock-pr $BISECT_FUTURE
+

.github/workflows/pages.yml

@@ -2,14 +2,8 @@
 name: Deploy docs
 
 on:
-  # Runs on pushes targeting the default branch
   push:
     branches: ["main"]
-  # TODO: needed ?
-  # schedule:
-  #   - cron: 0 0 * * 1
-
-  # Allows you to run this workflow manually from the Actions tab
   workflow_dispatch:
 
 # Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
@@ -35,10 +29,18 @@ jobs:
 
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v6
 
-      - name: Install nix
-        uses: cachix/install-nix-action@v20
+      - name: Install Nix
+        uses: cachix/install-nix-action@v31
+        with:
+          github_access_token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Setup Caching
+        uses: cachix/cachix-action@v17
+        with:
+          name: selfhostblocks
+          authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
 
       - name: Build docs
         run: |
@@ -57,13 +59,13 @@ jobs:
             public
 
       - name: Setup Pages
-        uses: actions/configure-pages@v3
+        uses: actions/configure-pages@v6
 
       - name: Upload artifact
-        uses: actions/upload-pages-artifact@v1
+        uses: actions/upload-pages-artifact@v5
         with:
           path: ./public
 
       - name: Deploy to GitHub Pages
         id: deployment
-        uses: actions/deploy-pages@v2
+        uses: actions/deploy-pages@v5

.github/workflows/update-flake-lock-pr.nix

@@ -0,0 +1,223 @@
+{
+  gh,
+  writeShellApplication,
+}:
+writeShellApplication {
+  name = "update-flake-lock-pr";
+  runtimeInputs = [
+    gh
+  ];
+  text = ''
+    branch=ci/nixpkgs-update
+    if [ "''${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
+      branch=ci/nixpkgs-update-auto
+    fi
+
+    goto_future=no
+    if [ -n "''${1:-}" ]; then
+      goto_future=yes
+    fi
+
+    main () {
+      PR=$(get_pr)
+
+      if [ -z "$PR" ]; then
+        create_pr
+        exit 0
+      fi
+
+      case "$(get_pr_check_status "$PR")" in
+        pending)
+          echo "Checks are running on PR $PR, nothing to do yet."
+          exit 0
+          ;;
+
+        succeeded)
+          echo "Checks succeeded, nothing to do."
+          exit 0
+          ;;
+
+        failing)
+          echo "Checks failed, updating PR to midpoint commit."
+
+          local CURRENT_COMMIT
+          local LAST_WORKING_COMMIT
+          local FAILING_COMMIT
+          local NEW_COMMIT
+
+          CURRENT_COMMIT="$(get_main_branch_commit)"
+          if [ "$goto_future" = "no" ]; then
+            LAST_WORKING_COMMIT="$CURRENT_COMMIT"
+            FAILING_COMMIT="$(get_pr_commit "$PR")"
+          else
+            LAST_WORKING_COMMIT="$(get_pr_commit "$PR")"
+            FAILING_COMMIT="$(get_latest_nixpkgs_commit)"
+          fi
+          NEW_COMMIT="$(midpoint_commit "$LAST_WORKING_COMMIT" "$FAILING_COMMIT")"
+
+          if [ "$NEW_COMMIT" = "$LAST_WORKING_COMMIT" ] || [ "$NEW_COMMIT" = "$FAILING_COMMIT" ]; then
+            NEW_COMMIT="$(get_latest_nixpkgs_commit)"
+          fi
+
+          if [ "$NEW_COMMIT" = "$FAILING_COMMIT" ]; then
+            echo "Latest nixpkgs commit is still same failing commit, will retry later."
+            exit 0
+          fi
+
+          update_pr "$PR" "$CURRENT_COMMIT" "$NEW_COMMIT"
+          exit 0
+          ;;
+      esac
+    }
+
+    get_main_branch_commit() {
+      git fetch origin main
+
+      git show origin/main:flake.lock \
+        | jq -r '.nodes.nixpkgs.locked.rev'
+    }
+
+    midpoint_commit() {
+      local good="$1"
+      local bad="$2"
+
+      mkdir -p .cache
+      if [ ! -d .cache/nixpkgs.git ]; then
+        git clone --mirror https://github.com/NixOS/nixpkgs.git .cache/nixpkgs.git
+      else
+        git --git-dir=.cache/nixpkgs.git fetch origin
+      fi
+
+      local commits=()
+
+      mapfile -t commits < <(
+        git --git-dir=.cache/nixpkgs.git \
+          rev-list \
+          --reverse \
+          --ancestry-path \
+          "''${good}..''${bad}"
+      )
+
+      local count="''${#commits[@]}"
+
+      if [ "$count" -eq 0 ]; then
+        echo "$bad"
+        return
+      fi
+
+      local midpoint_index=$((count / 2))
+
+      echo "''${commits[$midpoint_index]}"
+    }
+
+    get_latest_nixpkgs_commit () {
+      git ls-remote https://github.com/NixOS/nixpkgs nixos-unstable | cut -f1
+    }
+
+    get_pr () {
+      gh pr list \
+        --head "$branch" \
+        --state open \
+        --json number \
+        --jq '.[0].number // empty'
+    }
+
+    create_pr() {
+      local test_commit
+
+      test_commit="$(get_latest_nixpkgs_commit)"
+      echo "Creating PR on nixpkgs' latest commit $test_commit"
+
+      git checkout -B "$branch"
+
+      nix flake lock \
+        --override-input nixpkgs "github:NixOS/nixpkgs/$test_commit"
+
+      git add flake.lock
+
+      git commit -m "update nixpkgs to $test_commit"
+
+      git push -u origin "$branch" --force
+
+      current_commit="$(get_main_branch_commit)"
+      gh pr create \
+        --title "update nixpkgs to $test_commit" \
+        --body "$(printf "%s\n\n%s" "Automated nixpkgs update. Latest tries:" " - https://github.com/NixOS/nixpkgs/compare/$current_commit...$test_commit")" \
+        --label automerge-merge
+    }
+
+    append_pr_body() {
+      local pr="$1"
+      local text="$2"
+
+      local current_body
+
+      current_body="$(gh pr view "$pr" --json body --jq .body)"
+
+      gh pr edit "$pr" \
+        --body "$(printf '%s%b' "$current_body" "$text")"
+    }
+
+    update_pr() {
+      local pr="$1"
+      local current_commit="$2"
+      local test_commit="$3"
+
+      echo "Updating PR to nixpkgs' commit $test_commit"
+
+      git checkout "$branch"
+
+      nix flake lock \
+        --override-input nixpkgs "github:NixOS/nixpkgs/$test_commit"
+
+      git add flake.lock
+
+      git commit -m "update nixpkgs to $test_commit"
+
+      git push --force-with-lease origin "$branch"
+
+      local future_text
+      if [ "$goto_future" = "yes" ]; then
+        future_text="bisected in the future"
+      else
+        future_text="bisected in the past"
+      fi
+
+      gh pr edit "$pr" \
+        --title "update nixpkgs to $test_commit"
+
+      append_pr_body "$pr" \
+        " ❌\n - https://github.com/NixOS/nixpkgs/compare/$current_commit...$test_commit ($future_text)"
+    }
+
+    get_pr_check_status() {
+      local pr="$1"
+
+      local status
+      status="$(gh pr checks "$pr" --json state --jq '.[].state' || true)"
+
+      if echo "$status" | grep -qE 'PENDING|QUEUED|IN_PROGRESS'; then
+        echo "pending"
+        return
+      fi
+
+      if echo "$status" | grep -qE 'FAILURE|TIMED_OUT|CANCELLED'; then
+        echo "failing"
+        return
+      fi
+
+      echo "success"
+    }
+
+    get_pr_commit() {
+      local pr="$1"
+
+      git fetch origin "$branch"
+
+      git show "origin/$branch:flake.lock" \
+        | jq -r '.nodes.nixpkgs.locked.rev'
+    }
+
+    main
+  '';
+}

.github/workflows/version.yaml

@@ -0,0 +1,22 @@
+name: Version Bump
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - VERSION
+
+jobs:
+  create-tag:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 1
+      - name: Get version
+        id: vars
+        run: echo "version=v$(cat VERSION)" >> $GITHUB_OUTPUT
+      - uses: rickstaa/action-create-tag@v1.7.2
+        with:
+          tag: ${{ steps.vars.outputs.version }}

.gitignore

@@ -1,3 +1,6 @@
 *.qcow2
 result
-result-*
+result-*
+docs/redirects.json.backup
+.nixos-test-history
+\#*#

CHANGELOG.md

@@ -1,16 +1,496 @@
-# Upcoming Release
+<!---
+
+Template:
 
 ## Breaking Changes
 
+## New Features
+
+## User Facing Backwards Compatible Changes
+
+## Fixes
+
+## Other Changes
+
+-->
+
+# Upcoming Release
+
+# v0.9.0
+
+Commits: https://github.com/ibizaman/selfhostblocks/compare/v0.8.0...v0.9.0
+
+## Breaking Changes
+
+- Updated simple-nixos-mailserver. Update all options following this pattern:
+
+  ```diff
+  - mailserver.mailboxes.<name>.specialUse = "value"
+  + mailserver.mailboxes.<name>.special_use = "\\value"
+  ```
+
+  Note that simple-nixos-mailserver wanted to upgrade the location of the storage path
+  to include the ldap UID instead of the email but I kept the email address.
+  This means there is no migration to do.
+- Update nixpkgs [from 6201e2 to abd1ea](https://github.com/ibizaman/selfhostblocks/compare/6201e203d09599479a3b3450ed24fa81537ebc4e...abd1ea17fdbedd48cbca770847b39e3a7a09ab5b).
+
+## New Features
+
+- Add `shb.zfs.snapshotBeforeActivation` option to take a ZFS snapshot of given datasets before activation.
+  See [the manual](https://shb.skarabox.com/blocks-zfs.html) for more details.
+- Add [sanoid block](https://shb.skarabox.com/blocks-sanoid.html)
+- Add option to take snapshot before activation with ZFS block.
+
+## Fixes
+
+- Fix oidc_login build for Nextcloud
+- Fix mailserver build
+
+## Other Changes
+
+- Harden lldap configuration
+- Convert ZFS block to system and [add documentation](https://shb.skarabox.com/blocks-zfs.html)
+- Switch monitoring log fetching from deprecated promtail to fluent-bit
+- Add test to catch drift for Let's Encrypt DNS provider
+
+# [BROKEN] v0.8.0
+
+## Breaking Changes
+
+- Bump of Nextcloud version to 32 and 33 because of nixpkgs bump. All provided apps are verified compatible with Nextcloud 33 thanks to new tests.
+
+## New Features
+
+- Added Immich Public Proxy service
+- Add homepage service with dashboard contract implemented by all services
+- Add scrutiny service.
+- ZFS module now supports setting permissions
+- Add landing page for mailserver and dashboard contract integration
+
+## Bug Fixes
+
+- Use configurable dataDir in arr stack
+- Forgejo ensures ldap is setup when sso is configured
+- Add nixpkgs patches on aarch64-linux too
+- Self-signed certs are now idempotent
+- Prometheus scrapes metrics at 15s interval instead of 1m
+
+## Other Changes
+
+- Arr stack declares ldap groups, declare ApiKeys and bypasses auth for readarr when sso is enabled
+- Forgejo declares ldap group
+
+# v0.7.3
+
+## New Features
+
+- Add [mailserver module](https://shb.skarabox.com/services-mailserver.html) integrating with [Simple NixOS Mailserver](https://gitlab.com/simple-nixos-mailserver/nixos-mailserver) and allowing full backup of an email provider.
+- Bump nixpkgs from https://github.com/NixOS/nixpkgs/commit/5e2a59a5b1a82f89f2c7e598302a9cacebb72a67 to https://github.com/NixOS/nixpkgs/commit/bfc1b8a4574108ceef22f02bafcf6611380c100d. [Full diff](https://github.com/nixos/nixpkgs/compare/5e2a59a5b1a82f89f2c7e598302a9cacebb72a67...bfc1b8a4574108ceef22f02bafcf6611380c100d).
+  On top of minor changes, the most notable one was:
+  - Updated Jellyfin LDAP and SSO plugins and configuration. @Codys-Wright
+
+## Bug Fixes
+
+- Fix Restic and Authelia modules referencing systemd services without the `.service` suffix and leading to 
+
+# v0.7.2
+
+## New Features
+
+- Forgejo uses secrets contract for smtp password.
+- Add [Firefly-iii](https://shb.skarabox.com/services-firefly-iii.html) service.
+- Jellyfin can [install plugins declaratively](https://shb.skarabox.com/services-jellyfin.html#services-jellyfin-options-shb.jellyfin.plugins).
+  (Support is quite crude and WIP).
+- Jellyfin configures LDAP and SSO fully declaratively, including installing necessary plugins.
+- Nextcloud 32 is fully supported thanks to tests for version 31 and 32.
+
+## Fixes
+
+- Revert Authelia to continue using dots in systemd service names.
+  This caused issue with nginx name resolution.
+
+## Other Changes
+
+- Authelia uses non deprecated `smtp.address` option.
+- Add documentation for Nginx block
+- Now a user which is only member of the admin LDAP group of a service can login.
+  Before, some services required a user to be member of both the user and admin LDAP group.
+  This is ensured by regression tests going forward.
+
+# v0.7.1
+
+## New Features
+
+- Add a Grafana dashboard showing SSL certificate renewal jobs
+
+## Fixes
+
+- Fix let's encrypt certificate renewal jobs by removing duplicated domain name.
+  Also adds an assertion to catch these kinds of errors.
+
+## Other Changes
+
+- Reduce number of late SSL renewal alert by merging all metrics corresponding to one CN.
+
+# v0.7.0
+
+## Breaking Changes
+
+- Fix pkgs overrides not being passed to users of SelfHostBlocks.
+  This will require to update your flake to follow the example in the [Usage](https://shb.skarabox.com/usage.html) section.
+
+## New Features
+
+- Add a Grafana dashboard showing stats on backup jobs
+  and also an alert if a backup job did not run in the last 24 hours or never succeeded in the last 24 hours.
+- Add SSO integration in Grafana.
+- Add Paperless service.
+
+## Fixes
+
+- Allow to upload big files in Immich.
+- Only enable php-fpm Prometheus exporter if Nextcloud is enabled.
+
+## Other Changes
+
+- Add recipe to setup DNS server with DNSSEC.
+
+# v0.6.1
+
+## New Features
+
+- Implement backup and databasebackup contracts with BorgBackup block.
+
+## Fixes
+
+- Add back erroneously removed Prometheus collectors.
+
+# v0.6.0
+
+## Breaking Changes
+
+- Removed Nextcloud 30, update to Nextcloud 31 then after to 32.
+- Removed the `sops` module in the `default` NixOS module. Removed the `all` NixOS module.
+
+## New Features
+
+- Meilisearch configured with production environment and master key.
+
+## Other Changes
+
+- Only import hardcodedsecret module in tests.
+- Better usage section in manual.
+- Added new demo for minimal SelfHostBlocks setup, which is tested in CI.
+- Format all files in repo and make sure they are formatted in CI.
+
+# v0.5.1
+
+## New Features
+
+- Added Karakeep service with SSO integration.
+- Add SelfHostBlocks' `lib` into `pkgs.lib.shb`. Integrates with [Skarabox](https://github.com/ibizaman/skarabox/blob/631ff5af0b5c850bb63a3b3df451df9707c0af4e/template/flake.nix#L42-L43) too.
+
+## Other Changes
+
+- Moved implementation guide under contributing section.
+
+# v0.5.0
+
+## Breaking Changes
+
+- Modules in the `nixosModules` output field do not anymore have the `system` in their path.
+  `selfhostblocks.nixosModules.x86_64-linux.home-assistant` becomes `selfhostblocks.nixosModules.home-assistant`
+  like it always should have been.
+
+## Fixes
+
+- Added test case making sure a user belonging to a not authorized LDAP group cannot login.
+  Fixed Open WebUI module.
+- Now importing a single module, like `selfhostblocks.nixosModules.home-assistant`, will
+  import all needed block modules at the same time.
+
+## Other Changes
+
+- Nextcloud module can now setup SSO integration without setting up LDAP integration.
+
+# v0.4.4
+
+## New Features
+
+- Added Pinchflat service with SSO integration. Declarative user creation only supported through SSO integration.
+- Added Immich service with SSO integration.
+- Added Open WebUI service with SSO integration.
+
+# v0.4.3
+
+## New Features
+
+- Allow user to change their SSO password in Authelia.
+- Make Audiobookshelf SSO integration respect admin users.
+
+## Fixes
+
+- Fix permission on Nextcloud systemd service.
+- Delete Forgejo backups correctly to avoid them piling up.
+
+## Other Changes
+
+- Add recipes section to the documentation.
+
+# v0.4.2
+
+## New Features
+
+- The LLDAP and Authelia modules gain a debug mode where a mitmdump instance is added so all traffic is printed.
+
+## Fixes
+
+- By default, LLDAP module only enforces groups declaratively. Users that are not defined declaratively
+  are not anymore deleted by inadvertence.
+- SSO integration with most services got fixed. A recent incompatible change in upstream Authelia broke most of them.
+- Fixed PostgreSQL and Home Assistant modules after nixpkgs updates.
+- Fixed Nextcloud module SSO integration with Authelia.
+- Make Nextcloud SSO integration respect admin users.
+
+# v0.4.1
+
+## New Features
+
+- LLDAP now manages users, groups, user attributes and group attributes declaratively.
+- Individual modules are exposed in the flake output for each block and service.
+- A mitmdump block is added that can be placed between two services and print all requests and responses.
+- The SSO setup for Audiobookshelf is now a bit more declarative.
+
+## Other Changes
+
+- Forgejo got a new playwright test to check the LDAP integration.
+- Some renaming options have been added retroactively for jellyfin and forgejo.
+
+# v0.4.0
+
+## Breaking Changes
+
+- Rename ldap module to lldap as well as option name `shb.ldap` to `shb.lldap`.
+
+## New Features
+
+- Jellyfin service now waits for Jellyfin server to be fully available before starting.
+- Add debug option for Jellyfin.
+- Allow to choose port for Jellyfin.
+- Make Jellyfin setup including initial admin user declarative.
+
+## Fixes
+
+- Fix Jellyfin redirect URI scheme after update.
+
+## Other Changes
+
+- Add documentation for LLDAP and Authelia block and link to it from other docs.
+
+# v0.3.1
+
+## Breaking Changes
+
+- Default version of Nextcloud is now 30.
+- Disable memories app on Nextcloud because it is broken.
+
+## New Features
+
+- Add patchNixpkgs function and pre-patched patchedNixpkgs output.
+
+## Fixes
+
+- Fix secrets passing to Nextcloud service after update.
+
+## Other Changes
+
+- Bump nixpkgs to https://github.com/NixOS/nixpkgs/commit/216207b1e58325f3590277d9102b45273afe9878
+
+# v0.3.0
+
+## New Features
+
+- Add option to add extra args to hledger command.
+
+## Breaking Changes
+
+- Default version of Nextcloud is now 29.
+
+## Fixes
+
+- Home Assistant config gets correctly generated with secrets
+  even if LDAP integration is not enabled.
+- Fix Jellyfin SSO plugin which was left badly configured
+  after a code refactoring.
+
+## Other Changes
+
+- Add a lot of playwright tests for services.
+- Add service implementation manual page to document
+  how to integrate a service in SHB.
+- Add `update-redirects` command to manage the `redirect.json` page.
+- Add home-assistant manual.
+
+# v0.2.10
+
+## New Features
+
+- Add `shb.forgejo.users` option to create users declaratively.
+
+## Fixes
+
+- Make Nextcloud create the external storage if it's a local storage
+  and the directory does not exist yet.
+- Disable flow to change password on first login for admin Forgejo user.
+  This is not necessary since the password comes from some secret store.
+
+## Breaking Changes
+
+- Fix internal link for Home Assistant
+  which now points to the fqdn. This fixes Voice Assistant
+  onboarding. This is a breaking change if one relies on
+  reaching Home Assistant through the IP address but I
+  don't recommend that. It's much better to have a DNS
+  server running locally which redirects the fqdn to the
+  server running Home Assistant.
+
+## Other Changes
+
+- Refactor tests and add playwright tests for services.
+
+# v0.2.9
+
+## New Features
+
+- Add Memories Nextcloud app declaratively configured.
+- Add Recognize Nextcloud app declaratively configured.
+
+# v0.2.8
+
+## New Features
+
+- Add dashboard for SSL certificates validity
+  and alert they did not renew on time.
+
+## Fixes
+
+- Only enable php-fpm exporter when php-fpm is enabled.
+
+## Breaking Changes
+
+- Remove upgrade script from postgres 13 to 14 and 14 to 15.
+
+# v0.2.7
+
+## New Features
+
+- Add dashboard for Nextcloud with PHP-FPM exporter.
+- Add voice option to Home-Assistant.
+
+## User Facing Backwards Compatible Changes
+
+- Add hostname and domain labels for scraped Prometheus metrics and Loki logs.
+
+# v0.2.6
+
+## New Features
+
+- Add dashboard for deluge.
+
+# v0.2.5
+
+## Other Changes
+
+- Fix more modules using backup contract.
+
+# v0.2.4
+
+## Other Changes
+
+- Fix modules using backup contract.
+
+# v0.2.3
+
+## Breaking Changes
+
+- Options `before_backup` and `after_backup` for backup contract have been renamed to
+  `beforeBackup` and `afterBackup`.
+- All options using the backup and databasebackup contracts now use the new style.
+
+## Other Changes
+
+- Show how to pin Self Host Blocks flake input to a tag.
+
+# v0.2.2
+
+## User Facing Backwards Compatible Changes
+
+- Fix: add implementation for `sops.nix` module.
+
+## Other Changes
+
+- Use VERSION when rendering manual too.
+
+# v0.2.1
+
+## User Facing Backwards Compatible Changes
+
+- Add `sops.nix` module to `nixosModules.default`.
+
+## Other Changes
+
+- Auto-tagging of git repo when VERSION file gets updated.
+- Add VERSION file to track version.
+
+# v0.2.0
+
+## New Features
+
+- Backup:
+  - Add feature to backup databases with the database backup contract, implemented with `shb.restic.databases`.
+
+## Breaking Changes
+
+- Remove dependency on `sops-nix`.
 - Rename `shb.nginx.autheliaProtect` to `shb.nginx.vhosts`. Indeed, the option allows to define a vhost with _optional_ Authelia protection but the former name made it look like Authelia protection was enforced.
 - Rename all `shb.arr.*.APIKey` to `shb.arr.*.ApiKey`.
 - Remove `shb.vaultwarden.ldapEndpoint` option because it was not used in the implementation anyway.
 - Bump Nextcloud default version from 27 to 28. Add support for version 29.
 - Deluge config breaks the authFile into an attrset of user to password file. Also deluge has tests now.
 - Nextcloud now configures the LDAP app to use the `user_id` from LLDAP as the user ID used in Nextcloud. This makes all source of user - internal, LDAP and SSO - agree on the user ID.
+- Authelia options changed:
+  - `shb.authelia.oidcClients.id` -> `shb.authelia.oidcClients.client_id`
+  - `shb.authelia.oidcClients.description` -> `shb.authelia.oidcClients.client_name`
+  - `shb.authelia.oidcClients.secret` -> `shb.authelia.oidcClients.client_secret`
+  - `shb.authelia.ldapEndpoint` -> `shb.authelia.ldapHostname` and `shb.authelia.ldapPort`
+  - `shb.authelia.jwtSecretFile` -> `shb.authelia.jwtSecret.result.path`
+  - `shb.authelia.ldapAdminPasswordFile` -> `shb.authelia.ldapAdminPassword.result.path`
+  - `shb.authelia.sessionSecretFile` -> `shb.authelia.sessionSecret.result.path`
+  - `shb.authelia.storageEncryptionKeyFile` -> `shb.authelia.storageEncryptionKey.result.path`
+  - `shb.authelia.identityProvidersOIDCIssuerPrivateKeyFile` -> `shb.authelia.identityProvidersOIDCIssuerPrivateKey.result.path`
+  - `shb.authelia.smtp.passwordFile` -> `shb.authelia.smtp.password.result.path`
+- Make Nextcloud automatically disable maintenance mode upon service restart.
+- `shb.ldap.ldapUserPasswordFile` -> `shb.ldap.ldapUserPassword.result.path`
+- `shb.ldap.jwtSecretFile` -> `shb.ldap.jwtSecret.result.path`
+- Jellyfin changes:
+  - `shb.jellyfin.ldap.passwordFile` -> `shb.jellyfin.ldap.adminPassword.result.path`.
+  - `shb.jellyfin.sso.secretFile` -> `shb.jellyfin.ldap.sharedSecret.result.path`.
+  - + `shb.jellyfin.ldap.sharedSecretForAuthelia`.
+- Forgejo changes:
+  - `shb.forgejo.ldap.adminPasswordFile` -> `shb.forgejo.ldap.adminPassword.result.path`.
+  - `shb.forgejo.sso.secretFile` -> `shb.forgejo.ldap.sharedSecret.result.path`.
+  - `shb.forgejo.sso.secretFileForAuthelia` -> `shb.forgejo.ldap.sharedSecretForAuthelia.result.path`.
+  - `shb.forgejo.adminPasswordFile` -> `shb.forgejo.adminPassword.result.path`.
+  - `shb.forgejo.databasePasswordFile` -> `shb.forgejo.databasePassword.result.path`.
+- Backup:
+  - `shb.restic.instances` options has been split between `shb.restic.instances.request` and `shb.restic.instances.settings`, matching better with contracts.
+- Use of secret contract everywhere.
 
 ## User Facing Backwards Compatible Changes
 
+- Add mount contract.
+- Export torrent metrics.
+- Bump chunkSize in Nextcloud to boost performance.
 - Fix home-assistant onboarding file generation. Added new VM test.
 - OIDC and SMTP config are now optional in Vaultwarden. Added new VM test.
 - Add default OIDC config for Authelia. This way, Authelia can start even with no config or only forward auth configs.

README.md

@@ -1,35 +1,86 @@
-# Self Host Blocks
-
-*Building blocks for self-hosting with battery included.*
+![GitHub Release](https://img.shields.io/github/v/release/ibizaman/selfhostblocks)
+![GitHub commits since latest release (branch)](https://img.shields.io/github/commits-since/ibizaman/selfhostblocks/latest/main)
+![GitHub commit activity (branch)](https://img.shields.io/github/commit-activity/w/ibizaman/selfhostblocks/main)
+![GitHub Issues or Pull Requests](https://img.shields.io/github/issues-pr-raw/ibizaman/selfhostblocks)
+![GitHub Issues or Pull Requests](https://img.shields.io/github/issues-pr-closed-raw/ibizaman/selfhostblocks?label=closed)
+![GitHub Issues or Pull Requests](https://img.shields.io/github/issues-raw/ibizaman/selfhostblocks)
+![GitHub Issues or Pull Requests](https://img.shields.io/github/issues-closed-raw/ibizaman/selfhostblocks?label=closed)
 
 [![Documentation](https://github.com/ibizaman/selfhostblocks/actions/workflows/pages.yml/badge.svg)](https://github.com/ibizaman/selfhostblocks/actions/workflows/pages.yml)
-[![Tests](https://img.shields.io/endpoint.svg?url=https%3A%2F%2Fgarnix.io%2Fapi%2Fbadges%2Fibizaman%2Fselfhostblocks%3Fbranch%3Dmain)](https://garnix.io) (using Garnix)
+[![Tests](https://github.com/ibizaman/selfhostblocks/actions/workflows/build.yaml/badge.svg)](https://github.com/ibizaman/selfhostblocks/actions/workflows/build.yaml)
+[![Demo](https://github.com/ibizaman/selfhostblocks/actions/workflows/demo.yml/badge.svg)](https://github.com/ibizaman/selfhostblocks/actions/workflows/demo.yml)
+![Matrix](https://img.shields.io/matrix/selfhostblocks%3Amatrix.org)
 
-SHB's (Self Host Blocks) is yet another server management tool whose goal is to provide better
-building blocks for self-hosting. Indeed, SHB provides opinionated [building
-blocks](#available-blocks) fitting together to self-host any service you'd want. Some [common
-services](#provided-services) are provided out of the box.
+<hr />
 
-SHB's goal is to make these building blocks plug-and-play. To achieve this, SHB pioneers
-[contracts](https://shb.skarabox.com/contracts.html) which allows you, the final user, to be more in
-control of which pieces go where. The promise here is to let you choose, for example, any reverse
-proxy you want or any database you want, without requiring work from maintainers of the services you
-want to self host.
+# SelfHostBlocks
 
-To achieve all this, SHB is using the full power of NixOS modules and NixOS VM tests. Indeed, each
-building block and each service is a NixOS module using modules defined in
-[Nixpkgs](https://github.com/NixOS/nixpkgs/) and they are tested using full VMs on every commit.
+SelfHostBlocks is:
+
+- Your escape from the cloud, for privacy and data sovereignty enthusiast. [Why?](#why-self-hosting)
+- A groupware to self-host [all your data](#services): documents, pictures, calendars, contacts, etc.
+- An opinionated NixOS server management OS for a [safe self-hosting experience](#features).
+- A NixOS distribution making sure all services build and work correctly thanks to NixOS VM tests.
+- A collection of NixOS modules standardizing options so configuring services [look the same](#unified-interfaces).
+- A testing ground for [contracts](#contracts) which intents to make nixpkgs modules more modular.
+- [Upstreaming][] as much as possible.
+
+[upstreaming]: https://github.com/pulls?page=1&q=created%3A%3E2023-06-01+is%3Apr+author%3Aibizaman+archived%3Afalse+-repo%3Aibizaman%2Fselfhostblocks+-repo%3Aibizaman%2Fskarabox
+
+## Why Self-Hosting
+
+It is obvious by now that
+a deep dependency on proprietary service providers - "the cloud" -
+is a significant liability.
+One aspect often talked about is privacy
+which is inherently not guaranteed when using a proprietary service
+and is a valid concern.
+A more punishing issue is having your account closed or locked
+without prior warning
+When that happens,
+you get an instantaneous sinking feeling in your stomach
+at the realization you lost access to your data,
+possibly without recourse.
+
+Hosting services yourself is the obvious alternative
+to alleviate those concerns
+but it tends to require a lot of technical skills and time.
+SelfHostBlocks (together with its sibling project [Skarabox][])
+aims to lower the bar to self-hosting,
+and provides an opinionated server management system based on NixOS modules
+embedding best practices.
+Contrary to other server management projects,
+its main focus is ease of long term maintenance
+before ease of installation.
+To achieve this, it provides building blocks to setup services.
+Some are already provided out of the box,
+and customizing or adding additional ones is done easily.
+
+The building blocks fit nicely together thanks to [contracts](#contracts)
+which SelfHostBlocks sets out to introduce into nixpkgs.
+This will increase modularity, code reuse
+and empower end users to assemble components
+that fit together to build their server.
 
 ## TOC
 
 <!--toc:start-->
 - [Usage](#usage)
-- [Manual](#manual)
+  - [At a Glance](#at-a-glance)
+  - [Existing Installation](#existing-installation)
+  - [Installation From Scratch](#installation-from-scratch)
+- [Features](#features)
+  - [Services](#services)
+  - [Blocks](#blocks)
+  - [Unified Interfaces](#unified-interfaces)
+  - [Contracts](#contracts)
+  - [Interfacing With Other OSes](#interfacing-with-other-oses)
+  - [Sitting on the Shoulders of a Giant](#sitting-on-the-shoulders-of-a-giant)
+  - [Automatic Updates](#automatic-updates)
+  - [Demos](#demos)
 - [Roadmap](#roadmap)
-- [Available Blocks](#available-blocks)
-- [Provided Services](#provided-services)
-- [Demos](#demos)
 - [Community](#community)
+- [Funding](#funding)
 - [License](#license)
 <!--toc:end-->
 
@@ -39,121 +90,350 @@ building block and each service is a NixOS module using modules defined in
 > production server, this is really just a one person effort for now and there are most certainly
 > bugs that I didn't discover yet.
 
-Self Host Blocks is available as a flake. To use it in your project, add the following flake input:
+To get started using SelfHostBlocks, the following snippet is enough:
 
 ```nix
-inputs.selfhostblocks.url = "github:ibizaman/selfhostblocks";
+{
+  inputs.selfhostblocks.url = "github:ibizaman/selfhostblocks";
+
+  outputs = { selfhostblocks, ... }: let
+    system = "x86_64-linux";
+    nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
+  in
+    nixosConfigurations = {
+      myserver = nixpkgs'.nixosSystem {
+        inherit system;
+        modules = [
+          selfhostblocks.nixosModules.default
+          ./configuration.nix
+        ];
+      };
+    };
+}
 ```
 
-This is not quite enough though and more information is provided in [the
-manual](https://shb.skarabox.com/usage.html).
+SelfHostBlocks provides its own patched nixpkgs, so you are required to use it
+otherwise evaluation can quickly break.
+[The usage section](https://shb.skarabox.com/usage.html) of the manual has
+more details and goes over how to deploy with [Colmena][], [nixos-rebuild][] and [deploy-rs][]
+and also how to handle secrets management with [SOPS][].
 
-- You are new to self hosting and want pre-configured services to deploy easily. Look at the
-  [services section](https://shb.skarabox.com/services.html).
-- You are a seasoned self-hoster but want to enhance some services you deploy already. Go to the
-  [blocks section](https://shb.skarabox.com/blocks.html).
-- You are a user of Self Host Blocks but would like to use your own implementation for a block. Go
-  to the [contracts section](https://shb.skarabox.com/contracts.html).
+[Colmena]: https://shb.skarabox.com/usage.html#usage-example-colmena
+[nixos-rebuild]: https://shb.skarabox.com/usage.html#usage-example-nixosrebuild
+[deploy-rs]: https://shb.skarabox.com/usage.html#usage-example-deployrs
+[SOPS]: https://shb.skarabox.com/usage.html#usage-secrets
 
-Head over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org) for any remaining
-question, or just to say hi :)
+Then, to actually configure services, you can choose which one interests you in
+the [services section](https://shb.skarabox.com/services.html) of the manual.
 
-## Why yet another self hosting tool?
+The [recipes section](https://shb.skarabox.com/recipes.html) of the manual shows some other common use cases.
 
-By using Self Host Blocks, you get all the benefits of NixOS which are, for self hosted applications
-specifically:
+Head over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org)
+for any remaining question, or just to say hi :)
+
+### Installation From Scratch
+
+I do recommend for this my sibling project [Skarabox][]
+which bootstraps a new server and sets up a few tools:
+
+- Create a bootable ISO, installable on an USB key.
+- Handles one or two (in raid 1) SSDs for root partition.
+- Handles two (in raid 1) or more hard drives for data partition.
+- [nixos-anywhere](https://github.com/nix-community/nixos-anywhere) to install NixOS headlessly.
+- [disko](https://github.com/nix-community/disko) to format the drives using native ZFS encryption with remote unlocking through ssh.
+- [sops-nix](https://github.com/Mic92/sops-nix) to handle secrets.
+- [deploy-rs](https://github.com/serokell/deploy-rs) to deploy updates.
+
+[Skarabox]:  https://github.com/ibizaman/skarabox
+
+## Features
+
+SelfHostBlocks provides building blocks that take care of common self-hosting needs:
+
+- Backup for all services.
+- Automatic creation of ZFS datasets per service.
+- LDAP and SSO integration for most services.
+- Monitoring with Grafana and Prometheus stack with provided dashboards and integration with Scrutiny.
+- Automatic reverse proxy and certificate management for HTTPS.
+- VPN and proxy tunneling services.
+
+Great care is taken to make the proposed stack robust.
+This translates into a test suite comprised of automated NixOS VM tests
+which includes playwright tests to verify some important workflow
+like logging in.
+
+This test suite also serves as a guaranty that all services provided by SelfHostBlocks
+all evaluate, build and work correctly together. It works similarly as a distribution but here it's all [automated](#automatic-updates).
+
+Also, the stack fits together nicely thanks to [contracts](#contracts).
+
+### Services
+
+[Provided services](https://shb.skarabox.com/services.html) are:
+
+- Nextcloud
+- Audiobookshelf
+- Deluge + *arr stack
+- Simple NixOS Mailserver
+- Firefly-iii
+- Forgejo
+- Grocy
+- Hledger
+- Home-Assistant
+- Jellyfin
+- Karakeep
+- Open WebUI
+- Pinchflat
+- Vaultwarden
+
+Like explained above, those services all benefit from
+out of the box backup,
+LDAP and SSO integration,
+monitoring with Grafana,
+reverse proxy and certificate management
+and VPN integration for the *arr suite.
+
+Some services do not have an entry yet in the manual.
+To know options for those, the only way for now
+is to go to the [All Options][] section of the manual.
+
+[All Options]: https://shb.skarabox.com/options.html
+
+### Blocks
+
+The services above rely on the following [common blocks][]
+which altogether provides a solid foundation for self-hosting services:
+
+- Authelia
+- BorgBackup
+- Davfs
+- LDAP
+- Monitoring (Grafana - Prometheus - Loki stack + Scrutiny)
+- Nginx
+- PostgreSQL
+- Restic
+- Sops
+- SSL
+- Tinyproxy
+- VPN
+- ZFS
+
+Those blocks can be used with services
+not provided by SelfHostBlocks as shown [in the manual][common blocks].
+
+[common blocks]: https://shb.skarabox.com/blocks.html
+
+The manual also provides documentation for each individual blocks.
+
+### Unified Interfaces
+
+Thanks to the blocks,
+SelfHostBlocks provides an unified configuration interface
+for the services it provides.
+
+Compare the configuration for Nextcloud and Forgejo.
+The following snippets focus on similitudes and assume the relevant blocks - like secrets - are configured off-screen.
+It also does not show specific options for each service.
+These are still complete snippets that configure HTTPS,
+subdomain serving the service, LDAP and SSO integration.
+
+```nix
+shb.nextcloud = {
+  enable = true;
+  subdomain = "nextcloud";
+  domain = "example.com";
+
+  ssl = config.shb.certs.certs.letsencrypt.${domain};
+
+  apps.ldap = {
+    enable = true;
+    host = "127.0.0.1";
+    port = config.shb.lldap.ldapPort;
+    dcdomain = config.shb.lldap.dcdomain;
+    adminPassword.result = config.shb.sops.secret."nextcloud/ldap/admin_password".result;
+  };
+  apps.sso = {
+    enable = true;
+    endpoint = "https://${config.shb.authelia.subdomain}.${config.shb.authelia.domain}";
+
+    secret.result = config.shb.sops.secret."nextcloud/sso/secret".result;
+    secretForAuthelia.result = config.shb.sops.secret."nextcloud/sso/secretForAuthelia".result;
+  };
+};
+```
+
+```nix
+shb.forgejo = {
+  enable = true;
+  subdomain = "forgejo";
+  domain = "example.com";
+
+  ssl = config.shb.certs.certs.letsencrypt.${domain};
+
+  ldap = {
+    enable = true;
+    host = "127.0.0.1";
+    port = config.shb.lldap.ldapPort;
+    dcdomain = config.shb.lldap.dcdomain;
+    adminPassword.result = config.shb.sops.secret."nextcloud/ldap/admin_password".result;
+  };
+
+  sso = {
+    enable = true;
+    endpoint = "https://${config.shb.authelia.subdomain}.${config.shb.authelia.domain}";
+
+    secret.result = config.shb.sops.secret."forgejo/sso/secret".result;
+    secretForAuthelia.result = config.shb.sops.secret."forgejo/sso/secretForAuthelia".result;
+  };
+};
+```
+
+As you can see, they are pretty similar!
+This makes setting up a new service pretty easy and intuitive.
+
+SelfHostBlocks provides an ever growing list of [services](#services)
+that are configured in the same way.
+
+### Contracts
+
+To make building blocks that fit nicely together,
+SelfHostBlocks pioneers [contracts][] which allows you, the final user,
+to be more in control of which piece goes where.
+This lets you choose, for example,
+any reverse proxy you want or any database you want,
+without requiring work from maintainers of the services you want to self host.
+
+An [RFC][] exists to upstream this concept into `nixpkgs`.
+The [manual][contracts] also provides an explanation of the why and how of contracts.
+
+Also, two videos exist of me presenting the topic,
+the first at [NixCon North America in spring of 2024][NixConNA2024]
+and the second at [NixCon in Berlin in fall of 2024][NixConBerlin2024].
+
+[contracts]: https://shb.skarabox.com/contracts.html
+[RFC]: https://github.com/NixOS/rfcs/pull/189
+[NixConNA2024]: https://www.youtube.com/watch?v=lw7PgphB9qM
+[NixConBerlin2024]: https://www.youtube.com/watch?v=CP0hR6w1csc
+
+### Interfacing With Other OSes
+
+Thanks to [contracts](#contracts), one can interface NixOS
+with systems on other OSes.
+The [RFC][] explains how that works.
+
+### Sitting on the Shoulders of a Giant
+
+By using SelfHostBlocks, you get all the benefits of NixOS
+which are, for self hosted applications specifically:
 
 - declarative configuration;
 - atomic configuration rollbacks;
 - real programming language to define configurations;
-- user-defined abstractions (create your own functions or NixOS modules on top of SHB!);
+- create your own higher level abstractions on top of SelfHostBlocks;
 - integration with the rest of nixpkgs;
 - much fewer "works on my machine" type of issues.
 
-In no particular order, here are some aspects of SHB which I find interesting and differentiates it
-from other server management projects:
+### Automatic Updates
 
-- SHB intends to be a library, not a framework. You can either go all in and use SHB provided
-  services directly or use just one block in your existing infrastructure.
-- SHB introduces [contracts](https://shb.skarabox.com/contracts.html) to allow you to swap
-  implementation for each self-hosting need. For example, you should be able to use the reverse
-  proxy you want without modifying any services depending on it.
-- SHB contracts also allows you to use your own custom implementation instead of the provided one,
-  as long as it follows the contract and passes the tests.
-- SHB provides at least one implementation for each contract like backups, SSL certificates, reverse
-  proxy, VPN, etc. Those are called blocks here and are documented in [the
-  manual](https://shb.skarabox.com/blocks.html).
-- SHB provides several services out of the box fully using the blocks provided. Those can also be
-  found in [the manual](https://shb.skarabox.com/services.html).
-- SHB follows nixpkgs unstable branch closely. There is a GitHub action running daily that updates
-  the `nixpkgs` input in the root `flakes.nix`, runs the tests and merges a PR with the new input if
-  the tests pass.
+SelfHostBlocks follows nixpkgs unstable branch closely.
+There is a GitHub action running every couple of days that updates
+the `nixpkgs` input in the root `flakes.nix`,
+runs the tests and merges the PR automatically
+if the tests pass.
 
-## Manual
+A release is then made every few commits,
+whenever deemed sensible.
+On your side, to update I recommend pinning to a release
+with the following command,
+replacing the RELEASE with the one you want:
 
-The manual can be found at [shb.skarabox.com](https://shb.skarabox.com/).
+```bash
+RELEASE=0.2.4
+nix flake update \
+  --override-input selfhostblocks github:ibizaman/selfhostblocks/$RELEASE \
+  selfhostblocks
+```
 
-Work is in progress to document everything in the manual but I'm not there yet. For what's not yet
-documented, unfortunately the source code is the best place to read about them.
-[Here](./modules/services) for services and [here](./modules/blocks) for blocks.
+### Demos
+
+Demos that start and deploy a service
+on a Virtual Machine on your computer are located
+under the [demo](./demo/) folder.
+
+These show the onboarding experience you would get
+if you deployed one of the services on your own server.
 
 ## Roadmap
 
-Currently, the Nextcloud service and SSL block are the most advanced and most documented.
+Currently, the Nextcloud and Vaultwarden services
+and the SSL and backup blocks
+are the most advanced and most documented.
 
-Documenting all services and blocks will be done as I make all blocks and services use the
-contracts.
+Documenting all services and blocks will be done
+as I make all blocks and services use the contracts.
 
 Upstreaming changes is also on the roadmap.
 
-Check [the issues](https://github.com/ibizaman/selfhostblocks/issues) to see planned works. Feel
-free to add more!
+Check the [issues][] and the [milestones]() to see planned work.
+Feel free to add more or to contribute!
 
-That being said, I am personally using all the blocks and services in this project, so they do work
-to some extent.
+[issues]: (https://github.com/ibizaman/selfhostblocks/issues)
+[milestones]: https://github.com/ibizaman/selfhostblocks/milestones
 
-## Available Blocks
-
-- [`authelia.nix`](./modules/blocks/authelia.nix) for Single Sign On.
-- [`backup.nix`](./modules/blocks/backup.nix).
-- [`ldap.nix`](./modules/blocks/ldap.nix) for user management.
-- [`monitoring.nix`](./modules/blocks/monitoring.nix) for dashboards, logs and alerts.
-- [`nginx.nix`](./modules/blocks/nginx.nix) for reverse proxy with SSL termination.
-- [`postgresql.nix`](./modules/blocks/postgresql.nix) for database setup.
-- [`ssl.nix`](./modules/blocks/ssl.nix) for maintaining self-signed SSL certificates or certificates provided by Let's Encrypt.
-- [`tinyproxy.nix`](./modules/blocks/tinyproxy.nix) to forward traffic to a VPN tunnel.
-- [`vpn.nix`](./modules/blocks/vpn.nix) to setup a VPN tunnel.
-
-## Provided Services
-
-- [`arr.nix`](./modules/services/arr.nix) for finding media https://wiki.servarr.com/.
-- [`deluge.nix`](./modules/services/deluge.nix) for downloading linux isos https://deluge-torrent.org/.
-- [`hledger.nix`](./modules/services/hledger.nix) for managing finances https://hledger.org/.
-- [`home-assistant.nix`](./modules/services/home-assistant.nix) for private IoT https://www.home-assistant.io/.
-- [`jellyfin.nix`](./modules/services/jellyfin.nix) for watching media https://jellyfin.org/.
-- [Nextcloud Server](https://shb.skarabox.com/services-nextcloud.html) for private documents, contacts, calendar, etc https://nextcloud.com.
-- [`vaultwarden.nix`](./modules/services/vaultwarden.nix) for passwords https://github.com/dani-garcia/vaultwarden.
-- [`audiobookshelf.nix`](./modules/services/audiobookshelf.nix) for hosting podcasts and audio books https://www.audiobookshelf.org/.
-
-## Demos
-
-Demos that start and deploy a service on a Virtual Machine on your computer are located under the
-[demo](./demo/) folder. These show the onboarding experience you would get if you deployed one of
-the services on your own server.
+All blocks and services have NixOS tests.
+Also, I am personally using all the blocks and services in this project, so they do work to some extent.
 
 ## Community
 
-All issues and PRs are welcome. For PRs, if they are substantial changes, please open an issue to
-discuss the details first. More details in [here](https://shb.skarabox.com/contributing.html).
+This project has been the main focus
+of my (non work) life for the past 3 year now
+and I intend to continue working on this for a long time.
 
-Come hang out in the [Matrix channel](https://matrix.to/#/%23selfhostblocks%3Amatrix.org). :)
+All issues and PRs are welcome:
 
-One aspect that's close to my heart is I intent to make SHB the lightest layer on top of nixpkgs as
+- Use this project. Something does not make sense? Something's not working?
+- Documentation. Something is not clear?
+- New services. Have one of your preferred service not integrated yet?
+- Better patterns. See something weird in the code?
+
+For PRs, if they are substantial changes, please open an issue to
+discuss the details first. More details in [the contributing section](https://shb.skarabox.com/contributing.html)
+of the manual.
+
+Issues that are being worked on are labeled with the [in progress][] label.
+Before starting work on those, you might want to talk about it in the issue tracker
+or in the [matrix][] channel.
+
+The prioritized issues are those belonging to the [next milestone][milestone].
+Those issues are not set in stone and I'd be very happy to solve
+an issue an user has before scratching my own itch.
+
+[in progress]: https://github.com/ibizaman/selfhostblocks/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22in%20progress%22
+[matrix]: https://matrix.to/#/%23selfhostblocks%3Amatrix.org
+[milestone]: https://github.com/ibizaman/selfhostblocks/milestones
+
+One aspect that's close to my heart is I intent to make SelfHostBlocks the lightest layer on top of nixpkgs as
 possible. I want to upstream as much as possible. I will still take some time to experiment here but
 when I'm satisfied with how things look, I'll upstream changes.
 
+## Funding
+
+I was lucky to [obtain a grant][nlnet] from NlNet which is an European fund,
+under [NGI Zero Core][NGI0],
+to work on this project.
+This also funds the contracts RFC.
+
+Go apply for a grant too!
+
+[nlnet]: https://nlnet.nl/project/SelfHostBlocks
+[NGI0]: https://nlnet.nl/core/
+
+<p>
+<img alt="NlNet logo" src="https://nlnet.nl/logo/banner.svg" width="200" />
+<img alt="NGI Zero Core logo" src="https://nlnet.nl/image/logos/NGI0Core_tag.svg" width="200" />
+</p>
+
 ## License
 
-I'm following the [Nextcloud](https://github.com/nextcloud/server) license which is AGPLv3. See
-[this article](https://www.fsf.org/bulletin/2021/fall/the-fundamentals-of-the-agplv3) from the FSF that explains what this license adds to the GPL
-one.
+I'm following the [Nextcloud](https://github.com/nextcloud/server) license which is AGPLv3.
+See [this article](https://www.fsf.org/bulletin/2021/fall/the-fundamentals-of-the-agplv3) from the FSF that explains what this license adds to the GPL one.

VERSION

@@ -0,0 +1 @@
+0.9.0

demo/homeassistant/README.md

@@ -60,7 +60,7 @@ chmod 600 sshkey
 
 This is only needed because git mangles with the permissions. You will not even see this change in
 `git status`.
-### Deploy with Colmena {#demo-homeassitant-deploy-colmena}
+### Deploy with Colmena {#demo-homeassistant-deploy-colmena}
 
 If you deploy with Colmena, you must first build the VM and start it:
 
@@ -177,7 +177,7 @@ user and password you just created above.
 ### Files {#demo-homeassistant-files}
 
 - [`flake.nix`](./flake.nix): nix entry point, defines one target host for
-  [colmena](https://colmena.cli.rs) to deploy to as well as the selfhostblock's config for
+  [colmena](https://colmena.cli.rs) to deploy to as well as the selfhostblocks' config for
   setting up the home assistant server paired with the LDAP server.
 - [`configuration.nix`](./configuration.nix): defines all configuration required for colmena
   to deploy to the VM. The file has comments if you're interested.

demo/homeassistant/configuration.nix

@@ -5,17 +5,17 @@ let
   targetPort = 2222;
 in
 {
-  imports =
-    [ # Include the results of the hardware scan.
-      ./hardware-configuration.nix
-    ];
+  imports = [
+    # Include the results of the hardware scan.
+    ./hardware-configuration.nix
+  ];
 
   boot.loader.grub.enable = true;
   boot.kernelModules = [ "kvm-intel" ];
   system.stateVersion = "22.11";
 
   # Options above are generate by running nixos-generate-config on the VM.
-  
+
   # Needed otherwise deploy will say system won't be able to boot.
   boot.loader.grub.device = "/dev/vdb";
   # Needed to avoid getting into not available disk space in /boot.
@@ -26,7 +26,10 @@ in
 
   # Options above are needed to deploy in a VM.
 
-  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
 
   # We need to create the user we will deploy with.
   users.users.${targetUser} = {
@@ -41,9 +44,11 @@ in
 
   # The user we're deploying with must be able to run sudo without password.
   security.sudo.extraRules = [
-    { users = [ targetUser ];
+    {
+      users = [ targetUser ];
       commands = [
-        { command = "ALL";
+        {
+          command = "ALL";
           options = [ "NOPASSWD" ];
         }
       ];
@@ -59,7 +64,9 @@ in
   services.openssh = {
     enable = true;
     ports = [ targetPort ];
-    permitRootLogin = "no";
-    passwordAuthentication = false;
+    settings = {
+      PermitRootLogin = "no";
+      PasswordAuthentication = false;
+    };
   };
 }

demo/homeassistant/flake.lock

@@ -1,164 +0,0 @@
-{
-  "nodes": {
-    "flake-utils": {
-      "inputs": {
-        "systems": "systems"
-      },
-      "locked": {
-        "lastModified": 1709126324,
-        "narHash": "sha256-q6EQdSeUZOG26WelxqkmR7kArjgWCdw5sfJVHPH/7j8=",
-        "owner": "numtide",
-        "repo": "flake-utils",
-        "rev": "d465f4819400de7c8d874d50b982301f28a84605",
-        "type": "github"
-      },
-      "original": {
-        "owner": "numtide",
-        "repo": "flake-utils",
-        "type": "github"
-      }
-    },
-    "nix-flake-tests": {
-      "locked": {
-        "lastModified": 1677844186,
-        "narHash": "sha256-ErJZ/Gs1rxh561CJeWP5bohA2IcTq1rDneu1WT6CVII=",
-        "owner": "antifuchs",
-        "repo": "nix-flake-tests",
-        "rev": "bbd9216bd0f6495bb961a8eb8392b7ef55c67afb",
-        "type": "github"
-      },
-      "original": {
-        "owner": "antifuchs",
-        "repo": "nix-flake-tests",
-        "type": "github"
-      }
-    },
-    "nixpkgs": {
-      "locked": {
-        "lastModified": 1709150264,
-        "narHash": "sha256-HofykKuisObPUfj0E9CJVfaMhawXkYx3G8UIFR/XQ38=",
-        "owner": "nixos",
-        "repo": "nixpkgs",
-        "rev": "9099616b93301d5cf84274b184a3a5ec69e94e08",
-        "type": "github"
-      },
-      "original": {
-        "owner": "nixos",
-        "ref": "nixos-unstable",
-        "repo": "nixpkgs",
-        "type": "github"
-      }
-    },
-    "nixpkgs-stable": {
-      "locked": {
-        "lastModified": 1708819810,
-        "narHash": "sha256-1KosU+ZFXf31GPeCBNxobZWMgHsSOJcrSFA6F2jhzdE=",
-        "owner": "NixOS",
-        "repo": "nixpkgs",
-        "rev": "89a2a12e6c8c6a56c72eb3589982c8e2f89c70ea",
-        "type": "github"
-      },
-      "original": {
-        "owner": "NixOS",
-        "ref": "release-23.11",
-        "repo": "nixpkgs",
-        "type": "github"
-      }
-    },
-    "nixpkgs_2": {
-      "locked": {
-        "lastModified": 1708751719,
-        "narHash": "sha256-0uWOKSpXJXmXswOvDM5Vk3blB74apFB6rNGWV5IjoN0=",
-        "owner": "NixOS",
-        "repo": "nixpkgs",
-        "rev": "f63ce824cd2f036216eb5f637dfef31e1a03ee89",
-        "type": "github"
-      },
-      "original": {
-        "owner": "NixOS",
-        "ref": "nixpkgs-unstable",
-        "repo": "nixpkgs",
-        "type": "github"
-      }
-    },
-    "nmdsrc": {
-      "flake": false,
-      "locked": {
-        "lastModified": 1705050560,
-        "narHash": "sha256-x3zzcdvhJpodsmdjqB4t5mkVW22V3wqHLOun0KRBzUI=",
-        "ref": "refs/heads/master",
-        "rev": "66d9334933119c36f91a78d565c152a4fdc8d3d3",
-        "revCount": 66,
-        "type": "git",
-        "url": "https://git.sr.ht/~rycee/nmd"
-      },
-      "original": {
-        "type": "git",
-        "url": "https://git.sr.ht/~rycee/nmd"
-      }
-    },
-    "root": {
-      "inputs": {
-        "selfhostblocks": "selfhostblocks"
-      }
-    },
-    "selfhostblocks": {
-      "inputs": {
-        "flake-utils": "flake-utils",
-        "nix-flake-tests": "nix-flake-tests",
-        "nixpkgs": "nixpkgs",
-        "nmdsrc": "nmdsrc",
-        "sops-nix": "sops-nix"
-      },
-      "locked": {
-        "lastModified": 1709267447,
-        "narHash": "sha256-5Q467FhpS18L/+5iB3wsWaR9tBqdzNt0fpdkZJNqNxc=",
-        "owner": "ibizaman",
-        "repo": "selfhostblocks",
-        "rev": "fa206d0e1515fb0e49393e7ada6d7e5c6ec1df58",
-        "type": "github"
-      },
-      "original": {
-        "owner": "ibizaman",
-        "repo": "selfhostblocks",
-        "type": "github"
-      }
-    },
-    "sops-nix": {
-      "inputs": {
-        "nixpkgs": "nixpkgs_2",
-        "nixpkgs-stable": "nixpkgs-stable"
-      },
-      "locked": {
-        "lastModified": 1708987867,
-        "narHash": "sha256-k2lDaDWNTU5sBVHanYzjDKVDmk29RHIgdbbXu5sdzBA=",
-        "owner": "Mic92",
-        "repo": "sops-nix",
-        "rev": "a1c8de14f60924fafe13aea66b46157f0150f4cf",
-        "type": "github"
-      },
-      "original": {
-        "owner": "Mic92",
-        "repo": "sops-nix",
-        "type": "github"
-      }
-    },
-    "systems": {
-      "locked": {
-        "lastModified": 1681028828,
-        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
-        "owner": "nix-systems",
-        "repo": "default",
-        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
-        "type": "github"
-      },
-      "original": {
-        "owner": "nix-systems",
-        "repo": "default",
-        "type": "github"
-      }
-    }
-  },
-  "root": "root",
-  "version": 7
-}

demo/homeassistant/flake.nix

@@ -3,157 +3,163 @@
 
   inputs = {
     selfhostblocks.url = "github:ibizaman/selfhostblocks";
+    sops-nix.url = "github:Mic92/sops-nix";
   };
 
-  outputs = inputs@{ self, selfhostblocks, ... }:
+  outputs =
+    inputs@{
+      self,
+      selfhostblocks,
+      sops-nix,
+    }:
     let
-      basic = { config, ...  }: {
-        imports = [
-          ./configuration.nix
-          selfhostblocks.inputs.sops-nix.nixosModules.default
-          selfhostblocks.nixosModules.x86_64-linux.default
-        ];
+      system = "x86_64-linux";
+      nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
 
-        shb.home-assistant = {
-          enable = true;
-          domain = "example.com";
-          subdomain = "ha";
-          config = {
-            name = "SHB Home Assistant";
-            country.source = config.sops.secrets."home-assistant/country".path;
-            latitude.source = config.sops.secrets."home-assistant/latitude".path;
-            longitude.source = config.sops.secrets."home-assistant/longitude".path;
-            time_zone.source = config.sops.secrets."home-assistant/time_zone".path;
-            unit_system = "metric";
+      basic =
+        { config, ... }:
+        {
+          imports = [
+            ./configuration.nix
+            selfhostblocks.nixosModules.authelia
+            selfhostblocks.nixosModules.home-assistant
+            selfhostblocks.nixosModules.sops
+            selfhostblocks.nixosModules.ssl
+            sops-nix.nixosModules.default
+          ];
+
+          sops.defaultSopsFile = ./secrets.yaml;
+
+          shb.home-assistant = {
+            enable = true;
+            domain = "example.com";
+            subdomain = "ha";
+            config = {
+              name = "SHB Home Assistant";
+              country.source = config.shb.sops.secret."home-assistant/country".result.path;
+              latitude.source = config.shb.sops.secret."home-assistant/latitude".result.path;
+              longitude.source = config.shb.sops.secret."home-assistant/longitude".result.path;
+              time_zone.source = config.shb.sops.secret."home-assistant/time_zone".result.path;
+              unit_system = "metric";
+            };
+          };
+          shb.sops.secret."home-assistant/country".request = {
+            mode = "0440";
+            owner = "hass";
+            group = "hass";
+            restartUnits = [ "home-assistant.service" ];
+          };
+          shb.sops.secret."home-assistant/latitude".request = {
+            mode = "0440";
+            owner = "hass";
+            group = "hass";
+            restartUnits = [ "home-assistant.service" ];
+          };
+          shb.sops.secret."home-assistant/longitude".request = {
+            mode = "0440";
+            owner = "hass";
+            group = "hass";
+            restartUnits = [ "home-assistant.service" ];
+          };
+          shb.sops.secret."home-assistant/time_zone".request = {
+            mode = "0440";
+            owner = "hass";
+            group = "hass";
+            restartUnits = [ "home-assistant.service" ];
+          };
+
+          nixpkgs.config.permittedInsecurePackages = [
+            "openssl-1.1.1w"
+          ];
+        };
+
+      ldap =
+        { config, ... }:
+        {
+          shb.lldap = {
+            enable = true;
+            domain = "example.com";
+            subdomain = "ldap";
+            ldapPort = 3890;
+            webUIListenPort = 17170;
+            dcdomain = "dc=example,dc=com";
+            ldapUserPassword.result = config.shb.sops.secret."lldap/user_password".result;
+            jwtSecret.result = config.shb.sops.secret."lldap/jwt_secret".result;
+          };
+          shb.sops.secret."lldap/user_password".request = config.shb.lldap.ldapUserPassword.request;
+          shb.sops.secret."lldap/jwt_secret".request = config.shb.lldap.jwtSecret.request;
+
+          shb.home-assistant.ldap = {
+            enable = true;
+            host = "127.0.0.1";
+            port = config.shb.lldap.webUIListenPort;
+            userGroup = "homeassistant_user";
           };
         };
-        sops.secrets."home-assistant/country" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0440";
-          owner = "hass";
-          group = "hass";
-          restartUnits = [ "home-assistant.service" ];
-        };
-        sops.secrets."home-assistant/latitude" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0440";
-          owner = "hass";
-          group = "hass";
-          restartUnits = [ "home-assistant.service" ];
-        };
-        sops.secrets."home-assistant/longitude" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0440";
-          owner = "hass";
-          group = "hass";
-          restartUnits = [ "home-assistant.service" ];
-        };
-        sops.secrets."home-assistant/time_zone" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0440";
-          owner = "hass";
-          group = "hass";
-          restartUnits = [ "home-assistant.service" ];
-        };
-
-        nixpkgs.config.permittedInsecurePackages = [
-          "openssl-1.1.1w"
-        ];
-      };
-
-      ldap = { config, ...  }: {
-        shb.ldap = {
-          enable = true;
-          domain = "example.com";
-          subdomain = "ldap";
-          ldapPort = 3890;
-          webUIListenPort = 17170;
-          dcdomain = "dc=example,dc=com";
-          ldapUserPasswordFile = config.sops.secrets."lldap/user_password".path;
-          jwtSecretFile = config.sops.secrets."lldap/jwt_secret".path;
-        };
-        sops.secrets."lldap/user_password" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0440";
-          owner = "lldap";
-          group = "lldap";
-          restartUnits = [ "lldap.service" ];
-        };
-        sops.secrets."lldap/jwt_secret" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0440";
-          owner = "lldap";
-          group = "lldap";
-          restartUnits = [ "lldap.service" ];
-        };
-
-        shb.home-assistant.ldap = {
-          enable = true;
-          host = "127.0.0.1";
-          port = config.shb.ldap.webUIListenPort;
-          userGroup = "homeassistant_user";
-        };
-      };
 
       sopsConfig = {
         sops.age.keyFile = "/etc/sops/my_key";
         environment.etc."sops/my_key".source = ./keys.txt;
       };
     in
-      {
-        nixosConfigurations = {
-          basic = selfhostblocks.inputs.nixpkgs.lib.nixosSystem {
-            system = "x86_64-linux";
-            modules = [
-              basic
-              sopsConfig
-            ];
-          };
-
-          ldap = selfhostblocks.inputs.nixpkgs.lib.nixosSystem {
-            system = "x86_64-linux";
-            modules = [
-              basic
-              ldap
-              sopsConfig
-            ];
-          };
+    {
+      nixosConfigurations = {
+        basic = nixpkgs'.nixosSystem {
+          system = "x86_64-linux";
+          modules = [
+            basic
+            sopsConfig
+          ];
         };
 
-        colmena = {
-          meta = {
-            nixpkgs = import selfhostblocks.inputs.nixpkgs {
-              system = "x86_64-linux";
-            };
-            specialArgs = inputs;
-          };
-
-          basic = { config, ... }: {
-            imports = [
-              basic
-            ];
-
-            # Used by colmena to know which target host to deploy to.
-            deployment = {
-              targetHost = "example";
-              targetUser = "nixos";
-              targetPort = 2222;
-            };
-          };
-
-          ldap = { config, ... }: {
-            imports = [
-              basic
-              ldap
-            ];
-
-            # Used by colmena to know which target host to deploy to.
-            deployment = {
-              targetHost = "example";
-              targetUser = "nixos";
-              targetPort = 2222;
-            };
-          };
+        ldap = nixpkgs'.nixosSystem {
+          system = "x86_64-linux";
+          modules = [
+            basic
+            ldap
+            sopsConfig
+          ];
         };
       };
+
+      colmena = {
+        meta = {
+          nixpkgs = import nixpkgs' {
+            system = "x86_64-linux";
+          };
+          specialArgs = inputs;
+        };
+
+        basic =
+          { config, ... }:
+          {
+            imports = [
+              basic
+            ];
+
+            # Used by colmena to know which target host to deploy to.
+            deployment = {
+              targetHost = "example";
+              targetUser = "nixos";
+              targetPort = 2222;
+            };
+          };
+
+        ldap =
+          { config, ... }:
+          {
+            imports = [
+              basic
+              ldap
+            ];
+
+            # Used by colmena to know which target host to deploy to.
+            deployment = {
+              targetHost = "example";
+              targetUser = "nixos";
+              targetPort = 2222;
+            };
+          };
+      };
+    };
 }

demo/homeassistant/hardware-configuration.nix

@@ -3,52 +3,65 @@
 # Do not modify this file!  It was generated by ‘nixos-generate-config’
 # and may be overwritten by future invocations.  Please make changes
 # to /etc/nixos/configuration.nix instead.
-{ config, lib, pkgs, modulesPath, ... }:
+{
+  config,
+  lib,
+  pkgs,
+  modulesPath,
+  ...
+}:
 
 {
-  imports =
-    [ (modulesPath + "/profiles/qemu-guest.nix")
-    ];
+  imports = [
+    (modulesPath + "/profiles/qemu-guest.nix")
+  ];
 
-  boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" "virtio_pci" "floppy" "sr_mod" "virtio_blk" ];
+  boot.initrd.availableKernelModules = [
+    "ata_piix"
+    "uhci_hcd"
+    "virtio_pci"
+    "floppy"
+    "sr_mod"
+    "virtio_blk"
+  ];
   boot.initrd.kernelModules = [ ];
   boot.kernelModules = [ "kvm-intel" ];
   boot.extraModulePackages = [ ];
 
-  fileSystems."/" =
-    { device = "/dev/vda";
-      fsType = "ext4";
-    };
+  fileSystems."/" = {
+    device = "/dev/vda";
+    fsType = "ext4";
+  };
 
-  fileSystems."/nix/.ro-store" =
-    { device = "nix-store";
-      fsType = "9p";
-    };
+  fileSystems."/nix/.ro-store" = {
+    device = "nix-store";
+    fsType = "9p";
+  };
 
-  fileSystems."/nix/.rw-store" =
-    { device = "tmpfs";
-      fsType = "tmpfs";
-    };
+  fileSystems."/nix/.rw-store" = {
+    device = "tmpfs";
+    fsType = "tmpfs";
+  };
 
-  fileSystems."/tmp/shared" =
-    { device = "shared";
-      fsType = "9p";
-    };
+  fileSystems."/tmp/shared" = {
+    device = "shared";
+    fsType = "9p";
+  };
 
-  fileSystems."/tmp/xchg" =
-    { device = "xchg";
-      fsType = "9p";
-    };
+  fileSystems."/tmp/xchg" = {
+    device = "xchg";
+    fsType = "9p";
+  };
 
-  fileSystems."/nix/store" =
-    { device = "overlay";
-      fsType = "overlay";
-    };
+  fileSystems."/nix/store" = {
+    device = "overlay";
+    fsType = "overlay";
+  };
 
-  fileSystems."/boot" =
-    { device = "/dev/vdb2";
-      fsType = "vfat";
-    };
+  fileSystems."/boot" = {
+    device = "/dev/vdb2";
+    fsType = "vfat";
+  };
 
   swapDevices = [ ];
 

demo/minimal/flake.nix

@@ -0,0 +1,223 @@
+{
+  description = "Minimal example to setup SelfHostBlocks";
+
+  inputs = {
+    selfhostblocks.url = "github:ibizaman/selfhostblocks";
+
+    sops-nix = {
+      url = "github:Mic92/sops-nix";
+    };
+  };
+
+  outputs =
+    {
+      self,
+      selfhostblocks,
+      sops-nix,
+    }:
+    {
+      nixosConfigurations =
+        let
+          system = "x86_64-linux";
+          nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
+
+          # This module makes the assertions happy and the build succeed.
+          # This is of course wrong and will not work on any real system.
+          filesystemModule = {
+            fileSystems."/" = {
+              device = "/dev/null";
+              fsType = "none";
+            };
+            boot.loader.grub.devices = [ "/dev/null" ];
+          };
+        in
+        {
+          # Test with:
+          #   nix build .#nixosConfigurations.minimal.config.system.build.toplevel
+          minimal = nixpkgs'.nixosSystem {
+            inherit system;
+            modules = [
+              selfhostblocks.nixosModules.default
+              filesystemModule
+              # This modules showcases the use of SHB's lib.
+              (
+                {
+                  config,
+                  lib,
+                  shb,
+                  ...
+                }:
+                {
+                  options.myOption = lib.mkOption {
+                    # Using provided nixosSystem directly.
+                    # SHB's lib is available under `shb` thanks to the overlay.
+                    type = shb.secretFileType;
+                  };
+                  config = {
+                    myOption.source = "/a/path";
+                    # Use the option.
+                    environment.etc.myOption.text = config.myOption.source;
+                  };
+                }
+              )
+            ];
+          };
+
+          # Test with:
+          #   nix build .#nixosConfigurations.sops.config.system.build.toplevel
+          #   nix eval .#nixosConfigurations.sops.config.myOption
+          sops = nixpkgs'.nixosSystem {
+            inherit system;
+            modules = [
+              selfhostblocks.nixosModules.default
+              selfhostblocks.nixosModules.sops
+              sops-nix.nixosModules.default
+              filesystemModule
+              # This modules showcases the use of SHB's lib.
+              (
+                {
+                  config,
+                  lib,
+                  shb,
+                  ...
+                }:
+                {
+                  options.myOption = lib.mkOption {
+                    # Using provided nixosSystem directly.
+                    # SHB's lib is available under `shb` thanks to the overlay.
+                    type = shb.secretFileType;
+                  };
+                  config = {
+                    myOption.source = "/a/path";
+                    # Use the option.
+                    environment.etc.myOption.text = config.myOption.source;
+                  };
+                }
+              )
+            ];
+          };
+
+          # This example shows how to import the nixosSystem patches to nixpkgs manually.
+          #
+          # Test with:
+          #   nix build .#nixosConfigurations.lowlevel.config.system.build.toplevel
+          #   nix eval .#nixosConfigurations.lowlevel.config.myOption
+          lowlevel =
+            let
+              # We must import nixosSystem directly from the patched nixpkgs
+              # otherwise we do not get the patches.
+              nixosSystem' = import "${nixpkgs'}/nixos/lib/eval-config.nix";
+            in
+            nixosSystem' {
+              inherit system;
+              modules = [
+                selfhostblocks.nixosModules.default
+                filesystemModule
+                # This modules showcases the use of SHB's lib.
+                (
+                  {
+                    config,
+                    lib,
+                    shb,
+                    ...
+                  }:
+                  {
+                    options.myOption = lib.mkOption {
+                      # Using provided nixosSystem directly.
+                      # SHB's lib is available under `shb` thanks to the overlay.
+                      type = shb.secretFileType;
+                    };
+                    config = {
+                      myOption.source = "/a/path";
+                      # Use the option.
+                      environment.etc.myOption.text = config.myOption.source;
+                    };
+                  }
+                )
+              ];
+            };
+
+          # This example shows how to apply patches to nixpkgs manually.
+          #
+          # Test with:
+          #   nix build .#nixosConfigurations.manual.config.system.build.toplevel
+          #   nix eval .#nixosConfigurations.manual.config.myOption
+          manual =
+            let
+              pkgs = import selfhostblocks.inputs.nixpkgs {
+                inherit system;
+              };
+              nixpkgs' = pkgs.applyPatches {
+                name = "nixpkgs-patched";
+                src = selfhostblocks.inputs.nixpkgs;
+                patches = selfhostblocks.lib.${system}.patches;
+              };
+              # We must import nixosSystem directly from the patched nixpkgs
+              # otherwise we do not get the patches.
+              nixosSystem' = import "${nixpkgs'}/nixos/lib/eval-config.nix";
+            in
+            nixosSystem' {
+              inherit system;
+              modules = [
+                selfhostblocks.nixosModules.default
+                filesystemModule
+                # This modules showcases the use of SHB's lib.
+                (
+                  {
+                    config,
+                    lib,
+                    shb,
+                    ...
+                  }:
+                  {
+                    options.myOption = lib.mkOption {
+                      # Using provided nixosSystem directly.
+                      # SHB's lib is available under `shb` thanks to the overlay.
+                      type = shb.secretFileType;
+                    };
+                    config = {
+                      myOption.source = "/a/path";
+                      # Use the option.
+                      environment.etc.myOption.text = config.myOption.source;
+                    };
+                  }
+                )
+              ];
+            };
+
+          # Test with:
+          #   nix build .#nixosConfigurations.contractsDirect.config.system.build.toplevel
+          contractsDirect =
+            let
+              nixosSystem' = import "${selfhostblocks.inputs.nixpkgs}/nixos/lib/eval-config.nix";
+            in
+            nixosSystem' {
+              inherit system;
+              modules = [
+                filesystemModule
+                (import "${selfhostblocks}/lib/module.nix")
+                (
+                  {
+                    config,
+                    lib,
+                    shb,
+                    ...
+                  }:
+                  {
+                    options.myOption = lib.mkOption {
+                      # Using provided nixosSystem directly.
+                      # SHB's lib is available under `shb` thanks to the overlay.
+                      type = shb.secretFileType;
+                    };
+                    config = {
+                      myOption.source = "/a/path";
+                      # Use the option.
+                      environment.etc.myOption.text = config.myOption.source;
+                    };
+                  }
+                )
+              ];
+            };
+        };
+    };
+}

demo/nextcloud/README.md

@@ -112,7 +112,7 @@ ssh -F ssh_config example
 
 :::: {.note}
 This section corresponds to the `basic` section of the [Nextcloud
-manual](services-nextcloud.html#services-nextcloud-server-usage-basic).
+manual](services-nextcloud.html#services-nextcloudserver-usage-basic).
 ::::
 
 Assuming you already deployed the `basic` demo, now you must add the following entry to the
@@ -143,7 +143,7 @@ This is the admin user of Nextcloud and that's the end of the `basic` demo.
 
 :::: {.note}
 This section corresponds to the `ldap` section of the [Nextcloud
-manual](services-nextcloud.html#services-nextcloud-server-usage-ldap).
+manual](services-nextcloud.html#services-nextcloudserver-usage-ldap).
 ::::
 
 Assuming you already deployed the `ldap` demo, now you must add the following entry to the
@@ -172,6 +172,8 @@ Create the group `nextcloud_user` and a create a user and assign them to that gr
 
 Finally, go to [http://n.example.com/login:8080](http://n.example.com/login:8080) and login with the user and
 password you just created above.
+You might need to wait a minute or two until Nextcloud initialized correctly.
+Until then, you'll get a 502 Bad Gateway error.
 
 Nextcloud doesn't like being run without SSL protection, which this demo does not setup, so you
 might see errors loading scripts. See the `sso` demo for SSL.
@@ -182,11 +184,12 @@ This is the end of the `ldap` demo.
 
 :::: {.note}
 This section corresponds to the `sso` section of the [Nextcloud
-manual](services-nextcloud.html#services-nextcloud-server-usage-oidc).
+manual](services-nextcloud.html#services-nextcloudserver-usage-oidc).
 ::::
 
-At this point, it is assumed you already deployed the `sso` demo. There is no host to add to
-`/etc/hosts` here. Instead, there is a `dnsmasq` server running in the VM and you must create a
+At this point, it is assumed you already deployed the `sso` demo. This time, we cannot simply edit local
+`/etc/hosts`, because Nextcloud SSO addon must be able to connect to Authelia by domain name
+(`auth.example.com`). Instead, there is a `dnsmasq` server running in the VM and you must create a
 SOCKS proxy to connect to it like so:
 
 ```bash
@@ -217,7 +220,7 @@ This is the end of the `sso` demo.
 ### Files {#demo-nextcloud-tips-files}
 
 - [`flake.nix`](./flake.nix): nix entry point, defines the target hosts for
-  [colmena](https://colmena.cli.rs) to deploy to as well as the selfhostblock's config for setting
+  [colmena](https://colmena.cli.rs) to deploy to as well as the selfhostblocks' config for setting
   up Nextcloud and the auxiliary services.
 - [`configuration.nix`](./configuration.nix): defines all configuration required for colmena
   to deploy to the VM. The file has comments if you're interested.

demo/nextcloud/configuration.nix

@@ -5,17 +5,17 @@ let
   targetPort = 2222;
 in
 {
-  imports =
-    [ # Include the results of the hardware scan.
-      ./hardware-configuration.nix
-    ];
+  imports = [
+    # Include the results of the hardware scan.
+    ./hardware-configuration.nix
+  ];
 
   boot.loader.grub.enable = true;
   boot.kernelModules = [ "kvm-intel" ];
   system.stateVersion = "22.11";
 
   # Options above are generate by running nixos-generate-config on the VM.
-  
+
   # Needed otherwise deploy will say system won't be able to boot.
   boot.loader.grub.device = "/dev/vdb";
   # Needed to avoid getting into not available disk space in /boot.
@@ -27,7 +27,10 @@ in
 
   # Options above are needed to deploy in a VM.
 
-  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
 
   # We need to create the user we will deploy with.
   users.users.${targetUser} = {
@@ -42,9 +45,11 @@ in
 
   # The user we're deploying with must be able to run sudo without password.
   security.sudo.extraRules = [
-    { users = [ targetUser ];
+    {
+      users = [ targetUser ];
       commands = [
-        { command = "ALL";
+        {
+          command = "ALL";
           options = [ "NOPASSWD" ];
         }
       ];
@@ -60,7 +65,9 @@ in
   services.openssh = {
     enable = true;
     ports = [ targetPort ];
-    permitRootLogin = "no";
-    passwordAuthentication = false;
+    settings = {
+      PermitRootLogin = "no";
+      PasswordAuthentication = false;
+    };
   };
 }

demo/nextcloud/flake.lock

@@ -1,164 +0,0 @@
-{
-  "nodes": {
-    "flake-utils": {
-      "inputs": {
-        "systems": "systems"
-      },
-      "locked": {
-        "lastModified": 1710146030,
-        "narHash": "sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ=",
-        "owner": "numtide",
-        "repo": "flake-utils",
-        "rev": "b1d9ab70662946ef0850d488da1c9019f3a9752a",
-        "type": "github"
-      },
-      "original": {
-        "owner": "numtide",
-        "repo": "flake-utils",
-        "type": "github"
-      }
-    },
-    "nix-flake-tests": {
-      "locked": {
-        "lastModified": 1677844186,
-        "narHash": "sha256-ErJZ/Gs1rxh561CJeWP5bohA2IcTq1rDneu1WT6CVII=",
-        "owner": "antifuchs",
-        "repo": "nix-flake-tests",
-        "rev": "bbd9216bd0f6495bb961a8eb8392b7ef55c67afb",
-        "type": "github"
-      },
-      "original": {
-        "owner": "antifuchs",
-        "repo": "nix-flake-tests",
-        "type": "github"
-      }
-    },
-    "nixpkgs": {
-      "locked": {
-        "lastModified": 1714253743,
-        "narHash": "sha256-mdTQw2XlariysyScCv2tTE45QSU9v/ezLcHJ22f0Nxc=",
-        "owner": "nixos",
-        "repo": "nixpkgs",
-        "rev": "58a1abdbae3217ca6b702f03d3b35125d88a2994",
-        "type": "github"
-      },
-      "original": {
-        "owner": "nixos",
-        "ref": "nixos-unstable",
-        "repo": "nixpkgs",
-        "type": "github"
-      }
-    },
-    "nixpkgs-stable": {
-      "locked": {
-        "lastModified": 1713638189,
-        "narHash": "sha256-q7APLfB6FmmSMI1Su5ihW9IwntBsk2hWNXh8XtSdSIk=",
-        "owner": "NixOS",
-        "repo": "nixpkgs",
-        "rev": "74574c38577914733b4f7a775dd77d24245081dd",
-        "type": "github"
-      },
-      "original": {
-        "owner": "NixOS",
-        "ref": "release-23.11",
-        "repo": "nixpkgs",
-        "type": "github"
-      }
-    },
-    "nixpkgs_2": {
-      "locked": {
-        "lastModified": 1713596654,
-        "narHash": "sha256-LJbHQQ5aX1LVth2ST+Kkse/DRzgxlVhTL1rxthvyhZc=",
-        "owner": "NixOS",
-        "repo": "nixpkgs",
-        "rev": "fd16bb6d3bcca96039b11aa52038fafeb6e4f4be",
-        "type": "github"
-      },
-      "original": {
-        "owner": "NixOS",
-        "ref": "nixpkgs-unstable",
-        "repo": "nixpkgs",
-        "type": "github"
-      }
-    },
-    "nmdsrc": {
-      "flake": false,
-      "locked": {
-        "lastModified": 1705050560,
-        "narHash": "sha256-x3zzcdvhJpodsmdjqB4t5mkVW22V3wqHLOun0KRBzUI=",
-        "ref": "refs/heads/master",
-        "rev": "66d9334933119c36f91a78d565c152a4fdc8d3d3",
-        "revCount": 66,
-        "type": "git",
-        "url": "https://git.sr.ht/~rycee/nmd"
-      },
-      "original": {
-        "type": "git",
-        "url": "https://git.sr.ht/~rycee/nmd"
-      }
-    },
-    "root": {
-      "inputs": {
-        "selfhostblocks": "selfhostblocks"
-      }
-    },
-    "selfhostblocks": {
-      "inputs": {
-        "flake-utils": "flake-utils",
-        "nix-flake-tests": "nix-flake-tests",
-        "nixpkgs": "nixpkgs",
-        "nmdsrc": "nmdsrc",
-        "sops-nix": "sops-nix"
-      },
-      "locked": {
-        "dirtyRev": "710428c5a02a1b6c0789281c493e9555a16e4d22-dirty",
-        "dirtyShortRev": "710428c-dirty",
-        "lastModified": 1714374711,
-        "narHash": "sha256-bn8mImVyK0EQwWHeIK6bx6JvVVQVCnxaUt6aJqWigkY=",
-        "type": "git",
-        "url": "file:///home/timi/Projects/selfhostblocks"
-      },
-      "original": {
-        "owner": "ibizaman",
-        "repo": "selfhostblocks",
-        "type": "github"
-      }
-    },
-    "sops-nix": {
-      "inputs": {
-        "nixpkgs": "nixpkgs_2",
-        "nixpkgs-stable": "nixpkgs-stable"
-      },
-      "locked": {
-        "lastModified": 1713892811,
-        "narHash": "sha256-uIGmA2xq41vVFETCF1WW4fFWFT2tqBln+aXnWrvjGRE=",
-        "owner": "Mic92",
-        "repo": "sops-nix",
-        "rev": "f1b0adc27265274e3b0c9b872a8f476a098679bd",
-        "type": "github"
-      },
-      "original": {
-        "owner": "Mic92",
-        "repo": "sops-nix",
-        "type": "github"
-      }
-    },
-    "systems": {
-      "locked": {
-        "lastModified": 1681028828,
-        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
-        "owner": "nix-systems",
-        "repo": "default",
-        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
-        "type": "github"
-      },
-      "original": {
-        "owner": "nix-systems",
-        "repo": "default",
-        "type": "github"
-      }
-    }
-  },
-  "root": "root",
-  "version": 7
-}

demo/nextcloud/flake.nix

@@ -1,288 +1,260 @@
 {
-  description = "Home Assistant example for Self Host Blocks";
+  description = "Nextcloud example for Self Host Blocks";
 
   inputs = {
     selfhostblocks.url = "github:ibizaman/selfhostblocks";
+    sops-nix.url = "github:Mic92/sops-nix";
   };
 
-  outputs = inputs@{ self, selfhostblocks, ... }:
+  outputs =
+    inputs@{
+      self,
+      selfhostblocks,
+      sops-nix,
+    }:
     let
-      basic = { config, ... }: {
-        imports = [
-          ./configuration.nix
-          selfhostblocks.inputs.sops-nix.nixosModules.default
-          selfhostblocks.nixosModules.x86_64-linux.default
-        ];
+      system = "x86_64-linux";
+      nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
 
-        shb.nextcloud = {
-          enable = true;
-          domain = "example.com";
-          subdomain = "n";
-          dataDir = "/var/lib/nextcloud";
-          tracing = null;
-          defaultPhoneRegion = "US";
+      basic =
+        { config, ... }:
+        {
+          imports = [
+            ./configuration.nix
+            selfhostblocks.nixosModules.authelia
+            selfhostblocks.nixosModules.nextcloud-server
+            selfhostblocks.nixosModules.nginx
+            selfhostblocks.nixosModules.sops
+            selfhostblocks.nixosModules.ssl
+            sops-nix.nixosModules.default
+          ];
 
-          # This option is only needed because we do not access Nextcloud at the default port in the VM.
-          port = 8080;
+          sops.defaultSopsFile = ./secrets.yaml;
 
-          adminPassFile = config.sops.secrets."nextcloud/adminpass".path;
+          shb.nextcloud = {
+            enable = true;
+            domain = "example.com";
+            subdomain = "n";
+            dataDir = "/var/lib/nextcloud";
+            tracing = null;
+            defaultPhoneRegion = "US";
 
-          apps = {
-            previewgenerator.enable = true;
-          };
-        };
+            # This option is only needed because we do not access Nextcloud at the default port in the VM.
+            port = 8080;
 
-        # Secret needed for services.nextcloud.config.adminpassFile.
-        sops.secrets."nextcloud/adminpass" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0440";
-          owner = "nextcloud";
-          group = "nextcloud";
-          restartUnits = [ "phpfpm-nextcloud.service" ];
-        };
+            adminPass.result = config.shb.sops.secret."nextcloud/adminpass".result;
 
-        # Set to true for more debug info with `journalctl -f -u nginx`.
-        shb.nginx.accessLog = true;
-        shb.nginx.debugLog = false;
-      };
-
-      ldap = { config, ... }: {
-        shb.ldap = {
-          enable = true;
-          domain = "example.com";
-          subdomain = "ldap";
-          ldapPort = 3890;
-          webUIListenPort = 17170;
-          dcdomain = "dc=example,dc=com";
-          ldapUserPasswordFile = config.sops.secrets."lldap/user_password".path;
-          jwtSecretFile = config.sops.secrets."lldap/jwt_secret".path;
-        };
-        sops.secrets."lldap/user_password" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0440";
-          owner = "lldap";
-          group = "lldap";
-          restartUnits = [ "lldap.service" ];
-        };
-        sops.secrets."lldap/jwt_secret" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0440";
-          owner = "lldap";
-          group = "lldap";
-          restartUnits = [ "lldap.service" ];
-        };
-
-        shb.nextcloud.apps.ldap = {
-          enable = true;
-          host = "127.0.0.1";
-          port = config.shb.ldap.ldapPort;
-          dcdomain = config.shb.ldap.dcdomain;
-          adminName = "admin";
-          adminPasswordFile = config.sops.secrets."nextcloud/ldap_admin_password".path;
-          userGroup = "nextcloud_user";
-        };
-
-        # Secret needed for LDAP app.
-        sops.secrets."nextcloud/ldap_admin_password" = {
-          sopsFile = ./secrets.yaml;
-          key = "lldap/user_password";
-          mode = "0400";
-          owner = "nextcloud";
-          group = "nextcloud";
-          restartUnits = [ "nextcloud-setup.service" ];
-        };
-      };
-
-      sso = { config, ... }: {
-        shb.certs = {
-          cas.selfsigned.myca = {
-            name = "My CA";
-          };
-          certs.selfsigned = {
-            n = {
-              ca = config.shb.certs.cas.selfsigned.myca;
-              domain = "*.example.com";
+            apps = {
+              previewgenerator.enable = true;
             };
           };
+          shb.sops.secret."nextcloud/adminpass".request = config.shb.nextcloud.adminPass.request;
+
+          # Set to true for more debug info with `journalctl -f -u nginx`.
+          shb.nginx.accessLog = true;
+          shb.nginx.debugLog = false;
         };
 
-        services.dnsmasq = {
-          enable = true;
-          settings = {
-            domain-needed = true;
-            # no-resolv = true;
-            bogus-priv = true;
-            address =
-              map (hostname: "/${hostname}/127.0.0.1") [
+      ldap =
+        { config, ... }:
+        {
+          shb.lldap = {
+            enable = true;
+            domain = "example.com";
+            subdomain = "ldap";
+            ldapPort = 3890;
+            webUIListenPort = 17170;
+            dcdomain = "dc=example,dc=com";
+            ldapUserPassword.result = config.shb.sops.secret."lldap/user_password".result;
+            jwtSecret.result = config.shb.sops.secret."lldap/jwt_secret".result;
+          };
+          shb.sops.secret."lldap/user_password".request = config.shb.lldap.ldapUserPassword.request;
+          shb.sops.secret."lldap/jwt_secret".request = config.shb.lldap.jwtSecret.request;
+
+          shb.nextcloud.apps.ldap = {
+            enable = true;
+            host = "127.0.0.1";
+            port = config.shb.lldap.ldapPort;
+            dcdomain = config.shb.lldap.dcdomain;
+            adminName = "admin";
+            adminPassword.result = config.shb.sops.secret."nextcloud/ldap_admin_password".result;
+            userGroup = "nextcloud_user";
+          };
+          shb.sops.secret."nextcloud/ldap_admin_password" = {
+            request = config.shb.nextcloud.apps.ldap.adminPassword.request;
+            settings.key = "lldap/user_password";
+          };
+        };
+
+      sso =
+        { config, lib, ... }:
+        {
+          shb.certs = {
+            cas.selfsigned.myca = {
+              name = "My CA";
+            };
+            certs.selfsigned = {
+              n = {
+                ca = config.shb.certs.cas.selfsigned.myca;
+                domain = "*.example.com";
+                group = "nginx";
+              };
+            };
+          };
+          shb.nextcloud = {
+            port = lib.mkForce null;
+            ssl = config.shb.certs.certs.selfsigned.n;
+          };
+          shb.lldap.ssl = config.shb.certs.certs.selfsigned.n;
+
+          services.dnsmasq = {
+            enable = true;
+            settings = {
+              domain-needed = true;
+              # no-resolv = true;
+              bogus-priv = true;
+              address = map (hostname: "/${hostname}/127.0.0.1") [
                 "example.com"
                 "n.example.com"
                 "ldap.example.com"
                 "auth.example.com"
               ];
+            };
+          };
+
+          shb.authelia = {
+            enable = true;
+            domain = "example.com";
+            subdomain = "auth";
+            ssl = config.shb.certs.certs.selfsigned.n;
+            ldapPort = config.shb.lldap.ldapPort;
+            ldapHostname = "127.0.0.1";
+            dcdomain = config.shb.lldap.dcdomain;
+
+            secrets = {
+              jwtSecret.result = config.shb.sops.secret."authelia/jwt_secret".result;
+              ldapAdminPassword.result = config.shb.sops.secret."authelia/ldap_admin_password".result;
+              sessionSecret.result = config.shb.sops.secret."authelia/session_secret".result;
+              storageEncryptionKey.result = config.shb.sops.secret."authelia/storage_encryption_key".result;
+              identityProvidersOIDCHMACSecret.result = config.shb.sops.secret."authelia/hmac_secret".result;
+              identityProvidersOIDCIssuerPrivateKey.result = config.shb.sops.secret."authelia/private_key".result;
+            };
+          };
+          shb.sops.secret."authelia/jwt_secret".request = config.shb.authelia.secrets.jwtSecret.request;
+          shb.sops.secret."authelia/ldap_admin_password" = {
+            request = config.shb.authelia.secrets.ldapAdminPassword.request;
+            settings.key = "lldap/user_password";
+          };
+          shb.sops.secret."authelia/session_secret".request =
+            config.shb.authelia.secrets.sessionSecret.request;
+          shb.sops.secret."authelia/storage_encryption_key".request =
+            config.shb.authelia.secrets.storageEncryptionKey.request;
+          shb.sops.secret."authelia/hmac_secret".request =
+            config.shb.authelia.secrets.identityProvidersOIDCHMACSecret.request;
+          shb.sops.secret."authelia/private_key".request =
+            config.shb.authelia.secrets.identityProvidersOIDCIssuerPrivateKey.request;
+
+          shb.nextcloud.apps.sso = {
+            enable = true;
+            endpoint = "https://${config.shb.authelia.subdomain}.${config.shb.authelia.domain}";
+            clientID = "nextcloud";
+            fallbackDefaultAuth = true;
+
+            secret.result = config.shb.sops.secret."nextcloud/sso/secret".result;
+            secretForAuthelia.result = config.shb.sops.secret."authelia/nextcloud_sso_secret".result;
+          };
+          shb.sops.secret."nextcloud/sso/secret".request = config.shb.nextcloud.apps.sso.secret.request;
+          shb.sops.secret."authelia/nextcloud_sso_secret" = {
+            request = config.shb.nextcloud.apps.sso.secretForAuthelia.request;
+            settings.key = "nextcloud/sso/secret";
           };
         };
 
-        shb.authelia = {
-          enable = true;
-          domain = "example.com";
-          subdomain = "auth";
-          ssl = config.shb.certs.certs.selfsigned.n;
-
-          ldapEndpoint = "ldap://127.0.0.1:${builtins.toString config.shb.ldap.ldapPort}";
-          dcdomain = config.shb.ldap.dcdomain;
-
-          secrets = {
-            jwtSecretFile = config.sops.secrets."authelia/jwt_secret".path;
-            ldapAdminPasswordFile = config.sops.secrets."authelia/ldap_admin_password".path;
-            sessionSecretFile = config.sops.secrets."authelia/session_secret".path;
-            storageEncryptionKeyFile = config.sops.secrets."authelia/storage_encryption_key".path;
-            identityProvidersOIDCHMACSecretFile = config.sops.secrets."authelia/hmac_secret".path;
-            identityProvidersOIDCIssuerPrivateKeyFile = config.sops.secrets."authelia/private_key".path;
-          };
-        };
-        sops.secrets."authelia/jwt_secret" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0400";
-          owner = config.shb.authelia.autheliaUser;
-          restartUnits = [ "authelia.service" ];
-        };
-        # Here we use the password defined in the lldap/user_password field in the secrets.yaml file
-        # and sops-nix will write it to "/run/secrets/authelia/ldap_admin_password".
-        sops.secrets."authelia/ldap_admin_password" = {
-          sopsFile = ./secrets.yaml;
-          key = "lldap/user_password";
-          mode = "0400";
-          owner = config.shb.authelia.autheliaUser;
-          restartUnits = [ "authelia.service" ];
-        };
-        sops.secrets."authelia/session_secret" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0400";
-          owner = config.shb.authelia.autheliaUser;
-          restartUnits = [ "authelia.service" ];
-        };
-        sops.secrets."authelia/storage_encryption_key" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0400";
-          owner = config.shb.authelia.autheliaUser;
-          restartUnits = [ "authelia.service" ];
-        };
-        sops.secrets."authelia/hmac_secret" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0400";
-          owner = config.shb.authelia.autheliaUser;
-          restartUnits = [ "authelia.service" ];
-        };
-        sops.secrets."authelia/private_key" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0400";
-          owner = config.shb.authelia.autheliaUser;
-          restartUnits = [ "authelia.service" ];
-        };
-
-        shb.nextcloud.apps.sso = {
-          enable = true;
-          endpoint = "https://${config.shb.authelia.subdomain}.${config.shb.authelia.domain}";
-          clientID = "nextcloud";
-          fallbackDefaultAuth = true;
-
-          secretFile = config.sops.secrets."nextcloud/sso/secret".path;
-          secretFileForAuthelia = config.sops.secrets."authelia/nextcloud_sso_secret".path;
-        };
-
-        sops.secrets."nextcloud/sso/secret" = {
-          sopsFile = ./secrets.yaml;
-          mode = "0400";
-          owner = "nextcloud";
-          restartUnits = [ "nextcloud-setup.service" ];
-        };
-        sops.secrets."authelia/nextcloud_sso_secret" = {
-          sopsFile = ./secrets.yaml;
-          key = "nextcloud/sso/secret";
-          mode = "0400";
-          owner = config.shb.authelia.autheliaUser;
-        };
-      };
-
       sopsConfig = {
         sops.age.keyFile = "/etc/sops/my_key";
         environment.etc."sops/my_key".source = ./keys.txt;
       };
     in
-      {
-        nixosConfigurations = {
-          basic = selfhostblocks.inputs.nixpkgs.lib.nixosSystem {
-            system = "x86_64-linux";
-            modules = [
-              sopsConfig
-              basic
-            ];
-          };
-          ldap = selfhostblocks.inputs.nixpkgs.lib.nixosSystem {
-            system = "x86_64-linux";
-            modules = [
-              sopsConfig
-              basic
-              ldap
-            ];
-          };
-          sso = selfhostblocks.inputs.nixpkgs.lib.nixosSystem {
-            system = "x86_64-linux";
-            modules = [
-              sopsConfig
-              basic
-              ldap
-              sso
-            ];
-          };
+    {
+      nixosConfigurations = {
+        basic = nixpkgs'.nixosSystem {
+          system = "x86_64-linux";
+          modules = [
+            sopsConfig
+            basic
+          ];
         };
-
-        colmena = {
-          meta = {
-            nixpkgs = import selfhostblocks.inputs.nixpkgs {
-              system = "x86_64-linux";
-            };
-            specialArgs = inputs;
-          };
-
-          basic = { config, ... }: {
-            imports = [
-              basic
-            ];
-
-            deployment = {
-              targetHost = "example";
-              targetUser = "nixos";
-              targetPort = 2222;
-            };
-          };
-
-          ldap = { config, ... }: {
-            imports = [
-              basic
-              ldap
-            ];
-
-            deployment = {
-              targetHost = "example";
-              targetUser = "nixos";
-              targetPort = 2222;
-            };
-          };
-
-          sso = { config, ... }: {
-            imports = [
-              basic
-              ldap
-              sso
-            ];
-
-            deployment = {
-              targetHost = "example";
-              targetUser = "nixos";
-              targetPort = 2222;
-            };
-          };
+        ldap = nixpkgs'.nixosSystem {
+          system = "x86_64-linux";
+          modules = [
+            sopsConfig
+            basic
+            ldap
+          ];
+        };
+        sso = nixpkgs'.nixosSystem {
+          system = "x86_64-linux";
+          modules = [
+            sopsConfig
+            basic
+            ldap
+            sso
+          ];
         };
       };
+
+      colmena = {
+        meta = {
+          nixpkgs = import nixpkgs' {
+            system = "x86_64-linux";
+          };
+          specialArgs = inputs;
+        };
+
+        basic =
+          { config, ... }:
+          {
+            imports = [
+              basic
+            ];
+
+            deployment = {
+              targetHost = "example";
+              targetUser = "nixos";
+              targetPort = 2222;
+            };
+          };
+
+        ldap =
+          { config, ... }:
+          {
+            imports = [
+              basic
+              ldap
+            ];
+
+            deployment = {
+              targetHost = "example";
+              targetUser = "nixos";
+              targetPort = 2222;
+            };
+          };
+
+        sso =
+          { config, ... }:
+          {
+            imports = [
+              basic
+              ldap
+              sso
+            ];
+
+            deployment = {
+              targetHost = "example";
+              targetUser = "nixos";
+              targetPort = 2222;
+            };
+          };
+      };
+    };
 }

demo/nextcloud/hardware-configuration.nix

@@ -3,52 +3,65 @@
 # Do not modify this file!  It was generated by ‘nixos-generate-config’
 # and may be overwritten by future invocations.  Please make changes
 # to /etc/nixos/configuration.nix instead.
-{ config, lib, pkgs, modulesPath, ... }:
+{
+  config,
+  lib,
+  pkgs,
+  modulesPath,
+  ...
+}:
 
 {
-  imports =
-    [ (modulesPath + "/profiles/qemu-guest.nix")
-    ];
+  imports = [
+    (modulesPath + "/profiles/qemu-guest.nix")
+  ];
 
-  boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" "virtio_pci" "floppy" "sr_mod" "virtio_blk" ];
+  boot.initrd.availableKernelModules = [
+    "ata_piix"
+    "uhci_hcd"
+    "virtio_pci"
+    "floppy"
+    "sr_mod"
+    "virtio_blk"
+  ];
   boot.initrd.kernelModules = [ ];
   boot.kernelModules = [ "kvm-intel" ];
   boot.extraModulePackages = [ ];
 
-  fileSystems."/" =
-    { device = "/dev/vda";
-      fsType = "ext4";
-    };
+  fileSystems."/" = {
+    device = "/dev/vda";
+    fsType = "ext4";
+  };
 
-  fileSystems."/nix/.ro-store" =
-    { device = "nix-store";
-      fsType = "9p";
-    };
+  fileSystems."/nix/.ro-store" = {
+    device = "nix-store";
+    fsType = "9p";
+  };
 
-  fileSystems."/nix/.rw-store" =
-    { device = "tmpfs";
-      fsType = "tmpfs";
-    };
+  fileSystems."/nix/.rw-store" = {
+    device = "tmpfs";
+    fsType = "tmpfs";
+  };
 
-  fileSystems."/tmp/shared" =
-    { device = "shared";
-      fsType = "9p";
-    };
+  fileSystems."/tmp/shared" = {
+    device = "shared";
+    fsType = "9p";
+  };
 
-  fileSystems."/tmp/xchg" =
-    { device = "xchg";
-      fsType = "9p";
-    };
+  fileSystems."/tmp/xchg" = {
+    device = "xchg";
+    fsType = "9p";
+  };
 
-  fileSystems."/nix/store" =
-    { device = "overlay";
-      fsType = "overlay";
-    };
+  fileSystems."/nix/store" = {
+    device = "overlay";
+    fsType = "overlay";
+  };
 
-  fileSystems."/boot" =
-    { device = "/dev/vdb2";
-      fsType = "vfat";
-    };
+  fileSystems."/boot" = {
+    device = "/dev/vdb2";
+    fsType = "vfat";
+  };
 
   swapDevices = [ ];
 

demo/nextcloud/secrets.yaml

@@ -8,7 +8,6 @@ lldap:
     user_password: ENC[AES256_GCM,data:4ImmaC2T1hj6L8tzrxv4d7/I4F9xEA/uuc56QOqkY08=,iv:SljGhXi3SYoMNcR9onwqthOAyFX1D8KsegmWRypbblQ=,tag:Aw+juIV2AM0J+89itNDjVA==,type:str]
     jwt_secret: ENC[AES256_GCM,data:btABIOGRgioXmPe8QirhyozQzhVaAcF2sbB07hevz+Q=,iv:vBOq4Mab3RE69rOA8ZbMX72Gm3KEng6HaCveZrXsIrU=,tag:zkbJ+SeNnzQyAZxOjso8fg==,type:str]
 authelia:
-    ldap_admin_password: ENC[AES256_GCM,data:Ze1FJSl8ZJYCYrULlkwcbDFzxCS4MzujJbCGZasOiWU=,iv:X/su9ty9883+4qmrQhAIe6HDwjFoqHQ43aqd/4ZmtBw=,tag:QeLqUtYlZcHMR+bqRHCb2A==,type:str]
     jwt_secret: ENC[AES256_GCM,data:xom/W92DGS2RafO+olwG8oKAbKPbkPKyZ2mYv0lWqtVAWUFwSoCGLgxe4uHAoGcLosJmDxU/srq+HNPzYORY8+mHn9wMoQgYg2oceLw2xamYdkIzvswof6LoYAV7MaZReYgYXcqMy2LZuU3PnnE4wag3liSuEx4qtJrLKB52ljE=,iv:t5PsBdZDze3/4S8utfnkmiToaorqq5BiJn99JuRirXY=,tag:ZJCszIOpaSwl9Sua8VWHoA==,type:str]
     storage_encryption_key: ENC[AES256_GCM,data:wUmF+0etuhEr3FNy7x0LBJunn1vmWO+IExm/wgkh0CEDWzxblpylC/PGAGgHdlJMQOhUY6tDPD67sJgO2g+yTBB3lfOo/kql0gnGVKQjRMMHqfEEmXK56yXP+J2JePJ6DlaqzdAXko4Tmh4GnRKsswMQZVA5PDOuHHNRcVTCb0E=,iv:wz1Mry7jMwGvD9mF1/PbQsHb/jmm8WOWchLL95YADeY=,tag:AZp43iti+nxW0TYK7MlYNg==,type:str]
     session_secret: ENC[AES256_GCM,data:TSe2YEyXl0Ls8wAynUYRJBQL8mbC1i/31ueuCj7d7ouO9gCX/Igz6OM9EgWigxucsMVQkiUtDCI9DD9B8jFaYGMIiB9FrKQnixigptrIUj210zJ3Aer38GyFxSI541PaBzmnauEo1MtBykjSg93xyI6ivB8FJmmauQOMYNiTYvk=,iv:OBtUCw7BevaF3VQKLJ2HiB828IzJqS27SZUOoAqoD+E=,tag:WfCGlHi6a15AYeSFXnnOVw==,type:str]
@@ -29,8 +28,8 @@ sops:
             dTNrOUhzOSsvRnNSMC9VOTJaY1orWUEK8IcLk/4X7O+ZRosM7KNQNSEgyGkFklRw
             YSutsre5OOEUx1X+hxzu2GF9I4DGcSAbQtzPYBq7qcwxUR+oIXiJyQ==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2024-01-22T06:30:53Z"
-    mac: ENC[AES256_GCM,data:mdCpYLoaMcotuOU8qB7Gj+79ALG4d4HAR0Yw6Y3gf5SFUOc59B4WdK4A3+cgSm3dvRB8HCg9Vo9llEjiOBNVFpBgIjOvUeyAMYNi6ZndS/yr4x3NSL2rPz2s9c+0tm8Qg61T5RtYS/on+gWiUoA+lzXN2uFFWyo09fWF4N5EOQo=,iv:TgdI759YCkgmGAbUtgiV+NoT40Cg8+BcRGH0ZlQZ5SE=,tag:LGgFrlJPNpG+HzQLHDcDUQ==,type:str]
+    lastmodified: "2025-03-17T00:29:32Z"
+    mac: ENC[AES256_GCM,data:eE3F1K/brgKMnixJQo/A/VYjafNLAGKuSq1n8857yjsiNnro/hwDy9jNKLH3a6/5DX/aOjMfZJzgH3ycb7f4771IohrWoDLjymaVdgJXsTITXZaLQyN+QHoOTRbXAJwG1f4Mr2kEAdwK7JLtu9TqX82o2DmBWNRxkkn1Kv5NjiA=,iv:OSAI0b4H40xbzKQbD6F2B5Xu/8enUIclfds8uYH/q3o=,tag:fYTnxx8IQYMyXAeVTUiQ+A==,type:str]
     pgp: []
     unencrypted_suffix: _unencrypted
-    version: 3.8.1
+    version: 3.9.2

docs/blocks.md

@@ -1,43 +1,74 @@
+<!-- Read these docs at https://shb.skarabox.com -->
 # Blocks {#blocks}
 
 Blocks help you self-host apps or services. They implement a specific function like backup or secure
 access through a subdomain. Each block is designed to be usable on its own and to fit nicely with
 others.
 
-In practice, a block implements a [contract](contracts.html) that must be followed to implement a
-specific self-hosting function. It also comes with a unit test and NixOS VM test suite to ensure any
-implementation follows the contract.
+All blocks are implemented under the blocks folder [in the repository](@REPO@/modules/blocks).
 
-As an example, let's take the HTTPS access block which allows for a service to be accessible through
-a specific subdomain. In Nix terms, this block defines at minimum the inputs:
+All services in SHB document how to setup the various blocks provided here.
+For custom services or those not provided by SHB,
+the [Expose a service Recipe](recipes-exposeService.html) explains how to use the blocks here.
 
-- subdomain,
-- domain,
-- and upstream address of the service.
+## Authentication {#blocks-category-authentication}
 
-It defines no outputs but has one major side effect:
+```{=include=} chapters html:into-file=//blocks-authelia.html
+modules/blocks/authelia/docs/default.md
+```
 
-- the service should be accessible through HTTPS at `https://subdomain.domain`.
+```{=include=} chapters html:into-file=//blocks-lldap.html
+modules/blocks/lldap/docs/default.md
+```
 
-Anything that provides the inputs and expected outputs and side effects defined by the block can be
-used to fulfill its contract. In this example, we could use any of Nginx, Caddy, Haproxy or others.
+## Backup {#blocks-category-backup}
 
-Self Host Blocks provides at least one implementation for each block and allows you to use your own
-implementation if you want to, as long as it passes the tests. You can then use blocks to improve
-services you already have deployed.
+```{=include=} chapters html:into-file=//blocks-borgbackup.html
+modules/blocks/borgbackup/docs/default.md
+```
 
-::: {.note}
-Not all blocks are yet documented. You can find all available blocks [in the repository](@REPO@/modules/blocks).
-:::
+```{=include=} chapters html:into-file=//blocks-restic.html
+modules/blocks/restic/docs/default.md
+```
+
+```{=include=} chapters html:into-file=//blocks-sanoid.html
+modules/blocks/sanoid/docs/default.md
+```
+
+## Database {#blocks-category-database}
+
+```{=include=} chapters html:into-file=//blocks-postgresql.html
+modules/blocks/postgresql/docs/default.md
+```
+
+## Secrets {#blocks-category-secrets}
+
+```{=include=} chapters html:into-file=//blocks-sops.html
+modules/blocks/sops/docs/default.md
+```
+
+## Filesystem {#blocks-category-filesystem}
+
+```{=include=} chapters html:into-file=//blocks-zfs.html
+modules/blocks/zfs/docs/default.md
+```
+
+## Network {#blocks-category-network}
 
 ```{=include=} chapters html:into-file=//blocks-ssl.html
 modules/blocks/ssl/docs/default.md
 ```
 
-```{=include=} chapters html:into-file=//blocks-backup.html
-modules/blocks/backup/docs/default.md
+```{=include=} chapters html:into-file=//blocks-nginx.html
+modules/blocks/nginx/docs/default.md
 ```
 
+## Introspection {#blocks-category-introspection}
+
 ```{=include=} chapters html:into-file=//blocks-monitoring.html
 modules/blocks/monitoring/docs/default.md
 ```
+
+```{=include=} chapters html:into-file=//blocks-mitmdump.html
+modules/blocks/mitmdump/docs/default.md
+```

docs/contracts.md

@@ -1,15 +1,30 @@
+<!-- Read these docs at https://shb.skarabox.com -->
 # Contracts {#contracts}
 
+::: {.note}
+An [RFC][] has been created which is the most up-to-date version of contracts.
+The text here is still relevant although the implementation itself has changed a little bit.
+
+[RFC]: https://github.com/NixOS/rfcs/pull/189
+:::
+
 A contract decouples modules that use a functionality from modules that provide it. A first
 intuition for contracts is they are generally related to accessing a shared resource.
 
 A few examples of contracts are generating SSL certificates, creating a user or knowing which files
-and folders to backup. Indeed, when generating certificates, the service using those do not care how
-they were created. They just need to know where the certificate files are located.
+and folders to backup.
+Indeed, when generating certificates, the service using those do not care how they were created.
+They just need to know where the certificate files are located.
 
-In practice, a contract is a set of options that any user of a contract expects to exist. Also, the
-values of these options dictate the behavior of the implementation. This is enforced with NixOS VM
-tests.
+A contract is made between a `requester` module and a `provider` module.
+For example, a `backup` contract can be made between the [Nextcloud service][] and the [Restic service][].
+The former is the `requester` - the one wanted to be backed up -
+and the latter is the `provider` of the contract - the one backing up files.
+The `backup contract` would then say which set of options the `requester` and `provider` modules
+must use to talk to each other.
+
+[Nextcloud service]: ./services-nextcloud.html
+[Restic service]: ./blocks-restic.html
 
 ## Provided contracts {#contracts-provided}
 
@@ -19,55 +34,537 @@ as possible, reducing the quite thick layer that it is now.
 
 Provided contracts are:
 
-- [SSL generator contract](contracts-ssl.html) to generate SSL certificates. Two implementations are provided: self-signed and Let's Encrypt.
+- [SSL generator contract](contracts-ssl.html) to generate SSL certificates.
+  Two providers are implemented: self-signed and Let's Encrypt.
+- [Backup contract][] to backup directories.
+  Two providers are implemented: [BorgBackup][] and [Restic][].
+- [Database Backup contract](contracts-databasebackup.html) to backup database dumps.
+  Two providers are implemented: [BorgBackup][] and [Restic][].
+- [Dataset Backup contract](contracts-datasetbackup.html) to backup ZFS datasets.
+  One provider is implemented: [Sanoid][]
+- [Contract for Secrets](contracts-secret.html) to provide secrets that are deployed outside of the Nix store.
+  One provider is implemented: [SOPS][].
+- [Dashboard contract](contracts-dashboard.html) to show services in a nice user-facing dashboard.
+  One provider is implemented: [Homepage][].
+
+[backup contract]: contracts-backup.html
+[borgbackup]: blocks-borgbackup.html
+[homepage]: services-homepage.html
+[restic]: blocks-restic.html
+[sanoid]: blocks-sanoid.html
+[sops]: blocks-sops.html
 
 ```{=include=} chapters html:into-file=//contracts-ssl.html
 modules/contracts/ssl/docs/default.md
 ```
 
-## Why do we need this new concept? {#contracts-why}
+```{=include=} chapters html:into-file=//contracts-backup.html
+modules/contracts/backup/docs/default.md
+```
 
-Currently in nixpkgs, every module needing access to a shared resource must implement the logic
-needed to setup that resource themselves. Similarly, if the module is mature enough to let the user
-select a particular implementation, the code lives inside that module.
+```{=include=} chapters html:into-file=//contracts-databasebackup.html
+modules/contracts/databasebackup/docs/default.md
+```
+
+```{=include=} chapters html:into-file=//contracts-datasetbackup.html
+modules/contracts/datasetbackup/docs/default.md
+```
+
+```{=include=} chapters html:into-file=//contracts-secret.html
+modules/contracts/secret/docs/default.md
+```
+
+```{=include=} chapters html:into-file=//contracts-dashboard.html
+modules/contracts/dashboard/docs/default.md
+```
+
+## Problem Statement {#contracts-why}
+
+Currently in nixpkgs, every module accessing a shared resource
+must either implement the logic needed to setup that resource themselves
+or either instruct the user how to set it up themselves.
+
+For example, this is what the Nextcloud module looks like.
+It sets up the `nginx module` and a database,
+letting you choose between multiple databases.
 
 ![](./assets/contracts_before.png "A module composed of a core logic and a lot of peripheral logic.")
 
 This has a few disadvantages:
 
-- This leads to a lot of **duplicated code**. If a module wants to support a new implementation of a
-contract, the maintainers of that module must write code to make that happen.
-- This also leads to **tight coupling**. The code written by the maintainers cannot be reused in
-  other modules, apart from copy pasting.
-- There is also a **lack of separation of concerns**. The maintainers of a service must be experts
+_I'm using the Nextcloud module to make the following examples more concrete
+but this applies to all other modules._
+
+- This leads to a lot of **duplicated code**.
+  If the Nextcloud module wants to support a new type of database,
+  the maintainer of the Nextcloud module must do the work.
+  And if another module wants to support it too,
+  the maintainers of that module cannot re-use easily the work
+  of the Nextcloud maintainer,
+  apart from copy-pasting and adapting the code.
+- This also leads to **tight coupling**.
+  The code written to integrate Nextcloud with the Nginx reverse proxy
+  is hard to decouple and make generic.
+  Letting the user choose between Nginx and another reverse proxy
+  will require a lot of work.
+- There is also a **lack of separation of concerns**.
+  The maintainers of a service must be experts
   in all implementations they let the users choose from.
-- Finally, this is **not extensible**. If you, the user of the module, want to use another
-  implementation that is not supported, you are out of luck. You can always dive into the module's
-  code and extend it, but that is not an optimal experience.
+- This is **not extendable**.
+  If you, the user of the module, want to use another
+  implementation that is not supported, you are out of luck.
+  You can always dive into the module's code and extend it with a lot of `mkForce`,
+  but that is not an optimal experience.
+- Finally, there is **no interoperability**.
+  It is not currently possible to integrate the Nextcloud module
+  with an existing database or reverse proxy or other type of shared resource
+  that already exists on a non-NixOS machine.
 
 We do believe that the decoupling contracts provides helps alleviate all the issues outlined above
-which makes it an essential step towards more adoption of Nix, if only in the self hosting scene.
+which makes it an essential step towards better interoperability.
 
 ![](./assets/contracts_after.png "A module containing only logic using peripheral logic through contracts.")
 
 Indeed, contracts allow:
 
-- **Reuse of code**. Since the implementation of a contract lives outside of modules using it, using
-  that implementation elsewhere is trivial.
-- **Loose coupling**. Modules that use a contract do not care how they are implemented, as long as
-  the implementation follows the behavior outlined by the contract.
-- Full **separation of concerns** (see diagram below). Now, each party's concern is separated with a
-  clear boundary. The maintainer of a module using a contract can be different from the maintainers
-  of the implementation, allowing them to be experts in their own respective fields. But more
-  importantly, the contracts themselves can be created and maintained by the community.
-- Full **extensibility**. The final user themselves can choose an implementation, even new custom
-  implementations not available in nixpkgs, without changing existing code.
-- Last but not least, **Testability**. Thanks to NixOS VM test, we can even go one step further by
-  ensuring each implementation of a contract, even custom ones, provides required options and
-  behaves as the contract requires.
+- **Reuse of code**.
+  Since the implementation of a contract lives outside of modules using it,
+  using the same implementation and code elsewhere without copy-pasting is trivial.
+- **Loose coupling**.
+  Modules that use a contract do not care how they are implemented
+  as long as the implementation follows the behavior outlined by the contract.
+- Full **separation of concerns** (see diagram below).
+  Now, each party's concern is separated with a clear boundary.
+  The maintainer of a module using a contract can be different from the maintainers
+  of the implementation, allowing them to be experts in their own respective fields.
+  But more importantly, the contracts themselves can be created and maintained by the community.
+- Full **extensibility**.
+  The final user themselves can choose an implementation,
+  even new custom implementations not available in nixpkgs, without changing existing code.
+- **Incremental adoption**.
+  Contracts can help bridge a NixOS system with any non-NixOS one.
+  For that, one can hardcode a requester or provider module to match
+  how the non-NixOS system is configured.
+  The responsibility falls of course on the user to make sure both system agree on the configuration.
+- Last but not least, **Testability**.
+  Thanks to NixOS VM test, we can even go one step further
+  by ensuring each implementation of a contract, even custom ones,
+  provides required options and behaves as the contract requires
+  thanks to generic NixOS tests.
+  For an example, see the [generic backup contract test][generic backup test]
+  and the [instantiated NixOS tests][instantiated backup test]
+  ensuring the providers do implement the contract correctly.
 
 ![](./assets/contracts_separationofconcerns.png "Separation of concerns thanks to contracts.")
 
+## Concept {#contracts-concept}
+
+Conceptually, a contract is an attrset of options with a defined behavior.
+
+Let's take a reduced `secret` contract as example.
+This contract allows a `requester` module to ask for a secret
+and a `provider` module to generate that secret outside of the nix store
+and provide it back to the `requester`.
+In this case, the options for the contract could look like so:
+
+_The full secret contract can be found [here][secret contract]._
+
+[secret contract]: ./contracts-secret.html
+
+```nix
+{ lib, ... }:
+let
+  inherit (lib) mkOption;
+  inherit (lib.types) submodule str;
+in
+{
+  # Filled out by the requester module.
+  request = mkOption {
+    type = submodule {
+      options = {
+        owner = mkOption {
+          description = "Linux user owning the secret file.";
+          type = str;
+        };
+      };
+    };
+  };
+
+  # Filled out by the provider module.
+  result = mkOption {
+    type = submodule {
+      options = {
+        path = mkOption {
+          description = "Linux user owning the secret file.";
+          type = str;
+        };
+      };
+    };
+  };
+
+  # Options specific for each provider.
+  settings = mkOption {
+    type = submodule {
+      options = {
+        encryptedFile = mkOption {
+          description = "Encrypted file containing the secret.";
+          type = path;
+        };
+      };
+    };
+  };
+}
+```
+
+Unfortunately, the contract needs to be more complicated to handle several constraints.
+
+1. First, to fill out the contract,
+   the `requester` must set the defaults for the `request.*` options
+   and the `provider` for the `result.*` options.
+
+   Since one cannot do that after calling the `mkOption` function,
+   the `request` and `result` attributes must be functions
+   taking in the defaults as arguments.
+
+2. Another constraint is a `provider` module of a contract
+   will need to work for several `requester` modules.
+   This means that the option to provide the contract will be an
+   `attrsOf` of something, not just plainly the contract.
+
+   Think of a provider for the secret contract,
+   if it didn't use `attrsOf`, one could only create an unique secret
+   for all the modules, which is not useful.
+
+3. Also, one usually want the defaults
+   for the contract to be computed from some other option.
+   For a `provider` module, the options in the `result` could be computed
+   from the `name` provided in the `attrsOf`
+   or from a value given in the `request` or `setting` attrset.
+
+   For example, a `provider` module for the `secret` contract would want
+   something like the following in pseudo code:
+   
+   ```nix
+   services.provier = {
+     secret = mkOption {
+       type = attrsOf (submodule ({ name, ... }: {
+         result = {
+           path = mkOption {
+             type = str;
+             default = "/run/secrets/${name}";
+           };
+         };
+       }))
+     };
+   };
+   ```
+   
+   Another example is for a `provider` module for the `backup` contract
+   which would want the name of the restore script to depend on the path
+   to the repository it is backing up to.
+   This is necessary to differentiate which source to restore from
+   in case one wants to backup a same `requester` service
+   to multiple different repositories.
+   One could be local and another remote, for example.
+   
+   ```nix
+   services.provider = {
+     backup = mkOption {
+       type = attrsOf (submodule ({ name, config, ... }: {
+         settings = {
+         };
+   
+         result = {
+           restoreScript = {
+             type = str;
+             default = "provider-restore-${name}-${config.settings.repository.path}";
+           };
+         };
+       }));
+     };
+   };
+   ```
+
+4. Finally, the last constraint, which is also the more demanding,
+   is we want to generate the documentation
+   for the options with `nixos-generate-config`.
+   For that, the complicated `default` we give to options
+   that depend on other options break the documentation generation.
+   So instead of using only `default`,
+   we must also define `defaultText` attributes.
+
+   This means the actual `mkRequest` and `mkResult` functions
+   must take twice as many arguments as there are option.
+   One for the `default` and the other for the `defaultText`.
+   This will not be shown in the following snippets as it is
+   already complicated enough.
+
+These are all the justifications to why the final contract structure
+is as presented in the next section.
+It makes it harder to write, but much easier to use,
+which is nice property.
+
+## Schema {#contracts-schema}
+
+A contract for a version of the [backup contract][] with less options
+would look like so:
+
+```nix
+{ lib, ... }:
+let
+  inherit (lib) mkOption;
+  inherit (lib.types) submodule str;
+in
+{
+  mkRequest =
+    { owner ? "root",
+    }: mkOption {
+      default = {
+        inherit owner;
+      };
+
+      type = submodule {
+        options = {
+          owner = mkOption {
+            description = "Linux user owning the secret file.";
+            type = str;
+            default = owner;
+          };
+        };
+      };
+    };
+
+  mkResult =
+    { path ? "/run/secrets/secret",
+    }: mkOption {
+    type = submodule {
+      options = {
+        path = mkOption {
+          description = "Linux user owning the secret file.";
+          type = str;
+          default = path;
+        };
+      };
+    };
+  };
+}
+```
+
+Assuming the `services.requester` module needs to receive a password from the user
+and wants to use the `secret contract` for that,
+it would then setup the option like so:
+
+```nix
+{ pkgs, lib, ... }:
+let
+  inherit (lib) mkOption;
+  inherit (lib.types) submodule;
+
+  contracts = pkgs.callPackage ./modules/contracts {};
+
+  mkRequester = requestCfg: {
+    request = contracts.secret.mkRequest requestCfg;
+
+    result = contracts.secret.mkResult {};
+  };
+in
+{
+  options.services.requester = {
+    password = mkOption {
+      description = "Password for the service.";
+      type = submodule {
+        options = mkRequester {
+          owner = "requester";
+        };
+      };
+    };
+  };
+  config = {
+    // Use config.services.requester.password.result.path
+  };
+}
+```
+
+A provider that can create multiple secrets would have an `attrsOf` option
+and use the contract in it like so:
+
+```nix
+let
+  inherit (lib) mkOption;
+  inherit (lib.types) attrsOf submodule;
+
+  contracts = pkgs.callPackage ./modules/contracts {};
+
+  mkProvider =
+    module:
+    { resultCfg,
+      settings ? {},
+    }: {
+      request = contracts.secret.mkRequest {};
+
+      result = contracts.secret.mkResult resultCfg;
+    } // optionalAttrs (settings != {}) { inherit settings; };
+in
+{
+  options.services.provider = {
+    secrets = mkOption {
+      type = attrsOf (submodule ({ name, options, ... }: {
+        options = mkProvider {
+          resultCfg = {
+            path = "/run/secrets/${name}";
+          };
+
+          settings = mkOption {
+            description = "Settings specific to the Sops provider.";
+
+            type = attrsOf (submodule {
+              options = {
+                repository = mkOption {
+                };
+              };
+            });
+            default = {};
+          };
+        };
+      }));
+    };
+  };
+}
+```
+
+The `mkRequester` and `mkProvider` are provided by Self Host Blocks
+as they are generic, so the actual syntax is a little bit different.
+They were copied here that way so the snippets were self-contained.
+
+To see a full contract in action, the secret contract is a good example.
+It is composed of:
+
+- [the contract][secret contract ref],
+- [the mkRequester and mkProvider][contract lib] functions,
+- [a requester][],
+- [a provider][].
+
+[secret contract ref]: ./contracts-secret.html#contract-secret-options
+[contract lib]: @REPO@/modules/contracts/default.nix
+[a requester]: ./blocks-sops.html#blocks-sops-options-shb.sops.secret
+[a provider]: ./services-nextcloud.html#services-nextcloudserver-options-shb.nextcloud.adminPass
+
+## Contract Tests {#contracts-test}
+
+To make sure all providers module of a contract have the same behavior,
+generic NixOS VM tests exist per contract.
+They are generic because they work on any module,
+as long as the module implements the contract of course.
+
+A simplified test for a secret contract would look like the following.
+First, there is the generic test:
+
+```nix
+{ pkgs, lib, shb, ... }:
+let
+  inherit (lib) getAttrFromPath setAttrByPath;
+in
+  { name,
+    configRoot,
+    settingsCfg,
+    modules ? [],
+    owner ? "root",
+    content ? "secretPasswordA",
+  }: shb.test.runNixOSTest {
+    inherit name;
+    
+    nodes.machine = { config, ... }: {
+      imports = modules;
+      
+      config = setAttrByPath configRoot {
+        secretA = {
+          request = {
+            inherit owner;
+          };
+          settings = settingsCfg content;
+        };
+      };
+    };
+
+    testScript = { nodes, ... }:
+      let
+        result = (getAttrFromPath configRoot nodes.machine)."A".result;
+      in
+        ''
+          owner = machine.succeed("stat -c '%U' ${result.path}").strip()
+          if owner != "${owner}":
+              raise Exception(f"Owner should be '${owner}' but got '{owner}'")
+
+          content = machine.succeed("cat ${result.path}").strip()
+          if content != "${content}":
+              raise Exception(f"Content should be '${content}' but got '{content}'")
+        '';
+  }
+```
+
+This test is generic because it sets the `request` on an option
+whose path is not yet known.
+It achieves this by calling `setAttrByPath configRoot` where `configRoot`
+is a path to a module, for example `[ "services" "provider" ]` for a module
+whose root option is under `services.provider`.
+
+This test validates multiple aspects of the contract:
+
+- The provider must understand the options of the `request`.
+  Here `request.owner`.
+- The provider correctly provides the expected result.
+  Here the location of the secret in the `result.path` option.
+- The provider must behave as expected.
+  Here, the secret located at `result.path` must have the correct `owner`
+  and the correct `content`.
+
+Instantiating the test for a given provider looks like so:
+
+```nix
+{
+  hardcoded_root = contracts.test.secret {
+    name = "hardcoded_root";
+
+    modules = [ ./modules/blocks/hardcodedsecret.nix ];
+    configRoot = [ "shb" "hardcodedsecret" ];
+    settingsCfg = secret: {
+      content = secret;
+    };
+  };
+
+  hardcoded_user = contracts.test.secret {
+    name = "hardcoded_user";
+
+    owner = "user";
+    modules = [ ./modules/blocks/hardcodedsecret.nix ];
+    configRoot = [ "shb" "hardcodedsecret" ];
+    settingsCfg = secret: {
+      content = secret;
+    };
+  };
+}
+```
+
+Validating a new provider is then just a matter of extending the above snippet.
+
+To see a full contract test in action, the test for backup contract is a good example.
+It is composed of:
+
+- the [generic test][generic backup test]
+- and [instantiated tests][instantiated backup test] for some providers.
+
+[generic backup test]: @REPO@/modules/contracts/backup/test.nix
+[instantiated backup test]: @REPO@/test/contracts/backup.nix
+
+## Videos {#contracts-videos}
+
+Two videos exist of me presenting the topic,
+the first at [NixCon North America in spring of 2024][NixConNA2024]
+and the second at [NixCon in Berlin in fall of 2024][NixConBerlin2024].
+
+[NixConNA2024]: https://www.youtube.com/watch?v=lw7PgphB9qM
+[NixConBerlin2024]: https://www.youtube.com/watch?v=CP0hR6w1csc
+
 ## Are there contracts in nixpkgs already? {#contracts-nixpkgs}
 
 Actually not quite, but close. There are some ubiquitous options in nixpkgs. Those I found are:

docs/contributing.md

@@ -1,8 +1,28 @@
+<!-- Read these docs at https://shb.skarabox.com -->
 # Contributing {#contributing}
 
 All issues and Pull Requests are welcome!
 
-For Pull Requests, if they are substantial changes, please open an issue to discuss the details
+- Use this project. Something does not make sense? Something's not working?
+- Documentation. Something is not clear?
+- New services. Have one of your preferred service not integrated yet?
+- Better patterns. See something weird in the code?
+
+For PRs, if they are substantial changes, please open an issue to
+discuss the details first. More details in [the contributing section](https://shb.skarabox.com/contributing.html)
+of the manual.
+
+Issues that are being worked on are labeled with the [in progress][] label.
+Before starting work on those, you might want to talk about it in the issue tracker
+or in the [matrix][] channel.
+
+The prioritized issues are those belonging to the [next milestone][milestone].
+Those issues are not set in stone and I'd be very happy to solve
+an issue an user has before scratching my own itch.
+
+[in progress]: https://github.com/ibizaman/selfhostblocks/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22in%20progress%22
+[matrix]: https://matrix.to/#/%23selfhostblocks%3Amatrix.org
+[milestone]: https://github.com/ibizaman/selfhostblocks/milestones
 first.
 
 ## Chat Support {#contributing-chat}
@@ -21,8 +41,6 @@ gets updated. I intend to upstream to nixpkgs as much of those as makes sense.
 Run all tests:
 
 ```bash
-$ nix build .#checks.${system}.all
-# or
 $ nix flake check
 # or
 $ nix run github:Mic92/nix-fast-build -- --skip-cached --flake ".#checks.$(nix eval --raw --impure --expr builtins.currentSystem)"
@@ -35,15 +53,67 @@ $ nix build .#checks.${system}.modules
 $ nix build .#checks.${system}.vm_postgresql_peerAuth
 ```
 
-Run one VM test interactively:
+### Playwright Tests {#contributing-playwright-tests}
+
+If the test includes playwright tests, you can see the playwright trace with:
+
+```bash
+$ nix run .#playwright -- show-trace $(nix eval .#checks.x86_64-linux.vm_grocy_basic --raw)/trace/0.zip
+```
+
+If the test fails, you won't have the output directory available.
+Instead, run the test with `--keep-failed` and at the end you will see a line saying
+
+```
+Keeping build directory '/nix/var/nix/builds/nix-31776-4270051001/build'
+```
+
+Copy the path and run:
+
+```bash
+sudo cp -r /nix/var/nix/builds/nix-31776-4270051001/build/shared-xchg/trace trace && sudo chown -R $USER: trace
+```
+
+Now, open that file with playwright:
+
+```bash
+nix run .#playwright -- show-trace trace/0.zip
+```
+
+### Debug Tests {#contributing-debug-tests}
+
+Run the test in driver interactive mode:
 
 ```bash
 $ nix run .#checks.${system}.vm_postgresql_peerAuth.driverInteractive
 ```
 
-When you get to the shell, run either `start_all()` or `test_script()`. The former just starts all
-the VMs and service, then you can introspect. The latter also starts the VMs if they are not yet and
-then will run the test script.
+When you get to the shell, start the server and/or client with one of the following commands:
+
+```bash
+server.start()
+client.start()
+start_all()
+```
+
+To run the test from the shell, use `test_script()`.
+Note that if the test script ends in error,
+the shell will exit and you will need to restart the VMs.
+
+After the shell started, you will see lines like so:
+
+```
+SSH backdoor enabled, the machines can be accessed like this:
+Note: this requires systemd-ssh-proxy(1) to be enabled (default on NixOS 25.05 and newer).
+    client:  ssh -o User=root vsock/3
+    server:  ssh -o User=root vsock/4
+```
+
+With the following command, you can directly access the server's nginx instance with your browser at `http://localhost:8000`:
+
+```bash
+ssh-keygen -R vsock/4; ssh -o User=root -L 8000:localhost:80 vsock/4
+```
 
 ## Upload test results to CI {#contributing-upload}
 
@@ -56,6 +126,18 @@ After running the `nix-fast-build` command from the previous section, run:
 $ find . -type l -name "result-vm_*" | xargs readlink | nix run nixpkgs#cachix -- push selfhostblocks
 ```
 
+## Upload package to CI {#contributing-upload-package}
+
+In the rare case where a package must be built but cannot in CI,
+for example because of not enough memory,
+you can push the package directly to the cache with:
+
+```bash
+nix build .#checks.x86_64-linux.vm_karakeep_backup.nodes.server.services.karakeep.package
+readlink result | nix run nixpkgs#cachix -- push selfhostblocks
+
+```
+
 ## Deploy using colmena {#contributing-deploy-colmena}
 
 ```bash
@@ -140,6 +222,12 @@ $ nix run nixpkgs#nvd -- diff \
 $ nix run nixpkgs#openssl -- rand -hex 64
 ```
 
+## Write code {#contributing-code}
+
+```{=include=} chapters html:into-file=//service-implementation-guide.html
+service-implementation-guide.md
+```
+
 ## Links that helped {#contributing-links}
 
 While creating NixOS tests:

docs/default.nix

@@ -1,34 +1,49 @@
 # Taken nearly verbatim from https://github.com/nix-community/home-manager/pull/4673
-{ pkgs
-, buildPackages
-, lib
-, nmdsrc
-, stdenv
-, documentation-highlighter
-, nixos-render-docs
+# Read these docs online at https://shb.skarabox.com.
+{
+  pkgs,
+  buildPackages,
+  lib,
+  nmdsrc,
+  stdenv,
+  documentation-highlighter,
+  nixos-render-docs,
 
-, release
-, allModules
+  release,
+  allModules,
+
+  version ? builtins.readFile ../VERSION,
+  substituteVersionIn,
+
+  modules,
 }:
 
 let
   shbPath = toString ./..;
 
-  gitHubDeclaration = user: repo: subpath:
-    let urlRef = "main";
-        end = if subpath == "" then "" else "/" + subpath;
-    in {
+  gitHubDeclaration =
+    user: repo: subpath:
+    let
+      urlRef = "main";
+      end = if subpath == "" then "" else "/" + subpath;
+    in
+    {
       url = "https://github.com/${user}/${repo}/blob/${urlRef}${end}";
       name = "<${repo}${end}>";
     };
 
   ghRoot = (gitHubDeclaration "ibizaman" "selfhostblocks" "").url;
 
-  buildOptionsDocs = args@{ modules, includeModuleSystemOptions ? true, ... }:
+  buildOptionsDocs =
+    {
+      modules,
+      filterOptionPath ? null,
+    }:
+    args:
     let
       config = {
         _module.check = false;
-        _module.args = {};
+        _module.args = { };
         system.stateVersion = "22.11";
       };
 
@@ -45,38 +60,57 @@ let
         };
       };
 
-      options = if includeModuleSystemOptions
-                then eval.options
-                else builtins.removeAttrs eval.options [ "_module" ];
-    in buildPackages.nixosOptionsDoc ({
-      inherit options;
+      options = lib.setAttrByPath filterOptionPath (lib.getAttrFromPath filterOptionPath eval.options);
+    in
+    buildPackages.nixosOptionsDoc (
+      {
+        inherit options;
 
-      transformOptions = opt:
-        opt // {
-          # Clean up declaration sites to not refer to the Home Manager
-          # source tree.
-          declarations = map (decl:
-            gitHubDeclaration "ibizaman" "selfhostblocks"
-              (lib.removePrefix "/" (lib.removePrefix shbPath (toString decl)))) opt.declarations;
-        };
-    } // builtins.removeAttrs args [ "modules" "includeModuleSystemOptions" ]);
+        transformOptions =
+          opt:
+          opt
+          // {
+            # Clean up declaration sites to not refer to the Home Manager
+            # source tree.
+            declarations = map (
+              decl:
+              gitHubDeclaration "ibizaman" "selfhostblocks" (
+                lib.removePrefix "/" (lib.removePrefix shbPath (toString decl))
+              )
+            ) opt.declarations;
+          };
+      }
+      // builtins.removeAttrs args [ "includeModuleSystemOptions" ]
+    );
 
   scrubbedModule = {
     _module.args.pkgs = lib.mkForce (nmd.scrubDerivations "pkgs" pkgs);
     _module.check = false;
   };
 
-  optionsDocs = buildOptionsDocs {
-    modules = allModules ++ [ scrubbedModule ];
-    variablelistId = "selfhostblocks-options";
-    includeModuleSystemOptions = false;
-  };
+  allOptionsDocs =
+    paths:
+    (buildOptionsDocs
+      {
+        modules = paths ++ allModules ++ [ scrubbedModule ];
+        filterOptionPath = [ "shb" ];
+      }
+      {
+        variablelistId = "selfhostblocks-options";
+      }
+    ).optionsJSON;
 
-  individualModuleOptionsDocs = path: (buildOptionsDocs {
-    modules = [ path scrubbedModule ];
-    variablelistId = "selfhostblocks-options";
-    includeModuleSystemOptions = false;
-  }).optionsJSON;
+  individualModuleOptionsDocs =
+    filterOptionPath: paths:
+    (buildOptionsDocs
+      {
+        modules = paths ++ [ scrubbedModule ];
+        inherit filterOptionPath;
+      }
+      {
+        variablelistId = "selfhostblocks-options";
+      }
+    ).optionsJSON;
 
   nmd = import nmdsrc {
     inherit lib;
@@ -84,15 +118,15 @@ let
     # `nmd` uses to work around the broken stylesheets in
     # `docbook-xsl-ns`, so we restore the patched version here.
     pkgs = pkgs // {
-      docbook-xsl-ns =
-        pkgs.docbook-xsl-ns.override { withManOptDedupPatch = true; };
+      docbook-xsl-ns = pkgs.docbook-xsl-ns.override { withManOptDedupPatch = true; };
     };
   };
 
   outputPath = "share/doc/selfhostblocks";
 
-  manpage-urls = pkgs.writeText "manpage-urls.json" ''{}'';
-in stdenv.mkDerivation {
+  manpage-urls = pkgs.writeText "manpage-urls.json" "{}";
+in
+stdenv.mkDerivation {
   name = "self-host-blocks-manual";
 
   nativeBuildInputs = [ nixos-render-docs ];
@@ -124,41 +158,54 @@ in stdenv.mkDerivation {
       ${nmdsrc}/static/highlightjs/highlight.min.js \
       ${nmdsrc}/static/highlightjs/highlight.load.js
 
+  ''
+  + lib.concatStringsSep "\n" (
+    map (m: ''
+      substituteInPlace ${m} --replace '@VERSION@' ${version}
+    '') substituteVersionIn
+  )
+  + ''
     substituteInPlace ./options.md \
       --replace \
         '@OPTIONS_JSON@' \
-        ${optionsDocs.optionsJSON}/share/doc/nixos/options.json
-
-    substituteInPlace ./modules/blocks/ssl/docs/default.md \
-      --replace \
-        '@OPTIONS_JSON@' \
-        ${individualModuleOptionsDocs ../modules/blocks/ssl.nix}/share/doc/nixos/options.json
-
-    substituteInPlace ./modules/blocks/backup/docs/default.md \
-      --replace \
-        '@OPTIONS_JSON@' \
-        ${individualModuleOptionsDocs ../modules/blocks/backup.nix}/share/doc/nixos/options.json
-
-    substituteInPlace ./modules/services/nextcloud-server/docs/default.md \
-      --replace \
-        '@OPTIONS_JSON@' \
-       ${individualModuleOptionsDocs ../modules/services/nextcloud-server.nix}/share/doc/nixos/options.json
-
-    substituteInPlace ./modules/contracts/ssl/docs/default.md \
-      --replace \
-        '@OPTIONS_JSON@' \
-       ${individualModuleOptionsDocs ../modules/contracts/ssl/dummyModule.nix}/share/doc/nixos/options.json
-
+        ${
+          allOptionsDocs [
+            (pkgs.path + "/nixos/modules/services/misc/forgejo.nix")
+          ]
+        }/share/doc/nixos/options.json
+  ''
+  + lib.concatStringsSep "\n" (
+    lib.mapAttrsToList (
+      name: cfg':
+      let
+        cfg = if builtins.isAttrs cfg' then cfg' else { module = cfg'; };
+        module = if builtins.isList cfg.module then cfg.module else [ cfg.module ];
+        optionRoot =
+          cfg.optionRoot or [
+            "shb"
+            (lib.last (lib.splitString "/" name))
+          ];
+      in
+      ''
+        substituteInPlace ./modules/${name}/docs/default.md \
+          --replace-fail \
+            '@OPTIONS_JSON@' \
+            ${individualModuleOptionsDocs optionRoot module}/share/doc/nixos/options.json
+      ''
+    ) modules
+  )
+  + ''
     find . -name "*.md" -print0 | \
       while IFS= read -r -d ''' f; do
         substituteInPlace "''${f}" \
-          --replace \
+          --replace-quiet \
             '@REPO@' \
             "${ghRoot}" 2>/dev/null
       done
 
     nixos-render-docs manual html \
       --manpage-urls ${manpage-urls} \
+      --redirects ./redirects.json \
       --media-dir media \
       --revision ${lib.trivial.revisionWithDefault release} \
       --stylesheet static/style.css \

docs/demos.md

@@ -1,3 +1,4 @@
+<!-- Read these docs at https://shb.skarabox.com -->
 # Demos {#demos}
 
 These demos are showcasing what Self Host Blocks can do. They deploy a block or a service on a VM on
@@ -7,6 +8,6 @@ your local machine with minimal manual steps.
 demo/homeassistant/README.md
 ```
 
-```{=include=} chapters html:into-file=//demo-nextcloud-server.html
+```{=include=} chapters html:into-file=//demo-nextcloud.html
 demo/nextcloud/README.md
 ```

docs/generate-redirects-nixos-render-docs.py

@@ -0,0 +1,208 @@
+#!/usr/bin/env python3
+"""
+Generate redirects.json by scanning actual HTML files produced by nixos-render-docs.
+
+This script implements a runtime patching mechanism to automatically generate a
+complete redirects.json file by scanning generated HTML files for real anchor
+locations, eliminating manual maintenance and ensuring accuracy.
+
+ARCHITECTURE OVERVIEW:
+The script works by monkey-patching nixos-render-docs at runtime to:
+1. Disable redirect validation during HTML generation
+2. Generate HTML documentation normally
+3. Scan all generated HTML files to extract anchor IDs and their file locations
+4. Apply filtering logic to exclude system-generated anchors
+5. Generate and write redirects.json with accurate mappings
+
+KEY COMPONENTS:
+- Runtime patching: Modifies nixos-render-docs behavior without source changes
+- HTML scanning: Extracts anchor IDs using regex pattern matching
+- Filtering: Excludes NixOS options (opt-*) and extra options (selfhostblock*)
+- Output generation: Creates both debug information and production redirects.json
+
+IMPORTANT NOTES:
+- Uses atexit handler to ensure output is generated even if process is interrupted
+- Patches are applied on module import, making this a side-effect import
+- Error handling preserves original validation behavior in case of failure
+"""
+
+import sys
+import json
+import atexit
+import os
+import re
+
+# Global storage for anchor-to-file mappings discovered during HTML scanning
+# Structure: {anchor_id: html_filename}
+file_target_mapping = {}
+
+def scan_html_files(output_dir, html_files):
+    """
+    Scan HTML files to extract anchor IDs and build anchor-to-file mappings.
+    
+    Discovers all HTML files in output_dir and extracts id attributes to populate
+    the global file_target_mapping. Filters out NixOS system options during scanning.
+    
+    Args:
+        output_dir: Directory containing generated HTML files
+        html_files: Unused parameter (always discovers files from filesystem)
+    """
+    # Always discover HTML files from the output directory
+    if not os.path.exists(output_dir):
+        print(f"DEBUG: Output directory {output_dir} does not exist", file=sys.stderr)
+        return
+    
+    html_files = [f for f in os.listdir(output_dir) if f.endswith('.html')]
+    print(f"DEBUG: Discovered {len(html_files)} HTML files in {output_dir}", file=sys.stderr)
+    
+    # Process each HTML file to extract anchor IDs
+    for html_file in html_files:
+        html_path = os.path.join(output_dir, html_file)
+        try:
+            with open(html_path, 'r', encoding='utf-8') as f:
+                html_content = f.read()
+            
+            # Extract all id attributes using regex pattern matching
+            # Matches: id="anchor-name" and captures anchor-name
+            anchor_matches = re.findall(r'id="([^"]+)"', html_content)
+            
+            # Filter and record anchor mappings
+            non_opt_count = 0
+            for anchor_id in anchor_matches:
+                # Skip NixOS system option anchors (opt-* prefix)
+                if not anchor_id.startswith('opt-'):
+                    file_target_mapping[anchor_id] = html_file
+                    non_opt_count += 1
+            
+            if non_opt_count > 0:
+                print(f"Found {non_opt_count} anchors in {html_file}", file=sys.stderr)
+                
+        except Exception as e:
+            # Log errors but continue processing other files
+            print(f"Failed to scan {html_path}: {e}", file=sys.stderr)
+
+def output_collected_refs():
+    """
+    Generate and write the final redirects.json file from collected anchor mappings.
+    
+    This function is registered as an atexit handler to ensure output is generated
+    even if the process is interrupted. It processes the global file_target_mapping
+    to create the final redirects file with appropriate filtering.
+    
+    Output files:
+        - out/redirects.json: Production redirects mapping
+    """
+    import os
+    
+    # Generate redirects from discovered HTML anchor mappings
+    if file_target_mapping:
+        print(f"Creating redirects from {len(file_target_mapping)} HTML mappings", file=sys.stderr)
+        redirects = {}
+        filtered_count = 0
+        
+        # Apply filtering logic to exclude system-generated anchors
+        for anchor_id, html_file in file_target_mapping.items():
+            # Filter out:
+            # - opt-*: NixOS system options 
+            # - selfhostblock*: Extra options from this project
+            if not anchor_id.startswith('opt-') and not anchor_id.startswith('selfhostblock'):
+                redirects[anchor_id] = [f"{html_file}#{anchor_id}"]
+            else:
+                filtered_count += 1
+        
+        print(f"Generated {len(redirects)} redirects (filtered out {filtered_count} system options)", file=sys.stderr)
+    else:
+        # Fallback case - should not occur during normal operation
+        print("Warning: No HTML mappings available", file=sys.stderr)
+        redirects = {}
+    
+    # Ensure output directory exists
+    os.makedirs('out', exist_ok=True)
+    
+    # Write production redirects file
+    try:
+        redirects_file = 'out/redirects.json'
+        
+        with open(redirects_file, 'w') as f:
+            json.dump(redirects, f, indent=2, sort_keys=True)
+        
+        print(f"Generated redirects.json with {len(redirects)} redirects", file=sys.stderr)
+        
+    except Exception as e:
+        print(f"Failed to write redirects.json: {e}", file=sys.stderr)
+
+# Register output generation to run on process exit
+atexit.register(output_collected_refs)
+
+def apply_patches():
+    """
+    Apply runtime monkey patches to nixos-render-docs modules.
+    
+    This function modifies the behavior of nixos-render-docs by:
+    1. Hooking into the HTML generation CLI command
+    2. Temporarily disabling redirect validation during HTML generation
+    3. Scanning generated HTML files to extract anchor mappings
+    4. Restoring original validation behavior
+    
+    The patching approach allows us to extract anchor information without
+    modifying the nixos-render-docs source code directly.
+    
+    Raises:
+        ImportError: If nixos-render-docs modules cannot be imported
+    """
+    try:
+        # Import required nixos-render-docs modules
+        import nixos_render_docs.html as html_module
+        import nixos_render_docs.redirects as redirects_module
+        import nixos_render_docs.manual as manual_module
+        
+        # Store reference to original HTML CLI function
+        original_run_cli_html = manual_module._run_cli_html
+        
+        def patched_run_cli_html(args):
+            """
+            Patched version of _run_cli_html that disables validation and scans output.
+            
+            This wrapper function:
+            1. Temporarily disables redirect validation to prevent errors
+            2. Runs normal HTML generation
+            3. Scans generated HTML files for anchor mappings
+            4. Restores original validation behavior
+            """
+            print("Generating HTML documentation...", file=sys.stderr)
+            
+            # Temporarily disable redirect validation
+            original_validate = redirects_module.Redirects.validate
+            redirects_module.Redirects.validate = lambda self, targets: None
+            
+            try:
+                # Run original HTML generation
+                result = original_run_cli_html(args)
+                
+                # Determine output directory from CLI arguments
+                if hasattr(args, 'outfile') and args.outfile:
+                    output_dir = os.path.dirname(args.outfile)
+                else:
+                    output_dir = '.'
+                
+                # Scan generated HTML files for anchor mappings
+                scan_html_files(output_dir, None)
+                print(f"Scanned {len(file_target_mapping)} anchor mappings", file=sys.stderr)
+                
+            finally:
+                # Always restore original validation function
+                redirects_module.Redirects.validate = original_validate
+            
+            return result
+        
+        # Replace the original function with our patched version
+        manual_module._run_cli_html = patched_run_cli_html
+                
+        print("Applied patches to nixos-render-docs", file=sys.stderr)
+        
+    except ImportError as e:
+        print(f"Failed to apply patches: {e}", file=sys.stderr)
+
+# Apply patches immediately when this module is imported
+# This ensures the patches are active before nixos-render-docs CLI runs
+apply_patches()

docs/manual.md

@@ -1,6 +1,7 @@
+<!-- Read these docs at https://shb.skarabox.com -->
 # Self Host Blocks Manual {#self-host-blocks-manual}
 
-## Version 0.0.1
+## Version @VERSION@
 
 
 ```{=include=} preface
@@ -23,6 +24,14 @@ contracts.md
 blocks.md
 ```
 
+```{=include=} chapters html:into-file=//recipes.html
+recipes.md
+```
+
+```{=include=} chapters html:into-file=//misc.html
+misc.md
+```
+
 ```{=include=} chapters html:into-file=//demos.html
 demos.md
 ```

docs/misc.md

@@ -0,0 +1,39 @@
+<!-- Read these docs at https://shb.skarabox.com -->
+# Miscellaneous {#misc}
+
+Here goes meta-discussions around the repository and other topics not related directly to the usage of SHB.
+
+## Lock file update {#misc-lock-file-update}
+
+*SHB has an unusual strategy to keep up with its flake inputs. This section explains why.*
+
+SHB only depends on `nixpkgs` as far as flake inputs goes.
+There are others but they only matter for tasks around taking care of the repository itself.
+So only the `nixpkgs` input is important for downstream consumers of SHB.
+
+To keep up to date with nixpkgs unstable,
+initially a job was updating the nixpkgs input by simply updating the lock file and creating a PR.
+Now, this job failed most of the time because the tip of unstable often breaks modules and packages, resulting in failing SHB tests.
+To fix this, we usually needed to choose a commit around 2 weeks prior to the latest unstable commit.
+This usually did not lead to successful tests but the reason was then not
+because of failing packages but because nixpkgs modules evolved and SHB needs to adapt to that.
+The net effect was SHB was lagging too far behind unstable and updating it was becoming annoying.
+
+The new strategy now is as follows. On every job run:
+
+- If there is no PR, update nixpkgs input to latest unstable commit and create a PR which auto-merges when tests are successful.
+- If there is a PR:
+  - If the checks succeeded, do nothing because the PR will auto-merge.
+  - If the checks are pending, do nothing yet.
+  - If the checks failed, bisect and update nixpkgs input to between the commit on main and the commit in the PR.
+
+The effect here is as long as the tests are failing, we'll try a commit further in the past from nixpkgs unstable
+and closer to the tip of main branch.
+
+This could be made smarter by distinguishing between types of failures.
+We will see if this is needed.
+
+One can also run the bisect manually with `nix run .#update-flake-lock-pr`.
+This will create a PR on a different branch than the one used for the automated workflow.
+It accepts also one argument `future` which instead of trying a commit in between the tip of SHB main and the PR's failing one,
+it tries a commit between the PR's failing one and the nixpkgs unstable latest commit.

docs/options.md

@@ -1,7 +1,8 @@
-# All Options {#ch-options}
+<!-- Read these docs at https://shb.skarabox.com -->
+# All Options {#all-options}
 
 ```{=include=} options
 id-prefix: opt-
-list-id: selfhostblock-options
+list-id: selfhostblocks-options
 source: @OPTIONS_JSON@
 ```

docs/preface.md

@@ -1,28 +1,405 @@
+<!-- Read these docs at https://shb.skarabox.com -->
 # Preface {#preface}
 
 ::: {.note}
-Self Host Blocks is hosted on [GitHub](https://github.com/ibizaman/selfhostblocks). If you encounter
-problems or bugs then please report them on the [issue
+Self Host Blocks is hosted on [GitHub](https://github.com/ibizaman/selfhostblocks).
+If you encounter problems or bugs then please report them on the [issue
 tracker](https://github.com/ibizaman/selfhostblocks/issues).
 
 Feel free to join the dedicated Matrix room
 [matrix.org#selfhostblocks](https://matrix.to/#/#selfhostblocks:matrix.org).
 :::
 
-Self Host Blocks intends to help you self host any service you would like with best practices out of
-the box.
+SelfHostBlocks is:
 
-To achieve this, Self Host Blocks provides building blocks which each provide part of what a self
-hosted app should do (SSO, HTTPS, etc.). It also provides some services that are already integrated
-with all those building blocks.
+- Your escape from the cloud, for privacy and data sovereignty enthusiast. [Why?](#preface-why-self-hosting)
+- A groupware to self-host [all your data](#preface-services): documents, pictures, calendars, contacts, etc.
+- An opinionated NixOS server management OS for a [safe self-hosting experience](#preface-features).
+- A NixOS distribution making sure all services build and work correctly thanks to NixOS VM tests.
+- A collection of NixOS modules standardizing options so configuring services [look the same](#preface-unified-interfaces).
+- A testing ground for [contracts](#preface-contracts) which intents to make nixpkgs modules more modular.
+- [Upstreaming][] as much as possible.
 
-- You are new to self hosting and want pre-configured services to deploy easily. Look at the
-  [services section](services.html).
-- You are a seasoned self-hoster but want to enhance some services you deploy already. Go to the
-  [blocks section](blocks.html).
-- You are a user of Self Host Blocks but would like to use your own implementation for a block. Head
-  over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org) (the manual for this
-  is WIP).
+[upstreaming]: https://github.com/pulls?page=1&q=created%3A%3E2023-06-01+is%3Apr+author%3Aibizaman+archived%3Afalse+-repo%3Aibizaman%2Fselfhostblocks+-repo%3Aibizaman%2Fskarabox
 
-Self Host Blocks uses the full power of NixOS modules to achieve these goals. Blocks and service are
-both NixOS modules.
+## Why Self-Hosting {#preface-why-self-hosting}
+
+It is obvious by now that
+a deep dependency on proprietary service providers - "the cloud" -
+is a significant liability.
+One aspect often talked about is privacy
+which is inherently not guaranteed when using a proprietary service
+and is a valid concern.
+A more punishing issue is having your account closed or locked
+without prior warning
+When that happens,
+you get an instantaneous sinking feeling in your stomach
+at the realization you lost access to your data,
+possibly without recourse.
+
+Hosting services yourself is the obvious alternative
+to alleviate those concerns
+but it tends to require a lot of technical skills and time.
+SelfHostBlocks (together with its sibling project [Skarabox][])
+aims to lower the bar to self-hosting,
+and provides an opinionated server management system based on NixOS modules
+embedding best practices.
+Contrary to other server management projects,
+its main focus is ease of long term maintenance
+before ease of installation.
+To achieve this, it provides building blocks to setup services.
+Some are already provided out of the box,
+and customizing or adding additional ones is done easily.
+
+The building blocks fit nicely together thanks to [contracts](#contracts)
+which SelfHostBlocks sets out to introduce into nixpkgs.
+This will increase modularity, code reuse
+and empower end users to assemble components
+that fit together to build their server.
+
+## Usage {#preface-usage}
+
+> **Caution:** You should know that although I am using everything in this repo for my personal
+> production server, this is really just a one person effort for now and there are most certainly
+> bugs that I didn't discover yet.
+
+To get started using SelfHostBlocks, the following snippet is enough:
+
+```nix
+{
+  inputs.selfhostblocks.url = "github:ibizaman/selfhostblocks";
+
+  outputs = { selfhostblocks, ... }: let
+    system = "x86_64-linux";
+    nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
+  in
+    nixosConfigurations = {
+      myserver = nixpkgs'.nixosSystem {
+        inherit system;
+        modules = [
+          selfhostblocks.nixosModules.default
+          ./configuration.nix
+        ];
+      };
+    };
+}
+```
+
+SelfHostBlocks provides its own patched nixpkgs, so you are required to use it
+otherwise evaluation can quickly break.
+[The usage section](https://shb.skarabox.com/usage.html) of the manual has
+more details and goes over how to deploy with [Colmena][], [nixos-rebuild][] and [deploy-rs][]
+and also how to handle secrets management with [SOPS][].
+
+[Colmena]: https://shb.skarabox.com/usage.html#usage-example-colmena
+[nixos-rebuild]: https://shb.skarabox.com/usage.html#usage-example-nixosrebuild
+[deploy-rs]: https://shb.skarabox.com/usage.html#usage-example-deployrs
+[SOPS]: https://shb.skarabox.com/usage.html#usage-secrets
+
+Then, to actually configure services, you can choose which one interests you in
+the [services section](https://shb.skarabox.com/services.html) of the manual.
+
+The [recipes section](https://shb.skarabox.com/recipes.html) of the manual shows some other common use cases.
+
+Head over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org)
+for any remaining question, or just to say hi :)
+
+### Installation From Scratch {#preface-usage-installation-from-scratch}
+
+I do recommend for this my sibling project [Skarabox][]
+which bootstraps a new server and sets up a few tools:
+
+- Create a bootable ISO, installable on an USB key.
+- Handles one or two (in raid 1) SSDs for root partition.
+- Handles two (in raid 1) or more hard drives for data partition.
+- [nixos-anywhere](https://github.com/nix-community/nixos-anywhere) to install NixOS headlessly.
+- [disko](https://github.com/nix-community/disko) to format the drives using native ZFS encryption with remote unlocking through ssh.
+- [sops-nix](https://github.com/Mic92/sops-nix) to handle secrets.
+- [deploy-rs](https://github.com/serokell/deploy-rs) to deploy updates.
+
+[Skarabox]:  https://github.com/ibizaman/skarabox
+
+## Features {#preface-features}
+
+SelfHostBlocks provides building blocks that take care of common self-hosting needs:
+
+- Backup for all services.
+- Automatic creation of ZFS datasets per service.
+- LDAP and SSO integration for most services.
+- Monitoring with Grafana and Prometheus stack with provided dashboards.
+- Automatic reverse proxy and certificate management for HTTPS.
+- VPN and proxy tunneling services.
+
+Great care is taken to make the proposed stack robust.
+This translates into a test suite comprised of automated NixOS VM tests
+which includes playwright tests to verify some important workflow
+like logging in.
+
+This test suite also serves as a guaranty that all services provided by SelfHostBlocks
+all evaluate, build and work correctly together. It works similarly as a distribution but here it's all [automated](#preface-updates).
+
+Also, the stack fits together nicely thanks to [contracts](#preface-contracts).
+
+### Services {#preface-services}
+
+[Provided services](https://shb.skarabox.com/services.html) are:
+
+- Nextcloud
+- Audiobookshelf
+- Deluge + *arr stack
+- Forgejo
+- Grocy
+- Hledger
+- Home-Assistant
+- Jellyfin
+- Karakeep
+- Open WebUI
+- Pinchflat
+- Vaultwarden
+
+Like explained above, those services all benefit from
+out of the box backup,
+LDAP and SSO integration,
+monitoring with Grafana,
+reverse proxy and certificate management
+and VPN integration for the *arr suite.
+
+Some services do not have an entry yet in the manual.
+To know options for those, the only way for now
+is to go to the [All Options][] section of the manual.
+
+[All Options]: https://shb.skarabox.com/options.html
+
+### Blocks {#preface-blocks}
+
+The services above rely on the following [common blocks][]
+which altogether provides a solid foundation for self-hosting services:
+
+- Authelia
+- BorgBackup
+- Davfs
+- LDAP
+- Monitoring (Grafana - Prometheus - Loki stack)
+- Nginx
+- PostgreSQL
+- Restic
+- Sops
+- SSL
+- Tinyproxy
+- VPN
+- ZFS
+
+Those blocks can be used with services
+not provided by SelfHostBlocks as shown [in the manual][common blocks].
+
+[common blocks]: https://shb.skarabox.com/blocks.html
+
+The manual also provides documentation for each individual blocks.
+
+### Unified Interfaces {#preface-unified-interfaces}
+
+Thanks to the blocks,
+SelfHostBlocks provides an unified configuration interface
+for the services it provides.
+
+Compare the configuration for Nextcloud and Forgejo.
+The following snippets focus on similitudes and assume the relevant blocks - like secrets - are configured off-screen.
+It also does not show specific options for each service.
+These are still complete snippets that configure HTTPS,
+subdomain serving the service, LDAP and SSO integration.
+
+```nix
+shb.nextcloud = {
+  enable = true;
+  subdomain = "nextcloud";
+  domain = "example.com";
+
+  ssl = config.shb.certs.certs.letsencrypt.${domain};
+
+  apps.ldap = {
+    enable = true;
+    host = "127.0.0.1";
+    port = config.shb.lldap.ldapPort;
+    dcdomain = config.shb.lldap.dcdomain;
+    adminPassword.result = config.shb.sops.secret."nextcloud/ldap/admin_password".result;
+  };
+  apps.sso = {
+    enable = true;
+    endpoint = "https://${config.shb.authelia.subdomain}.${config.shb.authelia.domain}";
+
+    secret.result = config.shb.sops.secret."nextcloud/sso/secret".result;
+    secretForAuthelia.result = config.shb.sops.secret."nextcloud/sso/secretForAuthelia".result;
+  };
+};
+```
+
+```nix
+shb.forgejo = {
+  enable = true;
+  subdomain = "forgejo";
+  domain = "example.com";
+
+  ssl = config.shb.certs.certs.letsencrypt.${domain};
+
+  ldap = {
+    enable = true;
+    host = "127.0.0.1";
+    port = config.shb.lldap.ldapPort;
+    dcdomain = config.shb.lldap.dcdomain;
+    adminPassword.result = config.shb.sops.secret."nextcloud/ldap/admin_password".result;
+  };
+
+  sso = {
+    enable = true;
+    endpoint = "https://${config.shb.authelia.subdomain}.${config.shb.authelia.domain}";
+
+    secret.result = config.shb.sops.secret."forgejo/sso/secret".result;
+    secretForAuthelia.result = config.shb.sops.secret."forgejo/sso/secretForAuthelia".result;
+  };
+};
+```
+
+As you can see, they are pretty similar!
+This makes setting up a new service pretty easy and intuitive.
+
+SelfHostBlocks provides an ever growing list of [services](#preface-services)
+that are configured in the same way.
+
+### Contracts {#preface-contracts}
+
+To make building blocks that fit nicely together,
+SelfHostBlocks pioneers [contracts][] which allows you, the final user,
+to be more in control of which piece goes where.
+This lets you choose, for example,
+any reverse proxy you want or any database you want,
+without requiring work from maintainers of the services you want to self host.
+
+An [RFC][] exists to upstream this concept into `nixpkgs`.
+The [manual][contracts] also provides an explanation of the why and how of contracts.
+
+Also, two videos exist of me presenting the topic,
+the first at [NixCon North America in spring of 2024][NixConNA2024]
+and the second at [NixCon in Berlin in fall of 2024][NixConBerlin2024].
+
+[contracts]: https://shb.skarabox.com/contracts.html
+[RFC]: https://github.com/NixOS/rfcs/pull/189
+[NixConNA2024]: https://www.youtube.com/watch?v=lw7PgphB9qM
+[NixConBerlin2024]: https://www.youtube.com/watch?v=CP0hR6w1csc
+
+### Interfacing With Other OSes {#preface-interface}
+
+Thanks to [contracts](#contracts), one can interface NixOS
+with systems on other OSes.
+The [RFC][] explains how that works.
+
+### Sitting on the Shoulders of a Giant {#preface-giants}
+
+By using SelfHostBlocks, you get all the benefits of NixOS
+which are, for self hosted applications specifically:
+
+- declarative configuration;
+- atomic configuration rollbacks;
+- real programming language to define configurations;
+- create your own higher level abstractions on top of SelfHostBlocks;
+- integration with the rest of nixpkgs;
+- much fewer "works on my machine" type of issues.
+
+### Automatic Updates {#preface-updates}
+
+SelfHostBlocks follows nixpkgs unstable branch closely.
+There is a GitHub action running every couple of days that updates
+the `nixpkgs` input in the root `flakes.nix`,
+runs the tests and merges the PR automatically
+if the tests pass.
+
+A release is then made every few commits,
+whenever deemed sensible.
+On your side, to update I recommend pinning to a release
+with the following command,
+replacing the RELEASE with the one you want:
+
+```bash
+RELEASE=0.2.4
+nix flake update \
+  --override-input selfhostblocks github:ibizaman/selfhostblocks/$RELEASE \
+  selfhostblocks
+```
+
+### Demos {#preface-demos}
+
+Demos that start and deploy a service
+on a Virtual Machine on your computer are located
+under the [demo](./demo/) folder.
+
+These show the onboarding experience you would get
+if you deployed one of the services on your own server.
+
+## Roadmap {#preface-roadmap}
+
+Currently, the Nextcloud and Vaultwarden services
+and the SSL and backup blocks
+are the most advanced and most documented.
+
+Documenting all services and blocks will be done
+as I make all blocks and services use the contracts.
+
+Upstreaming changes is also on the roadmap.
+
+Check the [issues][] and the [milestones]() to see planned work.
+Feel free to add more or to contribute!
+
+[issues]: (https://github.com/ibizaman/selfhostblocks/issues)
+[milestones]: https://github.com/ibizaman/selfhostblocks/milestones
+
+All blocks and services have NixOS tests.
+Also, I am personally using all the blocks and services in this project, so they do work to some extent.
+
+## Community {#preface-community}
+
+This project has been the main focus
+of my (non work) life for the past 3 year now
+and I intend to continue working on this for a long time.
+
+All issues and PRs are welcome:
+
+- Use this project. Something does not make sense? Something's not working?
+- Documentation. Something is not clear?
+- New services. Have one of your preferred service not integrated yet?
+- Better patterns. See something weird in the code?
+
+For PRs, if they are substantial changes, please open an issue to
+discuss the details first. More details in [the contributing section](https://shb.skarabox.com/contributing.html)
+of the manual.
+
+Issues that are being worked on are labeled with the [in progress][] label.
+Before starting work on those, you might want to talk about it in the issue tracker
+or in the [matrix][] channel.
+
+The prioritized issues are those belonging to the [next milestone][milestone].
+Those issues are not set in stone and I'd be very happy to solve
+an issue an user has before scratching my own itch.
+
+[in progress]: https://github.com/ibizaman/selfhostblocks/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22in%20progress%22
+[matrix]: https://matrix.to/#/%23selfhostblocks%3Amatrix.org
+[milestone]: https://github.com/ibizaman/selfhostblocks/milestones
+
+One aspect that's close to my heart is I intent to make SelfHostBlocks the lightest layer on top of nixpkgs as
+possible. I want to upstream as much as possible. I will still take some time to experiment here but
+when I'm satisfied with how things look, I'll upstream changes.
+
+## Funding {#preface-funding}
+
+I was lucky to [obtain a grant][nlnet] from NlNet which is an European fund,
+under [NGI Zero Core][NGI0],
+to work on this project.
+This also funds the contracts RFC.
+
+Go apply for a grant too!
+
+[nlnet]: https://nlnet.nl/project/SelfHostBlocks
+[NGI0]: https://nlnet.nl/core/
+
+## License {#preface-license}
+
+I'm following the [Nextcloud](https://github.com/nextcloud/server) license which is AGPLv3.
+See [this article](https://www.fsf.org/bulletin/2021/fall/the-fundamentals-of-the-agplv3) from the FSF that explains what this license adds to the GPL one.

docs/recipes.md

@@ -0,0 +1,16 @@
+<!-- Read these docs at https://shb.skarabox.com -->
+# Recipes {#recipes}
+
+This section of the manual gives you easy to follow recipes for common use cases. 
+
+```{=include=} chapters html:into-file=//recipes-dnsServer.html
+recipes/dnsServer.md
+```
+
+```{=include=} chapters html:into-file=//recipes-exposeService.html
+recipes/exposeService.md
+```
+
+```{=include=} chapters html:into-file=//recipes-serveStaticPages.html
+recipes/serveStaticPages.md
+```

docs/recipes/dnsServer.md

@@ -0,0 +1,186 @@
+<!-- Read these docs at https://shb.skarabox.com -->
+# Self-Host a DNS server {#recipes-dnsServer}
+
+This recipe will show how to setup [dnsmasq][] as a local DNS server
+that forwards all queries to your own domain `example.com` to a local IP - your server running SelfHostBlocks for example.
+
+[dnsmasq]: https://dnsmasq.org/doc.html
+
+Other DNS queries will be forwarded to an external DNS server
+using [DNSSEC][] to encrypt your queries.
+
+[DNSSEC]: https://en.wikipedia.org/wiki/Domain_Name_System_Security_Extensions
+
+For this to work, you must configure the DHCP server of your network
+to set the DNS server to the IP of the host where the DNS server is running.
+Usually, your ISP's router can do this but probably easier is to disable completely that DHCP server
+and also self-host the DHCP server.
+This recipe shows how to do that too.
+
+## Why {#recipes-dnsServer-why}
+
+_You want to hide your DNS queries from your ISP or other prying eyes._
+
+Even if you use HTTPS to access an URL,
+DNS queries are by default made in plain text.
+Crazy, right?
+So, even if the actual communication is encrypted,
+everyone can see which site you're trying to access.
+Using DNSSEC means encrypting the traffic to your preferred external DNS server.
+Of course, that server will see what domain names you're trying to resolve,
+but at least intermediary hops will not be able to anymore.
+
+_You want more control on which DNS queries can be made._
+
+Self-hosting your own DNS server means you can block some domains or subdomains.
+This is done in practice by instructing your DNS server
+to fail resolving some domains or subdomains.
+Want to block Facebook for every host in the house?
+That's the way to go.
+
+Some routers allow this level of fine-tuning but if not,
+self-hosting your own DNS server is the way to go.
+
+## Drawbacks {#recipes-dnsServer-drawbacks}
+
+Although it has some nice advantages,
+self-hosting your own DNS server has one major drawback:
+if it goes down, the whole household will be impacted.
+By experience, it takes up to 5 minutes for others to notice something is wrong with internet.
+
+So be wary when you deploy a new config.
+
+## Recipe {#recipes-dnsServer-recipe}
+
+The following snippet:
+
+- Opens UDP port 53 in the firewall which is the ubiquitous (and hardcoded, crazy I know) port for DNS queries.
+- Disables the default DNS resolver.
+- Sets up dnsmasq as the DNS server.
+- Optionally sets up dnsmasq as the DHCP server.
+- Answers all DNS requests to your domain with the internal IP of the server.
+- Forwards all other DNS requests to an external DNS server using DNSSEC.
+  This is done using [stubby][].
+
+[stubby]: https://dnsprivacy.org/dns_privacy_daemon_-_stubby/
+
+For more information about options, read the dnsmasq [manual][].
+
+[manual]: https://dnsmasq.org/docs/dnsmasq-man.html
+
+```nix
+let
+  # Replace these values with what matches your network.
+  domain = "example.com";
+  serverIP = "192.168.1.30";
+
+  # This port is used internally for dnsmasq to talk to stubby on the loopback interface.
+  # Only change this if that port is already taken.
+  stubbyPort = 53000;
+in
+{
+  networking.firewall.allowedUDPPorts = [ 53 ];
+
+  services.resolved.enable = false;
+  services.dnsmasq = {
+    enable = true;
+    settings = {
+      inherit domain;
+
+      # Redirect queries to the stubby instance.
+      server = [
+        "127.0.0.1#${stubbyPort}"
+        "::1#${stubbyPort}"
+      ];
+      # We do trust our own instance of stubby
+      # so we can proxy DNSSEC stuff.
+      # I'm not sure how useful this is.
+      proxy-dnssec = true;
+
+      # Log all queries.
+      # This produces a lot of log lines
+      # and looking at those can be scary!
+      log-queries = true;
+
+      # Do not look at /etc/resolv.conf
+      no-resolv = true;
+
+      # Do not forward externally reverse DNS lookups for internal IPs.
+      bogus-priv = true;
+
+      address = [
+        "/.${domain}/${serverIP}"
+        # You can redirect anything anywhere too.
+        "/pikvm.${domain}/192.168.1.31"
+      ];
+    };
+  };
+
+  services.stubby = {
+    enable = true;
+    # It's a bit weird but default values comes from the examples settings hosted at
+    # https://github.com/getdnsapi/stubby/blob/develop/stubby.yml.example
+    settings = pkgs.stubby.passthru.settingsExample // {
+      listen_addresses = [
+        "127.0.0.1@${stubbyPort}"
+        "0::1@${stubbyPort}"
+      ];
+
+      # For more example of good DNS resolvers,
+      # head to https://dnsprivacy.org/public_resolvers/
+      #
+      # The digest comes from https://nixos.wiki/wiki/Encrypted_DNS#Stubby
+      upstream_recursive_servers = [
+        {
+          address_data = "9.9.9.9";
+          tls_auth_name = "dns.quad9.net";
+          tls_pubkey_pinset = [
+            {
+              digest = "sha256";
+              value = "i2kObfz0qIKCGNWt7MjBUeSrh0Dyjb0/zWINImZES+I=";
+            }
+          ];
+        }
+        {
+          address_data = "149.112.112.112";
+          tls_auth_name = "dns.quad9.net";
+          tls_pubkey_pinset = [
+            {
+              digest = "sha256";
+              value = "i2kObfz0qIKCGNWt7MjBUeSrh0Dyjb0/zWINImZES+I=";
+            }
+          ];
+        }
+      ];
+    };
+  };
+}
+```
+
+Optionally, to use dnsmasq as the DHCP server too,
+use the following snippet:
+
+```nix
+services.dnsmasq = {
+  settings = {
+    # When switching DNS server, accept old leases from previous server.
+    dhcp-authoritative = true;
+
+    # Adapt to your needs
+    # <ip-from>,<ip-to>,<mask>,<lease-ttl>
+    dhcp-range = "192.168.1.101,192.168.1.150,255.255.255.0,6h";
+
+    # Static DNS leases if needed.
+    # Choose an IP outside of the DHCP range
+    # <mac-address>,<DNS name>,<ip>,<lease-ttl>
+    dhcp-host = [
+      "12:34:56:78:9a:bc,server,192.168.1.50,infinite"
+    ];
+
+    # Set default route to the router that can acccess the internet.
+    dhcp-option = [
+      "3,192.168.1.1"
+    ];
+  };
+};
+```

docs/recipes/exposeService.md

@@ -0,0 +1,156 @@
+<!-- Read these docs at https://shb.skarabox.com -->
+# Expose a service {#recipes-exposeService}
+
+Let's see how one can use most of the blocks provided by SelfHostBlocks to make a service
+accessible through a reverse proxy with LDAP and SSO integration as well as backing up
+this service and creating a ZFS dataset to store the service's data.
+
+We'll use an hypothetical well made service found under `services.awesome` as our example.
+We're purposely not using a real service to avoid needing to deal with uninteresting particularities.
+
+## Service setup {#recipes-exposeService-service}
+
+Let's say our domain name is `example.com`,
+and we want to reach our service under the `awesome` subdomain:
+
+```nix
+let
+  domain = "example.com";
+  subdomain = "awesome";
+  fqdn = "${subdomain}.${domain}";
+  listenPort = 9000;
+  dataDir = "/var/lib/awesome";
+  ldapGroup = "awesome_user";
+in
+```
+
+We then `enable` the service and explicitly set the `listenPort` and `dataDir`,
+assuming those options exist:
+
+```nix
+services.awesome = {
+  enable = true;
+  inherit dataDir listenPort;
+};
+```
+
+## SSL Certificate {#recipes-exposeService-ssl}
+
+Requesting an SSL certificate from Let's Encrypt is done by adding an entry to
+the `extraDomains` option:
+
+```nix
+shb.certs.certs.letsencrypt.${domain}.extraDomains = [ fqdn ];
+```
+
+This assumes the `shb.certs` block has been configured:
+
+```nix
+shb.certs.certs.letsencrypt.${domain} = {
+  inherit domain;
+  group = "nginx";
+  reloadServices = [ "nginx.service" ];
+  adminEmail = "admin@${domain}";
+};
+```
+
+## LDAP group {#recipes-exposeService-ldap}
+
+We want only users of the group `calibre_user` to be able to access this subdomain.
+The following snippet creates the LDAP group:
+
+```nix
+shb.lldap.ensureGroups = {
+  calibre_user = {};
+};
+```
+
+## Reverse Proxy with Forward Auth {#recipes-exposeService-nginx}
+
+If our service does not integrate with OIDC, we can still protect it with SSO
+with forward authentication by letting the reverse proxy handle authentication.
+This is done by adding an entry to `shb.nginx.vhosts`:
+
+```nix
+shb.nginx.vhosts = [
+  {
+    inherit subdomain domain;
+    ssl = config.shb.certs.certs.letsencrypt.${domain};
+    upstream = "http://127.0.0.1:${toString listenPort}";
+    authEndpoint = "https://${config.shb.authelia.subdomain}.${config.shb.authelia.domain}";
+    autheliaRules = [{
+      policy = "one_factor";
+      subject = [ "group:${ldapGroup}" ];
+    }];
+  }
+];
+```
+
+## ZFS support {#recipes-exposeService-zfs}
+
+If you use ZFS, you can use SelfHostBlocks to create a dataset for you:
+
+```nix
+shb.zfs.datasets."safe/awesome".path = config.services.awesome.dataDir;
+```
+
+## Debugging {#recipes-exposeService-debug}
+
+Usually, the log level of the service can be increased with some option they provide.
+
+With SelfHostBlocks, you can also introspect any HTTP service by adding an
+`mitmdump` instance between the reverse proxy and the `awesome` service:
+
+```nix
+shb.mitmdump.awesome = {
+  inherit listenPort;
+  upstreamPort = listenPort + 1;
+};
+services.awesome.listenPort = lib.mkForce (listenPort + 1);
+```
+
+This creates a `mitmdump-awesome.service` systemd service which prints the requests' and responses' headers and bodies.
+
+## Backup {#recipes-exposeService-backup}
+
+The following snippet uses the `shb.restic` block to backup the `services.awesome.dataDir` directory:
+
+```nix
+shb.restic.instances.awesome = {
+  request.user = "awesome";
+  request.sourceDirectories = [ dataDir ];
+  settings.enable = true;
+  settings.passphrase.result = config.shb.sops.secret.awesome.result;
+  settings.repository.path = config.services.awesome.dataDir;
+};
+
+shb.sops.secret."awesome" = {
+  request = config.shb.restic.instances.awesome.settings.passphrase.request;
+};
+```
+
+## Impermanence {#recipes-exposeService-impermanence}
+
+To save the data folder in an impermanence setup, add:
+
+```nix
+{
+  shb.zfs.datasets."safe/awesome".path = config.services.awesome.dataDir;
+}
+```
+
+## Application Dashboard {#recipes-exposeService-applicationdashboard}
+
+For example using the [Homepage](services-homepage.html) service:
+
+```nix
+{
+  shb.homepage.servicesGroups.MyServices.services.Awesome = {
+    sortOrder = 1;
+    dashboard.request = {
+      externalUrl = "https://${fqdn}";
+      internalUrl = "http://127.0.0.1:${toString listenPort}";
+    };
+  };
+}
+```

docs/recipes/serveStaticPages.md

@@ -0,0 +1,75 @@
+<!-- Read these docs at https://shb.skarabox.com -->
+# Serve Static Pages {#recipes-serveStaticPages}
+
+This recipe shows how to use SelfHostBlocks blocks to serve static web pages using the Nginx reverse proxy with SSL termination.
+
+In this recipe, we'll assume the pages to serve are found under the `/srv/my-website` path and will be served under the `my-website.example.com` fqdn.
+
+```nix
+let
+  name = "my-website";
+  subdomain = name;
+  domain = "example.com";
+  fqdn = "${subdomain}.${domain}";
+  user = "me";
+in
+```
+
+We also assume the static web pages are owned and updated by the user named `me`.
+
+## ZFS dataset {#recipes-serveStaticPages-zfs}
+
+We can create a ZFS dataset with:
+
+```nix
+shb.zfs.datasets."safe/${name}".path = "/srv/${name}";
+```
+
+## SSL Certificate {#recipes-serveStaticPages-ssl}
+
+Requesting an SSL certificate from Let's Encrypt is done by adding an entry to
+the `extraDomains` option:
+
+```nix
+shb.certs.certs.letsencrypt.${domain}.extraDomains = [ fqdn ];
+```
+
+This assumes the `shb.certs` block has been configured:
+
+```nix
+shb.certs.certs.letsencrypt.${domain} = {
+  inherit domain;
+  group = "nginx";
+  reloadServices = [ "nginx.service" ];
+  adminEmail = "admin@${domain}";
+};
+```
+
+## Reverse Proxy {#recipes-serveStaticPages-nginx}
+
+First, we make the parent directory owned by the user which will upload them and `nginx`:
+
+```nix
+systemd.tmpfiles.rules = lib.mkBefore [
+  "d '/srv/${name}' 0750 ${user} nginx - -"
+];
+```
+
+Now, we can setup nginx. The following snippet serves files from the `/srv/${name}/` directory.
+
+```nix
+services.nginx.enable = true;
+
+services.nginx.virtualHosts."skarabox.${domain}" = {
+  forceSSL = true;
+  sslCertificate = config.shb.certs.certs.letsencrypt."${domain}".paths.cert;
+  sslCertificateKey = config.shb.certs.certs.letsencrypt."${domain}".paths.key;
+  locations."/" = {
+    root = "/srv/${name}/";
+    extraConfig = ''
+      add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
+      add_header Cache-Control "max-age=604800, stale-while-revalidate=86400, stale-if-error=86400, must-revalidate, public";
+    '';
+  };
+};
+```

docs/service-implementation-guide.md

@@ -0,0 +1,629 @@
+# SelfHostBlocks Service Implementation Guide {#service-implementation-guide}
+
+This guide documents the complete process for implementing a new service in SelfHostBlocks, based on lessons learned from the nzbget implementation and analysis of existing service patterns.
+
+**Note**: SelfHostBlocks aims to be "the smallest amount of code above what is available in nixpkgs" (see `docs/contributing.md`). Services should leverage existing nixpkgs options when possible and focus on providing contract integrations rather than reimplementing configuration.
+
+## What Makes a "Complete" SHB Service {#complete-shb-service}
+
+According to the project maintainer's criteria, a service is considered fully supported if it includes:
+
+1. **SSL block integration** - HTTPS/TLS certificate management
+2. **Backup block integration** - Automated backup of service data
+3. **Monitoring integration** - Prometheus metrics and health checks
+4. **LDAP (LLDAP) integration** - Directory-based authentication
+5. **SSO (Authelia) integration** - Single sign-on authentication
+6. **Comprehensive tests** - All integration variants tested
+
+## Pre-Implementation Research {#pre-implementation-research}
+
+### 1. Analyze Existing Services {#analyze-existing-services}
+
+Before starting, study existing services to understand patterns:
+
+```bash
+# Study service patterns
+ls modules/services/          # List all services
+cat modules/services/deluge.nix   # Best practice example
+cat modules/services/vaultwarden.nix  # Another good example
+```
+
+**Key patterns to identify:**
+- Configuration structure and options
+- How contracts are used (SSL, backup, monitoring, secrets)
+- Authentication integration approaches
+- Service-specific settings and defaults
+
+### 2. Understand the Target Service {#understand-target-service}
+
+Research the service you're implementing:
+- **Configuration format** (YAML, INI, JSON, etc.)
+- **Authentication methods** (built-in users, LDAP, OIDC/OAuth)
+- **API endpoints** (for monitoring/health checks)
+- **Data directories** (what needs backing up)
+- **Network requirements** (ports, protocols)
+- **Dependencies** (databases, external tools)
+
+### 3. Check NixOS Integration {#check-nixos-integration}
+
+Verify nixpkgs support:
+```bash
+# Check if NixOS service exists
+nix eval --impure --expr '(import <nixpkgs/nixos> { configuration = {...}: {}; }).options.services' --apply 'builtins.attrNames' --json | jq -r '.[]' | grep -i servicename
+# or search online: https://search.nixos.org/options?query=services.servicename
+```
+
+If no nixpkgs integration exists, you may need to:
+- Package the service first
+- Use containerized approach
+- Request upstream nixpkgs integration
+
+## Implementation Steps {#implementation-steps}
+
+### 1. Create the Service Module {#create-service-module}
+
+Location: `modules/services/servicename.nix`
+
+**Basic structure:**
+```nix
+{ config, pkgs, lib, ... }:
+
+let
+  cfg = config.shb.servicename;
+  contracts = pkgs.callPackage ../contracts {};
+  fqdn = "${cfg.subdomain}.${cfg.domain}";
+  
+  # Choose appropriate format based on service config
+  settingsFormat = pkgs.formats.yaml {};  # or .ini, .json, etc.
+in
+{
+  options.shb.servicename = {
+    # Core options (always required)
+    enable = lib.mkEnableOption "selfhostblocks.servicename";
+    subdomain = lib.mkOption { ... };
+    domain = lib.mkOption { ... };
+    
+    # SSL integration (always include)
+    ssl = lib.mkOption {
+      description = "Path to SSL files";
+      type = lib.types.nullOr contracts.ssl.certs;
+      default = null;
+    };
+    
+    # Service-specific options
+    port = lib.mkOption { ... };
+    dataDir = lib.mkOption { ... };
+    settings = lib.mkOption {
+      type = lib.types.submodule {
+        freeformType = settingsFormat.type;
+        options = {
+          # Define key options with descriptions
+        };
+      };
+    };
+    
+    # Authentication options
+    authEndpoint = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      description = "OIDC endpoint for SSO";
+      default = null;
+    };
+    
+    ldap = lib.mkOption { ... };  # LDAP integration
+    users = lib.mkOption { ... }; # Local user management
+    
+    # Integration options
+    backup = lib.mkOption {
+      type = lib.types.submodule {
+        options = contracts.backup.mkRequester {
+          user = "servicename";
+          sourceDirectories = [ cfg.dataDir ];
+        };
+      };
+    };
+    
+    monitoring = lib.mkOption {
+      type = lib.types.nullOr (lib.types.submodule {
+        options = {
+          # Service-specific monitoring options
+        };
+      });
+      default = null;
+    };
+    
+    # System options
+    extraServiceConfig = lib.mkOption { ... };
+    logLevel = lib.mkOption { ... };
+  };
+
+  config = lib.mkIf cfg.enable (lib.mkMerge [
+    {
+      # Base service configuration
+      services.servicename = {
+        enable = true;
+        # Map SHB options to nixpkgs service options
+      };
+      
+      # Nginx reverse proxy
+      shb.nginx.vhosts = [{
+        inherit (cfg) subdomain domain ssl;
+        upstream = "http://127.0.0.1:${toString cfg.port}";
+        
+        # SSO integration
+        autheliaRules = lib.mkIf (cfg.authEndpoint != null) [
+          {
+            domain = fqdn;
+            policy = "bypass";
+            resources = [ "^/api" ];  # API endpoints
+          }
+          {
+            domain = fqdn;
+            policy = "two_factor";
+            resources = [ "^.*" ];    # Everything else
+          }
+        ];
+      }];
+      
+      # User/group setup
+      users.users.servicename = {
+        extraGroups = [ "media" ];  # If needed for file access
+      };
+      
+      # Directory permissions
+      systemd.tmpfiles.rules = [
+        "d ${cfg.dataDir} 0755 servicename servicename - -"
+      ];
+    }
+    
+    # Monitoring configuration (conditional)
+    (lib.mkIf (cfg.monitoring != null) {
+      services.prometheus.scrapeConfigs = [{
+        job_name = "servicename";
+        static_configs = [{
+          targets = [ "127.0.0.1:${toString cfg.port}" ];
+          labels = {
+            hostname = config.networking.hostName;
+            domain = cfg.domain;
+          };
+        }];
+        metrics_path = "/metrics";  # or appropriate endpoint
+        scrape_interval = "30s";
+      }];
+    })
+  ]);
+}
+```
+
+### 2. Key Implementation Considerations {#implementation-considerations}
+
+#### Configuration Management {#configuration-management}
+- **Use freeform settings** when possible: `freeformType = settingsFormat.type`
+- **Provide sensible defaults** for common options
+- **Use lib.mkDefault** for user-overridable settings
+- **Use lib.mkForce** for security-critical settings
+
+#### Authentication Integration {#authentication-integration}
+- **SSO (Authelia)**: Use `autheliaRules` with appropriate bypass policies
+- **LDAP**: Follow the patterns from existing services
+- **Local users**: Use SHB secret contracts for password management
+
+#### Security Best Practices {#security-best-practices}
+- **Bind to localhost**: Services should listen on `127.0.0.1` only
+- **Use nginx for TLS**: Don't configure TLS in the service itself
+- **Proper file permissions**: Use systemd.tmpfiles.rules
+- **Secret management**: Always use SHB secret contracts
+
+### 3. Monitoring Implementation {#monitoring-implementation}
+
+Choose the appropriate monitoring approach:
+
+#### Option A: Native Prometheus Metrics {#native-prometheus-metrics}
+If the service supports Prometheus natively:
+```nix
+services.prometheus.scrapeConfigs = [{
+  job_name = "servicename";
+  static_configs = [{ targets = [ "127.0.0.1:${toString cfg.port}" ]; }];
+  metrics_path = "/metrics";
+}];
+```
+
+#### Option B: API Health Check {#api-health-check}
+If no native metrics, monitor API endpoints:
+```nix
+services.prometheus.scrapeConfigs = [{
+  job_name = "servicename";
+  static_configs = [{ targets = [ "127.0.0.1:${toString cfg.port}" ]; }];
+  metrics_path = "/api/status";  # or appropriate endpoint
+}];
+```
+
+#### Option C: External Exporter {#external-exporter}
+For services requiring dedicated exporters (like Deluge):
+```nix
+services.prometheus.exporters.servicename = {
+  enable = true;
+  # exporter-specific configuration
+};
+```
+
+### 4. Create Comprehensive Tests {#create-comprehensive-tests}
+
+Location: `test/services/servicename.nix`
+
+**Test structure:**
+```nix
+{ pkgs, ... }:
+let
+  testLib = pkgs.callPackage ../common.nix {};
+  
+  # Common test scripts
+  commonTestScript = testLib.mkScripts {
+    hasSSL = { node, ... }: !(isNull node.config.shb.servicename.ssl);
+    waitForServices = { ... }: [ "nginx.service" "servicename.service" ];
+    waitForPorts = { node, ... }: [ node.config.services.servicename.port ];
+    
+    # Service-specific connectivity test
+    extraScript = { node, proto_fqdn, ... }: ''
+      with subtest("service connectivity"):
+          response = curl(client, "", "${proto_fqdn}/api/health")
+          # Add service-specific checks
+    '';
+  };
+  
+  # Monitoring test script
+  prometheusTestScript = { nodes, ... }: ''
+    server.wait_for_open_port(${toString nodes.server.config.services.servicename.port})
+    with subtest("prometheus monitoring"):
+        # Test the actual monitoring endpoint
+        response = server.succeed("curl -sSf http://localhost:${port}/metrics")
+        # Validate response format
+  '';
+  
+  # Base configuration
+  basic = { config, ... }: {
+    imports = [
+      testLib.baseModule
+      ../../modules/services/servicename.nix
+    ];
+    
+    shb.servicename = {
+      enable = true;
+      inherit (config.test) domain subdomain;
+      # Basic configuration
+    };
+  };
+  
+in {
+  # Test variants (all 6 required)
+  basic = lib.shb.test.runNixOSTest { ... };
+  backup = lib.shb.test.runNixOSTest { ... };
+  https = lib.shb.test.runNixOSTest { ... };
+  ldap = lib.shb.test.runNixOSTest { ... };
+  monitoring = lib.shb.test.runNixOSTest { ... };
+  sso = lib.shb.test.runNixOSTest { ... };
+}
+```
+
+#### Required Test Variants {#required-test-variants}
+
+1. **basic**: Core functionality without authentication
+2. **backup**: Tests backup integration
+3. **https**: Tests SSL/TLS integration  
+4. **ldap**: Tests LDAP authentication
+5. **monitoring**: Tests Prometheus integration
+6. **sso**: Tests Authelia SSO integration
+
+### 5. Update Flake Configuration {#update-flake-configuration}
+
+Add to `flake.nix`:
+
+```nix
+allModules = [
+  # ... existing modules
+  modules/services/servicename.nix
+];
+```
+
+```nix
+checks = {
+  # ... existing checks
+  // (vm_test "servicename" ./test/services/servicename.nix)
+};
+```
+
+### 6. Create Service Documentation {#create-service-documentation}
+
+Create comprehensive documentation for the new service:
+
+**Location**: `modules/services/servicename/docs/default.md`
+
+```markdown
+# ServiceName Service {\#services-servicename}
+
+Brief description of what the service does.
+
+## Features {\#services-servicename-features}
+
+- Feature 1
+- Feature 2
+
+## Usage {\#services-servicename-usage}
+
+### Basic Configuration {\#services-servicename-basic}
+
+shb.servicename = {
+  enable = true;
+  domain = "example.com";
+  subdomain = "servicename";
+};
+
+### SSL Configuration {\#services-servicename-ssl}
+
+shb.servicename.ssl.paths = {
+  cert = /path/to/cert;
+  key = /path/to/key;
+};
+
+## Options Reference {\#services-servicename-options}
+
+{=include=} options
+id-prefix: services-servicename-options-
+list-id: selfhostblocks-servicename-options
+source: @OPTIONS_JSON@
+```
+
+**Important**: Use consistent heading ID patterns:
+- Service overview: `{\#services-servicename}`  
+- Features: `{\#services-servicename-features}`
+- Usage sections: `{\#services-servicename-basic}`, `{\#services-servicename-ssl}`, etc.
+- Options: `{\#services-servicename-options}`
+
+Note: Replace `servicename` with your actual service name (e.g., `nzbget`, `jellyfin`).
+
+For the `@OPTIONS_JSON@` to work, a line must be added
+in the `flake.nix` file:
+
+```nix
+packages.manualHtml = pkgs.callPackage ./docs {
+  modules = {
+    "blocks/authelia" = ./modules/blocks/authelia.nix;
+    // Add line and keep in alphabetical order.
+  };
+};
+```
+
+### 7. Update Redirects Automatically {#update-redirects-automatically}
+
+After creating documentation, generate the required redirects:
+
+```bash
+# Scan documentation and add missing redirects
+nix run .#update-redirects
+
+# Review the changes
+git diff docs/redirects.json
+
+# The tool will show what redirects were added
+```
+
+The automation will:
+- Find all heading IDs in your documentation
+- Generate appropriate redirect entries
+- Add them to `docs/redirects.json`
+- Follow established naming patterns
+
+### 8. Handle Unfree Dependencies {#handle-unfree-dependencies}
+
+If the service requires unfree packages:
+
+```nix
+# In flake.nix
+config = {
+  allowUnfree = true;
+  permittedInsecurePackages = [
+    # List any required insecure packages
+  ];
+};
+```
+
+Update CI workflow if needed:
+```yaml
+# In .github/workflows/build.yaml
+- name: Setup Nix
+  uses: cachix/install-nix-action@v31
+  with:
+    extra_nix_config: |
+      allow-unfree = true
+```
+
+## Testing and Validation {#testing-and-validation}
+
+### Local Testing {#local-testing}
+```bash
+# Test redirect automation
+nix run .#update-redirects
+
+# Test all service variants (replace ${system} with your system, e.g., x86_64-linux)
+nix build .#checks.${system}.vm_servicename_basic
+nix build .#checks.${system}.vm_servicename_backup
+nix build .#checks.${system}.vm_servicename_https
+nix build .#checks.${system}.vm_servicename_ldap
+nix build .#checks.${system}.vm_servicename_monitoring
+nix build .#checks.${system}.vm_servicename_sso
+
+# Or run all tests (as recommended in docs/contributing.md)
+nix flake check
+
+# For interactive testing and debugging, see docs/contributing.md:
+# nix run .#checks.${system}.vm_servicename_basic.driverInteractive
+
+# Test documentation build (includes redirect validation)
+nix build .#manualHtml
+```
+
+To continuously rebuild the documentation of file change, run the following command.
+To exit, you'll need to do Ctrl-C twice in a row.
+
+```bash
+nix run .#manualHtml-watch
+```
+
+### Iterative Development Approach {#iterative-development-approach}
+
+1. **Start with basic functionality** - get core service working
+2. **Add SSL integration** - enable HTTPS
+3. **Add backup integration** - ensure data protection
+4. **Add monitoring** - implement health checks
+5. **Add authentication** - LDAP and SSO integration
+6. **Create documentation** - write service documentation with heading IDs
+7. **Update redirects** - run `nix run .#update-redirects` to generate redirects
+8. **Comprehensive testing** - all 6 test variants
+9. **Final validation** - ensure documentation builds correctly
+
+## Common Pitfalls and Solutions {#common-pitfalls-and-solutions}
+
+### Configuration Issues {#configuration-issues}
+- **Problem**: Service doesn't start due to config validation
+- **Solution**: Use `lib.mkDefault` for user settings, `lib.mkForce` for security settings
+
+### Authentication Integration {#authentication-integration-pitfalls}
+- **Problem**: SSO redirect loops or access denied
+- **Solution**: Check `autheliaRules` bypass patterns for API endpoints
+
+### Monitoring Failures {#monitoring-failures}
+- **Problem**: Prometheus scraping fails with 404
+- **Solution**: Verify the actual API endpoints the service provides
+
+### Test Failures {#test-failures}
+- **Problem**: VM tests timeout or fail connectivity
+- **Solution**: Check `waitForServices` and `waitForPorts` configurations
+
+### Nixpkgs Integration {#nixpkgs-integration}
+- **Problem**: Service options don't match SHB needs
+- **Solution**: Map SHB options to nixpkgs options, use `extraConfig` for overrides
+
+## Best Practices Summary {#best-practices-summary}
+
+1. **Follow existing patterns** - study deluge.nix and vaultwarden.nix
+2. **Use freeform configuration** - maximum flexibility with typed key options
+3. **Implement all contracts** - SSL, backup, monitoring, secrets
+4. **Test comprehensively** - all 6 integration variants
+5. **Security first** - localhost binding, proper permissions, secret management
+6. **Document thoroughly** - clear descriptions for all options
+7. **Iterative development** - build complexity gradually
+8. **CI/CD validation** - ensure all tests pass before submission
+
+## Redirect Management {#redirect-management}
+
+SelfHostBlocks uses `nixos-render-docs` for documentation generation, which includes built-in redirect validation. The `docs/redirects.json` file maps documentation identifiers to their target URLs.
+
+### Automated Redirect Generation {#automated-redirect-generation}
+
+SelfHostBlocks includes an automated redirect management tool that leverages the official `nixos-render-docs` ecosystem:
+
+```bash
+# Generate fresh redirects from HTML documentation
+nix run .#update-redirects
+```
+
+This tool:
+- **Generates HTML documentation** using `nixos-render-docs` with redirect collection enabled
+- **Scans actual HTML files** for anchor IDs to ensure perfect accuracy
+- **Creates fresh redirects** from scratch by mapping anchors to their real file locations
+- **Filters system-generated anchors** (excludes `opt-*` and `selfhostblock*` entries)
+- **Provides interactive confirmation** before updating `docs/redirects.json`
+
+### How Redirects Work {#how-redirects-work}
+
+1. **nixos-render-docs validation**: During documentation builds, `nixos-render-docs` automatically validates that all heading IDs have corresponding redirect entries
+2. **Automated maintenance**: The `update-redirects` tool automatically maintains `redirects.json` by:
+   - Building HTML documentation with patched `nixos-render-docs`
+   - Scanning generated HTML files for actual anchor IDs and their file locations
+   - Creating accurate redirect mappings without guesswork or pattern matching
+3. **Manual override**: You can still manually edit `docs/redirects.json` for special cases
+
+### Redirect Patterns {#redirect-patterns}
+
+The automation follows these patterns when mapping headings to redirect targets:
+
+| Heading ID | Source File | Redirect Target | 
+|------------|-------------|-----------------|
+| `services-nzbget-basic` | `modules/services/nzbget/docs/default.md` | `["services-nzbget.html#services-nzbget-basic"]` |
+| `blocks-monitoring` | `modules/blocks/monitoring/docs/default.md` | `["blocks-monitoring.html#blocks-monitoring"]` |
+| `demo-nextcloud` | `demo/nextcloud/README.md` | `["demo-nextcloud.html#demo-nextcloud"]` |
+| `contracts` | `docs/contracts.md` | `["contracts.html#contracts"]` |
+
+Note: Redirects always include the anchor link (`#heading-id`) to jump to the specific heading within the target page.
+
+### Adding New Service Documentation {#adding-new-service-documentation}
+
+When implementing a new service, the redirect workflow is now automated:
+
+1. **Write documentation** with heading IDs:
+   ```markdown
+   # NewService {\#services-newservice}
+   ## Basic Configuration {\#services-newservice-basic}
+   ```
+
+2. **Update redirects automatically**:
+   ```bash
+   nix run .#update-redirects
+   ```
+
+3. **Review and commit** the changes:
+   ```bash
+   git add docs/redirects.json modules/services/newservice/docs/default.md
+   git commit -m "Add newservice documentation"
+   ```
+
+### Build-time Validation {#build-time-validation}
+
+The documentation build process will fail if:
+- Any documentation heading ID lacks a corresponding redirect entry
+- Redirect targets point to non-existent content  
+- There are formatting errors in the redirects file
+
+This ensures documentation links remain functional when content is moved or reorganized.
+
+## Resources {#resources}
+
+- **Contributing guide**: `docs/contributing.md` for authoritative development workflows and testing procedures
+- **Existing services**: `modules/services/` for patterns and implementation examples
+- **Contracts documentation**: `modules/contracts/` for understanding integration interfaces
+- **Test framework**: `test/common.nix` for testing utilities and patterns
+- **NixOS options**: https://search.nixos.org/options for upstream service options
+- **SHB documentation**: Generated docs showing existing service patterns
+- **Redirect automation**: `nix run .#update-redirects` for automated redirect management
+- **nixos-render-docs**: Built-in redirect validation and documentation generation
+
+## Quick Reference {#quick-reference}
+
+### Complete Workflow {#complete-workflow}
+```bash
+# 1. Implement service module
+vim modules/services/SERVICENAME.nix
+
+# 2. Create tests  
+vim test/services/SERVICENAME.nix
+
+# 3. Update flake
+vim flake.nix  # Add to allModules and checks
+
+# 4. Write documentation
+vim modules/services/SERVICENAME/docs/default.md
+
+# 5. Generate redirects
+nix run .#update-redirects
+
+# 6. Test everything  
+nix flake check  # Run all tests (recommended)
+# Or test specific variants:
+# nix build .#checks.${system}.vm_SERVICENAME_basic
+nix build .#manualHtml
+
+# 7. Commit changes
+git add .
+git commit -m "Add SERVICENAME with full integration"
+```
+
+This guide provides a complete roadmap for implementing production-ready SelfHostBlocks services that meet the project's quality standards.

docs/services.md

@@ -1,7 +1,10 @@
+<!-- Read these docs at https://shb.skarabox.com -->
 # Services {#services}
 
-Services are usually web applications that SHB help you self-host. Configuration of those is
-purposely made opinionated and require as few nix options as can make sense. That is possible thanks to the extensive use of blocks provided by SHB.
+Services are usually web applications that SHB help you self-host some of your data.
+Configuration of those is purposely made more opinionated than the upstream nixpkgs modules
+in exchange for an uniformized configuration experience.
+That is possible thanks to the extensive use of blocks provided by SHB.
 
 ::: {.note}
 Not all services are yet documented. You can find all available services [in the repository](@REPO@/modules/services).
@@ -10,17 +13,105 @@ Not all services are yet documented. You can find all available services [in the
 The following table summarizes for each documented service what features it provides. More
 information is provided in the respective manual sections.
 
-| Service               | Backup | Reverse Proxy | SSO | LDAP  | Monitoring | Profiling |
-|-----------------------|--------|---------------|-----|-------|------------|-----------|
-| [Nextcloud Server][1] | P (1)  | Y             | Y   | Y     | Y          | P (2)     |
+| Service                     | Backup | Reverse Proxy | SSO | LDAP  | Monitoring | Profiling |
+|-----------------------------|--------|---------------|-----|-------|------------|-----------|
+| [*Arr][]                    | Y (1)  | Y             | Y   | Y (4) | Y (2)      | N         |
+| [Firefly-iii][]             | Y (1)  | Y             | Y   | Y     | Y (2)      | N         |
+| [Forgejo][]                 | Y (1)  | Y             | Y   | Y     | Y (2)      | N         |
+| [Home-Assistant][]          | Y (1)  | Y             | N   | Y     | Y (2)      | N         |
+| [Homepage][]                | Y (1)  | Y             | N   | Y     | Y (2)      | N         |
+| [Jellyfin][]                | Y (1)  | Y             | Y   | Y     | Y (2)      | N         |
+| [Karakeep][]                | Y (1)  | Y             | Y   | Y     | Y (2)      | N         |
+| [Nextcloud Server][]        | Y (1)  | Y             | Y   | Y     | Y (2)      | P (3)     |
+| [Open WebUI][]              | Y (1)  | Y             | Y   | Y     | Y (2)      | N         |
+| [Pinchflat][]               | Y      | Y             | Y   | Y (4) | Y (5)      | N         |
+| [Simple NixOS Mailserver][] | Y      | Y             | N   | Y     | Y          | N         |
+| [Vaultwarden][]             | Y (1)  | Y             | Y   | Y     | Y (2)      | N         |
 
 Legend: **N**: no but WIP; **P**: partial; **Y**: yes
 
-1. Does not backup the database yet.
-2. Works but the traces are not exported to Grafana yet.
+1. Database and data files are backed up separately.
+   This could lead to backups not being in sync.
+   Any idea on how to fix this is welcomed!
+2. Dashboard is common to all services.
+3. Works but the traces are not exported to Grafana yet.
+4. Uses LDAP indirectly through forward auth.
 
-[1]: services-nextcloud.html
+[*Arr]: services-arr.html
+[Firefly-iii]: services-firefly-iii.html
+[Forgejo]: services-forgejo.html
+[Home-Assistant]: services-home-assistant.html
+[Homepage]: services-homepage.html
+[Jellyfin]: services-jellyfin.html
+[Karakeep]: services-karakeep.html
+[Nextcloud Server]: services-nextcloud.html
+[Open WebUI]: services-open-webui.html
+[Pinchflat]: services-pinchflat.html
+[Simple NixOS Mailserver]: services-mailserver.html
+[Vaultwarden]: services-vaultwarden.html
+
+## Dashboard {#services-category-dashboard}
+
+```{=include=} chapters html:into-file=//services-homepage.html
+modules/services/homepage/docs/default.md
+```
+
+## Documents {#services-category-documents}
 
 ```{=include=} chapters html:into-file=//services-nextcloud.html
 modules/services/nextcloud-server/docs/default.md
 ```
+
+## Emails {#services-category-emails}
+
+```{=include=} chapters html:into-file=//services-mailserver.html
+modules/services/mailserver/docs/default.md
+```
+
+## Passwords {#services-category-passwords}
+
+```{=include=} chapters html:into-file=//services-vaultwarden.html
+modules/services/vaultwarden/docs/default.md
+```
+
+## Automation {#services-category-automation}
+
+```{=include=} chapters html:into-file=//services-home-assistant.html
+modules/services/home-assistant/docs/default.md
+```
+
+## AI {#services-category-ai}
+
+```{=include=} chapters html:into-file=//services-karakeep.html
+modules/services/karakeep/docs/default.md
+```
+
+```{=include=} chapters html:into-file=//services-open-webui.html
+modules/services/open-webui/docs/default.md
+```
+
+## Code {#services-category-code}
+
+```{=include=} chapters html:into-file=//services-forgejo.html
+modules/services/forgejo/docs/default.md
+```
+
+## Media {#services-category-media}
+
+```{=include=} chapters html:into-file=//services-arr.html
+modules/services/arr/docs/default.md
+```
+
+```{=include=} chapters html:into-file=//services-jellyfin.html
+modules/services/jellyfin/docs/default.md
+```
+
+```{=include=} chapters html:into-file=//services-pinchflat.html
+modules/services/pinchflat/docs/default.md
+```
+
+## Finance {#services-category-finance}
+
+```{=include=} chapters html:into-file=//services-firefly-iii.html
+modules/services/firefly-iii/docs/default.md
+```

docs/usage.md

@@ -1,21 +1,138 @@
+<!-- Read these docs at https://shb.skarabox.com -->
+
 # Usage {#usage}
 
 ## Flake {#usage-flake}
 
-Self Host Blocks is available as a flake. To use it in your project, add the following flake input:
+Self Host Blocks (SHB) is available as a flake. It also uses its own `pkgs.lib` and
+`nixpkgs` and it is required to use the provided ones as input for your
+deployments, otherwise you might end up blocked when SHB patches a
+module, function or package. The following snippet is thus required to use Self
+Host Blocks:
 
 ```nix
-inputs.selfhostblocks.url = "github:ibizaman/selfhostblocks";
+{
+  inputs.selfhostblocks.url = "github:ibizaman/selfhostblocks";
+
+  outputs = { selfhostblocks, ... }: let
+    system = "x86_64-linux";
+    nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
+  in
+    nixosConfigurations = {
+      myserver = nixpkgs'.nixosSystem {
+        inherit system;
+        modules = [
+          selfhostblocks.nixosModules.default
+          ./configuration.nix
+        ];
+      };
+    };
+}
 ```
 
-Then, in your `nixosConfigurations`, import the module with:
+::: {.note}
+In case somehow this documentation became stale,
+look at the examples in [`./demo/minimal/flake.nix`](@REPO@/demo/minimal/flake.nix)
+which provides examples tested in CI - so assured to always be up to date -
+on how to use SHB.
+:::
 
-```nix
-imports = [
-  inputs.selfhostblocks.nixosModules.x86_64-linux.default
-];
+### Modules {#usage-flake-modules}
+
+The `default` module imports all modules except the SOPS module.
+That module is only needed if you want to use [sops-nix](#usage-secrets) to manage secrets.
+
+You can also import each module individually.
+You might want to do this to only import SHB overlays if you actually intend to use them.
+Importing the `nextcloud` module for example will anyway transitively import needed support modules
+so you can't go wrong:
+
+```diff
+         modules = [
+-          selfhostblocks.nixosModules.default
++          selfhostblocks.nixosModules.nextcloud
+           ./configuration.nix
+         ];
 ```
 
+To list all modules, run:
+
+```bash
+$ nix flake show github:ibizaman/selfhostblocks --allow-import-from-derivation
+
+...
+
+├───nixosModules
+│   ├───arr: NixOS module
+│   ├───audiobookshelf: NixOS module
+│   ├───authelia: NixOS module
+│   ├───borgbackup: NixOS module
+│   ├───davfs: NixOS module
+│   ├───default: NixOS module
+│   ├───deluge: NixOS module
+│   ├───forgejo: NixOS module
+│   ├───grocy: NixOS module
+│   ├───hardcodedsecret: NixOS module
+│   ├───hledger: NixOS module
+│   ├───home-assistant: NixOS module
+│   ├───immich: NixOS module
+│   ├───jellyfin: NixOS module
+│   ├───karakeep: NixOS module
+│   ├───lib: NixOS module
+│   ├───lldap: NixOS module
+│   ├───mitmdump: NixOS module
+│   ├───monitoring: NixOS module
+│   ├───nextcloud-server: NixOS module
+│   ├───nginx: NixOS module
+│   ├───open-webui: NixOS module
+│   ├───paperless: NixOS module
+│   ├───pinchflat: NixOS module
+│   ├───postgresql: NixOS module
+│   ├───restic: NixOS module
+│   ├───sops: NixOS module
+│   ├───ssl: NixOS module
+│   ├───tinyproxy: NixOS module
+│   ├───vaultwarden: NixOS module
+│   ├───vpn: NixOS module
+│   └───zfs: NixOS module
+
+...
+```
+
+### Patches {#usage-flake-patches}
+
+To add your own patches on top of the patches provided by SHB,
+you can remove the `patchedNixpkgs` line and instead apply the patches yourself:
+
+```diff
+-    nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
++    pkgs = import selfhostblocks.inputs.nixpkgs {
++      inherit system;
++    };
++    nixpkgs' = pkgs.applyPatches {
++      name = "nixpkgs-patched";
++      src = selfhostblocks.inputs.nixpkgs;
++      patches = selfhostblocks.lib.${system}.patches;
++    };
++    nixosSystem' = import "${nixpkgs'}/nixos/lib/eval-config.nix";
+   in
+     nixosConfigurations = {
+-      myserver = nixpkgs'.nixosSystem {
++      myserver = nixosSystem' {
+```
+
+### Overlays {#usage-flake-overlays}
+
+SHB applies its own overlays using `nixpkgs.overlays`.
+Each module provided by SHB set that option if needed.
+
+If you don't want to have those overlays applied for modules you don't intend to use SHB for,
+you will want to avoid importing the `default` module
+and instead import only the module for the services or blocks you intend to use,
+like shows in the [Modules](#usage-flake-modules) section.
+
+### Substituter {#usage-flake-substituter}
+
 You can also use the public cache as a substituter with:
 
 ```nix
@@ -28,36 +145,97 @@ nix.settings.substituters = [
 ];
 ```
 
-For now, Self Host Blocks has a hard dependency on `sops-nix`. I am [working on removing
-that](https://github.com/ibizaman/selfhostblocks/issues/24) so you can use any secrets manager you
-want. Until then, you also need to import the `sops-nix` module:
+### Unfree {#usage-flake-unfree}
+
+SHB does not necessarily attempt to provide only free packages.
+Currently, the only module using unfree modules is the [Open WebUI](@REPO@/modules/services/open-webui.nix) one.
+
+To be able to use that module, you can follow the [nixpkgs manual](https://nixos.org/manual/nixpkgs/stable/#sec-allow-unfree)
+and set either:
 
 ```nix
-imports = [
-  inputs.selfhostblocks.inputs.sops-nix.nixosModules.default
-];
+{
+  nixpkgs.config.allowUnfree = true;
+}
 ```
 
-Self Host Blocks provides its own `nixpkgs` input so both can be updated in lock step, ensuring
-maximum compatibility. It is recommended to use the following `nixpkgs` as input for your deployments:
+or the option `nixpkgs.config.allowUnfreePredicate`.
+
+### Tag Updates {#usage-flake-tag}
+
+To pin SHB to a release/tag, you can either use an implicit or explicit way.
+
+#### Implicit {#usage-flake-tag-implicit}
+
+Here, use the usual `inputs` form:
 
 ```nix
-inputs.selfhostblocks.inputs.nixpkgs
+{
+  inputs.selfhostblocks.url = "github:ibizaman/selfhostblocks";
+}
 ```
 
-Advanced users can if they wish use a version of `nixpkgs` of their choosing but then we cannot
-guarantee Self Host Block won't use a non-existing option from `nixpkgs`.
+then use the `flake update --override-input` command:
 
-To avoid manually updating the `nixpkgs` version, the [GitHub repository][1] for Self Host Blocks
-tries to update the `nixpkgs` input daily, verifying all tests pass before accepting this new
-`nixpkgs` version. The setup is explained in [this blog post][2].
+```bash
+nix flake update selfhostblocks \
+  --override-input selfhostblocks github:ibizaman/selfhostblocks/@VERSION@
+```
 
-[1]: https://github.com/ibizaman/selfhostblocks
-[2]: https://blog.tiserbox.com/posts/2023-12-25-automated-flake-lock-update-pull-requests-and-merging.html
+Note that running `nix flake update` will update the version of SHB to the latest from the main branch,
+canceling the override you just did above.
+So beware when running that command.
 
-## Example Deployment with Nixos-Rebuild {#usage-example-nixosrebuild}
+#### Explicit {#usage-flake-tag-explicit}
 
-The following snippets show how to deploy Self Host Blocks using the standard deployment system `nixos-rebuild`.
+Here, set the version in the input directly:
+
+```nix
+{
+  inputs.selfhostblocks.url = "github:ibizaman/selfhostblocks?ref=@VERSION@";
+}
+```
+
+Note that running `nix flake update` in this case will not update SHB,
+you must update the tag explicitly then run `nix flake update`.
+
+### Auto Updates {#usage-flake-autoupdate}
+
+To avoid burden on the maintainers to keep `nixpkgs` input updated with
+upstream, the [GitHub repository][repo] for SHB updates the
+`nixpkgs` input every couple days, and verifies all tests pass before
+automatically merging the new `nixpkgs` version. The setup is explained in
+[this blog post][automerge].
+
+[repo]: https://github.com/ibizaman/selfhostblocks
+[automerge]: https://blog.tiserbox.com/posts/2023-12-25-automated-flake-lock-update-pull-requests-and-merging.html
+
+### Lib {#usage-flake-lib}
+
+The `selfhostblocks.nixosModules.lib` module
+adds a module argument called `shb` by setting the
+`_module.args.shb` option.
+It is imported by nearly all other SHB modules
+but you could still import it on its own
+if you want to access SHB's functions and no other module.
+
+The library of functions is also available under the traditional
+`selfhostblocks.lib` flake output.
+
+The functions layout is, in pseudo-code:
+
+- `shb.*` all functions from [`./lib/default.nix`](@REPO@/lib/default.nix).
+- `shb.contracts.*` all functions from [`./modules/contracts/default.nix`](@REPO@/modules/contracts/default.nix).
+- `shb.test.*` all functions from [`./test/common.nix`](@REPO@/test/common.nix).
+
+## Example Deployments {#usage-examples}
+
+### With Nixos-Rebuild {#usage-examples-nixosrebuild}
+
+The following snippets show how to deploy SHB using the standard
+deployment system [nixos-rebuild][nixos-rebuild].
+
+[nixos-rebuild]: https://nixos.org/manual/nixos/stable/#sec-changing-config
 
 ```nix
 {
@@ -65,54 +243,25 @@ The following snippets show how to deploy Self Host Blocks using the standard de
     selfhostblocks.url = "github:ibizaman/selfhostblocks";
   };
 
-  outputs = { self, selfhostblocks }: {
+  outputs = { self, selfhostblocks }: let
+    system = "x86_64-linux";
+    nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
+  in {
     nixosConfigurations = {
-      machine = selfhostblocks.inputs.nixpkgs.lib.nixosSystem {
-        system = "x86_64-linux";
+      machine = nixpkgs'.nixosSystem {
+        inherit system;
         modules = [
-          selfhostblocks.nixosModules.${system}.default
+          selfhostblocks.nixosModules.default
         ];
-
-        # Machine specific configuration goes here.
       };
     };
   };
 }
 ```
 
-The above snippet is very minimal as it assumes you have only one machine to deploy to, so `nixpkgs`
-is defined exclusively by the `selfhostblocks.inputs.nixpkgs` input. If some machines are not using
-Self Host Blocks, you can do the following:
-
-```nix
-{
-  inputs = {
-    selfhostblocks.url = "github:ibizaman/selfhostblocks";
-  };
-
-  outputs = { self, selfhostblocks }: {
-    nixosConfigurations = {
-      machine1 = nixpkgs.lib.nixosSystem {
-      };
-
-      machine2 = selfhostblocks.inputs.nixpkgs.lib.nixosSystem {
-        system = "x86_64-linux";
-        modules = [
-          selfhostblocks.nixosModules.${system}.default
-        ];
-
-        # Machine specific configuration goes here.
-      };
-    };
-  };
-}
-```
-
-## Example Deployment With Colmena {#usage-example-colmena}
-
-The following snippets show how to deploy Self Host Blocks using the deployment system [Colmena][3].
-
-[3]: https://colmena.cli.rs
+The above snippet assumes one machine to deploy to, so `nixpkgs` is defined
+exclusively by the `selfhostblocks` input. It is more likely that you have
+multiple machines, some not using SHB, then you can do the following:
 
 ```nix
 {
@@ -123,28 +272,68 @@ The following snippets show how to deploy Self Host Blocks using the deployment
   };
 
   outputs = { self, selfhostblocks }: {
-    colmena =
-      let
-        system = "x86_64-linux";
-      in {
+    let
+      system = "x86_64-linux";
+      nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
+    in
+      nixosConfigurations = {
+        machine1 = nixpkgs.lib.nixosSystem {
+        };
+
+        machine2 = nixpkgs'.lib.nixosSystem {
+          system = "x86_64-linux";
+          modules = [
+            selfhostblocks.nixosModules.default
+          ];
+        };
+      };
+  };
+}
+```
+
+In the above snippet, `machine1` will use the `nixpkgs` version from your inputs
+while `machine2` will use the `nixpkgs` version from `selfhostblocks`.
+
+### With Colmena {#usage-examples-colmena}
+
+The following snippets show how to deploy SHB using the deployment
+system [Colmena][Colmena].
+
+[colmena]: https://colmena.cli.rs
+
+```nix
+{
+  inputs = {
+    selfhostblocks.url = "github:ibizaman/selfhostblocks";
+  };
+
+  outputs = { self, selfhostblocks }: {
+    let
+      system = "x86_64-linux";
+      nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
+      pkgs' = import nixpkgs' {
+        inherit system;
+      };
+    in
+      colmena = {
         meta = {
-          nixpkgs = import selfhostblocks.inputs.nixpkgs { inherit system; };
+          nixpkgs = pkgs';
         };
 
         machine = { selfhostblocks, ... }: {
           imports = [
-            selfhostblocks.nixosModules.${system}.default
+            selfhostblocks.nixosModules.default
           ];
-
-          # Machine specific configuration goes here.
         };
       };
   };
 }
 ```
 
-The above snippet is very minimal as it assumes you have only one machine to deploy to, so `nixpkgs`
-is defined exclusively by the `selfhostblocks` input. It is more likely that you have multiple machines, in this case you can use the `colmena.meta.nodeNixpkgs` option:
+The above snippet assumes one machine to deploy to, so `nixpkgs` is defined
+exclusively by the `selfhostblocks` input. It is more likely that you have
+multiple machines, some not using SHB, in this case you can use the
+`colmena.meta.nodeNixpkgs` option:
 
 ```nix
 {
@@ -155,14 +344,23 @@ is defined exclusively by the `selfhostblocks` input. It is more likely that you
   };
 
   outputs = { self, selfhostblocks }: {
-    colmena = {
-      let
-        system = "x86_64-linux";
-      in {
-        meta =
-          nixpkgs = import nixpkgs { inherit system; };
+    let
+      system = "x86_64-linux";
+      pkgs = import nixpkgs {
+        inherit system;
+      };
+
+      nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
+      pkgs' = import nixpkgs' {
+        inherit system;
+      };
+    in
+      colmena = {
+        meta = {
+          nixpkgs = pkgs;
+
           nodeNixpkgs = {
-            machine2 = import selfhostblocks.inputs.nixpkgs { inherit system; };
+            machine2 = pkgs';
           };
         };
 
@@ -170,49 +368,199 @@ is defined exclusively by the `selfhostblocks` input. It is more likely that you
 
         machine2 = { selfhostblocks, ... }: {
           imports = [
-            selfhostblocks.nixosModules.${system}.default
+            selfhostblocks.nixosModules.default
           ];
 
           # Machine specific configuration goes here.
         };
-    };
+      };
   };
 }
 ```
 
-In the above snippet, `machine1` will use the `nixpkgs` version from your inputs while `machine2`
-will use the `nixpkgs` version from `selfhostblocks`.
+In the above snippet, `machine1` will use the `nixpkgs` version from your inputs
+while `machine2` will use the `nixpkgs` version from `selfhostblocks`.
+
+### With Deploy-rs {#usage-examples-deploy-rs}
+
+The following snippets show how to deploy SHB using the deployment
+system [deploy-rs][deploy-rs].
+
+[deploy-rs]: https://github.com/serokell/deploy-rs
+
+```nix
+{
+  inputs = {
+    selfhostblocks.url = "github:ibizaman/selfhostblocks";
+  };
+
+  outputs = { self, selfhostblocks }: {
+    let
+      system = "x86_64-linux";
+      nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
+      pkgs' = import nixpkgs' {
+        inherit system;
+      };
+
+      deployPkgs = import selfhostblocks.inputs.nixpkgs {
+        inherit system;
+        overlays = [
+          deploy-rs.overlay
+          (self: super: {
+            deploy-rs = {
+              inherit (pkgs') deploy-rs;
+              lib = super.deploy-rs.lib;
+            };
+          })
+        ];
+      };
+    in
+      nixosModules.machine = {
+        imports = [
+          selfhostblocks.nixosModules.default
+        ];
+      };
+
+      nixosConfigurations.machine = nixpkgs'.nixosSystem {
+        inherit system;
+        modules = [
+          self.nixosModules.machine
+        ];
+      };
+
+      deploy.nodes.machine = {
+        hostname = ...;
+        sshUser = ...;
+        sshOpts = [ ... ];
+        profiles = {
+          system = {
+            user = "root";
+            path = deployPkgs.deploy-rs.lib.activate.nixos self.nixosConfigurations.machine;
+          };
+        };
+      };
+
+      # From https://github.com/serokell/deploy-rs?tab=readme-ov-file#overall-usage
+      checks = builtins.mapAttrs (system: deployLib: deployLib.deployChecks self.deploy) deploy-rs.lib;
+  };
+}
+```
+
+The above snippet assumes one machine to deploy to, so `nixpkgs` is defined
+exclusively by the `selfhostblocks` input. It is more likely that you have
+multiple machines, some not using SHB, in this case you can do:
+
+```nix
+{
+  inputs = {
+    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
+
+    selfhostblocks.url = "github:ibizaman/selfhostblocks";
+  };
+
+  outputs = { self, selfhostblocks }: {
+    let
+      system = "x86_64-linux";
+      nixpkgs' = selfhostblocks.lib.${system}.patchedNixpkgs;
+      pkgs' = import nixpkgs' {
+        inherit system;
+      };
+
+      deployPkgs = import selfhostblocks.inputs.nixpkgs {
+        inherit system;
+        overlays = [
+          deploy-rs.overlay
+          (self: super: {
+            deploy-rs = {
+              inherit (pkgs') deploy-rs;
+              lib = super.deploy-rs.lib;
+            };
+          })
+        ];
+      };
+    in
+      nixosModules.machine1 = {
+        # ...
+      };
+
+      nixosModules.machine2 = {
+        imports = [
+          selfhostblocks.nixosModules.default
+        ];
+      };
+
+      nixosConfigurations.machine1 = nixpkgs.lib.nixosSystem {
+        inherit system;
+        modules = [
+          self.nixosModules.machine1
+        ];
+      };
+
+      nixosConfigurations.machine2 = nixpkgs'.nixosSystem {
+        inherit system;
+        modules = [
+          self.nixosModules.machine2
+        ];
+      };
+
+      deploy.nodes.machine1 = {
+        hostname = ...;
+        sshUser = ...;
+        sshOpts = [ ... ];
+        profiles = {
+          system = {
+            user = "root";
+            path = deployPkgs.deploy-rs.lib.activate.nixos self.nixosConfigurations.machine1;
+          };
+        };
+      };
+
+      deploy.nodes.machine2 = # Similar here
+
+      # From https://github.com/serokell/deploy-rs?tab=readme-ov-file#overall-usage
+      checks = builtins.mapAttrs (system: deployLib: deployLib.deployChecks self.deploy) deploy-rs.lib;
+  };
+}
+```
+
+In the above snippet, `machine1` will use the `nixpkgs` version from your inputs
+while `machine2` will use the `nixpkgs` version from `selfhostblocks`.
 
 ## Secrets with sops-nix {#usage-secrets}
 
-This section complements the official [sops-nix](https://github.com/Mic92/sops-nix) guide.
+This section complements the official
+[sops-nix](https://github.com/Mic92/sops-nix) guide.
 
-Managing secrets is an important aspect of deploying. You cannot store your secrets in nix directly
-because they get stored unencrypted and you don't want that. We need to use another system that
-encrypts secrets when storing in the nix store and then decrypts them on the target host upon system
-activation. `sops-nix` is one of such system.
+Managing secrets is an important aspect of deploying. You cannot store your
+secrets in nix directly because they get stored unencrypted and you don't want
+that. We need to use another system that encrypts secrets when storing in the
+nix store and then decrypts them on the target host upon system activation.
+`sops-nix` is one of such system.
 
-Sops-nix works by encrypting the secrets file with at least 2 keys. Your private key and a private
-key from the target host. This way, you can edit the secrets and the target host can decrypt the
-secrets. Separating the keys this way is good practice because it reduces the impact of having one
-being compromised.
+Sops-nix works by encrypting the secrets file with at least 2 keys. Your private
+key and a private key from the target host. This way, you can edit the secrets
+and the target host can decrypt the secrets. Separating the keys this way is
+good practice because it reduces the impact of having one being compromised.
 
 One way to setup secrets management using `sops-nix`:
 
-1. Create your own private key that will be located in `keys.txt`. The public key will be printed on stdout.
+1. Create your own private key that will be located in `keys.txt`. The public
+   key will be printed on stdout.
    ```bash
    $ nix shell nixpkgs#age --command age-keygen -o keys.txt
    Public key: age1algdv9xwjre3tm7969eyremfw2ftx4h8qehmmjzksrv7f2qve9dqg8pug7
    ```
-2. Get the target host's public key. We will use the key derived from the ssh key of the host.
+2. Get the target host's public key. We will use the key derived from the ssh
+   key of the host.
    ```bash
    $ nix shell nixpkgs#ssh-to-age --command \
        sh -c 'ssh-keyscan -t ed25519 -4 <target_host> | ssh-to-age'
    # localhost:2222 SSH-2.0-OpenSSH_9.6
    age13wgyyae8epyw894ugd0rjjljh0rm98aurvzmsapcv7d852g9r5lq0pqfx8
    ```
-3. Create a `sops.yaml` file that explains how sops-nix should encrypt the - yet to be created -
-   `secrets.yaml` file. You can be creative here, but a basic snippet is:
+3. Create a `sops.yaml` file that explains how sops-nix should encrypt the - yet
+   to be created - `secrets.yaml` file. You can be creative here, but a basic
+   snippet is:
    ```bash
    keys:
      - &me age1algdv9xwjre3tm7969eyremfw2ftx4h8qehmmjzksrv7f2qve9dqg8pug7
@@ -224,45 +572,47 @@ One way to setup secrets management using `sops-nix`:
          - *me
          - *target
    ```
-4. Create a `secrets.yaml` file that will contain the encrypted secrets as a Yaml file:
+4. Create a `secrets.yaml` file that will contain the encrypted secrets as a
+   Yaml file:
    ```bash
    $ SOPS_AGE_KEY_FILE=keys.txt nix run --impure nixpkgs#sops -- \
      secrets.yaml
    ```
-   This will open your preferred editor. An example of yaml file is the following (secrets are elided for brevity):
+   This will open your preferred editor. An example of yaml file is the
+   following (secrets are elided for brevity):
    ```yaml
    nextcloud:
-       adminpass: 43bb4b...
-       onlyoffice:
-           jwt_secret: 3a10fce3...
+     adminpass: 43bb4b...
+     onlyoffice:
+       jwt_secret: 3a10fce3...
    ```
    The actual file on your filesystem will look like so, again with data elided:
    ```yaml
    nextcloud:
-       adminpass: ENC[AES256_GCM,data:Tt99...GY=,tag:XlAqRYidkOMRZAPBsoeEMw==,type:str]
-       onlyoffice:
-           jwt_secret: ENC[AES256_GCM,data:f87a...Yg=,tag:Y1Vg2WqDnJbl1Xg2B6W1Hg==,type:str]
+     adminpass: ENC[AES256_GCM,data:Tt99...GY=,tag:XlAqRYidkOMRZAPBsoeEMw==,type:str]
+     onlyoffice:
+       jwt_secret: ENC[AES256_GCM,data:f87a...Yg=,tag:Y1Vg2WqDnJbl1Xg2B6W1Hg==,type:str]
    sops:
-       kms: []
-       gcp_kms: []
-       azure_kv: []
-       hc_vault: []
-       age:
-           - recipient: age1algdv9xwjre3tm7969eyremfw2ftx4h8qehmmjzksrv7f2qve9dqg8pug7
-             enc: |
-               -----BEGIN AGE ENCRYPTED FILE-----
-               YWdl...6g==
-               -----END AGE ENCRYPTED FILE-----
-           - recipient: age13wgyyae8epyw894ugd0rjjljh0rm98aurvzmsapcv7d852g9r5lq0pqfx8
-             enc: |
-               -----BEGIN AGE ENCRYPTED FILE-----
-               YWdl...RA==
-               -----END AGE ENCRYPTED FILE-----
-       lastmodified: "2024-01-28T06:07:02Z"
-       mac: ENC[AES256_GCM,data:lDJh...To=,tag:Opon9lMZBv5S7rRhkGFuQQ==,type:str]
-       pgp: []
-       unencrypted_suffix: _unencrypted
-       version: 3.8.1
+     kms: []
+     gcp_kms: []
+     azure_kv: []
+     hc_vault: []
+     age:
+       - recipient: age1algdv9xwjre3tm7969eyremfw2ftx4h8qehmmjzksrv7f2qve9dqg8pug7
+         enc: |
+           -----BEGIN AGE ENCRYPTED FILE-----
+           YWdl...6g==
+           -----END AGE ENCRYPTED FILE-----
+       - recipient: age13wgyyae8epyw894ugd0rjjljh0rm98aurvzmsapcv7d852g9r5lq0pqfx8
+         enc: |
+           -----BEGIN AGE ENCRYPTED FILE-----
+           YWdl...RA==
+           -----END AGE ENCRYPTED FILE-----
+     lastmodified: "2024-01-28T06:07:02Z"
+     mac: ENC[AES256_GCM,data:lDJh...To=,tag:Opon9lMZBv5S7rRhkGFuQQ==,type:str]
+     pgp: []
+     unencrypted_suffix: _unencrypted
+     version: 3.8.1
    ```
 
    To actually create random secrets, you can use:
@@ -272,18 +622,20 @@ One way to setup secrets management using `sops-nix`:
 5. Use `sops-nix` module in nix:
    ```bash
    imports = [
-       selfhostblocks.inputs.sops-nix.nixosModules.default
+     inputs.sops-nix.nixosModules.default
+     inputs.selfhostblocks.nixosModules.sops
    ];
    ```
-6. Reference the secrets in nix:
-   ```nix
-   shb.nextcloud.adminPassFile = config.sops.secrets."nextcloud/adminpass".path;
-
-   sops.secrets."nextcloud/adminpass" = {
-     sopsFile = ./secrets.yaml;
-     mode = "0440";
-     owner = "nextcloud";
-     group = "nextcloud";
-     restartUnits = [ "phpfpm-nextcloud.service" ];
-   };
+   Import also the `sops` module provided by SHB.
+6. Set default sops file:
+   ```bash
+   sops.defaultSopsFile = ./secrets.yaml;
    ```
+   Setting the default this way makes all sops instances use that same file.
+7. Reference the secrets in nix:
+   ```nix
+   shb.sops.secret."nextcloud/adminpass".request = config.shb.nextcloud.adminPass.request;
+   shb.nextcloud.adminPass.result = config.shb.sops.secret."nextcloud/adminpass".result;
+   ```
+   The above snippet uses the [secrets contract](./contracts-secret.html) and
+   [sops block](./blocks-sops.html) to ease the configuration.

flake.lock

@@ -5,11 +5,11 @@
         "systems": "systems"
       },
       "locked": {
-        "lastModified": 1710146030,
-        "narHash": "sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ=",
+        "lastModified": 1731533236,
+        "narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
         "owner": "numtide",
         "repo": "flake-utils",
-        "rev": "b1d9ab70662946ef0850d488da1c9019f3a9752a",
+        "rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
         "type": "github"
       },
       "original": {
@@ -20,11 +20,11 @@
     },
     "nix-flake-tests": {
       "locked": {
-        "lastModified": 1677844186,
-        "narHash": "sha256-ErJZ/Gs1rxh561CJeWP5bohA2IcTq1rDneu1WT6CVII=",
+        "lastModified": 1775571237,
+        "narHash": "sha256-1f5Uvgcy3gx/eyRp4rqfDRBjcZc9uiHoIlfcFIL7GvI=",
         "owner": "antifuchs",
         "repo": "nix-flake-tests",
-        "rev": "bbd9216bd0f6495bb961a8eb8392b7ef55c67afb",
+        "rev": "86b789cff8aecbd7c1bed7cd421dd837c3f62201",
         "type": "github"
       },
       "original": {
@@ -35,11 +35,11 @@
     },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1716769173,
-        "narHash": "sha256-7EXDb5WBw+d004Agt+JHC/Oyh/KTUglOaQ4MNjBbo5w=",
+        "lastModified": 1781074563,
+        "narHash": "sha256-md8WlXOlfnIeHeOScMTTHFyf2d6iaTwPl2apR5EQ3P4=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "9ca3f649614213b2aaf5f1e16ec06952fe4c2632",
+        "rev": "9ae611a455b90cf061d8f332b977e387bda8e1ca",
         "type": "github"
       },
       "original": {
@@ -49,46 +49,14 @@
         "type": "github"
       }
     },
-    "nixpkgs-stable": {
-      "locked": {
-        "lastModified": 1716655032,
-        "narHash": "sha256-kQ25DAiCGigsNR/Quxm3v+JGXAEXZ8I7RAF4U94bGzE=",
-        "owner": "NixOS",
-        "repo": "nixpkgs",
-        "rev": "59a450646ec8ee0397f5fa54a08573e8240eb91f",
-        "type": "github"
-      },
-      "original": {
-        "owner": "NixOS",
-        "ref": "release-23.11",
-        "repo": "nixpkgs",
-        "type": "github"
-      }
-    },
-    "nixpkgs_2": {
-      "locked": {
-        "lastModified": 1716651315,
-        "narHash": "sha256-iMgzIeedMqf30TXZ439zW3Yvng1Xm9QTGO+ZwG1IWSw=",
-        "owner": "NixOS",
-        "repo": "nixpkgs",
-        "rev": "c5187508b11177ef4278edf19616f44f21cc8c69",
-        "type": "github"
-      },
-      "original": {
-        "owner": "NixOS",
-        "ref": "nixpkgs-unstable",
-        "repo": "nixpkgs",
-        "type": "github"
-      }
-    },
     "nmdsrc": {
       "flake": false,
       "locked": {
-        "lastModified": 1705050560,
-        "narHash": "sha256-x3zzcdvhJpodsmdjqB4t5mkVW22V3wqHLOun0KRBzUI=",
+        "lastModified": 1757277932,
+        "narHash": "sha256-eW2AzE4HeJXHKxSGDcp35/VQyxeESBt4p9i8QIFrSE8=",
         "ref": "refs/heads/master",
-        "rev": "66d9334933119c36f91a78d565c152a4fdc8d3d3",
-        "revCount": 66,
+        "rev": "055e814ac35379dece164a8413d336b57c5d2e0c",
+        "revCount": 67,
         "type": "git",
         "url": "https://git.sr.ht/~rycee/nmd"
       },
@@ -102,27 +70,7 @@
         "flake-utils": "flake-utils",
         "nix-flake-tests": "nix-flake-tests",
         "nixpkgs": "nixpkgs",
-        "nmdsrc": "nmdsrc",
-        "sops-nix": "sops-nix"
-      }
-    },
-    "sops-nix": {
-      "inputs": {
-        "nixpkgs": "nixpkgs_2",
-        "nixpkgs-stable": "nixpkgs-stable"
-      },
-      "locked": {
-        "lastModified": 1716692524,
-        "narHash": "sha256-sALodaA7Zkp/JD6ehgwc0UCBrSBfB4cX66uFGTsqeFU=",
-        "owner": "Mic92",
-        "repo": "sops-nix",
-        "rev": "962797a8d7f15ed7033031731d0bb77244839960",
-        "type": "github"
-      },
-      "original": {
-        "owner": "Mic92",
-        "repo": "sops-nix",
-        "type": "github"
+        "nmdsrc": "nmdsrc"
       }
     },
     "systems": {

flake.nix

@@ -3,7 +3,6 @@
 
   inputs = {
     nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
-    sops-nix.url = "github:Mic92/sops-nix";
     nix-flake-tests.url = "github:antifuchs/nix-flake-tests";
     flake-utils.url = "github:numtide/flake-utils";
     nmdsrc = {
@@ -12,126 +11,504 @@
     };
   };
 
-  outputs = { nixpkgs, nix-flake-tests, flake-utils, nmdsrc, ... }: flake-utils.lib.eachDefaultSystem (system:
+  outputs =
+    inputs@{
+      self,
+      nixpkgs,
+      nix-flake-tests,
+      flake-utils,
+      nmdsrc,
+      ...
+    }:
     let
-      originPkgs = nixpkgs.legacyPackages.${system};
-      patches = [
-        (originPkgs.fetchpatch {
-          url = "https://patch-diff.githubusercontent.com/raw/NixOS/nixpkgs/pull/315018.patch";
-          hash = "sha256-8jcGyO/d+htfv/ZajxXh89S3OiDZAr7/fsWC1JpGczM=";
-        })
-        (originPkgs.fetchpatch {
-          url = "https://github.com/NixOS/nixpkgs/pull/317107.patch";
-          hash = "sha256-6SfqnPLPxJHckXNU03HA0X03u9Ynn3baQ2HHMq5FkIc=";
-        })
-      ];
-      patchedNixpkgs = originPkgs.applyPatches {
-        name = "nixpkgs-patched";
-        src = nixpkgs;
-        inherit patches;
-      };
+      shbPatches =
+        system:
+        nixpkgs.legacyPackages.${system}.lib.optionals
+          (system == "x86_64-linux" || system == "aarch64-linux")
+          [
+            # Get rid of lldap patches when https://github.com/NixOS/nixpkgs/pull/425923 is merged.
+            ./patches/lldap.patch
+            ./patches/0001-nixos-borgbackup-add-option-to-override-state-direct.patch
 
-      pkgs = import patchedNixpkgs {
-        inherit system;
-      };
+            # Leaving commented out as an example.
+            # (originPkgs.fetchpatch {
+            #   url = "https://github.com/NixOS/nixpkgs/pull/317107.patch";
+            #   hash = "sha256-hoLrqV7XtR1hP/m0rV9hjYUBtrSjay0qcPUYlKKuVWk=";
+            # })
+          ];
 
-      allModules = [
-        modules/blocks/authelia.nix
-        modules/blocks/backup.nix
-        modules/blocks/davfs.nix
-        modules/blocks/ldap.nix
-        modules/blocks/monitoring.nix
-        modules/blocks/nginx.nix
-        modules/blocks/postgresql.nix
-        modules/blocks/ssl.nix
-        modules/blocks/tinyproxy.nix
-        modules/blocks/vpn.nix
-
-        modules/services/arr.nix
-        modules/services/audiobookshelf.nix
-        modules/services/deluge.nix
-        modules/services/grocy.nix
-        modules/services/hledger.nix
-        modules/services/home-assistant.nix
-        modules/services/jellyfin.nix
-        modules/services/nextcloud-server.nix
-        modules/services/vaultwarden.nix
-      ];
-
-      # Only used for documentation.
-      contractDummyModules = [
-        modules/contracts/ssl/dummyModule.nix
-      ];
-    in
-      {
-        nixosModules.default = { config, ... }: {
-          imports = allModules;
+      patchNixpkgs =
+        {
+          nixpkgs,
+          patches,
+          system,
+        }:
+        nixpkgs.legacyPackages.${system}.applyPatches {
+          name = "nixpkgs-patched";
+          src = nixpkgs;
+          inherit patches;
         };
+      patchedNixpkgs =
+        system:
+        let
+          patched = patchNixpkgs {
+            nixpkgs = inputs.nixpkgs;
+            patches = shbPatches system;
+            inherit system;
+          };
+        in
+        patched
+        // {
+          nixosSystem = args: import "${patched}/nixos/lib/eval-config.nix" args;
+        };
+      pkgs' =
+        system:
+        import (patchedNixpkgs system) {
+          inherit system;
+          config.allowUnfree = true;
+        };
+    in
+    flake-utils.lib.eachDefaultSystem (
+      system:
+      let
+        pkgs = pkgs' system;
+
+        # The contract dummies are used to show options for contracts.
+        contractDummyModules = [
+          modules/contracts/backup/dummyModule.nix
+          modules/contracts/dashboard/dummyModule.nix
+          modules/contracts/databasebackup/dummyModule.nix
+          modules/contracts/datasetbackup/dummyModule.nix
+          modules/contracts/secret/dummyModule.nix
+          modules/contracts/ssl/dummyModule.nix
+        ];
+      in
+      {
+        formatter = pkgs.nixfmt-tree;
 
         packages.manualHtml = pkgs.callPackage ./docs {
           inherit nmdsrc;
-          allModules = allModules ++ contractDummyModules;
-          release = "0.0.1";
+          allModules =
+            self.nixosModules.default.imports
+            ++ [
+              self.nixosModules.sops
+            ]
+            ++ contractDummyModules;
+          release = builtins.readFile ./VERSION;
+
+          substituteVersionIn = [
+            "./manual.md"
+            "./usage.md"
+          ];
+          modules = {
+            "blocks/authelia" = ./modules/blocks/authelia.nix;
+            "blocks/borgbackup" = ./modules/blocks/borgbackup.nix;
+            "blocks/lldap" = ./modules/blocks/lldap.nix;
+            "blocks/mitmdump" = ./modules/blocks/mitmdump.nix;
+            "blocks/monitoring" = ./modules/blocks/monitoring.nix;
+            "blocks/nginx" = ./modules/blocks/nginx.nix;
+            "blocks/postgresql" = ./modules/blocks/postgresql.nix;
+            "blocks/restic" = ./modules/blocks/restic.nix;
+            "blocks/sanoid" = ./modules/blocks/sanoid.nix;
+            "blocks/sops" = ./modules/blocks/sops.nix;
+            "blocks/ssl" = {
+              module = ./modules/blocks/ssl.nix;
+              optionRoot = [
+                "shb"
+                "certs"
+              ];
+            };
+            "blocks/zfs" = ./modules/blocks/zfs.nix;
+
+            "services/arr" = ./modules/services/arr.nix;
+            "services/firefly-iii" = ./modules/services/firefly-iii.nix;
+            "services/forgejo" = [
+              ./modules/services/forgejo.nix
+              (pkgs.path + "/nixos/modules/services/misc/forgejo.nix")
+            ];
+            "services/home-assistant" = ./modules/services/home-assistant.nix;
+            "services/homepage" = ./modules/services/homepage.nix;
+            "services/jellyfin" = ./modules/services/jellyfin.nix;
+            "services/karakeep" = ./modules/services/karakeep.nix;
+            "services/mailserver" = ./modules/services/mailserver.nix;
+            "services/nextcloud-server" = {
+              module = ./modules/services/nextcloud-server.nix;
+              optionRoot = [
+                "shb"
+                "nextcloud"
+              ];
+            };
+            "services/open-webui" = ./modules/services/open-webui.nix;
+            "services/pinchflat" = ./modules/services/pinchflat.nix;
+            "services/vaultwarden" = ./modules/services/vaultwarden.nix;
+
+            "contracts/backup" = {
+              module = ./modules/contracts/backup/dummyModule.nix;
+              optionRoot = [
+                "shb"
+                "contracts"
+                "backup"
+              ];
+            };
+            "contracts/dashboard" = {
+              module = ./modules/contracts/dashboard/dummyModule.nix;
+              optionRoot = [
+                "shb"
+                "contracts"
+                "dashboard"
+              ];
+            };
+            "contracts/databasebackup" = {
+              module = ./modules/contracts/databasebackup/dummyModule.nix;
+              optionRoot = [
+                "shb"
+                "contracts"
+                "databasebackup"
+              ];
+            };
+            "contracts/datasetbackup" = {
+              module = ./modules/contracts/datasetbackup/dummyModule.nix;
+              optionRoot = [
+                "shb"
+                "contracts"
+                "datasetbackup"
+              ];
+            };
+            "contracts/secret" = {
+              module = ./modules/contracts/secret/dummyModule.nix;
+              optionRoot = [
+                "shb"
+                "contracts"
+                "secret"
+              ];
+            };
+            "contracts/ssl" = {
+              module = ./modules/contracts/ssl/dummyModule.nix;
+              optionRoot = [
+                "shb"
+                "contracts"
+                "ssl"
+              ];
+            };
+          };
         };
 
-        lib.contracts = pkgs.callPackage ./modules/contracts {};
-
-        checks =
+        # Documentation redirect generation tool - scans HTML files for anchor mappings
+        packages.generateRedirects =
           let
-            importFiles = files:
-              map (m: pkgs.callPackage m {}) files;
+            # Python patch to inject redirect collector
+            pythonPatch = pkgs.writeText "nixos-render-docs-patch.py" ''
+              # Load redirect collector patch
+              try:
+                  import sys, os
+                  sys.path.insert(0, os.path.dirname(__file__) + '/..')
+                  import missing_refs_collector
+              except Exception as e:
+                  print(f"Warning: Failed to load redirect collector: {e}", file=sys.stderr)
+            '';
 
-            mergeTests = pkgs.lib.lists.foldl pkgs.lib.trivial.mergeAttrs {};
+            # Patched nixos-render-docs that collects redirects during HTML generation
+            nixos-render-docs-patched = pkgs.writeShellApplication {
+              name = "nixos-render-docs";
+              runtimeInputs = [ pkgs.nixos-render-docs ];
+              text = ''
+                TEMP_DIR=$(mktemp -d); trap 'rm -rf "$TEMP_DIR"' EXIT
 
-            flattenAttrs = root: attrset: pkgs.lib.attrsets.foldlAttrs (acc: name: value: acc // {
-              "${root}_${name}" = value;
-            }) {} attrset;
+                cp -r ${pkgs.nixos-render-docs}/${pkgs.python3.sitePackages}/nixos_render_docs "$TEMP_DIR/"
+                chmod -R +w "$TEMP_DIR"
+                cp ${./docs/generate-redirects-nixos-render-docs.py} "$TEMP_DIR/missing_refs_collector.py"
+                echo '{}' > "$TEMP_DIR/empty_redirects.json"
+                cat ${pythonPatch} >> "$TEMP_DIR/nixos_render_docs/__init__.py"
 
-            vm_test = name: path: flattenAttrs "vm_${name}" (
-              import path {
-                inherit pkgs;
-                inherit (pkgs) lib;
-              }
-            );
+                ARGS=()
+                while [[ $# -gt 0 ]]; do
+                  case $1 in
+                    --redirects) ARGS+=("$1" "$TEMP_DIR/empty_redirects.json"); shift 2 ;;
+                    *) ARGS+=("$1"); shift ;;
+                  esac
+                done
 
-            shblib = pkgs.callPackage ./lib {};
-          in (rec {
-            all = mergeTests [
-              modules
+                export PYTHONPATH="$TEMP_DIR:''${PYTHONPATH:-}"
+                nixos-render-docs "''${ARGS[@]}"
+              '';
+            };
+          in
+          (self.packages.${system}.manualHtml.override {
+            nixos-render-docs = nixos-render-docs-patched;
+          }).overrideAttrs
+            (old: {
+              installPhase = ''
+                ${old.installPhase}
+                ln -sf share/doc/selfhostblocks/redirects.json $out/redirects.json
+              '';
+            });
+
+        packages.manualHtml-watch = pkgs.writeShellApplication {
+          name = "manualHtml-watch";
+          runtimeInputs = [
+            pkgs.findutils
+            pkgs.entr
+          ];
+          text = ''
+            while sleep 1; do
+              find . -name "*.nix" -o -name "*.md" \
+                | entr -d sh -c '(nix run --offline .#update-redirects && nix build --offline .#manualHtml)' || :
+            done
+          '';
+        };
+
+        packages.update-flake-lock-pr = pkgs.callPackage ./.github/workflows/update-flake-lock-pr.nix { };
+
+        lib = (pkgs.callPackage ./lib { }) // {
+          test = pkgs.callPackage ./test/common.nix { };
+          contracts = pkgs.callPackage ./modules/contracts {
+            shb = self.lib.${system};
+          };
+          patches = shbPatches system;
+          inherit patchNixpkgs;
+          patchedNixpkgs = patchedNixpkgs system;
+        };
+
+        # To see the traces, run:
+        #   nix run .#playwright -- show-trace $(nix eval .#checks.x86_64-linux.vm_grocy_basic --raw)/trace/0.zip
+        packages.playwright = pkgs.callPackage (
+          {
+            stdenvNoCC,
+            makeWrapper,
+            playwright,
+          }:
+          stdenvNoCC.mkDerivation {
+            name = "playwright";
+
+            src = playwright;
+
+            nativeBuildInputs = [
+              makeWrapper
             ];
 
-            modules = shblib.check {
-              inherit pkgs;
-              tests =
-                mergeTests (importFiles [
-                  ./test/modules/arr.nix
-                  ./test/modules/davfs.nix
-                  ./test/modules/lib.nix
-                  ./test/modules/nginx.nix
-                  ./test/modules/postgresql.nix
-                ]);
-            };
-
-            lib = nix-flake-tests.lib.check {
-              inherit pkgs;
-              tests = pkgs.callPackage ./test/modules/lib.nix {};
-            };
+            # No quotes around the value for LLDAP_PASSWORD because we want the value to not be enclosed in quotes.
+            installPhase = ''
+              makeWrapper ${pkgs.python3Packages.playwright}/bin/playwright $out/bin/playwright \
+                --set PLAYWRIGHT_BROWSERS_PATH ${pkgs.playwright-driver.browsers} \
+                --set PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS true
+            '';
           }
-          // (vm_test "arr" ./test/vm/arr.nix)
-          // (vm_test "audiobookshelf" ./test/vm/audiobookshelf.nix)
-          // (vm_test "authelia" ./test/vm/authelia.nix)
-          // (vm_test "deluge" ./test/vm/deluge.nix)
-          // (vm_test "grocy" ./test/vm/grocy.nix)
-          // (vm_test "home-assistant" ./test/vm/home-assistant.nix)
-          // (vm_test "jellyfin" ./test/vm/jellyfin.nix)
-          // (vm_test "ldap" ./test/vm/ldap.nix)
-          // (vm_test "lib" ./test/vm/lib.nix)
-          // (vm_test "monitoring" ./test/vm/monitoring.nix)
-          // (vm_test "nextcloud" ./test/vm/nextcloud.nix)
-          // (vm_test "postgresql" ./test/vm/postgresql.nix)
-          // (vm_test "ssl" ./test/vm/ssl.nix)
-          // (vm_test "vaultwarden" ./test/vm/vaultwarden.nix)
-          );
+        ) { };
+
+        # Run "nix run .#update-redirects" to regenerate docs/redirects.json
+        apps.update-redirects = {
+          type = "app";
+          program = "${
+            pkgs.writeShellApplication {
+              name = "update-redirects";
+              runtimeInputs = [
+                pkgs.nix
+                pkgs.jq
+              ];
+              text = ''
+                echo "=== SelfHostBlocks Redirects Updater ==="
+                echo "Generating fresh ./docs/redirects.json..."
+
+                nix build .#generateRedirects || { echo "Error: Failed to generate redirects" >&2; exit 1; }
+                [[ -f result/redirects.json ]] || { echo "Error: Generated redirects file not found" >&2; exit 1; }
+
+                echo "Generated $(jq 'keys | length' result/redirects.json) redirects"
+
+                [[ -f docs/redirects.json ]] && cp docs/redirects.json docs/redirects.json.backup && echo "Created backup"
+                cp result/redirects.json docs/redirects.json
+                echo "  Updated docs/redirects.json"
+                echo "To verify: nix build .#manualHtml"
+              '';
+            }
+          }/bin/update-redirects";
+        };
       }
-  );
+    )
+    // flake-utils.lib.eachSystem [ "x86_64-linux" ] (
+      system:
+      let
+        pkgs = pkgs' system;
+      in
+      {
+        checks =
+          let
+            inherit (pkgs.lib)
+              foldl
+              foldlAttrs
+              mergeAttrs
+              ;
+
+            importFiles =
+              files:
+              map (
+                m:
+                pkgs.callPackage m {
+                  shb = self.lib.${system};
+                }
+              ) files;
+
+            mergeTests = foldl mergeAttrs { };
+
+            flattenAttrs =
+              root: attrset:
+              foldlAttrs (
+                acc: name: value:
+                acc
+                // {
+                  "${root}_${name}" = value;
+                }
+              ) { } attrset;
+
+            vm_test =
+              name: path:
+              flattenAttrs "vm_${name}" (
+                removeAttrs
+                  (pkgs.callPackage path {
+                    shb = self.lib.${system};
+                  })
+                  [
+                    "override"
+                    "overrideDerivation"
+                  ]
+              );
+          in
+          (
+            {
+              modules = self.lib.${system}.check {
+                inherit pkgs;
+                tests = mergeTests (importFiles [
+                  ./test/modules/davfs.nix
+                  # TODO: Make this not use IFD
+                  ./test/modules/lib.nix
+                ]);
+              };
+
+              # TODO: Make this not use IFD
+              lib = nix-flake-tests.lib.check {
+                inherit pkgs;
+                tests = pkgs.callPackage ./test/modules/lib.nix {
+                  shb = self.lib.${system};
+                };
+              };
+            }
+            // (vm_test "arr" ./test/services/arr.nix)
+            // (vm_test "audiobookshelf" ./test/services/audiobookshelf.nix)
+            // (vm_test "deluge" ./test/services/deluge.nix)
+            // (vm_test "firefly-iii" ./test/services/firefly-iii.nix)
+            // (vm_test "forgejo" ./test/services/forgejo.nix)
+            // (vm_test "grocy" ./test/services/grocy.nix)
+            // (vm_test "hledger" ./test/services/hledger.nix)
+            // (vm_test "immich" ./test/services/immich.nix)
+            // (vm_test "homeassistant" ./test/services/home-assistant.nix)
+            // (vm_test "homepage" ./test/services/homepage.nix)
+            // (vm_test "jellyfin" ./test/services/jellyfin.nix)
+            // (vm_test "karakeep" ./test/services/karakeep.nix)
+            // (vm_test "mailserver" ./test/services/mailserver.nix)
+            // (vm_test "nextcloud" ./test/services/nextcloud.nix)
+            // (vm_test "open-webui" ./test/services/open-webui.nix)
+            // (vm_test "paperless" ./test/services/paperless.nix)
+            // (vm_test "pinchflat" ./test/services/pinchflat.nix)
+            // (vm_test "vaultwarden" ./test/services/vaultwarden.nix)
+
+            // (vm_test "authelia" ./test/blocks/authelia.nix)
+            // (vm_test "borgbackup" ./test/blocks/borgbackup.nix)
+            // (vm_test "lib" ./test/blocks/lib.nix)
+            // (vm_test "lldap" ./test/blocks/lldap.nix)
+            // (vm_test "mitmdump" ./test/blocks/mitmdump.nix)
+            // (vm_test "monitoring" ./test/blocks/monitoring.nix)
+            // (vm_test "postgresql" ./test/blocks/postgresql.nix)
+            // (vm_test "restic" ./test/blocks/restic.nix)
+            // (vm_test "ssl" ./test/blocks/ssl.nix)
+            // (vm_test "zfs" ./test/blocks/zfs.nix)
+
+            // (vm_test "contracts-backup" ./test/contracts/backup.nix)
+            // (vm_test "contracts-databasebackup" ./test/contracts/databasebackup.nix)
+            // (vm_test "contracts-datasetbackup" ./test/contracts/datasetbackup.nix)
+            // (vm_test "contracts-secret" ./test/contracts/secret.nix)
+          );
+
+      }
+    )
+    // {
+      herculesCI.ciSystems = [ "x86_64-linux" ];
+
+      nixosModules.default = {
+        imports = [
+          # blocks
+          self.nixosModules.authelia
+          self.nixosModules.borgbackup
+          self.nixosModules.davfs
+          self.nixosModules.hardcodedsecret
+          self.nixosModules.lldap
+          self.nixosModules.mitmdump
+          self.nixosModules.monitoring
+          self.nixosModules.nginx
+          self.nixosModules.postgresql
+          self.nixosModules.restic
+          self.nixosModules.sanoid
+          self.nixosModules.ssl
+          self.nixosModules.tinyproxy
+          self.nixosModules.vpn
+          self.nixosModules.zfs
+
+          # services
+          self.nixosModules.arr
+          self.nixosModules.audiobookshelf
+          self.nixosModules.deluge
+          self.nixosModules.firefly-iii
+          self.nixosModules.forgejo
+          self.nixosModules.grocy
+          self.nixosModules.hledger
+          self.nixosModules.immich
+          self.nixosModules.home-assistant
+          self.nixosModules.homepage
+          self.nixosModules.jellyfin
+          self.nixosModules.karakeep
+          self.nixosModules.mailserver
+          self.nixosModules.nextcloud-server
+          self.nixosModules.open-webui
+          self.nixosModules.pinchflat
+          self.nixosModules.paperless
+          self.nixosModules.vaultwarden
+        ];
+      };
+
+      nixosModules.lib = lib/module.nix;
+
+      nixosModules.authelia = modules/blocks/authelia.nix;
+      nixosModules.borgbackup = modules/blocks/borgbackup.nix;
+      nixosModules.davfs = modules/blocks/davfs.nix;
+      nixosModules.hardcodedsecret = modules/blocks/hardcodedsecret.nix;
+      nixosModules.lldap = modules/blocks/lldap.nix;
+      nixosModules.mitmdump = modules/blocks/mitmdump.nix;
+      nixosModules.monitoring = modules/blocks/monitoring.nix;
+      nixosModules.nginx = modules/blocks/nginx.nix;
+      nixosModules.postgresql = modules/blocks/postgresql.nix;
+      nixosModules.restic = modules/blocks/restic.nix;
+      nixosModules.sanoid = modules/blocks/sanoid.nix;
+      nixosModules.sops = modules/blocks/sops.nix;
+      nixosModules.ssl = modules/blocks/ssl.nix;
+      nixosModules.tinyproxy = modules/blocks/tinyproxy.nix;
+      nixosModules.vpn = modules/blocks/vpn.nix;
+      nixosModules.zfs = modules/blocks/zfs.nix;
+
+      nixosModules.arr = modules/services/arr.nix;
+      nixosModules.audiobookshelf = modules/services/audiobookshelf.nix;
+      nixosModules.deluge = modules/services/deluge.nix;
+      nixosModules.firefly-iii = modules/services/firefly-iii.nix;
+      nixosModules.forgejo = modules/services/forgejo.nix;
+      nixosModules.grocy = modules/services/grocy.nix;
+      nixosModules.hledger = modules/services/hledger.nix;
+      nixosModules.immich = modules/services/immich.nix;
+      nixosModules.home-assistant = modules/services/home-assistant.nix;
+      nixosModules.homepage = modules/services/homepage.nix;
+      nixosModules.jellyfin = modules/services/jellyfin.nix;
+      nixosModules.karakeep = modules/services/karakeep.nix;
+      nixosModules.mailserver = modules/services/mailserver.nix;
+      nixosModules.nextcloud-server = modules/services/nextcloud-server.nix;
+      nixosModules.open-webui = modules/services/open-webui.nix;
+      nixosModules.paperless = modules/services/paperless.nix;
+      nixosModules.pinchflat = modules/services/pinchflat.nix;
+      nixosModules.vaultwarden = modules/services/vaultwarden.nix;
+    };
 }

lib/default.nix

@@ -1,235 +1,460 @@
 { pkgs, lib }:
-rec {
-  # Replace secrets in a file.
-  # - userConfig is an attrset that will produce a config file.
-  # - resultPath is the location the config file should have on the filesystem.
-  # - generator is a function taking two arguments name and value and returning path in the nix
-  #   nix store where the
-  replaceSecrets = { userConfig, resultPath, generator }:
-    let
-      configWithTemplates = withReplacements userConfig;
+let
+  inherit (builtins) isAttrs hasAttr;
+  inherit (lib)
+    any
+    concatMapStringsSep
+    concatStringsSep
+    escapeShellArg
+    ;
+  shb = rec {
+    # Replace secrets in a file.
+    # - userConfig is an attrset that will produce a config file.
+    # - resultPath is the location the config file should have on the filesystem.
+    # - generator is a function taking two arguments name and value and returning path in the nix
+    #   nix store where the
+    replaceSecrets =
+      {
+        userConfig,
+        resultPath,
+        generator,
+        user ? null,
+        permissions ? "u=r,g=r,o=",
+      }:
+      let
+        configWithTemplates = withReplacements userConfig;
 
-      nonSecretConfigFile = generator "template" configWithTemplates;
+        nonSecretConfigFile = generator "template" configWithTemplates;
 
-      replacements = getReplacements userConfig;
-    in
+        replacements = getReplacements userConfig;
+      in
       replaceSecretsScript {
         file = nonSecretConfigFile;
         inherit resultPath replacements;
+        inherit user permissions;
       };
 
-  replaceSecretsFormatAdapter = format: format.generate;
-  replaceSecretsGeneratorAdapter = generator: name: value: pkgs.writeText "generator " (generator value);
+    replaceSecretsFormatAdapter = format: format.generate;
+    replaceSecretsGeneratorAdapter =
+      generator: name: value:
+      pkgs.writeText "generator " (generator value);
+    toEnvVar = replaceSecretsGeneratorAdapter (
+      v: (lib.generators.toINIWithGlobalSection { } { globalSection = v; })
+    );
 
-  template = file: newPath: replacements: replaceSecretsScript {
-    inherit file replacements;
-    resultPath = newPath;
-  };
+    template =
+      file: newPath: replacements:
+      replaceSecretsScript {
+        inherit file replacements;
+        resultPath = newPath;
+      };
 
-  replaceSecretsScript = { file, resultPath, replacements }:
-    let
-      templatePath = resultPath + ".template";
-      sedPatterns = lib.strings.concatStringsSep " " (lib.attrsets.mapAttrsToList (from: to: "-e \"s|${from}|${to}|\"") replacements);
-      sedCmd = if replacements == {}
-               then "cat"
-               else "${pkgs.gnused}/bin/sed ${sedPatterns}";
-    in
+    genReplacement =
+      secret:
+      let
+        t =
+          {
+            transform ? null,
+            ...
+          }:
+          if isNull transform then x: x else transform;
+      in
+      lib.attrsets.nameValuePair (secretName secret.name) ((t secret) "$(cat ${toString secret.source})");
+
+    replaceSecretsScript =
+      {
+        file,
+        resultPath,
+        replacements,
+        user ? null,
+        permissions ? "u=r,g=r,o=",
+      }:
+      let
+        templatePath = resultPath + ".template";
+
+        # We check that the files containing the secrets have the
+        # correct permissions for us to read them in this separate
+        # step. Otherwise, the $(cat ...) commands inside the sed
+        # replacements could fail but not fail individually but
+        # not fail the whole script.
+        checkPermissions = concatMapStringsSep "\n" (
+          pattern: "cat ${pattern.source} > /dev/null"
+        ) replacements;
+
+        generatedReplacements = map genReplacement replacements;
+
+        replaceScript = pkgs.writers.writePython3 "replace-secret" { } ''
+          import argparse
+          import pathlib
+
+
+          def parse_args() -> argparse.Namespace:
+              parser = argparse.ArgumentParser()
+              parser.add_argument("--file", required=True, type=pathlib.Path)
+              parser.add_argument("--marker", required=True)
+              parser.add_argument("--replacement", required=True)
+              return parser.parse_args()
+
+
+          args = parse_args()
+          content = args.file.read_text()
+          args.file.write_text(content.replace(args.marker, args.replacement))
+        '';
+
+        replaceCommands = concatMapStringsSep "\n" (pattern: ''
+          ${replaceScript} \
+            --file ${escapeShellArg resultPath} \
+            --marker ${escapeShellArg pattern.name} \
+            --replacement "${pattern.value}"
+        '') generatedReplacements;
+
+        replaceCmd =
+          if replacements == [ ] then
+            "cat ${escapeShellArg templatePath} > ${escapeShellArg resultPath}"
+          else
+            ''
+              cat ${escapeShellArg templatePath} > ${escapeShellArg resultPath}
+              ${replaceCommands}
+            '';
+      in
       ''
-      set -euo pipefail
+        set -euo pipefail
 
-      mkdir -p $(dirname ${templatePath})
-      ln -fs ${file} ${templatePath}
-      rm -f ${resultPath}
-      ${sedCmd} ${templatePath} > ${resultPath}
+        ${checkPermissions}
+
+        mkdir -p $(dirname ${templatePath})
+        ln -fs ${file} ${templatePath}
+        rm -f ${resultPath}
+        touch ${resultPath}
+      ''
+      + (lib.optionalString (user != null) ''
+        chown ${user} ${resultPath}
+      '')
+      + ''
+        ${replaceCmd}
+        chmod ${permissions} ${resultPath}
       '';
 
-  secretFileType = lib.types.submodule {
-    options = {
-      source = lib.mkOption {
-        type = lib.types.path;
-        description = "File containing the value.";
-      };
+    secretFileType = lib.types.submodule {
+      options = {
+        source = lib.mkOption {
+          type = lib.types.path;
+          description = "File containing the value.";
+        };
 
-      transform = lib.mkOption {
-        type = lib.types.raw;
-        description = "An optional function to transform the secret.";
-        default = null;
-        example = lib.literalExpression ''
-        v: "prefix-$${v}-suffix"
-        '';
+        transform = lib.mkOption {
+          type = lib.types.raw;
+          description = "An optional function to transform the secret.";
+          default = null;
+          example = lib.literalExpression ''
+            v: "prefix-$${v}-suffix"
+          '';
+        };
       };
     };
-  };
 
-  secretName = name:
-      "%SECRET${lib.strings.toUpper (lib.strings.concatMapStrings (s: "_" + s) name)}%";
+    secretName =
+      names: "%SECRET${lib.strings.toUpper (lib.strings.concatMapStrings (s: "_" + s) names)}%";
 
-  withReplacements = attrs:
-    let
-      valueOrReplacement = name: value:
-        if !(builtins.isAttrs value && value ? "source")
-        then value
-        else secretName name;
-    in
-      mapAttrsRecursiveCond (v: ! v ? "source") valueOrReplacement attrs;
+    withReplacements =
+      attrs:
+      let
+        valueOrReplacement =
+          name: value: if !(builtins.isAttrs value && value ? "source") then value else secretName name;
+      in
+      mapAttrsRecursiveCond (v: !v ? "source") valueOrReplacement attrs;
 
-  getReplacements = attrs:
-    let
-      addNameField = name: value:
-        if !(builtins.isAttrs value && value ? "source")
-        then value
-        else value // { name = name; };
+    getReplacements =
+      attrs:
+      let
+        addNameField =
+          name: value:
+          if !(builtins.isAttrs value && value ? "source") then value else value // { name = name; };
 
-      secretsWithName = mapAttrsRecursiveCond (v: ! v ? "source") addNameField attrs;
+        secretsWithName = mapAttrsRecursiveCond (v: !v ? "source") addNameField attrs;
+      in
+      collect (v: builtins.isAttrs v && v ? "source") secretsWithName;
 
-      allSecrets = collect (v: builtins.isAttrs v && v ? "source") secretsWithName;
+    # Inspired lib.attrsets.mapAttrsRecursiveCond but also recurses on lists.
+    mapAttrsRecursiveCond =
+      # A function, given the attribute set the recursion is currently at, determine if to recurse deeper into that attribute set.
+      cond:
+      # A function, given a list of attribute names and a value, returns a new value.
+      f:
+      # Attribute set or list to recursively map over.
+      set:
+      let
+        recurse =
+          path: val:
+          if builtins.isAttrs val && cond val then
+            lib.attrsets.mapAttrs (n: v: recurse (path ++ [ n ]) v) val
+          else if builtins.isList val && cond val then
+            lib.lists.imap0 (i: v: recurse (path ++ [ (builtins.toString i) ]) v) val
+          else
+            f path val;
+      in
+      recurse [ ] set;
 
-      t = { transform ? null, ... }: if isNull transform then x: x else transform;
+    # Like lib.attrsets.collect but also recurses on lists.
+    collect =
+      # Given an attribute's value, determine if recursion should stop.
+      pred:
+      # The attribute set to recursively collect.
+      attrs:
+      if pred attrs then
+        [ attrs ]
+      else if builtins.isAttrs attrs then
+        lib.lists.concatMap (collect pred) (lib.attrsets.attrValues attrs)
+      else if builtins.isList attrs then
+        lib.lists.concatMap (collect pred) attrs
+      else
+        [ ];
 
-      genReplacement = secret:
-        lib.attrsets.nameValuePair (secretName secret.name) ((t secret) "$(cat ${toString secret.source})");
-    in
-      lib.attrsets.listToAttrs (map genReplacement allSecrets);
-      
-  # Inspired lib.attrsets.mapAttrsRecursiveCond but also recurses on lists.
-  mapAttrsRecursiveCond =
-    # A function, given the attribute set the recursion is currently at, determine if to recurse deeper into that attribute set.
-    cond:
-    # A function, given a list of attribute names and a value, returns a new value.
-    f:
-    # Attribute set or list to recursively map over.
-    set:
-    let
-      recurse = path: val:
-        if builtins.isAttrs val && cond val
-        then lib.attrsets.mapAttrs (n: v: recurse (path ++ [n]) v) val
-        else if builtins.isList val && cond val
-        then lib.lists.imap0 (i: v: recurse (path ++ [(builtins.toString i)]) v) val
-        else f path val;
-    in recurse [] set;
+    indent =
+      i: str:
+      lib.concatMapStringsSep "\n" (x: (lib.strings.replicate i " ") + x) (lib.splitString "\n" str);
 
-  # Like lib.attrsets.collect but also recurses on lists.
-  collect =
-  # Given an attribute's value, determine if recursion should stop.
-  pred:
-  # The attribute set to recursively collect.
-  attrs:
-    if pred attrs then
-      [ attrs ]
-    else if builtins.isAttrs attrs then
-      lib.lists.concatMap (collect pred) (lib.attrsets.attrValues attrs)
-    else if builtins.isList attrs then
-      lib.lists.concatMap (collect pred) attrs
-    else
-      [];
+    # Generator for XML
+    formatXML =
+      {
+        enclosingRoot ? null,
+      }:
+      {
+        type =
+          with lib.types;
+          let
+            valueType =
+              nullOr (oneOf [
+                bool
+                int
+                float
+                str
+                path
+                (attrsOf valueType)
+                (listOf valueType)
+              ])
+              // {
+                description = "XML value";
+              };
+          in
+          valueType;
+
+        generate =
+          name: value:
+          pkgs.callPackage (
+            { runCommand, python3 }:
+            runCommand "config"
+              {
+                value = builtins.toJSON (if enclosingRoot == null then value else { ${enclosingRoot} = value; });
+                passAsFile = [ "value" ];
+              }
+              (
+                pkgs.writers.writePython3 "dict2xml"
+                  {
+                    libraries = with python3.pkgs; [
+                      python
+                      dict2xml
+                    ];
+                  }
+                  ''
+                    import os
+                    import json
+                    from dict2xml import dict2xml
+
+                    with open(os.environ["valuePath"]) as f:
+                        content = json.loads(f.read())
+                        if content is None:
+                            print("Could not parse env var valuePath as json")
+                            os.exit(2)
+                        with open(os.environ["out"], "w") as out:
+                            out.write(dict2xml(content))
+                  ''
+              )
+          ) { };
 
-  # Generator for XML
-  formatXML = {
-    enclosingRoot ? null
-  }: {
-    type = with lib.types; let
-      valueType = nullOr (oneOf [
-        bool
-        int
-        float
-        str
-        path
-        (attrsOf valueType)
-        (listOf valueType)
-      ]) // {
-        description = "XML value";
       };
-    in valueType;
 
-    generate = name: value: pkgs.callPackage ({ runCommand, python3 }: runCommand "config" {
-      value = builtins.toJSON (
-        if enclosingRoot == null then
-          value
-        else
-          { ${enclosingRoot} = value; });
-      passAsFile = [ "value" ];
-    } (pkgs.writers.writePython3 "dict2xml" {
-      libraries = with python3.pkgs; [ python dict2xml ];
-    } ''
-      import os
-      import json
-      from dict2xml import dict2xml
-
-      with open(os.environ["valuePath"]) as f:
-          content = json.loads(f.read())
-          if content is None:
-              print("Could not parse env var valuePath as json")
-              os.exit(2)
-          with open(os.environ["out"], "w") as out:
-              out.write(dict2xml(content))
-    '')) {};
-
-  };
-
-  parseXML = xml:
-    let
-      xmlToJsonFile = pkgs.callPackage ({ runCommand, python3 }: runCommand "config" {
-        inherit xml;
-        passAsFile = [ "xml" ];
-      } (pkgs.writers.writePython3 "xml2json" {
-        libraries = with python3.pkgs; [ python ];
-      } ''
-        import os
-        import json
-        from collections import ChainMap
-        from xml.etree import ElementTree
-        
-
-        def xml_to_dict_recursive(root):
-            all_descendants = list(root)
-            if len(all_descendants) == 0:
-                return {root.tag: root.text}
-            else:
-                merged_dict = ChainMap(*map(xml_to_dict_recursive, all_descendants))
-                return {root.tag: dict(merged_dict)}
+    parseXML =
+      xml:
+      let
+        xmlToJsonFile = pkgs.callPackage (
+          { runCommand, python3 }:
+          runCommand "config"
+            {
+              inherit xml;
+              passAsFile = [ "xml" ];
+            }
+            (
+              pkgs.writers.writePython3 "xml2json"
+                {
+                  libraries = with python3.pkgs; [ python ];
+                }
+                ''
+                  import os
+                  import json
+                  from collections import ChainMap
+                  from xml.etree import ElementTree
 
 
-        with open(os.environ["xmlPath"]) as f:
-            root = ElementTree.XML(f.read())
-            xml = xml_to_dict_recursive(root)
-            j = json.dumps(xml)
+                  def xml_to_dict_recursive(root):
+                      all_descendants = list(root)
+                      if len(all_descendants) == 0:
+                          return {root.tag: root.text}
+                      else:
+                          merged_dict = ChainMap(*map(xml_to_dict_recursive, all_descendants))
+                          return {root.tag: dict(merged_dict)}
 
-            with open(os.environ["out"], "w") as out:
-                out.write(j)
-      '')) {};
-    in
+
+                  with open(os.environ["xmlPath"]) as f:
+                      root = ElementTree.XML(f.read())
+                      xml = xml_to_dict_recursive(root)
+                      j = json.dumps(xml)
+
+                      with open(os.environ["out"], "w") as out:
+                          out.write(j)
+                ''
+            )
+        ) { };
+      in
       builtins.fromJSON (builtins.readFile xmlToJsonFile);
 
-  renameAttrName = attrset: from: to:
-    (lib.attrsets.filterAttrs (name: v: name == from) attrset) // {
-      ${to} = attrset.${from};
-    };
+    renameAttrName =
+      attrset: from: to:
+      (lib.attrsets.filterAttrs (name: v: name == from) attrset)
+      // {
+        ${to} = attrset.${from};
+      };
 
-  # Taken from https://github.com/antifuchs/nix-flake-tests/blob/main/default.nix
-  # with a nicer diff display function.
-  check = { pkgs, tests }:
-    let
-      formatValue = val:
-        if (builtins.isList val || builtins.isAttrs val) then builtins.toJSON val
-        else builtins.toString val;
+    # Taken from https://github.com/antifuchs/nix-flake-tests/blob/main/default.nix
+    # with a nicer diff display function.
+    check =
+      { pkgs, tests }:
+      let
+        formatValue =
+          val:
+          if (builtins.isList val || builtins.isAttrs val) then
+            builtins.toJSON val
+          else
+            builtins.toString val;
 
-      resultToString = { name, expected, result }:
-        builtins.readFile (pkgs.runCommand "nix-flake-tests-error" {
-          expected = formatValue expected;
-          result = formatValue result;
-          passAsFile = [ "expected" "result" ];
-        } ''
-          echo "${name} failed (- expected, + result)" > $out
-          cp ''${expectedPath} ''${expectedPath}.json
-          cp ''${resultPath} ''${resultPath}.json
-          ${pkgs.deepdiff}/bin/deep diff ''${expectedPath}.json ''${resultPath}.json >> $out
-        '');
+        resultToString =
+          {
+            name,
+            expected,
+            result,
+          }:
+          builtins.readFile (
+            pkgs.runCommand "nix-flake-tests-error"
+              {
+                expected = formatValue expected;
+                result = formatValue result;
+                passAsFile = [
+                  "expected"
+                  "result"
+                ];
 
-      results = pkgs.lib.runTests tests;
-    in
-    if results != [ ] then
-      builtins.throw (builtins.concatStringsSep "\n" (map resultToString results))
-    else
-      pkgs.runCommand "nix-flake-tests-success" { } "echo > $out";
+                nativeBuildInputs = [
+                  (pkgs.python3.withPackages (ps: [ ps.deepdiff ] ++ ps.deepdiff.optional-dependencies.cli))
+                ];
+              }
+              ''
+                echo "${name} failed (- expected, + result)" > $out
+                cp ''${expectedPath} ''${expectedPath}.json
+                cp ''${resultPath} ''${resultPath}.json
+                deep diff ''${expectedPath}.json ''${resultPath}.json >> $out
+              ''
+          );
 
+        results = pkgs.lib.runTests tests;
+      in
+      if results != [ ] then
+        builtins.throw (concatStringsSep "\n" (map resultToString (lib.traceValSeq results)))
+      else
+        pkgs.runCommand "nix-flake-tests-success" { } "echo > $out";
+
+    genConfigOutOfBandSystemd =
+      {
+        config,
+        configLocation,
+        generator,
+        user ? null,
+        permissions ? "u=r,g=r,o=",
+      }:
+      {
+        loadCredentials = getLoadCredentials "source" config;
+        preStart = lib.mkBefore (replaceSecrets {
+          userConfig = updateToLoadCredentials "source" "$CREDENTIALS_DIRECTORY" config;
+          resultPath = configLocation;
+          inherit generator;
+          inherit user permissions;
+        });
+      };
+
+    updateToLoadCredentials =
+      sourceField: rootDir: attrs:
+      let
+        hasPlaceholderField = v: isAttrs v && hasAttr sourceField v;
+
+        valueOrLoadCredential =
+          path: value:
+          if !(hasPlaceholderField value) then
+            value
+          else
+            value // { ${sourceField} = rootDir + "/" + concatStringsSep "_" path; };
+      in
+      mapAttrsRecursiveCond (v: !(hasPlaceholderField v)) valueOrLoadCredential attrs;
+
+    getLoadCredentials =
+      sourceField: attrs:
+      let
+        hasPlaceholderField = v: isAttrs v && hasAttr sourceField v;
+
+        addPathField =
+          path: value: if !(hasPlaceholderField value) then value else value // { inherit path; };
+
+        secretsWithPath = mapAttrsRecursiveCond (v: !(hasPlaceholderField v)) addPathField attrs;
+
+        allSecrets = collect (v: hasPlaceholderField v) secretsWithPath;
+
+        genLoadCredentials = secret: "${concatStringsSep "_" secret.path}:${secret.${sourceField}}";
+      in
+      map genLoadCredentials allSecrets;
+
+    anyNotNull = any (x: x != null);
+
+    mkJellyfinPlugin =
+      {
+        pname,
+        version,
+        hash,
+        url,
+      }:
+      pkgs.callPackage (
+        { stdenv, fetchzip }:
+        stdenv.mkDerivation (finalAttrs: {
+          inherit pname version;
+
+          src = fetchzip {
+            inherit url hash;
+            stripRoot = false;
+          };
+
+          dontBuild = true;
+
+          installPhase = ''
+            mkdir $out
+            cp -r . $out
+          '';
+        })
+      ) { };
+
+    update =
+      attr: fn: attrset:
+      attrset // { ${attr} = fn attrset.${attr}; };
+
+  };
+in
+shb
+// {
+  homepage = pkgs.callPackage ./homepage.nix { inherit shb; };
 }

lib/homepage.nix

@@ -0,0 +1,92 @@
+{ lib, shb }:
+let
+  sort =
+    attr: vs:
+    map (v: { ${v.name} = v.${attr}; }) (
+      lib.sortOn (v: v.sortOrder) (lib.mapAttrsToList (n: v: v // { name = n; }) vs)
+    );
+
+  slufigy = builtins.replaceStrings [ "-" ] [ "_" ];
+
+  mkService =
+    groupName: serviceName:
+    {
+      request,
+      ...
+    }:
+    apiKey: settings:
+    lib.recursiveUpdate (
+      {
+        href = request.externalUrl;
+        siteMonitor = if (request.internalUrl == null) then null else request.internalUrl;
+        icon = "sh-${lib.toLower serviceName}";
+      }
+      // lib.optionalAttrs (apiKey != null) {
+        widget = {
+          # Duplicating because widgets call the api key various names
+          # and duplicating is a hacky but easy solution.
+          key = "{{HOMEPAGE_FILE_${slufigy groupName}_${slufigy serviceName}}}";
+          password = "{{HOMEPAGE_FILE_${slufigy groupName}_${slufigy serviceName}}}";
+          type = lib.toLower serviceName;
+          url = if (request.internalUrl != null) then request.internalUrl else request.externalUrl;
+        };
+      }
+    ) settings;
+
+  asServiceGroup =
+    cfg:
+    sort "services" (
+      lib.mapAttrs (
+        groupName: groupCfg:
+        shb.update "services" (
+          services:
+          sort "dashboard" (
+            lib.mapAttrs (
+              serviceName: serviceCfg:
+              shb.update "dashboard" (
+                dashboard:
+                (mkService groupName serviceName) dashboard serviceCfg.apiKey (serviceCfg.settings or { })
+              ) serviceCfg
+            ) services
+          )
+        ) groupCfg
+      ) cfg
+    );
+
+  allKeys =
+    cfg:
+    let
+      flat = lib.flatten (
+        lib.mapAttrsToList (
+          groupName: groupCfg:
+          lib.mapAttrsToList (
+            serviceName: serviceCfg:
+            lib.optionalAttrs (serviceCfg.apiKey != null) {
+              inherit serviceName groupName;
+              inherit (serviceCfg.apiKey.result) path;
+            }
+          ) groupCfg.services
+        ) cfg
+      );
+
+      flatWithApiKey = builtins.filter (v: v != { }) flat;
+    in
+    builtins.listToAttrs (
+      map (
+        {
+          groupName,
+          serviceName,
+          path,
+        }:
+        lib.nameValuePair "${slufigy groupName}_${slufigy serviceName}" path
+      ) flatWithApiKey
+    );
+in
+{
+  inherit
+    allKeys
+    asServiceGroup
+    mkService
+    sort
+    ;
+}

lib/module.nix

@@ -0,0 +1,10 @@
+{ pkgs, lib, ... }:
+let
+  shb = (import ./default.nix { inherit pkgs lib; });
+in
+{
+  _module.args.shb = shb // {
+    test = pkgs.callPackage ../test/common.nix { };
+    contracts = pkgs.callPackage ../modules/contracts { inherit shb; };
+  };
+}

modules/blocks/authelia.nix

@@ -1,17 +1,33 @@
-{ config, pkgs, lib, ... }:
+{
+  config,
+  options,
+  pkgs,
+  lib,
+  shb,
+  ...
+}:
 
 let
   cfg = config.shb.authelia;
-
-  contracts = pkgs.callPackage ../contracts {};
-  shblib = pkgs.callPackage ../../lib {};
+  opt = options.shb.authelia;
 
   fqdn = "${cfg.subdomain}.${cfg.domain}";
   fqdnWithPort = if isNull cfg.port then fqdn else "${fqdn}:${toString cfg.port}";
 
   autheliaCfg = config.services.authelia.instances.${fqdn};
+
+  inherit (lib) hasPrefix;
+
+  listenPort = if cfg.debug then 9090 else 9091;
 in
 {
+  imports = [
+    ../../lib/module.nix
+    ./lldap.nix
+    ./mitmdump.nix
+    ./postgresql.nix
+  ];
+
   options.shb.authelia = {
     enable = lib.mkEnableOption "selfhostblocks.authelia";
 
@@ -35,16 +51,22 @@ in
 
     ssl = lib.mkOption {
       description = "Path to SSL files";
-      type = lib.types.nullOr contracts.ssl.certs;
+      type = lib.types.nullOr shb.contracts.ssl.certs;
       default = null;
     };
 
-    ldapEndpoint = lib.mkOption {
+    ldapHostname = lib.mkOption {
       type = lib.types.str;
-      description = "Endpoint for LDAP authentication backend.";
+      description = "Hostname of the LDAP authentication backend.";
       example = "ldap.example.com";
     };
 
+    ldapPort = lib.mkOption {
+      type = lib.types.port;
+      description = "Port of the LDAP authentication backend.";
+      example = "389";
+    };
+
     dcdomain = lib.mkOption {
       type = lib.types.str;
       description = "dc domain for ldap.";
@@ -61,96 +83,193 @@ in
       description = "Secrets needed by Authelia";
       type = lib.types.submodule {
         options = {
-          jwtSecretFile = lib.mkOption {
-            type = lib.types.path;
-            description = "File containing the JWT secret.";
+          jwtSecret = lib.mkOption {
+            description = "JWT secret.";
+            type = lib.types.submodule {
+              options = shb.contracts.secret.mkRequester {
+                mode = "0400";
+                owner = cfg.autheliaUser;
+                restartUnits = [ "authelia-${opt.subdomain}.${opt.domain}.service" ];
+              };
+            };
           };
-          ldapAdminPasswordFile = lib.mkOption {
-            type = lib.types.path;
-            description = "File containing the LDAP admin user password.";
+          ldapAdminPassword = lib.mkOption {
+            description = "LDAP admin user password.";
+            type = lib.types.submodule {
+              options = shb.contracts.secret.mkRequester {
+                mode = "0400";
+                owner = cfg.autheliaUser;
+                restartUnits = [ "authelia-${opt.subdomain}.${opt.domain}.service" ];
+              };
+            };
           };
-          sessionSecretFile = lib.mkOption {
-            type = lib.types.path;
-            description = "File containing the session secret.";
+          sessionSecret = lib.mkOption {
+            description = "Session secret.";
+            type = lib.types.submodule {
+              options = shb.contracts.secret.mkRequester {
+                mode = "0400";
+                owner = cfg.autheliaUser;
+                restartUnits = [ "authelia-${opt.subdomain}.${opt.domain}.service" ];
+              };
+            };
           };
-          storageEncryptionKeyFile = lib.mkOption {
-            type = lib.types.path;
-            description = "File containing the storage encryption key.";
+          storageEncryptionKey = lib.mkOption {
+            description = "Storage encryption key. Must be >= 20 characters.";
+            type = lib.types.submodule {
+              options = shb.contracts.secret.mkRequester {
+                mode = "0400";
+                owner = cfg.autheliaUser;
+                restartUnits = [ "authelia-${opt.subdomain}.${opt.domain}.service" ];
+              };
+            };
           };
-          identityProvidersOIDCHMACSecretFile = lib.mkOption {
-            type = lib.types.path;
-            description = "File containing the identity provider OIDC HMAC secret.";
+          identityProvidersOIDCHMACSecret = lib.mkOption {
+            description = "Identity provider OIDC HMAC secret. Must be >= 40 characters.";
+            type = lib.types.submodule {
+              options = shb.contracts.secret.mkRequester {
+                mode = "0400";
+                owner = cfg.autheliaUser;
+                restartUnits = [ "authelia-${opt.subdomain}.${opt.domain}.service" ];
+              };
+            };
           };
-          identityProvidersOIDCIssuerPrivateKeyFile = lib.mkOption {
-            type = lib.types.path;
+          identityProvidersOIDCIssuerPrivateKey = lib.mkOption {
             description = ''
-              File containing the identity provider OIDC issuer private key.
+              Identity provider OIDC issuer private key.
 
               Generate one with `nix run nixpkgs#openssl -- genrsa -out keypair.pem 2048`
             '';
+            type = lib.types.submodule {
+              options = shb.contracts.secret.mkRequester {
+                mode = "0400";
+                owner = cfg.autheliaUser;
+                restartUnits = [ "authelia-${opt.subdomain}.${opt.domain}.service" ];
+              };
+            };
           };
         };
       };
     };
 
+    extraOidcClaimsPolicies = lib.mkOption {
+      description = "Extra OIDC claims policies.";
+      type = lib.types.attrsOf lib.types.attrs;
+      default = { };
+    };
+
+    extraOidcScopes = lib.mkOption {
+      description = "Extra OIDC scopes.";
+      type = lib.types.attrsOf lib.types.attrs;
+      default = { };
+    };
+
+    extraOidcAuthorizationPolicies = lib.mkOption {
+      description = "Extra OIDC authorization policies.";
+      type = lib.types.attrsOf lib.types.attrs;
+      default = { };
+    };
+
+    extraDefinitions = lib.mkOption {
+      description = "Extra definitions.";
+      type = lib.types.attrsOf lib.types.attrs;
+      default = { };
+    };
+
     oidcClients = lib.mkOption {
       description = "OIDC clients";
       default = [
         {
-          id = "dummy_client";
-          description = "Dummy Client so Authelia can start";
-          secret.source = pkgs.writeText "dummy.secret" "dummy_client_secret";
+          client_id = "dummy_client";
+          client_name = "Dummy Client so Authelia can start";
+          client_secret.source = pkgs.writeText "dummy.secret" "dummy_client_secret";
           public = false;
           authorization_policy = "one_factor";
-          redirect_uris = [];
+          redirect_uris = [ ];
         }
       ];
-      type = lib.types.listOf (lib.types.submodule {
-        freeformType = lib.types.attrsOf lib.types.anything;
+      type = lib.types.listOf (
+        lib.types.submodule {
+          freeformType = lib.types.attrsOf lib.types.anything;
 
-        options = {
-          id = lib.mkOption {
-            type = lib.types.str;
-            description = "Unique identifier of the OIDC client.";
-          };
+          options = {
+            client_id = lib.mkOption {
+              type = lib.types.str;
+              description = "Unique identifier of the OIDC client.";
+            };
 
-          description = lib.mkOption {
-            type = lib.types.nullOr lib.types.str;
-            description = "Human readable description of the OIDC client.";
-            default = null;
-          };
+            client_name = lib.mkOption {
+              type = lib.types.nullOr lib.types.str;
+              description = "Human readable description of the OIDC client.";
+              default = null;
+            };
 
-          secret = lib.mkOption {
-            type = shblib.secretFileType;
-            description = "File containing the shared secret with the OIDC client.";
-          };
+            client_secret = lib.mkOption {
+              type = shb.secretFileType;
+              description = ''
+                File containing the shared secret with the OIDC client.
 
-          public = lib.mkOption {
-            type = lib.types.bool;
-            description = "If the OIDC client is public or not.";
-            default = false;
-            apply = v: if v then "true" else "false";
-          };
+                Generate with:
 
-          authorization_policy = lib.mkOption {
-            type = lib.types.enum [ "one_factor" "two_factor" ];
-            description = "Require one factor (password) or two factor (device) authentication.";
-            default = "one_factor";
-          };
+                ```
+                nix run nixpkgs#authelia -- \
+                    crypto hash generate pbkdf2 \
+                    --variant sha512 \
+                    --random \
+                    --random.length 72 \
+                    --random.charset rfc3986
+                ```
+              '';
+            };
 
-          redirect_uris = lib.mkOption {
-            type = lib.types.listOf lib.types.str;
-            description = "List of uris that are allowed to be redirected to.";
-          };
+            public = lib.mkOption {
+              type = lib.types.bool;
+              description = "If the OIDC client is public or not.";
+              default = false;
+              apply = v: if v then "true" else "false";
+            };
 
-          scopes = lib.mkOption {
-            type = lib.types.listOf lib.types.str;
-            description = "Scopes to ask for";
-            example = [ "openid" "profile" "email" "groups" ];
-            default = [];
+            authorization_policy = lib.mkOption {
+              type = lib.types.enum (
+                [
+                  "one_factor"
+                  "two_factor"
+                ]
+                ++ lib.attrNames cfg.extraOidcAuthorizationPolicies
+              );
+              description = "Require one factor (password) or two factor (device) authentication.";
+              default = "one_factor";
+            };
+
+            redirect_uris = lib.mkOption {
+              type = lib.types.listOf lib.types.str;
+              description = "List of uris that are allowed to be redirected to.";
+            };
+
+            scopes = lib.mkOption {
+              type = lib.types.listOf lib.types.str;
+              description = "Scopes to ask for. See https://www.authelia.com/integration/openid-connect/openid-connect-1.0-claims";
+              example = [
+                "openid"
+                "profile"
+                "email"
+                "groups"
+              ];
+              default = [ ];
+            };
+
+            claims_policy = lib.mkOption {
+              type = lib.types.nullOr lib.types.str;
+              description = ''
+                Claim policy.
+
+                Defaults to 'default' to provide a backwards compatible experience.
+                Read [this document](https://www.authelia.com/integration/openid-connect/openid-connect-1.0-claims/#restore-functionality-prior-to-claims-parameter) for more information.
+              '';
+              default = "default";
+            };
           };
-        };
-      });
+        }
+      );
     };
 
     smtp = lib.mkOption {
@@ -163,44 +282,127 @@ in
       default = "/tmp/authelia-notifications";
       type = lib.types.oneOf [
         lib.types.str
-        (lib.types.nullOr (lib.types.submodule {
-          options = {
-            from_address = lib.mkOption {
-              type = lib.types.str;
-              description = "SMTP address from which the emails originate.";
-              example = "authelia@mydomain.com";
+        (lib.types.nullOr (
+          lib.types.submodule {
+            options = {
+              from_address = lib.mkOption {
+                type = lib.types.str;
+                description = "SMTP address from which the emails originate.";
+                example = "authelia@mydomain.com";
+              };
+              from_name = lib.mkOption {
+                type = lib.types.str;
+                description = "SMTP name from which the emails originate.";
+                default = "Authelia";
+              };
+              scheme = lib.mkOption {
+                description = "The protocl must be smtp, submission, or submissions. The only difference between these schemes are the default ports and submissions requires a TLS transport per SMTP Ports Security Measures, whereas submission and smtp use a standard TCP transport and typically enforce StartTLS.";
+                type = lib.types.enum [
+                  "smtp"
+                  "submission"
+                  "submissions"
+                ];
+                default = "smtp";
+              };
+              host = lib.mkOption {
+                type = lib.types.str;
+                description = "SMTP host to send the emails to.";
+              };
+              port = lib.mkOption {
+                type = lib.types.port;
+                description = "SMTP port to send the emails to.";
+                default = 25;
+              };
+              username = lib.mkOption {
+                type = lib.types.str;
+                description = "Username to connect to the SMTP host.";
+              };
+              password = lib.mkOption {
+                description = "File containing the password to connect to the SMTP host.";
+                type = lib.types.submodule {
+                  options = shb.contracts.secret.mkRequester {
+                    mode = "0400";
+                    owner = cfg.autheliaUser;
+                    restartUnits = [ "authelia-${fqdn}.service" ];
+                  };
+                };
+              };
             };
-            from_name = lib.mkOption {
-              type = lib.types.str;
-              description = "SMTP name from which the emails originate.";
-              default = "Authelia";
-            };
-            host = lib.mkOption {
-              type = lib.types.str;
-              description = "SMTP host to send the emails to.";
-            };
-            port = lib.mkOption {
-              type = lib.types.port;
-              description = "SMTP port to send the emails to.";
-              default = 25;
-            };
-            username = lib.mkOption {
-              type = lib.types.str;
-              description = "Username to connect to the SMTP host.";
-            };
-            passwordFile = lib.mkOption {
-              type = lib.types.str;
-              description = "File containing the password to connect to the SMTP host.";
-            };
-          };
-        }))
+          }
+        ))
       ];
     };
 
     rules = lib.mkOption {
       type = lib.types.listOf lib.types.anything;
       description = "Rule based clients";
-      default = [];
+      default = [ ];
+    };
+
+    mount = lib.mkOption {
+      type = shb.contracts.mount;
+      description = ''
+        Mount configuration. This is an output option.
+
+        Use it to initialize a block implementing the "mount" contract.
+        For example, with a zfs dataset:
+
+        ```
+        shb.zfs.datasets."authelia" = {
+          poolName = "root";
+        } // config.shb.authelia.mount;
+        ```
+      '';
+      readOnly = true;
+      default = {
+        path = "/var/lib/authelia-authelia.${cfg.domain}";
+      };
+      defaultText = {
+        path = "/var/lib/authelia-authelia.example.com";
+      };
+    };
+
+    mountRedis = lib.mkOption {
+      type = shb.contracts.mount;
+      description = ''
+        Mount configuration for Redis. This is an output option.
+
+        Use it to initialize a block implementing the "mount" contract.
+        For example, with a zfs dataset:
+
+        ```
+        shb.zfs.datasets."redis-authelia" = {
+          poolName = "root";
+        } // config.shb.authelia.mountRedis;
+        ```
+      '';
+      readOnly = true;
+      default = {
+        path = "/var/lib/redis-authelia";
+      };
+    };
+
+    debug = lib.mkOption {
+      type = lib.types.bool;
+      default = false;
+      description = ''
+        Set logging level to debug and add a mitmdump instance
+        to see exactly what Authelia receives and sends back.
+      '';
+    };
+
+    dashboard = lib.mkOption {
+      description = ''
+        Dashboard contract consumer
+      '';
+      default = { };
+      type = lib.types.submodule {
+        options = shb.contracts.dashboard.mkRequester {
+          externalUrl = "https://${cfg.subdomain}.${cfg.domain}";
+          externalUrlText = "https://\${config.shb.authelia.subdomain}.\${config.shb.authelia.domain}";
+          internalUrl = "http://127.0.0.1:${toString listenPort}";
+        };
+      };
     };
   };
 
@@ -210,11 +412,15 @@ in
         assertion = builtins.length cfg.oidcClients > 0;
         message = "Must have at least one oidc client otherwise Authelia refuses to start.";
       }
+      {
+        assertion = !(hasPrefix "ldap://" cfg.ldapHostname);
+        message = "LDAP hostname should be the bare host name and not start with ldap://";
+      }
     ];
 
     # Overriding the user name so we don't allow any weird characters anywhere. For example, postgres users do not accept the '.'.
     users = {
-      groups.${autheliaCfg.user} = {};
+      groups.${autheliaCfg.user} = { };
       users.${autheliaCfg.user} = {
         isSystemUser = true;
         group = autheliaCfg.user;
@@ -226,45 +432,38 @@ in
       user = cfg.autheliaUser;
 
       secrets = {
-        inherit (cfg.secrets) jwtSecretFile storageEncryptionKeyFile;
+        jwtSecretFile = cfg.secrets.jwtSecret.result.path;
+        storageEncryptionKeyFile = cfg.secrets.storageEncryptionKey.result.path;
+        sessionSecretFile = cfg.secrets.sessionSecret.result.path;
+        oidcIssuerPrivateKeyFile = cfg.secrets.identityProvidersOIDCIssuerPrivateKey.result.path;
+        oidcHmacSecretFile = cfg.secrets.identityProvidersOIDCHMACSecret.result.path;
       };
       # See https://www.authelia.com/configuration/methods/secrets/
       environmentVariables = {
-        AUTHELIA_AUTHENTICATION_BACKEND_LDAP_PASSWORD_FILE = toString cfg.secrets.ldapAdminPasswordFile;
-        AUTHELIA_SESSION_SECRET_FILE = toString cfg.secrets.sessionSecretFile;
+        AUTHELIA_AUTHENTICATION_BACKEND_LDAP_PASSWORD_FILE = toString cfg.secrets.ldapAdminPassword.result.path;
         # Not needed since we use peer auth.
         # AUTHELIA_STORAGE_POSTGRES_PASSWORD_FILE = "/run/secrets/authelia/postgres_password";
-        AUTHELIA_STORAGE_ENCRYPTION_KEY_FILE = toString cfg.secrets.storageEncryptionKeyFile;
-        AUTHELIA_IDENTITY_PROVIDERS_OIDC_HMAC_SECRET_FILE = toString cfg.secrets.identityProvidersOIDCHMACSecretFile;
-        AUTHELIA_IDENTITY_PROVIDERS_OIDC_ISSUER_PRIVATE_KEY_FILE = toString cfg.secrets.identityProvidersOIDCIssuerPrivateKeyFile;
-
-        AUTHELIA_NOTIFIER_SMTP_PASSWORD_FILE = lib.mkIf (!(builtins.isString cfg.smtp)) (toString cfg.smtp.passwordFile);
+        AUTHELIA_NOTIFIER_SMTP_PASSWORD_FILE = lib.mkIf (!(builtins.isString cfg.smtp)) (
+          toString cfg.smtp.password.result.path
+        );
+        X_AUTHELIA_CONFIG_FILTERS = "template";
       };
       settings = {
-        server.host = "127.0.0.1";
-        server.port = 9091;
+        server.address = "tcp://127.0.0.1:${toString listenPort}";
 
         # Inspired from https://github.com/lldap/lldap/blob/7d1f5abc137821c500de99c94f7579761fc949d8/example_configs/authelia_config.yml
         authentication_backend = {
           refresh_interval = "5m";
-          password_reset = {
-            disable = "false";
-          };
+          # We allow password reset and change because the ldap user we use allows it.
+          password_reset.disable = "false";
+          password_change.disable = "false";
           ldap = {
-            implementation = "custom";
-            url = cfg.ldapEndpoint;
+            implementation = "lldap";
+            address = "ldap://${cfg.ldapHostname}:${toString cfg.ldapPort}";
             timeout = "5s";
             start_tls = "false";
             base_dn = cfg.dcdomain;
-            username_attribute = "uid";
-            additional_users_dn = "ou=people";
-            # Sign in with username or email.
-            users_filter = "(&(|({username_attribute}={input})({mail_attribute}={input}))(objectClass=person))";
-            additional_groups_dn = "ou=groups";
-            groups_filter = "(member={dn})";
-            group_name_attribute = "cn";
-            mail_attribute = "mail";
-            display_name_attribute = "displayName";
+            # TODO: use user with less privilege and with lldap_password_manager group to be able to change passwords.
             user = "uid=admin,ou=people,${cfg.dcdomain}";
           };
         };
@@ -280,11 +479,16 @@ in
         # Inspired from https://www.authelia.com/configuration/session/introduction/ and https://www.authelia.com/configuration/session/redis
         session = {
           name = "authelia_session";
-          domain = if isNull cfg.port then cfg.domain else "${cfg.domain}:${toString cfg.port}";
+          cookies = [
+            {
+              domain = if isNull cfg.port then cfg.domain else "${cfg.domain}:${toString cfg.port}";
+              authelia_url = "https://${cfg.subdomain}.${cfg.domain}";
+            }
+          ];
           same_site = "lax";
           expiration = "1h";
           inactivity = "5m";
-          remember_me_duration = "1M";
+          remember_me = "1M";
           redis = {
             host = config.services.redis.servers.authelia.unixSocket;
             port = 0;
@@ -292,10 +496,9 @@ in
         };
         storage = {
           postgres = {
-            host = "/run/postgresql";
+            address = "unix:///run/postgresql";
             username = autheliaCfg.user;
             database = autheliaCfg.user;
-            port = config.services.postgresql.port;
             # Uses peer auth for local users, so we don't need a password.
             password = "test";
           };
@@ -305,8 +508,7 @@ in
             filename = cfg.smtp;
           };
           smtp = lib.mkIf (!(builtins.isString cfg.smtp)) {
-            host = cfg.smtp.host;
-            port = cfg.smtp.port;
+            address = "${cfg.smtp.scheme}://${cfg.smtp.host}:${toString cfg.smtp.port}";
             username = cfg.smtp.username;
             sender = "${cfg.smtp.from_name} <${cfg.smtp.from_address}>";
             subject = "[Authelia] {title}";
@@ -318,7 +520,11 @@ in
           networks = [
             {
               name = "internal";
-              networks = [ "10.0.0.0/8" "172.16.0.0/12" "192.168.0.0/18" ];
+              networks = [
+                "10.0.0.0/8"
+                "172.16.0.0/12"
+                "192.168.0.0/18"
+              ];
             }
           ];
           rules = [
@@ -329,7 +535,8 @@ in
                 "^/api/.*"
               ];
             }
-          ] ++ cfg.rules;
+          ]
+          ++ cfg.rules;
         };
         telemetry = {
           metrics = {
@@ -337,6 +544,28 @@ in
             address = "tcp://127.0.0.1:9959";
           };
         };
+
+        log.level = if cfg.debug then "debug" else "info";
+      }
+      // {
+        identity_providers.oidc = {
+          claims_policies = {
+            # This default claim should go away at some point.
+            # https://www.authelia.com/integration/openid-connect/openid-connect-1.0-claims/#restore-functionality-prior-to-claims-parameter
+            default.id_token = [
+              "email"
+              "preferred_username"
+              "name"
+              "groups"
+            ];
+          }
+          // cfg.extraOidcClaimsPolicies;
+          scopes = cfg.extraOidcScopes;
+          authorization_policies = cfg.extraOidcAuthorizationPolicies;
+        };
+      }
+      // lib.optionalAttrs (cfg.extraDefinitions != { }) {
+        definitions = cfg.extraDefinitions;
       };
 
       settingsFiles = [ "/var/lib/authelia-${fqdn}/oidc_clients.yaml" ];
@@ -344,16 +573,22 @@ in
 
     systemd.services."authelia-${fqdn}".preStart =
       let
-        mkCfg = clients:
-          shblib.replaceSecrets {
+        mkCfg =
+          clients:
+          shb.replaceSecrets {
             userConfig = {
               identity_providers.oidc.clients = clients;
             };
             resultPath = "/var/lib/authelia-${fqdn}/oidc_clients.yaml";
-            generator = shblib.replaceSecretsGeneratorAdapter (lib.generators.toYAML {});
+            generator = shb.replaceSecretsGeneratorAdapter (lib.generators.toYAML { });
           };
       in
-        lib.mkBefore (mkCfg cfg.oidcClients);
+      lib.mkBefore (
+        mkCfg cfg.oidcClients
+        + ''
+          ${pkgs.bash}/bin/bash -c '(while ! ${pkgs.netcat-openbsd}/bin/nc -z -v -w1 ${cfg.ldapHostname} ${toString cfg.ldapPort}; do echo "Waiting for port ${cfg.ldapHostname}:${toString cfg.ldapPort} to open..."; sleep 2; done); sleep 2'
+        ''
+      );
 
     services.nginx.virtualHosts.${fqdn} = {
       forceSSL = !(isNull cfg.ssl);
@@ -374,19 +609,21 @@ in
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host $http_host;
+        proxy_set_header X-Forwarded-Uri $request_uri;
         proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection "upgrade";
         proxy_cache_bypass $http_upgrade;
 
-        proxy_pass http://127.0.0.1:${toString autheliaCfg.settings.server.port};
+        proxy_pass http://127.0.0.1:9091;
         proxy_intercept_errors on;
         if ($request_method !~ ^(POST)$){
             error_page 401 = /error/401;
             error_page 403 = /error/403;
             error_page 404 = /error/404;
         }
-        '';
+      '';
 
       locations."/api/verify".extraConfig = ''
         add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
@@ -398,8 +635,23 @@ in
         add_header X-Permitted-Cross-Domain-Policies none;
 
         proxy_set_header Host $http_x_forwarded_host;
-        proxy_pass http://127.0.0.1:${toString autheliaCfg.settings.server.port};
-        '';
+        proxy_pass http://127.0.0.1:9091;
+      '';
+    };
+
+    # I would like this to live outside of the Authelia module.
+    # This will require a reverse proxy contract.
+    # Actually, not sure a full reverse proxy contract is needed.
+    shb.mitmdump.instances."authelia-${fqdn}" = lib.mkIf cfg.debug {
+      listenPort = 9091;
+      upstreamPort = 9090;
+      timeout = 30;
+      after = [ "authelia-${fqdn}.service" ];
+      enabledAddons = [ config.shb.mitmdump.addons.logger ];
+      extraArgs = [
+        "--set"
+        "verbose_pattern=/api"
+      ];
     };
 
     services.redis.servers.authelia = {
@@ -419,10 +671,30 @@ in
         job_name = "authelia";
         static_configs = [
           {
-            targets = ["127.0.0.1:9959"];
+            targets = [ "127.0.0.1:9959" ];
+            labels = {
+              "hostname" = config.networking.hostName;
+              "domain" = cfg.domain;
+            };
           }
         ];
       }
     ];
+
+    systemd.targets."authelia-${fqdn}" =
+      let
+        services = [
+          "authelia-${fqdn}.service"
+        ]
+        ++ lib.optionals cfg.debug [
+          config.shb.mitmdump.instances."authelia-${fqdn}".serviceName
+        ];
+      in
+      {
+        after = services;
+        requires = services;
+
+        wantedBy = [ "multi-user.target" ];
+      };
   };
 }

modules/blocks/authelia/docs/default.md

@@ -0,0 +1,244 @@
+# Authelia Block {#blocks-authelia}
+
+Defined in [`/modules/blocks/authelia.nix`](@REPO@/modules/blocks/authelia.nix).
+
+This block sets up an [Authelia][] service for Single-Sign On integration.
+
+[Authelia]: https://www.authelia.com/
+
+Compared to the upstream nixpkgs module, this module is tightly integrated
+with SHB which allows easy configuration of SSO with [OIDC integration](#blocks-authelia-shb-oidc)
+as well as some extensive [troubleshooting](#blocks-authelia-troubleshooting) features.
+
+Note that forward authentication is configured with the [nginx block](blocks-nginx.html#blocks-nginx-usage-shbforwardauth).
+
+## Features {#services-authelia-features}
+
+- Integration with the [dashboard contract](contracts-dashboard.html) for displaying user facing application in a dashboard. [Manual](#services-authelia-usage-applicationdashboard)
+
+## Usage {#services-authelia-usage}
+
+### Initial Configuration {#blocks-authelia-usage-configuration}
+
+Authelia cannot work without SSL and LDAP.
+So setting up the Authelia block requires to setup the [SSL block][] first
+and the [LLDAP block][] first.
+
+[SSL block]: blocks-ssl.html
+[LLDAP block]: blocks-lldap.html
+
+SSL is required to encrypt the communication and LDAP is used to handle users and group assignments.
+Authelia will allow access to a given resource only if the user that is authenticated
+is a member of the corresponding LDAP group.
+
+Afterwards, assuming the LDAP service runs on the same machine,
+the Authelia configuration can be done with:
+
+```nix
+shb.authelia = {
+  enable = true;
+  domain = "example.com";
+  subdomain = "auth";
+  ssl = config.shb.certs.certs.letsencrypt."example.com";
+
+  ldapHostname = "127.0.0.1";
+  ldapPort = config.shb.lldap.ldapPort;
+  dcdomain = config.shb.lldap.dcdomain;
+
+  smtp = {
+    host = "smtp.eu.mailgun.org";
+    port = 587;
+    username = "postmaster@mg.example.com";
+    from_address = "authelia@example.com";
+    password.result = config.shb.sops.secret."authelia/smtp_password".result;
+  };
+
+  secrets = {
+    jwtSecret.result = config.shb.sops.secret."authelia/jwt_secret".result;
+    ldapAdminPassword.result = config.shb.sops.secret."authelia/ldap_admin_password".result;
+    sessionSecret.result = config.shb.sops.secret."authelia/session_secret".result;
+    storageEncryptionKey.result = config.shb.sops.secret."authelia/storage_encryption_key".result;
+    identityProvidersOIDCHMACSecret.result = config.shb.sops.secret."authelia/hmac_secret".result;
+    identityProvidersOIDCIssuerPrivateKey.result = config.shb.sops.secret."authelia/private_key".result;
+  };
+};
+
+shb.certs.certs.letsencrypt."example.com".extraDomains = [ "auth.example.com" ];
+
+shb.sops.secret."authelia/jwt_secret".request = config.shb.authelia.secrets.jwtSecret.request;
+shb.sops.secret."authelia/ldap_admin_password" = {
+  request = config.shb.authelia.secrets.ldapAdminPassword.request;
+  settings.key = "lldap/user_password";
+};
+shb.sops.secret."authelia/session_secret".request = config.shb.authelia.secrets.sessionSecret.request;
+shb.sops.secret."authelia/storage_encryption_key".request = config.shb.authelia.secrets.storageEncryptionKey.request;
+shb.sops.secret."authelia/hmac_secret".request = config.shb.authelia.secrets.identityProvidersOIDCHMACSecret.request;
+shb.sops.secret."authelia/private_key".request = config.shb.authelia.secrets.identityProvidersOIDCIssuerPrivateKey.request;
+shb.sops.secret."authelia/smtp_password".request = config.shb.authelia.smtp.password.request;
+```
+
+This assumes secrets are setup with SOPS
+as mentioned in [the secrets setup section](usage.html#usage-secrets) of the manual.
+It's a bit annoying to setup all those secrets but it's only necessary once.
+Use `nix run nixpkgs#openssl -- rand -hex 64` to generate them.
+
+Crucially, the `shb.authelia.secrets.ldapAdminPasswordFile` must be the same
+as the `shb.lldap.ldapUserPassword` defined for the [LLDAP block][].
+This is done using Sops' `key` option.
+
+### Application Dashboard {#services-authelia-usage-applicationdashboard}
+
+Integration with the [dashboard contract](contracts-dashboard.html) is provided
+by the [dashboard option](#blocks-authelia-options-shb.authelia.dashboard).
+
+For example using the [Homepage](services-homepage.html) service:
+
+```nix
+{
+  shb.homepage.servicesGroups.Admin.services.Authelia = {
+    sortOrder = 2;
+    dashboard.request = config.shb.authelia.dashboard.request;
+  };
+}
+```
+
+## SHB OIDC integration {#blocks-authelia-shb-oidc}
+
+For services [provided by SelfHostBlocks][services] that handle [OIDC integration][OIDC],
+integrating with this block is done by configuring the service itself
+and linking it to this Authelia block through the `endpoint` option
+and by sharing a secret:
+
+[services]: services.html
+[OIDC]: https://openid.net/developers/how-connect-works/
+
+```nix
+shb.<service>.sso = {
+  enable = true;
+  endpoint = "https://${config.shb.authelia.subdomain}.${config.shb.authelia.domain}";
+
+  secret.result = config.shb.sops.secret."<service>/sso/secret".result;
+  secretForAuthelia.result = config.shb.sops.secret."<service>/sso/secretForAuthelia".result;
+};
+
+shb.sops.secret."<service>/sso/secret".request = config.shb.<service>.sso.secret.request;
+shb.sops.secret."<service>/sso/secretForAuthelia" = {
+  request = config.shb.<service>.sso.secretForAuthelia.request;
+  settings.key = "<service>/sso/secret";
+};
+```
+
+To share a secret between the service and Authelia,
+we generate a secret with `nix run nixpkgs#openssl -- rand -hex 64` under `<service>/sso/secret`
+then we ask Sops to use the same password for `<service>/sso/secretForAuthelia`
+thanks to the `settings.key` option.
+The difference between both secrets is one if owned by the `authelia` user
+while the other is owned by the user of the `<service`> we are configuring.
+
+## OIDC Integration {#blocks-authelia-oidc}
+
+To integrate a service handling OIDC integration not provided by SelfHostBlocks with this Authelia block,
+the necessary configuration is:
+
+```nix
+shb.authelia.oidcClients = [
+  {
+    client_id = "<service>";
+    client_secret.source = config.shb.sops.secret."<service>/sso/secretForAuthelia".response.path;
+    scopes = [ "openid" "email" "profile" ];
+    redirect_uris = [
+      "<provided by service documentation>"
+    ];
+  }
+];
+
+shb.sops.secret."<service>/sso/secret".request = {
+  owner = "<service_user>";
+};
+shb.sops.secret."<service>/sso/secretForAuthelia" = {
+  request.owner = "authelia";
+  settings.key = "<service>/sso/secret";
+};
+```
+
+As in the previous section, we create a shared secret using Sops'
+`settings.key` option.
+
+The configuration for the service itself is much dependent on the service itself.
+For example for [open-webui][], the configuration looks like so:
+
+[open-webui]: https://search.nixos.org/options?query=services.open-webui
+
+```nix
+services.open-webui.environment = {
+  ENABLE_SIGNUP = "False";
+  WEBUI_AUTH = "True";
+  ENABLE_FORWARD_USER_INFO_HEADERS = "True";
+  ENABLE_OAUTH_SIGNUP = "True";
+  OAUTH_UPDATE_PICTURE_ON_LOGIN = "True";
+  OAUTH_CLIENT_ID = "open-webui";
+  OAUTH_CLIENT_SECRET = "<raw secret>";
+  OPENID_PROVIDER_URL = "https://${config.shb.authelia.subdomain}.${config.shb.authelia.domain}/.well-known/openid-configuration";
+  OAUTH_PROVIDER_NAME = "Single Sign-On";
+  OAUTH_SCOPES = "openid email profile";
+  OAUTH_ALLOWED_ROLES = "open-webui_user";
+  OAUTH_ADMIN_ROLES = "open-webui_admin";
+  ENABLE_OAUTH_ROLE_MANAGEMENT = "True";
+};
+
+shb.authelia.oidcClients = [
+  {
+    client_id = "open-webui";
+    client_secret.source = config.shb.sops.secret."open-webui/sso/secretForAuthelia".response.path;
+    scopes = [ "openid" "email" "profile" ];
+    redirect_uris = [
+      "<provided by service documentation>"
+    ];
+  }
+];
+
+shb.sops.secret."open-webui/sso/secret".request = {
+  owner = "open-webui";
+};
+shb.sops.secret."open-webui/sso/secretForAuthelia" = {
+  request.owner = "authelia";
+  settings.key = "open-webui/sso/secret";
+};
+```
+
+Here, there is no way to give a path for the `OAUTH_CLIENT_SECRET`,
+we are obligated to pass the raw secret which is a very bad idea.
+There are ways around this but they are out of scope for this section.
+Inspiration can be taken from SelfHostBlocks' source code.
+
+To access the UI, we will need to create an `open-webui_user` and
+`open-webui_admin` LDAP group and assign our user to it.
+
+## Forward Auth {#blocks-authelia-forward-auth}
+
+Forward authentication is provided by the [nginx block](blocks-nginx.html#blocks-nginx-usage-ssl).
+
+## Troubleshooting {#blocks-authelia-troubleshooting}
+
+Set the [debug][opt-debug] option to `true` to:
+
+[opt-debug]: #blocks-authelia-options-shb.authelia.debug
+
+- Set logging level to `"debug"`.
+- Add an [shb.mitmdump][] instance in front of Authelia
+  which prints all requests and responses headers and body
+  to the systemd service `mitmdump-authelia-${config.shb.authelia.subdomain}.${config.shb.authelia.domain}.service`.
+
+[shb.mitmdump]: ./blocks-mitmdump.html
+
+## Tests {#blocks-authelia-tests}
+
+Specific integration tests are defined in [`/test/blocks/authelia.nix`](@REPO@/test/blocks/authelia.nix).
+
+## Options Reference {#blocks-authelia-options}
+
+```{=include=} options
+id-prefix: blocks-authelia-options-
+list-id: selfhostblocks-block-authelia-options
+source: @OPTIONS_JSON@
+```

modules/blocks/backup.nix

@@ -1,352 +0,0 @@
-{ config, pkgs, lib, utils, ... }:
-
-let
-  cfg = config.shb.backup;
-
-  instanceOptions = {
-    enable = lib.mkEnableOption "shb backup instance";
-
-    backend = lib.mkOption {
-      description = "What program to use to make the backups.";
-      type = lib.types.enum [ "borgmatic" "restic" ];
-      example = "borgmatic";
-    };
-
-    keySopsFile = lib.mkOption {
-      description = "Sops file that holds this instance's Borgmatic repository key and passphrase.";
-      type = lib.types.path;
-      example = "secrets/backup.yaml";
-    };
-
-    sourceDirectories = lib.mkOption {
-      description = "Borgmatic source directories.";
-      type = lib.types.nonEmptyListOf lib.types.str;
-    };
-
-    excludePatterns = lib.mkOption {
-      description = "Borgmatic exclude patterns.";
-      type = lib.types.listOf lib.types.str;
-      default = [];
-    };
-
-    secretName = lib.mkOption {
-      description = "Secret name, if null use the name of the backup instance.";
-      type = lib.types.nullOr lib.types.str;
-      default = null;
-    };
-
-    repositories = lib.mkOption {
-      description = "Repositories to back this instance to.";
-      type = lib.types.nonEmptyListOf (lib.types.submodule {
-        options = {
-          path = lib.mkOption {
-            type = lib.types.str;
-            description = "Repository location";
-          };
-
-          timerConfig = lib.mkOption {
-            type = lib.types.attrsOf utils.systemdUtils.unitOptions.unitOption;
-            default = {
-              OnCalendar = "daily";
-              Persistent = true;
-            };
-            description = ''When to run the backup. See {manpage}`systemd.timer(5)` for details.'';
-            example = {
-              OnCalendar = "00:05";
-              RandomizedDelaySec = "5h";
-              Persistent = true;
-            };
-          };
-        };
-      });
-    };
-
-    retention = lib.mkOption {
-      description = "Retention options.";
-      type = lib.types.attrsOf (lib.types.oneOf [ lib.types.int lib.types.nonEmptyStr ]);
-      default = {
-        keep_within = "1d";
-        keep_hourly = 24;
-        keep_daily = 7;
-        keep_weekly = 4;
-        keep_monthly = 6;
-      };
-    };
-
-    consistency = lib.mkOption {
-      description = "Consistency frequency options. Only applicable for borgmatic";
-      type = lib.types.attrsOf lib.types.nonEmptyStr;
-      default = {};
-      example = {
-        repository = "2 weeks";
-        archives = "1 month";
-      };
-    };
-
-    hooks = lib.mkOption {
-      description = "Borgmatic hooks.";
-      default = {};
-      type = lib.types.submodule {
-        options = {
-          before_backup = lib.mkOption {
-            description = "Hooks to run before backup";
-            type = lib.types.listOf lib.types.str;
-            default = [];
-          };
-
-          after_backup = lib.mkOption {
-            description = "Hooks to run after backup";
-            type = lib.types.listOf lib.types.str;
-            default = [];
-          };
-        };
-      };
-    };
-
-    environmentFile = lib.mkOption {
-      type = lib.types.bool;
-      description = "Add environment file to be read by the systemd service.";
-      default = false;
-      example = true;
-    };
-  };
-
-  repoSlugName = name: builtins.replaceStrings ["/" ":"] ["_" "_"] (lib.strings.removePrefix "/" name);
-
-in
-{
-  options.shb.backup = {
-    onlyOnAC = lib.mkOption {
-      description = "Run backups only if AC power is plugged in.";
-      default = true;
-      example = false;
-      type = lib.types.bool;
-    };
-
-    user = lib.mkOption {
-      description = "Unix user doing the backups.";
-      type = lib.types.str;
-      default = "backup";
-    };
-
-    group = lib.mkOption {
-      description = "Unix group doing the backups.";
-      type = lib.types.str;
-      default = "backup";
-    };
-
-    instances = lib.mkOption {
-      description = "Each instance is a backup setting";
-      default = {};
-      type = lib.types.attrsOf (lib.types.submodule {
-        options = instanceOptions;
-      });
-    };
-
-    borgServer = lib.mkOption {
-      description = "Add borgbackup package so external backups can use this server as a remote.";
-      default = false;
-      example = true;
-      type = lib.types.bool;
-    };
-
-    # Taken from https://github.com/HubbeKing/restic-kubernetes/blob/73bfbdb0ba76939a4c52173fa2dbd52070710008/README.md?plain=1#L23
-    performance = lib.mkOption {
-      description = "Reduce performance impact of backup jobs.";
-      default = {};
-      type = lib.types.submodule {
-        options = {
-          niceness = lib.mkOption {
-            type = lib.types.ints.between (-20) 19;
-            description = "nice priority adjustment, defaults to 15 for ~20% CPU time of normal-priority process";
-            default = 15;
-          };
-          ioSchedulingClass = lib.mkOption {
-            type = lib.types.enum [ "idle" "best-effort" "realtime" ];
-            description = "ionice scheduling class, defaults to best-effort IO. Only used for `restic backup`, `restic forget` and `restic check` commands.";
-            default = "best-effort";
-          };
-          ioPriority = lib.mkOption {
-            type = lib.types.nullOr (lib.types.ints.between 0 7);
-            description = "ionice priority, defaults to 7 for lowest priority IO. Only used for `restic backup`, `restic forget` and `restic check` commands.";
-            default = 7;
-          };
-        };
-      };
-    };
-  };
-
-  config = lib.mkIf (cfg.instances != {}) (
-    let
-      enabledInstances = lib.attrsets.filterAttrs (k: i: i.enable) cfg.instances;
-      borgmaticInstances = lib.attrsets.filterAttrs (k: i: i.backend == "borgmatic") enabledInstances;
-      resticInstances = lib.attrsets.filterAttrs (k: i: i.backend == "restic") enabledInstances;
-    in lib.mkMerge [
-      # Secrets configuration
-      {
-        users.users = {
-          ${cfg.user} = {
-            name = cfg.user;
-            group = cfg.group;
-            home = "/var/lib/backup";
-            createHome = true;
-            isSystemUser = true;
-            extraGroups = [ "keys" ];
-          };
-        };
-        users.groups = {
-          ${cfg.group} = {
-            name = cfg.group;
-          };
-        };
-
-        sops.secrets =
-          let
-            mkSopsSecret = name: instance: (
-              [
-                {
-                  "${instance.backend}/passphrases/${if isNull instance.secretName then name else instance.secretName}" = {
-                    sopsFile = instance.keySopsFile;
-                    mode = "0440";
-                    owner = cfg.user;
-                    group = cfg.group;
-                  };
-                }
-              ] ++ lib.optional ((lib.filter ({path, ...}: lib.strings.hasPrefix "s3" path) instance.repositories) != []) {
-                "${instance.backend}/environmentfiles/${if isNull instance.secretName then name else instance.secretName}" = {
-                  sopsFile = instance.keySopsFile;
-                  mode = "0440";
-                  owner = cfg.user;
-                  group = cfg.group;
-                };
-              } ++ lib.optionals (instance.backend == "borgmatic") (lib.flatten (map ({path, ...}: {
-                "${instance.backend}/keys/${repoSlugName path}" = {
-                  key = "${instance.backend}/keys/${if isNull instance.secretName then name else instance.secretName}";
-                  sopsFile = instance.keySopsFile;
-                  mode = "0440";
-                  owner = cfg.user;
-                  group = cfg.group;
-                };
-              }) instance.repositories))
-            );
-          in
-            lib.mkMerge (lib.flatten (lib.attrsets.mapAttrsToList mkSopsSecret enabledInstances));
-      }
-      # Borgmatic configuration
-      {
-        systemd.timers.borgmatic = lib.mkIf (borgmaticInstances != {}) {
-          timerConfig = {
-            OnCalendar = "hourly";
-          };
-        };
-
-        systemd.services.borgmatic = lib.mkIf (borgmaticInstances != {}) {
-          serviceConfig = {
-            User = cfg.user;
-            Group = cfg.group;
-            ExecStartPre = [ "" ]; # Do not sleep before starting.
-            ExecStart = [ "" "${pkgs.borgmatic}/bin/borgmatic --verbosity -1 --syslog-verbosity 1" ];
-            # For borgmatic, since we have only one service, we need to merge all environmentFile
-            # from all instances.
-            EnvironmentFile = lib.mapAttrsToList (name: value: value.environmentFile) enabledInstances;
-          };
-        };
-
-        systemd.packages = lib.mkIf (borgmaticInstances != {}) [ pkgs.borgmatic ];
-        environment.systemPackages = (
-          lib.optionals cfg.borgServer [ pkgs.borgbackup ]
-          ++ lib.optionals (borgmaticInstances != {}) [ pkgs.borgbackup pkgs.borgmatic ]
-        );
-
-        environment.etc =
-          let
-            mkSettings = name: instance: {
-              "borgmatic.d/${name}.yaml".text = lib.generators.toYAML {} {
-                location =
-                  {
-                    source_directories = instance.sourceDirectories;
-                    repositories = map ({path, ...}: path) instance.repositories;
-                  }
-                  // (lib.attrsets.optionalAttrs (builtins.length instance.excludePatterns > 0) {
-                    excludePatterns = instance.excludePatterns;
-                  });
-
-                storage = {
-                  encryption_passcommand = "cat /run/secrets/borgmatic/passphrases/${if isNull instance.secretName then name else instance.secretName}";
-                  borg_keys_directory = "/run/secrets/borgmatic/keys";
-                };
-
-                retention = instance.retention;
-                consistency.checks =
-                  let
-                    mkCheck = name: frequency: {
-                      inherit name frequency;
-                    };
-                  in
-                    lib.attrsets.mapAttrsToList mkCheck instance.consistency;
-
-                # hooks = lib.mkMerge [
-                #   lib.optionalAttrs (builtins.length instance.hooks.before_backup > 0) {
-                #     inherit (instance.hooks) before_backup;
-                #   }
-                #   lib.optionalAttrs (builtins.length instance.hooks.after_backup > 0) {
-                #     inherit (instance.hooks) after_backup;
-                #   }
-                # ];
-              };
-            };
-          in
-            lib.mkMerge (lib.attrsets.mapAttrsToList mkSettings borgmaticInstances);
-      }
-      # Restic configuration
-      {
-        environment.systemPackages = lib.optionals (resticInstances != {}) [ pkgs.restic ];
-
-        services.restic.backups =
-          let
-            mkRepositorySettings = name: instance: repository: {
-              "${name}_${repoSlugName repository.path}" = {
-                inherit (cfg) user;
-                repository = repository.path;
-
-                paths = instance.sourceDirectories;
-
-                passwordFile = "/run/secrets/${instance.backend}/passphrases/${name}";
-
-                initialize = true;
-
-                inherit (repository) timerConfig;
-
-                pruneOpts = lib.mapAttrsToList (name: value:
-                  "--${builtins.replaceStrings ["_"] ["-"] name} ${builtins.toString value}"
-                ) instance.retention;
-
-                backupPrepareCommand = lib.strings.concatStringsSep "\n" instance.hooks.before_backup;
-
-                backupCleanupCommand = lib.strings.concatStringsSep "\n" instance.hooks.after_backup;
-              } // lib.attrsets.optionalAttrs (instance.environmentFile) {
-                environmentFile = "/run/secrets/${instance.backend}/environmentfiles/${name}";
-              } // lib.attrsets.optionalAttrs (builtins.length instance.excludePatterns > 0) {
-                exclude = instance.excludePatterns;
-              };
-            };
-
-            mkSettings = name: instance: builtins.map (mkRepositorySettings name instance) instance.repositories;
-          in
-            lib.mkMerge (lib.flatten (lib.attrsets.mapAttrsToList mkSettings resticInstances));
-
-        systemd.services =
-          let
-            mkRepositorySettings = name: instance: repository: {
-              "restic-backups-${name}_${repoSlugName repository.path}".serviceConfig = {
-                Nice = cfg.performance.niceness;
-                IOSchedulingClass = cfg.performance.ioSchedulingClass;
-                IOSchedulingPriority = cfg.performance.ioPriority;
-              };
-            };
-            mkSettings = name: instance: builtins.map (mkRepositorySettings name instance) instance.repositories;
-          in
-            lib.mkMerge (lib.flatten (lib.attrsets.mapAttrsToList mkSettings resticInstances));
-      }
-    ]);
-}

modules/blocks/backup/docs/default.md

@@ -1,230 +0,0 @@
-# Backup Block {#blocks-backup}
-
-Defined in [`/modules/blocks/backup.nix`](@REPO@/modules/blocks/backup.nix).
-
-This block sets up backup jobs for Self Host Blocks.
-
-## Features {#blocks-backup-features}
-Two implementations for this block are provided:
-- [Restic](https://restic.net/)
-- [Borgmatic](https://torsion.org/borgmatic/)
-
-No integration tests are provided yet.
-
-## Usage {#blocks-backup-usage}
-
-### One folder backed up to mounted hard drives {#blocks-backup-usage-one}
-
-The following snippet shows how to configure backup of 1 folder using the Restic implementation to 1
-repository.
-
-Assumptions:
-- 1 hard drive pool is used for backup and is mounted on `/srv/pool1`.
-
-```nix
-shb.backup.instances.myfolder = {
-  enable = true;
-
-  backend = "restic";
-
-  keySopsFile = ./secrets.yaml;
-
-  repositories = [{
-    path = "/srv/pool1/backups/myfolder";
-    timerConfig = {
-      OnCalendar = "00:00:00";
-      RandomizedDelaySec = "3h";
-    };
-  }];
-
-  sourceDirectories = [
-    "/var/lib/myfolder"
-  ];
-
-  retention = {
-    keep_within = "1d";
-    keep_hourly = 24;
-    keep_daily = 7;
-    keep_weekly = 4;
-    keep_monthly = 6;
-  };
-
-  consistency = {
-    repository = "2 weeks";
-    archives = "1 month";
-  };
-};
-```
-
-The referenced Sops file must follow this structure:
-
-```yaml
-restic:
-    passphrases:
-        myfolder: <secret>
-```
-
-To generate a secret, use: `nix run nixpkgs#openssl -- rand -hex 64`.
-
-With the borgmatic implementation, the structure should be:
-
-```yaml
-borgmatic:
-    keys:
-        myfolder: |
-            BORG_KEY <key>
-    passphrases:
-        myfolder: <secret>
-```
-
-You can have both borgmatic and restic implementations working at the same time.
-
-### One folder backed up to S3 {#blocks-backup-usage-remote}
-
-> This is only supported by the Restic implementation. 
-
-Here we will only highlight the differences with the previous configuration.
-
-This assumes you have access to such a remote S3 store, for example by using Backblaze.
-
-```diff
-  shb.backup.instances.myfolder = {
-
-    repositories = [{
--     path = "/srv/pool1/backups/myfolder";
-+     path = "s3:s3.us-west-000.backblazeb2.com/backups/myfolder";
-      timerConfig = {
-        OnCalendar = "00:00:00";
-        RandomizedDelaySec = "3h";
-      };
-    }];
-
-
-+   environmentFile = true; # Needed for s3
-  }
-```
-
-The Sops file has a new required field:
-
-```yaml
-
-  restic:
-      passphrases:
-          myfolder: <secret>
-+     environmentfiles:
-+         myfolder: |-
-+             AWS_ACCESS_KEY_ID=<aws_key_id>
-+             AWS_SECRET_ACCESS_KEY=<aws_secret_key>
-```
-
-### Multiple folder to multiple destinations {#blocks-backup-usage-multiple}
-
-The following snippet shows how to configure backup of any number of folders using the Restic
-implementation to 3 repositories, each happening at different times to avoid contending for I/O
-time.
-
-We will also make sure to be able to re-use as much as the configuration as possible.
-
-A few assumptions:
-- 2 hard drive pools used for backup are mounted respectively on `/srv/pool1` and `/srv/pool2`.
-- You have a backblaze account.
-
-First, let's define a variable to hold all our repositories you want to back up to:
-
-```nix
-repos = [
-  {
-    path = "/srv/pool1/backups";
-    timerConfig = {
-      OnCalendar = "00:00:00";
-      RandomizedDelaySec = "3h";
-    };
-  }
-  {
-    path = "/srv/pool2/backups";
-    timerConfig = {
-      OnCalendar = "08:00:00";
-      RandomizedDelaySec = "3h";
-    };
-  }
-  {
-    path = "s3:s3.us-west-000.backblazeb2.com/backups";
-    timerConfig = {
-      OnCalendar = "16:00:00";
-      RandomizedDelaySec = "3h";
-    };
-  }
-];
-```
-
-Compared to the previous examples, we do not include the name of what we will back up in the
-repository paths.
-
-Now, let's define a function to create a backup configuration. It will take a list of repositories,
-a name identifying the backup and a list of folders to back up.
-
-```nix
-backupcfg = repositories: name: sourceDirectories {
-  enable = true;
-
-  backend = "restic";
-
-  keySopsFile = ../secrets/backup.yaml;
-
-  repositories = builtins.map (r: {
-    path = "${r.path}/${name}";
-    inherit (r) timerConfig;
-  }) repositories;
-
-  inherit sourceDirectories;
-
-  retention = {
-    keep_within = "1d";
-    keep_hourly = 24;
-    keep_daily = 7;
-    keep_weekly = 4;
-    keep_monthly = 6;
-  };
-
-  consistency = {
-    repository = "2 weeks";
-    archives = "1 month";
-  };
-
-  environmentFile = true;
-};
-```
-
-Now, we can define multiple backup jobs to backup different folders:
-
-```nix
-shb.backup.instances.myfolder1 = backupcfg repos ["/var/lib/myfolder1"];
-shb.backup.instances.myfolder2 = backupcfg repos ["/var/lib/myfolder2"];
-```
-
-The difference between the above snippet and putting all the folders into one configuration (shown
-below) is the former splits the backups into sub-folders on the repositories.
-
-```nix
-shb.backup.instances.all = backupcfg repos ["/var/lib/myfolder1" "/var/lib/myfolder2"];
-```
-
-## Demo {#blocks-backup-demo}
-
-[WIP]
-
-## Monitoring {#blocks-backup-monitoring}
-
-[WIP]
-
-## Maintenance {#blocks-backup-maintenance}
-
-[WIP]
-
-## Options Reference {#blocks-backup-options}
-
-```{=include=} options
-id-prefix: blocks-backup-options-
-list-id: selfhostblocks-block-backup-options
-source: @OPTIONS_JSON@
-```

modules/blocks/borgbackup.nix

@@ -0,0 +1,501 @@
+{
+  config,
+  pkgs,
+  lib,
+  utils,
+  shb,
+  ...
+}:
+
+let
+  cfg = config.shb.borgbackup;
+
+  inherit (lib)
+    concatStringsSep
+    filterAttrs
+    flatten
+    literalExpression
+    optionals
+    listToAttrs
+    mapAttrsToList
+    mkOption
+    mkMerge
+    ;
+  inherit (lib)
+    mkIf
+    nameValuePair
+    optionalAttrs
+    removePrefix
+    ;
+  inherit (lib.types)
+    attrsOf
+    int
+    oneOf
+    nonEmptyStr
+    nullOr
+    str
+    submodule
+    ;
+
+  commonOptions =
+    {
+      name,
+      prefix,
+      config,
+      ...
+    }:
+    {
+      enable = lib.mkEnableOption ''
+        SelfHostBlocks' BorgBackup block;
+
+        A disabled instance will not backup data anymore
+        but still provides the helper tool to restore snapshots
+      '';
+
+      passphrase = lib.mkOption {
+        description = "Encryption key for the backup repository.";
+        type = lib.types.submodule {
+          options = shb.contracts.secret.mkRequester {
+            mode = "0400";
+            owner = config.request.user;
+            ownerText = "[shb.borgbackup.${prefix}.<name>.request.user](#blocks-borgbackup-options-shb.borgbackup.${prefix}._name_.request.user)";
+            restartUnits = [ (fullName name config.settings.repository) ];
+            restartUnitsText = "[ [shb.borgbackup.${prefix}.<name>.settings.repository](#blocks-borgbackup-options-shb.borgbackup.${prefix}._name_.settings.repository) ]";
+          };
+        };
+      };
+
+      repository = lib.mkOption {
+        description = "Repository to send the backups to.";
+        type = submodule {
+          options = {
+            path = mkOption {
+              type = str;
+              description = "Repository location";
+            };
+
+            secrets = mkOption {
+              type = attrsOf shb.secretFileType;
+              default = { };
+              description = ''
+                Secrets needed to access the repository where the backups will be stored.
+
+                See [s3 config](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3) for an example
+                and [list](https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables) for the list of all secrets.
+
+              '';
+              example = literalExpression ''
+                {
+                  AWS_ACCESS_KEY_ID.source = <path/to/secret>;
+                  AWS_SECRET_ACCESS_KEY.source = <path/to/secret>;
+                }
+              '';
+            };
+
+            timerConfig = mkOption {
+              type = attrsOf utils.systemdUtils.unitOptions.unitOption;
+              default = {
+                OnCalendar = "daily";
+                Persistent = true;
+              };
+              description = "When to run the backup. See {manpage}`systemd.timer(5)` for details.";
+              example = {
+                OnCalendar = "00:05";
+                RandomizedDelaySec = "5h";
+                Persistent = true;
+              };
+            };
+          };
+        };
+      };
+
+      retention = lib.mkOption {
+        description = "Retention options. See {command}`borg help prune` for the available options.";
+        type = attrsOf (oneOf [
+          int
+          nonEmptyStr
+        ]);
+        default = {
+          within = "1d";
+          hourly = 24;
+          daily = 7;
+          weekly = 4;
+          monthly = 6;
+        };
+      };
+
+      consistency = lib.mkOption {
+        description = "Consistency frequency options.";
+        type = lib.types.attrsOf lib.types.nonEmptyStr;
+        default = { };
+        example = {
+          repository = "2 weeks";
+          archives = "1 month";
+        };
+      };
+
+      limitUploadKiBs = mkOption {
+        type = nullOr int;
+        description = "Limit upload bandwidth to the given KiB/s amount.";
+        default = null;
+        example = 8000;
+      };
+
+      stateDir = mkOption {
+        type = nullOr lib.types.str;
+        description = ''
+          Override the directory in which {command}`borg` stores its
+          configuration and cache. By default it uses the user's
+          home directory but is some cases this can cause conflicts.
+        '';
+        default = null;
+      };
+    };
+
+  repoSlugName = name: builtins.replaceStrings [ "/" ":" ] [ "_" "_" ] (removePrefix "/" name);
+  fullName = name: repository: "borgbackup-job-${name}_${repoSlugName repository.path}";
+in
+{
+  imports = [
+    ../../lib/module.nix
+    ../blocks/monitoring.nix
+  ];
+
+  options.shb.borgbackup = {
+    enableDashboard = lib.mkEnableOption "the Backups SHB dashboard" // {
+      default = true;
+    };
+
+    instances = mkOption {
+      description = "Files to backup following the [backup contract](./shb.contracts-backup.html).";
+      default = { };
+      type = attrsOf (
+        submodule (
+          { name, config, ... }:
+          {
+            options = shb.contracts.backup.mkProvider {
+              settings = mkOption {
+                description = ''
+                  Settings specific to the BorgBackup provider.
+                '';
+
+                type = submodule {
+                  options = commonOptions {
+                    inherit name config;
+                    prefix = "instances";
+                  };
+                };
+              };
+
+              resultCfg = {
+                restoreScript = fullName name config.settings.repository;
+                restoreScriptText = "${fullName "<name>" { path = "path/to/repository"; }}";
+
+                backupService = "${fullName name config.settings.repository}.service";
+                backupServiceText = "${fullName "<name>" { path = "path/to/repository"; }}.service";
+              };
+            };
+          }
+        )
+      );
+    };
+
+    databases = mkOption {
+      description = "Databases to backup following the [database backup contract](./shb.contracts-databasebackup.html).";
+      default = { };
+      type = attrsOf (
+        submodule (
+          { name, config, ... }:
+          {
+            options = shb.contracts.databasebackup.mkProvider {
+              settings = mkOption {
+                description = ''
+                  Settings specific to the BorgBackup provider.
+                '';
+
+                type = submodule {
+                  options = commonOptions {
+                    inherit name config;
+                    prefix = "databases";
+                  };
+                };
+              };
+
+              resultCfg = {
+                restoreScript = fullName name config.settings.repository;
+                restoreScriptText = "${fullName "<name>" { path = "path/to/repository"; }}";
+
+                backupService = "${fullName name config.settings.repository}.service";
+                backupServiceText = "${fullName "<name>" { path = "path/to/repository"; }}.service";
+              };
+            };
+          }
+        )
+      );
+    };
+
+    borgServer = lib.mkOption {
+      description = "Add borgbackup package to `environment.systemPackages` so external backups can use this server as a remote.";
+      default = false;
+      example = true;
+      type = lib.types.bool;
+    };
+
+    # Taken from https://github.com/HubbeKing/restic-kubernetes/blob/73bfbdb0ba76939a4c52173fa2dbd52070710008/README.md?plain=1#L23
+    performance = lib.mkOption {
+      description = "Reduce performance impact of backup jobs.";
+      default = { };
+      type = lib.types.submodule {
+        options = {
+          niceness = lib.mkOption {
+            type = lib.types.ints.between (-20) 19;
+            description = "nice priority adjustment, defaults to 15 for ~20% CPU time of normal-priority process";
+            default = 15;
+          };
+          ioSchedulingClass = lib.mkOption {
+            type = lib.types.enum [
+              "idle"
+              "best-effort"
+              "realtime"
+            ];
+            description = "ionice scheduling class, defaults to best-effort IO.";
+            default = "best-effort";
+          };
+          ioPriority = lib.mkOption {
+            type = lib.types.nullOr (lib.types.ints.between 0 7);
+            description = "ionice priority, defaults to 7 for lowest priority IO.";
+            default = 7;
+          };
+        };
+      };
+    };
+  };
+
+  config = lib.mkIf (cfg.instances != { } || cfg.databases != { }) (
+    let
+      enabledInstances = filterAttrs (k: i: i.settings.enable) cfg.instances;
+      enabledDatabases = filterAttrs (k: i: i.settings.enable) cfg.databases;
+    in
+    lib.mkMerge [
+      {
+        environment.systemPackages =
+          optionals (cfg.borgServer || enabledInstances != { } || enabledDatabases != { })
+            [
+              pkgs.borgbackup
+            ];
+      }
+      {
+        services.borgbackup.jobs =
+          let
+            mkJob = name: instance: {
+              "${name}_${repoSlugName instance.settings.repository.path}" = {
+                inherit (instance.request) user;
+
+                repo = instance.settings.repository.path;
+
+                paths = instance.request.sourceDirectories;
+
+                encryption.mode = "repokey-blake2";
+                # We do not set encryption.passphrase here, we set BORG_PASSPHRASE_FD further down.
+                encryption.passCommand = "cat ${instance.settings.passphrase.result.path}";
+
+                doInit = true;
+                failOnWarnings = true;
+                stateDir = instance.settings.stateDir;
+
+                persistentTimer = instance.settings.repository.timerConfig.Persistent or false;
+                startAt = ""; # Some non-empty string value tricks the upstream module in creating the systemd timer.
+
+                prune.keep = instance.settings.retention;
+
+                preHook = concatStringsSep "\n" instance.request.hooks.beforeBackup;
+
+                postHook = concatStringsSep "\n" instance.request.hooks.afterBackup;
+
+                extraArgs = (
+                  optionals (instance.settings.limitUploadKiBs != null) [
+                    "--upload-ratelimit=${toString instance.settings.limitUploadKiBs}"
+                  ]
+                );
+
+                exclude = instance.request.excludePatterns;
+              };
+            };
+          in
+          mkMerge (mapAttrsToList mkJob enabledInstances);
+      }
+      {
+        services.borgbackup.jobs =
+          let
+            mkJob = name: instance: {
+              "${name}_${repoSlugName instance.settings.repository.path}" = {
+                inherit (instance.request) user;
+
+                repo = instance.settings.repository.path;
+
+                dumpCommand = lib.getExe (
+                  pkgs.writeShellApplication {
+                    name = "dump-command";
+                    text = instance.request.backupCmd;
+                  }
+                );
+
+                encryption.mode = "repokey-blake2";
+                # We do not set encryption.passphrase here, we set BORG_PASSPHRASE_FD further down.
+                encryption.passCommand = "cat ${instance.settings.passphrase.result.path}";
+
+                doInit = true;
+                failOnWarnings = true;
+                stateDir = instance.settings.stateDir;
+
+                persistentTimer = instance.settings.repository.timerConfig.Persistent or false;
+                startAt = ""; # Some non-empty list value that tricks upstream in creating the systemd timer.
+
+                prune.keep = instance.settings.retention;
+
+                extraArgs = (
+                  optionals (instance.settings.limitUploadKiBs != null) [
+                    "--upload-ratelimit=${toString instance.settings.limitUploadKiBs}"
+                  ]
+                );
+              };
+            };
+          in
+          mkMerge (mapAttrsToList mkJob enabledDatabases);
+      }
+      {
+        systemd.timers =
+          let
+            mkTimer = name: instance: {
+              ${fullName name instance.settings.repository} = {
+                timerConfig = lib.mkForce instance.settings.repository.timerConfig;
+              };
+            };
+          in
+          mkMerge (mapAttrsToList mkTimer (enabledInstances // enabledDatabases));
+      }
+      {
+        systemd.services =
+          let
+            mkSettings =
+              name: instance:
+              let
+                serviceName = fullName name instance.settings.repository;
+              in
+              {
+                ${serviceName} = mkMerge [
+                  {
+                    serviceConfig = {
+                      # Purposely not a oneshot systemd service otherwise
+                      # the service waits on the completion the backup before deactivating.
+                      # This seems like a nice property at first but there is one annoying
+                      # edge case when deploying. If a backup job somehow is started when
+                      # the deploy happens, the deploy will wait on the service to finish
+                      # before considering the deploy done. And worse, it will consider the
+                      # deploy as failed if the backup fails, which is not what you want.
+                      Nice = lib.mkForce cfg.performance.niceness;
+                      IOSchedulingClass = lib.mkForce cfg.performance.ioSchedulingClass;
+                      IOSchedulingPriority = lib.mkForce cfg.performance.ioPriority;
+                      # BindReadOnlyPaths = instance.sourceDirectories;
+                    };
+                  }
+                  (optionalAttrs (instance.settings.repository.secrets != { }) {
+                    serviceConfig.EnvironmentFile = [
+                      "/run/secrets_borgbackup/${serviceName}"
+                    ];
+                    after = [ "${serviceName}-pre.service" ];
+                    requires = [ "${serviceName}-pre.service" ];
+                  })
+                ];
+
+                "${serviceName}-pre" = mkIf (instance.settings.repository.secrets != { }) (
+                  let
+                    script = shb.genConfigOutOfBandSystemd {
+                      config = instance.settings.repository.secrets;
+                      configLocation = "/run/secrets_borgbackup/${serviceName}";
+                      generator = shb.toEnvVar;
+                      user = instance.request.user;
+                    };
+                  in
+                  {
+                    script = script.preStart;
+                    # Makes the systemd service wait for the backup to be done before changing state to inactive.
+                    serviceConfig.Type = "oneshot";
+                    serviceConfig.LoadCredential = script.loadCredentials;
+                  }
+                );
+              };
+          in
+          mkMerge (flatten (mapAttrsToList mkSettings (enabledInstances // enabledDatabases)));
+      }
+      {
+        systemd.services =
+          let
+            mkEnv =
+              name: instance:
+              nameValuePair "${fullName name instance.settings.repository}_restore_gen" {
+                enable = true;
+                wantedBy = [ "multi-user.target" ];
+                serviceConfig.Type = "oneshot";
+                script = (
+                  shb.replaceSecrets {
+                    userConfig = instance.settings.repository.secrets // {
+                      BORG_PASSCOMMAND = ''"cat ${instance.settings.passphrase.result.path}"'';
+                      BORG_REPO = instance.settings.repository.path;
+                    };
+                    resultPath = "/run/secrets_borgbackup_env/${fullName name instance.settings.repository}";
+                    generator = shb.toEnvVar;
+                    user = instance.request.user;
+                  }
+                );
+              };
+          in
+          listToAttrs (flatten (mapAttrsToList mkEnv (cfg.instances // cfg.databases)));
+      }
+      {
+        environment.systemPackages =
+          let
+            mkBorgBackupBinary =
+              name: instance:
+              shb.contracts.backup.mkRestoreScript {
+                name = fullName name instance.settings.repository;
+                user = instance.request.user;
+                sudoPreserveEnvs = [
+                  "BORG_REPO"
+                  "BORG_PASSCOMMAND"
+                ];
+                secretsFile = "/run/secrets_borgbackup_env/${fullName name instance.settings.repository}";
+                restoreCmd = ''(cd / && ${pkgs.borgbackup}/bin/borg extract \"$BORG_REPO::$snapshot\")'';
+                listCmd = ''if [ -e \"$BORG_REPO/data\" ]; then borg list --short \"$BORG_REPO\"; fi'';
+              };
+          in
+          flatten (mapAttrsToList mkBorgBackupBinary cfg.instances);
+      }
+      {
+        environment.systemPackages =
+          let
+            mkBorgBackupBinary =
+              name: instance:
+              shb.contracts.backup.mkRestoreScript {
+                name = fullName name instance.settings.repository;
+                user = instance.request.user;
+                sudoPreserveEnvs = [
+                  "BORG_REPO"
+                  "BORG_PASSCOMMAND"
+                ];
+                secretsFile = "/run/secrets_borgbackup_env/${fullName name instance.settings.repository}";
+                restoreCmd = ''${pkgs.borgbackup}/bin/borg extract \"$BORG_REPO::$snapshot\" --stdout | ${instance.request.restoreCmd}'';
+                listCmd = ''if [ -e \"$BORG_REPO/data\" ]; then borg list --short \"$BORG_REPO\"; fi'';
+              };
+          in
+          flatten (mapAttrsToList mkBorgBackupBinary cfg.databases);
+      }
+
+      (lib.mkIf (cfg.enableDashboard && (cfg.instances != { } || cfg.databases != { })) {
+        shb.monitoring.dashboards = [
+          ./backup/dashboard/Backups.json
+        ];
+      })
+    ]
+  );
+}

modules/blocks/borgbackup/docs/default.md

@@ -0,0 +1,302 @@
+# Borgbackup Block {#blocks-borgbackup}
+
+Defined in [`/modules/blocks/borgbackup.nix`](@REPO@/modules/blocks/borgbackup.nix).
+
+This block sets up a backup job using [BorgBackup][].
+It is heavily based on the nixpkgs BorgBackup module.
+
+[borgbackup]: https://www.borgbackup.org/
+
+## Provider Contracts {#blocks-borgbackup-contract-provider}
+
+This block provides the following contracts:
+
+- [backup contract](contracts-backup.html) under the [`shb.borgbackup.instances`][instances] option.
+  It is tested with [contract tests][backup contract tests].
+- [database backup contract](contracts-databasebackup.html) under the [`shb.borgbackup.databases`][databases] option.
+  It is tested with [contract tests][database backup contract tests].
+
+[instances]: #blocks-borgbackup-options-shb.borgbackup.instances
+[databases]: #blocks-borgbackup-options-shb.borgbackup.databases
+[backup contract tests]: @REPO@/test/contracts/backup.nix
+[database backup contract tests]: @REPO@/test/contracts/databasebackup.nix
+
+As requested by those two contracts, when setting up a backup with BorgBackup,
+a backup Systemd service and a [restore script](#blocks-borgbackup-maintenance) are provided.
+
+## Usage {#blocks-borgbackup-usage}
+
+The following examples assume usage of the [sops block][] to provide secrets
+although any blocks providing the [secrets contract][] works too.
+
+[sops block]: ./blocks-sops.html
+[secrets contract]: ./contracts-secrets.html
+
+### One folder backed up manually {#blocks-borgbackup-usage-provider-manual}
+
+The following snippet shows how to configure
+the backup of 1 folder to 1 repository.
+We assume that the folder `/var/lib/myfolder` of the service `myservice` must be backed up.
+
+```nix
+shb.borgbackup.instances."myservice" = {
+  request = {
+    user = "myservice";
+
+    sourceDirectories = [
+      "/var/lib/myfolder"
+    ];
+  };
+
+  settings = {
+    enable = true;
+
+    passphrase.result = config.shb.sops.secret."passphrase".result;
+
+    repository = {
+      path = "/srv/backups/myservice";
+      timerConfig = {
+        OnCalendar = "00:00:00";
+        RandomizedDelaySec = "3h";
+      };
+    };
+
+    retention = {
+      within = "1d";
+      hourly = 24;
+      daily = 7;
+      weekly = 4;
+      monthly = 6;
+    };
+  };
+};
+
+shb.sops.secret."passphrase".request =
+  config.shb.borgbackup.instances."myservice".settings.passphrase.request;
+```
+
+### One folder backed up with contract {#blocks-borgbackup-usage-provider-contract}
+
+With the same example as before but assuming the `myservice` service
+has a `myservice.backup` option that is a requester for the backup contract,
+the snippet above becomes:
+
+```nix
+shb.borgbackup.instances."myservice" = {
+  request = config.myservice.backup;
+
+  settings = {
+    enable = true;
+
+    passphrase.result = config.shb.sops.secret."passphrase".result;
+
+    repository = {
+      path = "/srv/backups/myservice";
+      timerConfig = {
+        OnCalendar = "00:00:00";
+        RandomizedDelaySec = "3h";
+      };
+    };
+
+    retention = {
+      within = "1d";
+      hourly = 24;
+      daily = 7;
+      weekly = 4;
+      monthly = 6;
+    };
+  };
+};
+
+shb.sops.secret."passphrase".request =
+  config.shb.borgbackup.instances."myservice".settings.passphrase.request;
+```
+
+### One folder backed up to S3 {#blocks-borgbackup-usage-provider-remote}
+
+Here we will only highlight the differences with the previous configuration.
+
+This assumes you have access to such a remote S3 store, for example by using [Backblaze](https://www.backblaze.com/).
+
+```diff
+  shb.test.backup.instances.myservice = {
+
+    repository = {
+-     path = "/srv/pool1/backups/myfolder";
++     path = "s3:s3.us-west-000.backblazeb2.com/backups/myfolder";
+      timerConfig = {
+        OnCalendar = "00:00:00";
+        RandomizedDelaySec = "3h";
+      };
+
++     extraSecrets = {
++       AWS_ACCESS_KEY_ID.source="<path/to/access_key_id>";
++       AWS_SECRET_ACCESS_KEY.source="<path/to/secret_access_key>";
++     };
+    };
+  }
+```
+
+### Multiple directories to multiple destinations {#blocks-borgbackup-usage-multiple}
+
+The following snippet shows how to configure backup of any number of folders to 3 repositories,
+each happening at different times to avoid I/O contention.
+
+We will also make sure to be able to re-use as much as the configuration as possible.
+
+A few assumptions:
+- 2 hard drive pools used for backup are mounted respectively on `/srv/pool1` and `/srv/pool2`.
+- You have a backblaze account.
+
+First, let's define a variable to hold all the repositories we want to back up to:
+
+```nix
+repos = [
+  {
+    path = "/srv/pool1/backups";
+    timerConfig = {
+      OnCalendar = "00:00:00";
+      RandomizedDelaySec = "3h";
+    };
+  }
+  {
+    path = "/srv/pool2/backups";
+    timerConfig = {
+      OnCalendar = "08:00:00";
+      RandomizedDelaySec = "3h";
+    };
+  }
+  {
+    path = "s3:s3.us-west-000.backblazeb2.com/backups";
+    timerConfig = {
+      OnCalendar = "16:00:00";
+      RandomizedDelaySec = "3h";
+    };
+  }
+];
+```
+
+Compared to the previous examples, we do not include the name of what we will back up in the
+repository paths.
+
+Now, let's define a function to create a backup configuration. It will take a list of repositories,
+a name identifying the backup and a list of folders to back up.
+
+```nix
+backupcfg = repositories: name: sourceDirectories {
+  enable = true;
+
+  backend = "borgbackup";
+
+  keySopsFile = ../secrets/backup.yaml;
+
+  repositories = builtins.map (r: {
+    path = "${r.path}/${name}";
+    inherit (r) timerConfig;
+  }) repositories;
+
+  inherit sourceDirectories;
+
+  retention = {
+    within = "1d";
+    hourly = 24;
+    daily = 7;
+    weekly = 4;
+    monthly = 6;
+  };
+
+  environmentFile = true;
+};
+```
+
+Now, we can define multiple backup jobs to backup different folders:
+
+```nix
+shb.test.backup.instances.myfolder1 = backupcfg repos ["/var/lib/myfolder1"];
+shb.test.backup.instances.myfolder2 = backupcfg repos ["/var/lib/myfolder2"];
+```
+
+The difference between the above snippet and putting all the folders into one configuration (shown
+below) is the former splits the backups into sub-folders on the repositories.
+
+```nix
+shb.test.backup.instances.all = backupcfg repos ["/var/lib/myfolder1" "/var/lib/myfolder2"];
+```
+
+## Monitoring {#blocks-borgbackup-monitoring}
+
+A generic dashboard for all backup solutions is provided.
+See [Backups Dashboard and Alert](blocks-monitoring.html#blocks-monitoring-backup) section in the monitoring chapter.
+
+## Maintenance {#blocks-borgbackup-maintenance}
+
+### Manual Backup {#blocks-borgbackup-maintenance-manuql}
+
+To launch a backup manually, just run:
+
+```bash
+systemctl start <systemd-service-name>
+```
+
+You can easily discover the systemd service name you need by either listing the units:
+
+```bash
+systemctl list-units 'borgbackup*'
+```
+
+Or by autocompleting the unit name with `<TAB>`:
+
+```bash
+systemctl start borgbackup<TAB><TAB>
+```
+
+Note that the systemd services are of `Type=simple` which means the systemd service
+will not wait for the backup completion to terminate.
+If you want instead to wait for the backup to complete, use the `--wait` flag:
+
+```bash
+systemctl start --wait <systemd-service-name>
+```
+
+### Restore {#blocks-borgbackup-maintenance-restore}
+
+One command-line helper is provided per backup instance and repository pair to automatically supply the needed secrets.
+
+The restore script has all the secrets needed to access the repo,
+it will run `sudo` automatically
+and the user running it needs to have correct permissions for privilege escalation
+
+In the [multiple directories example](#blocks-borgbackup-usage-multiple) above, the following 6 helpers are provided in the `$PATH`:
+
+```bash
+borgbackup-job-myfolder1_srv_pool1_backups
+borgbackup-job-myfolder1_srv_pool2_backups
+borgbackup-job-myfolder1_s3_s3.us-west-000.backblazeb2.com_backups
+borgbackup-job-myfolder2_srv_pool1_backups
+borgbackup-job-myfolder2_srv_pool2_backups
+borgbackup-job-myfolder2_s3_s3.us-west-000.backblazeb2.com_backups
+```
+
+Discovering those is easy thanks to tab-completion.
+
+One can then restore a backup from a given repository with:
+
+```bash
+borgbackup-job-myfolder1_srv_pool1_backups restore latest
+```
+
+### Troubleshooting {#blocks-borgbackup-maintenance-troubleshooting}
+
+In case something bad happens with a backup, the [official documentation](https://borgbackup.readthedocs.io/en/stable/077_troubleshooting.html) has a lot of tips.
+
+## Tests {#blocks-borgbackup-tests}
+
+Specific integration tests are defined in [`/test/blocks/borgbackup.nix`](@REPO@/test/blocks/borgbackup.nix).
+
+## Options Reference {#blocks-borgbackup-options}
+
+```{=include=} options
+id-prefix: blocks-borgbackup-options-
+list-id: selfhostblocks-block-borgbackup-options
+source: @OPTIONS_JSON@
+```

modules/blocks/davfs.nix

@@ -1,80 +1,104 @@
-{ config, pkgs, lib, ... }:
+{ config, lib, ... }:
 
 let
   cfg = config.shb.davfs;
-
-  shblib = pkgs.callPackage ../../lib {};
 in
 {
   options.shb.davfs = {
     mounts = lib.mkOption {
       description = "List of mounts.";
-      default = [];
-      type = lib.types.listOf (lib.types.submodule {
-        options = {
-          remoteUrl = lib.mkOption {
-            type = lib.types.str;
-            description = "Webdav endpoint to connect to.";
-            example = "https://my.domain.com/dav";
-          };
+      default = [ ];
+      type = lib.types.listOf (
+        lib.types.submodule {
+          options = {
+            remoteUrl = lib.mkOption {
+              type = lib.types.str;
+              description = "Webdav endpoint to connect to.";
+              example = "https://my.domain.com/dav";
+            };
 
-          mountPoint = lib.mkOption {
-            type = lib.types.str;
-            description = "Mount point to mount the webdav endpoint on.";
-            example = "/mnt";
-          };
+            mountPoint = lib.mkOption {
+              type = lib.types.str;
+              description = "Mount point to mount the webdav endpoint on.";
+              example = "/mnt";
+            };
 
-          username = lib.mkOption {
-            type = lib.types.str;
-            description = "Username to connect to the webdav endpoint.";
-          };
+            username = lib.mkOption {
+              type = lib.types.str;
+              description = "Username to connect to the webdav endpoint.";
+            };
 
-          passwordFile = lib.mkOption {
-            type = lib.types.str;
-            description = "Password to connect to the webdav endpoint.";
-          };
+            passwordFile = lib.mkOption {
+              type = lib.types.str;
+              description = "Password to connect to the webdav endpoint.";
+            };
 
-          uid = lib.mkOption {
-            type = lib.types.nullOr lib.types.int;
-            description = "User owner of the mount point.";
-            example = 1000;
-            default = null;
-          };
+            uid = lib.mkOption {
+              type = lib.types.nullOr lib.types.int;
+              description = "User owner of the mount point.";
+              example = 1000;
+              default = null;
+            };
 
-          gid = lib.mkOption {
-            type = lib.types.nullOr lib.types.int;
-            description = "Group owner of the mount point.";
-            example = 1000;
-            default = null;
-          };
+            gid = lib.mkOption {
+              type = lib.types.nullOr lib.types.int;
+              description = "Group owner of the mount point.";
+              example = 1000;
+              default = null;
+            };
 
-          fileMode = lib.mkOption {
-            type = lib.types.nullOr lib.types.str;
-            description = "File creation mode";
-            example = "0664";
-            default = null;
-          };
+            fileMode = lib.mkOption {
+              type = lib.types.nullOr lib.types.str;
+              description = "File creation mode";
+              example = "0664";
+              default = null;
+            };
 
-          directoryMode = lib.mkOption {
-            type = lib.types.nullOr lib.types.str;
-            description = "Directory creation mode";
-            example = "2775";
-            default = null;
-          };
+            directoryMode = lib.mkOption {
+              type = lib.types.nullOr lib.types.str;
+              description = "Directory creation mode";
+              example = "2775";
+              default = null;
+            };
 
-          automount = lib.mkOption {
-            type = lib.types.bool;
-            description = "Create a systemd automount unit";
-            default = true;
+            automount = lib.mkOption {
+              type = lib.types.bool;
+              description = "Create a systemd automount unit";
+              default = true;
+            };
           };
-        };
-      });
+        }
+      );
     };
   };
 
   config = {
     services.davfs2.enable = builtins.length cfg.mounts > 0;
 
+    systemd.services = lib.optionalAttrs (builtins.length cfg.mounts > 0) {
+      davfs2-password-generation = {
+        wantedBy = [ "multi-user.target" ];
+        serviceConfig.Type = "oneshot";
+        script = ''
+          mkdir -p /etc/davfs2
+          [ -f /etc/davfs2/secrets ] && mv /etc/davfs2/secrets /etc/davfs2/secrets.prev
+          touch /etc/davfs2/secrets
+          chown root: /etc/davfs2
+          chown root: /etc/davfs2/secrets
+          chmod 700 /etc/davfs2
+          chmod 600 /etc/davfs2/secrets
+        ''
+        + (
+          let
+            mkPasswordCmd = cfg': ''
+              printf "%s %s %s\n" "${cfg'.mountPoint}" "${cfg'.username}" "$(cat ${cfg'.passwordFile})" >> /etc/davfs2/secrets
+            '';
+          in
+          lib.concatStringsSep "\n" (map mkPasswordCmd cfg.mounts)
+        );
+      };
+    };
+
     systemd.mounts =
       let
         mkMountCfg = c: {
@@ -82,12 +106,13 @@ in
           description = "Webdav mount point";
           after = [ "network-online.target" ];
           wants = [ "network-online.target" ];
+          wantedBy = [ "multi-user.target" ];
 
           what = c.remoteUrl;
           where = c.mountPoint;
           options = lib.concatStringsSep "," (
             (lib.optional (!(isNull c.uid)) "uid=${toString c.uid}")
-            ++ (lib.optional (!(isNull c.gid)) "gid=${toString c.uid}")
+            ++ (lib.optional (!(isNull c.gid)) "gid=${toString c.gid}")
             ++ (lib.optional (!(isNull c.fileMode)) "file_mode=${toString c.fileMode}")
             ++ (lib.optional (!(isNull c.directoryMode)) "dir_mode=${toString c.directoryMode}")
           );
@@ -95,6 +120,6 @@ in
           mountConfig.TimeoutSec = 15;
         };
       in
-        map mkMountCfg cfg.mounts;
+      map mkMountCfg cfg.mounts;
   };
 }

modules/blocks/hardcodedsecret.nix

@@ -0,0 +1,105 @@
+{
+  config,
+  lib,
+  pkgs,
+  shb,
+  ...
+}:
+let
+  cfg = config.shb.hardcodedsecret;
+
+  inherit (lib) mapAttrs' mkOption nameValuePair;
+  inherit (lib.types)
+    attrsOf
+    nullOr
+    str
+    submodule
+    ;
+  inherit (pkgs) writeText;
+in
+{
+  imports = [
+    ../../lib/module.nix
+  ];
+
+  options.shb.hardcodedsecret = mkOption {
+    default = { };
+    description = ''
+      Hardcoded secrets. These should only be used in tests.
+    '';
+    example = lib.literalExpression ''
+      {
+        mySecret = {
+          request = {
+            user = "me";
+            mode = "0400";
+            restartUnits = [ "myservice.service" ];
+          };
+          settings.content = "My Secret";
+        };
+      }
+    '';
+    type = attrsOf (
+      submodule (
+        { name, ... }:
+        {
+          options = shb.contracts.secret.mkProvider {
+            settings = mkOption {
+              description = ''
+                Settings specific to the hardcoded secret module.
+
+                Give either `content` or `source`.
+              '';
+
+              type = submodule {
+                options = {
+                  content = mkOption {
+                    type = nullOr str;
+                    description = ''
+                      Content of the secret as a string.
+
+                      This will be stored in the nix store and should only be used for testing or maybe in dev.
+                    '';
+                    default = null;
+                  };
+
+                  source = mkOption {
+                    type = nullOr str;
+                    description = ''
+                      Source of the content of the secret as a path in the nix store.
+                    '';
+                    default = null;
+                  };
+                };
+              };
+            };
+
+            resultCfg = {
+              path = "/run/hardcodedsecrets/hardcodedsecret_${name}";
+            };
+          };
+        }
+      )
+    );
+  };
+
+  config = {
+    system.activationScripts = mapAttrs' (
+      n: cfg':
+      let
+        source =
+          if cfg'.settings.source != null then
+            cfg'.settings.source
+          else
+            writeText "hardcodedsecret_${n}_content" cfg'.settings.content;
+      in
+      nameValuePair "hardcodedsecret_${n}" ''
+        mkdir -p "$(dirname "${cfg'.result.path}")"
+        touch "${cfg'.result.path}"
+        chmod ${cfg'.request.mode} "${cfg'.result.path}"
+        chown ${cfg'.request.owner}:${cfg'.request.group} "${cfg'.result.path}"
+        cp ${source} "${cfg'.result.path}"
+      ''
+    ) cfg;
+  };
+}

modules/blocks/ldap.nix

@@ -1,135 +0,0 @@
-{ config, pkgs, lib, ... }:
-
-let
-  cfg = config.shb.ldap;
-
-  contracts = pkgs.callPackage ../contracts {};
-
-  fqdn = "${cfg.subdomain}.${cfg.domain}";
-in
-{
-  options.shb.ldap = {
-    enable = lib.mkEnableOption "the LDAP service";
-
-    dcdomain = lib.mkOption {
-      type = lib.types.str;
-      description = "dc domain to serve.";
-      example = "dc=mydomain,dc=com";
-    };
-
-    subdomain = lib.mkOption {
-      type = lib.types.str;
-      description = "Subdomain under which the LDAP service will be served.";
-      example = "grafana";
-    };
-
-    domain = lib.mkOption {
-      type = lib.types.str;
-      description = "Domain under which the LDAP service will be served.";
-      example = "mydomain.com";
-    };
-
-    ldapPort = lib.mkOption {
-      type = lib.types.port;
-      description = "Port on which the server listens for the LDAP protocol.";
-      default = 3890;
-    };
-
-    ssl = lib.mkOption {
-      description = "Path to SSL files";
-      type = lib.types.nullOr contracts.ssl.certs;
-      default = null;
-    };
-
-    webUIListenPort = lib.mkOption {
-      type = lib.types.port;
-      description = "Port on which the web UI is exposed.";
-      default = 17170;
-    };
-
-    ldapUserPasswordFile = lib.mkOption {
-      type = lib.types.path;
-      description = "File containing the LDAP admin user password.";
-    };
-
-    jwtSecretFile = lib.mkOption {
-      type = lib.types.path;
-      description = "File containing the JWT secret.";
-    };
-
-    restrictAccessIPRange = lib.mkOption {
-      type = lib.types.nullOr lib.types.str;
-      description = "Set a local network range to restrict access to the UI to only those IPs.";
-      example = "192.168.1.1/24";
-      default = null;
-    };
-
-    debug = lib.mkOption {
-      description = "Enable debug logging.";
-      type = lib.types.bool;
-      default = false;
-    };
-  };
-
-  
-  config = lib.mkIf cfg.enable {
-    services.nginx = {
-      enable = true;
-
-      virtualHosts.${fqdn} = {
-        forceSSL = !(isNull cfg.ssl);
-        sslCertificate = lib.mkIf (!(isNull cfg.ssl)) cfg.ssl.paths.cert;
-        sslCertificateKey = lib.mkIf (!(isNull cfg.ssl)) cfg.ssl.paths.key;
-        locations."/" = {
-          extraConfig = ''
-            proxy_set_header Host $host;
-          '' + (if isNull cfg.restrictAccessIPRange then "" else ''
-            allow ${cfg.restrictAccessIPRange};
-            deny all;
-          '');
-          proxyPass = "http://${toString config.services.lldap.settings.http_host}:${toString config.services.lldap.settings.http_port}/";
-        };
-      };
-    };
-
-    users.users.lldap = {
-      name = "lldap";
-      group = "lldap";
-      isSystemUser = true;
-    };
-
-    users.groups.lldap = {
-      members = [ "backup" ];
-    };
-
-    services.lldap = {
-      enable = true;
-
-      environment = {
-        LLDAP_JWT_SECRET_FILE = toString cfg.jwtSecretFile;
-        LLDAP_LDAP_USER_PASS_FILE = toString cfg.ldapUserPasswordFile;
-
-        RUST_LOG = lib.mkIf cfg.debug "debug";
-      };
-
-      settings = {
-        http_url = "https://${fqdn}";
-        http_host = "127.0.0.1";
-        http_port = cfg.webUIListenPort;
-
-        ldap_host = "127.0.0.1";
-        ldap_port = cfg.ldapPort;
-
-        ldap_base_dn = cfg.dcdomain;
-
-        verbose = cfg.debug;
-      };
-    };
-
-    shb.backup.instances.lldap = {
-      sourceDirectories = [
-        "/var/lib/lldap"
-      ];
-    };
-  };
-}

modules/blocks/lldap.nix

@@ -0,0 +1,467 @@
+{
+  config,
+  pkgs,
+  lib,
+  shb,
+  ...
+}:
+
+let
+  cfg = config.shb.lldap;
+
+  fqdn = "${cfg.subdomain}.${cfg.domain}";
+
+  inherit (lib) mkOption types;
+
+  ensureFormat = pkgs.formats.json { };
+
+  ensureFieldsOptions = name: {
+    name = mkOption {
+      type = types.str;
+      description = "Name of the field.";
+      default = name;
+    };
+
+    attributeType = mkOption {
+      type = types.enum [
+        "STRING"
+        "INTEGER"
+        "JPEG"
+        "DATE_TIME"
+      ];
+      description = "Attribute type.";
+    };
+
+    isEditable = mkOption {
+      type = types.bool;
+      description = "Is field editable.";
+      default = true;
+    };
+
+    isList = mkOption {
+      type = types.bool;
+      description = "Is field a list.";
+      default = false;
+    };
+
+    isVisible = mkOption {
+      type = types.bool;
+      description = "Is field visible in UI.";
+      default = true;
+    };
+  };
+in
+{
+  imports = [
+    ../../lib/module.nix
+    ./mitmdump.nix
+
+    (lib.mkRenamedOptionModule [ "shb" "ldap" ] [ "shb" "lldap" ])
+  ];
+
+  options.shb.lldap = {
+    enable = lib.mkEnableOption "the LDAP service";
+
+    dcdomain = lib.mkOption {
+      type = lib.types.str;
+      description = "dc domain to serve.";
+      example = "dc=mydomain,dc=com";
+    };
+
+    subdomain = lib.mkOption {
+      type = lib.types.str;
+      description = "Subdomain under which the LDAP service will be served.";
+      example = "grafana";
+    };
+
+    domain = lib.mkOption {
+      type = lib.types.str;
+      description = "Domain under which the LDAP service will be served.";
+      example = "mydomain.com";
+    };
+
+    ldapPort = lib.mkOption {
+      type = lib.types.port;
+      description = "Port on which the server listens for the LDAP protocol.";
+      default = 3890;
+    };
+
+    ssl = lib.mkOption {
+      description = "Path to SSL files";
+      type = lib.types.nullOr shb.contracts.ssl.certs;
+      default = null;
+    };
+
+    webUIListenPort = lib.mkOption {
+      type = lib.types.port;
+      description = "Port on which the web UI is exposed.";
+      default = 17170;
+    };
+
+    ldapUserPassword = lib.mkOption {
+      description = "LDAP admin user secret. Must be >= 8 characters.";
+      type = lib.types.submodule {
+        options = shb.contracts.secret.mkRequester {
+          mode = "0440";
+          owner = "lldap";
+          group = "lldap";
+          restartUnits = [ "lldap.service" ];
+        };
+      };
+    };
+
+    jwtSecret = lib.mkOption {
+      description = "JWT secret.";
+      type = lib.types.submodule {
+        options = shb.contracts.secret.mkRequester {
+          mode = "0440";
+          owner = "lldap";
+          group = "lldap";
+          restartUnits = [ "lldap.service" ];
+        };
+      };
+    };
+
+    restrictAccessIPRange = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      description = "Set a local network range to restrict access to the UI to only those IPs.";
+      example = "192.168.1.1/24";
+      default = null;
+    };
+
+    debug = lib.mkOption {
+      description = "Enable debug logging.";
+      type = lib.types.bool;
+      default = false;
+    };
+
+    mount = lib.mkOption {
+      type = shb.contracts.mount;
+      description = ''
+        Mount configuration. This is an output option.
+
+        Use it to initialize a block implementing the "mount" contract.
+        For example, with a zfs dataset:
+
+        ```
+        shb.zfs.datasets."ldap" = {
+          poolName = "root";
+        } // config.shb.lldap.mount;
+        ```
+      '';
+      readOnly = true;
+      default = {
+        path = "/var/lib/lldap";
+      };
+    };
+
+    backup = lib.mkOption {
+      description = ''
+        Backup configuration.
+      '';
+      type = lib.types.submodule {
+        options = shb.contracts.backup.mkRequester {
+          # TODO: is there a workaround that avoid needing to use root?
+          # root because otherwise we cannot access the private StateDiretory
+          user = "root";
+          # /private because the systemd service uses DynamicUser=true
+          sourceDirectories = [
+            "/var/lib/private/lldap"
+          ];
+        };
+      };
+    };
+
+    ensureUsers = mkOption {
+      description = ''
+        Create the users defined here on service startup.
+
+        If `enforceUsers` option is `true`, the groups
+        users belong to must be present in the `ensureGroups` option.
+
+        Non-default options must be added to the `ensureGroupFields` option.
+      '';
+      default = { };
+      type = types.attrsOf (
+        types.submodule (
+          { name, ... }:
+          {
+            freeformType = ensureFormat.type;
+
+            options = {
+              id = mkOption {
+                type = types.str;
+                description = "Username.";
+                default = name;
+              };
+
+              email = mkOption {
+                type = types.str;
+                description = "Email.";
+              };
+
+              password = mkOption {
+                description = "Password.";
+                type = lib.types.submodule {
+                  options = shb.contracts.secret.mkRequester {
+                    mode = "0440";
+                    owner = "lldap";
+                    group = "lldap";
+                    restartUnits = [ "lldap.service" ];
+                  };
+                };
+              };
+
+              displayName = mkOption {
+                type = types.nullOr types.str;
+                default = null;
+                description = "Display name.";
+              };
+
+              firstName = mkOption {
+                type = types.nullOr types.str;
+                default = null;
+                description = "First name.";
+              };
+
+              lastName = mkOption {
+                type = types.nullOr types.str;
+                default = null;
+                description = "Last name.";
+              };
+
+              avatar_file = mkOption {
+                type = types.nullOr types.str;
+                default = null;
+                description = "Avatar file. Must be a valid path to jpeg file (ignored if avatar_url specified)";
+              };
+
+              avatar_url = mkOption {
+                type = types.nullOr types.str;
+                default = null;
+                description = "Avatar url. must be a valid URL to jpeg file (ignored if gravatar_avatar specified)";
+              };
+
+              gravatar_avatar = mkOption {
+                type = types.nullOr types.str;
+                default = null;
+                description = "Get avatar from Gravatar using the email.";
+              };
+
+              weser_avatar = mkOption {
+                type = types.nullOr types.str;
+                default = null;
+                description = "Convert avatar retrieved by gravatar or the URL.";
+              };
+
+              groups = mkOption {
+                type = types.listOf types.str;
+                default = [ ];
+                description = "Groups the user would be a member of (all the groups must be specified in group config files).";
+              };
+            };
+          }
+        )
+      );
+    };
+
+    ensureGroups = mkOption {
+      description = ''
+        Create the groups defined here on service startup.
+
+        Non-default options must be added to the `ensureGroupFields` option.
+      '';
+      default = { };
+      type = types.attrsOf (
+        types.submodule (
+          { name, ... }:
+          {
+            freeformType = ensureFormat.type;
+
+            options = {
+              name = mkOption {
+                type = types.str;
+                description = "Name of the group.";
+                default = name;
+              };
+            };
+          }
+        )
+      );
+    };
+
+    ensureUserFields = mkOption {
+      description = "Extra fields for users";
+      default = { };
+      type = types.attrsOf (
+        types.submodule (
+          { name, ... }:
+          {
+            options = ensureFieldsOptions name;
+          }
+        )
+      );
+    };
+
+    ensureGroupFields = mkOption {
+      description = "Extra fields for groups";
+      default = { };
+      type = types.attrsOf (
+        types.submodule (
+          { name, ... }:
+          {
+            options = ensureFieldsOptions name;
+          }
+        )
+      );
+    };
+
+    enforceUsers = mkOption {
+      description = "Delete users not set declaratively.";
+      type = types.bool;
+      default = false;
+    };
+
+    enforceUserMemberships = mkOption {
+      description = "Remove users from groups not set declaratively.";
+      type = types.bool;
+      default = false;
+    };
+
+    enforceGroups = mkOption {
+      description = "Remove groups not set declaratively.";
+      type = types.bool;
+      default = false;
+    };
+
+    dashboard = lib.mkOption {
+      description = ''
+        Dashboard contract consumer
+      '';
+      default = { };
+      type = lib.types.submodule {
+        options = shb.contracts.dashboard.mkRequester {
+          externalUrl = "https://${cfg.subdomain}.${cfg.domain}";
+          externalUrlText = "https://\${config.shb.lldap.subdomain}.\${config.shb.lldap.domain}";
+          internalUrl = "http://127.0.0.1:${toString cfg.webUIListenPort}";
+        };
+      };
+    };
+  };
+
+  config = lib.mkIf cfg.enable {
+
+    services.nginx = {
+      enable = true;
+
+      virtualHosts.${fqdn} = {
+        forceSSL = !(isNull cfg.ssl);
+        sslCertificate = lib.mkIf (!(isNull cfg.ssl)) cfg.ssl.paths.cert;
+        sslCertificateKey = lib.mkIf (!(isNull cfg.ssl)) cfg.ssl.paths.key;
+        locations."/" = {
+          extraConfig = ''
+            proxy_set_header Host $host;
+          ''
+          + (
+            if isNull cfg.restrictAccessIPRange then
+              ""
+            else
+              ''
+                allow ${cfg.restrictAccessIPRange};
+                deny all;
+              ''
+          );
+          proxyPass = "http://${toString config.services.lldap.settings.http_host}:${toString config.shb.lldap.webUIListenPort}/";
+        };
+      };
+    };
+
+    users.users.lldap = {
+      name = "lldap";
+      group = "lldap";
+      isSystemUser = true;
+    };
+    users.groups.lldap = { };
+
+    services.lldap = {
+      enable = true;
+
+      inherit (cfg) enforceUsers enforceUserMemberships enforceGroups;
+
+      environment = {
+        RUST_LOG = lib.mkIf cfg.debug "debug";
+      };
+
+      settings = {
+        http_url = "https://${fqdn}";
+        http_host = "127.0.0.1";
+        http_port = if !cfg.debug then cfg.webUIListenPort else cfg.webUIListenPort + 1;
+
+        ldap_host = "127.0.0.1";
+        ldap_port = cfg.ldapPort; # Would be great to be able to inspect this but it requires tcpdump instead of mitmproxy.
+        ldap_base_dn = cfg.dcdomain;
+
+        ldap_user_pass_file = toString cfg.ldapUserPassword.result.path;
+        force_ldap_user_pass_reset = "always";
+        jwt_secret_file = toString cfg.jwtSecret.result.path;
+
+        verbose = cfg.debug;
+      };
+
+      inherit (cfg) ensureGroups ensureUserFields ensureGroupFields;
+      ensureUsers = lib.mapAttrs (
+        n: v:
+        (lib.removeAttrs v [ "password" ])
+        // {
+          "password_file" = toString v.password.result.path;
+        }
+      ) cfg.ensureUsers;
+    };
+
+    # Harden lldap following https://github.com/NixOS/nixpkgs/pull/487933
+    systemd.services.lldap.serviceConfig = {
+      RemoveIPC = true;
+      RestrictNamespaces = true;
+      RestrictRealtime = true;
+      RestrictSUIDSGID = true;
+      RestrictAddressFamilies = [
+        "AF_UNIX"
+        "AF_INET"
+        "AF_INET6"
+      ];
+      SystemCallFilter = [
+        "@system-service"
+        "~@privileged"
+        "~@resources"
+      ];
+      SystemCallArchitectures = "native";
+      CapabilityBoundingSet = "";
+      LockPersonality = true;
+      NoNewPrivileges = true;
+      PrivateTmp = true;
+      PrivateDevices = true;
+      ProtectClock = true;
+      ProtectControlGroups = true;
+      ProtectHome = true;
+      ProtectHostname = true;
+      ProtectKernelLogs = true;
+      ProtectKernelModules = true;
+      ProtectKernelTunables = true;
+      ProtectSystem = "strict";
+      ProtectProc = "invisible";
+      ProcSubset = "pid";
+      MemoryDenyWriteExecute = true;
+    };
+
+    shb.mitmdump.instances."lldap-web" = lib.mkIf cfg.debug {
+      listenPort = config.shb.lldap.webUIListenPort;
+      upstreamPort = config.shb.lldap.webUIListenPort + 1;
+      after = [ "lldap.service" ];
+      enabledAddons = [ config.shb.mitmdump.addons.logger ];
+      extraArgs = [
+        "--set"
+        "verbose_pattern=/api"
+      ];
+    };
+  };
+}

modules/blocks/lldap/docs/default.md

@@ -0,0 +1,212 @@
+# LLDAP Block {#blocks-lldap}
+
+Defined in [`/modules/blocks/lldap.nix`](@REPO@/modules/blocks/lldap.nix).
+
+This block sets up an [LLDAP][] service for user and group management
+across services.
+
+[LLDAP]: https://github.com/lldap/lldap
+
+## Features {#blocks-lldap-features}
+
+- Integration with the [dashboard contract](contracts-dashboard.html) for displaying user facing application in a dashboard. [Manual](#blocks-lldap-usage-applicationdashboard)
+
+## Usage {#blocks-lldap-usage}
+
+### Initial Configuration {#blocks-lldap-usage-configuration}
+
+```nix
+shb.lldap = {
+  enable = true;
+  subdomain = "ldap";
+  domain = "example.com";
+  dcdomain = "dc=example,dc=com";
+
+  ldapPort = 3890;
+  webUIListenPort = 17170;
+
+  jwtSecret.result = config.shb.sops.secret."lldap/jwt_secret".result;
+  ldapUserPassword.result = config.shb.sops.secret."lldap/user_password".result;
+};
+shb.sops.secret."lldap/jwt_secret".request = config.shb.lldap.jwtSecret.request;
+shb.sops.secret."lldap/user_password".request = config.shb.lldap.ldapUserPassword.request;
+```
+
+This assumes secrets are setup with SOPS
+as mentioned in [the secrets setup section](usage.html#usage-secrets) of the manual.
+
+### SSL {#blocks-lldap-usage-ssl}
+
+Using SSL is an important security practice, like always.
+Using the [SSL block][], the configuration to add to the one above is:
+
+[SSL block]: blocks-ssl.html
+
+```nix
+shb.certs.certs.letsencrypt.${domain}.extraDomains = [
+  "${config.shb.lldap.subdomain}.${config.shb.lldap.domain}"
+];
+
+shb.lldap.ssl = config.shb.certs.certs.letsencrypt.${config.shb.lldap.domain};
+```
+
+### Restrict Access By IP {#blocks-lldap-usage-restrict-access-by-ip}
+
+For added security, you can restrict access to the LLDAP UI
+by adding the following line:
+
+```nix
+shb.lldap.restrictAccessIPRange = "192.168.50.0/24";
+```
+
+### Application Dashboard {#blocks-lldap-usage-applicationdashboard}
+
+Integration with the [dashboard contract](contracts-dashboard.html) is provided
+by the [dashboard option](#blocks-lldap-options-shb.lldap.dashboard).
+
+For example using the [Homepage](services-homepage.html) service:
+
+```nix
+{
+  shb.homepage.servicesGroups.Admin.services.LLDAP = {
+    sortOrder = 2;
+    dashboard.request = config.shb.lldap.dashboard.request;
+  };
+}
+```
+
+## Manage Groups {#blocks-lldap-manage-groups}
+
+The following snippet will create group named "family" if it does not exist yet.
+Also, all other groups will be deleted and only the "family" group will remain.
+
+Note that the `lldap_admin`, `lldap_password_manager` and `lldap_strict_readonly` groups, which are internal to LLDAP, will always exist.
+
+If you want existing groups not declared in the `shb.lldap.ensureGroups` to be deleted,
+set [`shb.lldap.enforceGroups`](#blocks-lldap-options-shb.lldap.enforceGroups) to `true`.
+
+```nix
+{
+  shb.lldap.ensureGroups = {
+   family = {};
+  };
+}
+```
+
+Changing the configuration to the following will add a new group "friends":
+
+```nix
+{
+  shb.lldap.ensureGroups = {
+    family = {};
+    friends = {};
+  };
+}
+```
+
+Switching back the configuration to the previous one will delete the group "friends":
+
+```nix
+{
+  shb.lldap.ensureGroups = {
+    family = {};
+  };
+}
+```
+
+Custom fields can be added to groups as long as they are added to the `ensureGroupFields` field:
+
+```nix
+shb.lldap = {
+  ensureGroupFields = {
+    mygroupattribute = {
+      attributeType = "STRING";
+    };
+  };
+
+  ensureGroups = {
+    family = {
+      mygroupattribute = "Managed by NixOS";
+    };
+  };
+};
+```
+
+## Manage Users {#blocks-lldap-manage-users}
+
+The following snippet creates a user and makes it a member of the "family" group.
+
+Note the following behavior:
+
+- New users will be created following the `shb.lldap.ensureUsers` option.
+- Existing users will be updated, their password included, if they are mentioned in the `shb.lldap.ensureUsers` option.
+- Existing users not declared in the `shb.lldap.ensureUsers` will be left as-is.
+- User memberships to groups not declared in their respective `shb.lldap.ensureUsers.<name>.groups`.
+
+If you want existing users not declared in the `shb.lldap.ensureUsers` to be deleted,
+set [`shb.lldap.enforceUsers`](#blocks-lldap-options-shb.lldap.enforceUsers) to `true`.
+
+If you want memberships to groups not declared in the respective
+`shb.lldap.ensureUsers.<name>.groups` option to be deleted,
+set [`shb.lldap.enforceUserMemberships`](#blocks-lldap-options-shb.lldap.enforceUserMemberships) `true`.
+
+```nix
+{
+  shb.lldap.ensureUsers = {
+    dad = {
+      email = "dad@example.com";
+      displayName = "Dad";
+      firstName = "First Name";
+      lastName = "Last Name";
+      groups = [ "family" ];
+      password.result = config.shb.sops.secret."dad".result;
+    };
+  };
+
+  shb.sops.secret."dad".request =
+    config.shb.lldap.ensureUsers.dad.password.request;
+}
+```
+
+The password field assumes usage of the [sops block][] to provide secrets
+although any blocks providing the [secrets contract][] works too.
+
+[sops block]: blocks-sops.html
+[secrets contract]: contracts-secrets.html
+
+The user is still editable through the UI.
+That being said, any change will be overwritten next time the configuration is applied.
+
+## Troubleshooting {#blocks-lldap-troubleshooting}
+
+To increase logging verbosity and see the trace of the GraphQL queries, add:
+
+```nix
+shb.lldap.debug = true;
+```
+
+Note that verbosity is truly verbose here
+so you will want to revert this at some point.
+
+To see the logs, then run `journalctl -u lldap.service`.
+
+Setting the `debug` option to `true` will also
+add an [shb.mitmdump][] instance in front of the LLDAP [web UI port](#blocks-lldap-options-shb.lldap.webUIListenPort)
+which prints all requests and responses headers and body
+to the systemd service `mitmdump-lldap.service`. Note the you won't
+see the query done using something like `ldapsearch` since those
+go through the [`LDAP` port](#blocks-lldap-options-shb.lldap.ldapPort).
+
+[shb.mitmdump]: ./blocks-mitmdump.html
+
+## Tests {#blocks-lldap-tests}
+
+Specific integration tests are defined in [`/test/blocks/lldap.nix`](@REPO@/test/blocks/lldap.nix).
+
+## Options Reference {#blocks-lldap-options}
+
+```{=include=} options
+id-prefix: blocks-lldap-options-
+list-id: selfhostblocks-block-lldap-options
+source: @OPTIONS_JSON@
+```

modules/blocks/mitmdump.nix

@@ -0,0 +1,307 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+let
+  inherit (lib)
+    mapAttrs'
+    mkOption
+    nameValuePair
+    types
+    ;
+  inherit (types)
+    attrsOf
+    listOf
+    port
+    submodule
+    str
+    ;
+
+  cfg = config.shb.mitmdump;
+
+  mitmdumpScript =
+    pkgs.writers.writePython3Bin "mitmdump"
+      {
+        libraries =
+          let
+            p = pkgs.python3Packages;
+          in
+          [
+            p.systemd-python
+            p.mitmproxy
+          ];
+        flakeIgnore = [ "E501" ];
+      }
+      ''
+        from systemd.daemon import notify
+        import argparse
+        import logging
+        import os
+        import subprocess
+        import socket
+        import sys
+        import time
+
+
+        logging.basicConfig(level=logging.INFO, format='%(message)s')
+
+
+        def wait_for_port(host, port, timeout):
+            deadline = time.time() + timeout
+            while time.time() < deadline:
+                try:
+                    with socket.create_connection((host, port), timeout=0.5):
+                        return True
+                except Exception:
+                    time.sleep(0.1)
+            return False
+
+
+        def flatten(xss):
+            return [x for xs in xss for x in xs]
+
+
+        parser = argparse.ArgumentParser()
+        parser.add_argument("--listen_host", default="127.0.0.1", help="Host mitmdump will listen on")
+        parser.add_argument("--listen_port", required=True, help="Port mitmdump will listen on")
+        parser.add_argument("--upstream_host", default="http://127.0.0.1", help="Host mitmdump will connect to for upstream. Example: http://127.0.0.1 or https://otherhost")
+        parser.add_argument("--upstream_port", required=True, help="Port mitmdump will connect to for upstream")
+        parser.add_argument("--timeout", required=False, type=int, default=10, help="Timeout to wait for port availability")
+        args, rest = parser.parse_known_args()
+
+        MITMDUMP_BIN = os.environ.get("MITMDUMP_BIN")
+        if MITMDUMP_BIN is None:
+            raise Exception("MITMDUMP_BIN env var must be set to the path of the mitmdump binary")
+
+        logging.info(f"Waiting for upstream address '{args.upstream_host}:{args.upstream_port}' to be up.")
+        wait_for_port(args.upstream_host, args.upstream_port, timeout=args.timeout)
+        logging.info(f"Upstream address '{args.upstream_host}:{args.upstream_port}' is up.")
+
+        proc = subprocess.Popen(
+            [
+                MITMDUMP_BIN,
+                "--listen-host", args.listen_host,
+                "-p", args.listen_port,
+                "--mode", f"reverse:{args.upstream_host}:{args.upstream_port}",
+            ] + rest,
+            stdout=sys.stdout,
+            stderr=sys.stderr,
+        )
+
+        logging.info(f"Waiting for mitmdump instance to start on port {args.listen_port}.")
+        if wait_for_port("127.0.0.1", args.listen_port, timeout=args.timeout):
+            logging.info(f"Mitmdump is started on port {args.listen_port}.")
+            notify("READY=1")
+        else:
+            print(f"Mitmdump instance did not start before the timeout of {args.timeout} seconds, consider increasing the timeout")
+            proc.terminate()
+            exit(1)
+
+        proc.wait()
+      '';
+
+  logger = toString (
+    pkgs.writers.writeText "loggerAddon.py" ''
+      import logging
+      from collections.abc import Sequence
+      from mitmproxy import ctx, http
+      import re
+
+
+      logger = logging.getLogger(__name__)
+
+
+      class RegexLogger:
+          def __init__(self):
+              self.verbose_patterns = None
+
+          def load(self, loader):
+              loader.add_option(
+                  name="verbose_pattern",
+                  typespec=Sequence[str],
+                  default=[],
+                  help="Regex patterns for verbose logging",
+              )
+
+          def response(self, flow: http.HTTPFlow):
+              if self.verbose_patterns is None:
+                  self.verbose_patterns = [re.compile(p) for p in ctx.options.verbose_pattern]
+
+              matched = any(p.search(flow.request.path) for p in self.verbose_patterns)
+              if matched:
+                  logger.info(format_flow(flow))
+
+
+      def format_flow(flow: http.HTTPFlow) -> str:
+          return (
+              "\n"
+              "RequestHeaders:\n"
+              f"    {format_headers(flow.request.headers.items())}\n"
+              f"RequestBody:     {flow.request.get_text()}\n"
+              f"Status:          {flow.response.data.status_code}\n"
+              "ResponseHeaders:\n"
+              f"    {format_headers(flow.response.headers.items())}\n"
+              f"ResponseBody:    {flow.response.get_text()}\n"
+          )
+
+
+      def format_headers(headers) -> str:
+          return "\n    ".join(k + ": " + v for k, v in headers)
+
+
+      addons = [RegexLogger()]
+    ''
+  );
+in
+{
+  options.shb.mitmdump = {
+    addons = mkOption {
+      type = attrsOf str;
+      default = [ ];
+      description = ''
+        Addons available to the be added to the mitmdump instance.
+
+        To enabled them, add them to the `enabledAddons` option.
+      '';
+    };
+
+    instances = mkOption {
+      default = { };
+      description = "Mitmdump instance.";
+      type = attrsOf (
+        submodule (
+          { name, ... }:
+          {
+            options = {
+              package = lib.mkPackageOption pkgs "mitmproxy" { };
+
+              serviceName = mkOption {
+                type = str;
+                description = ''
+                  Name of the mitmdump system service.
+                '';
+                default = "mitmdump-${name}.service";
+                readOnly = true;
+              };
+
+              listenHost = mkOption {
+                type = str;
+                default = "127.0.0.1";
+                description = ''
+                  Host the mitmdump instance will connect on.
+                '';
+              };
+
+              listenPort = mkOption {
+                type = port;
+                description = ''
+                  Port the mitmdump instance will listen on.
+
+                  The upstream port from the client's perspective.
+                '';
+              };
+
+              upstreamHost = mkOption {
+                type = str;
+                default = "http://127.0.0.1";
+                description = ''
+                  Host the mitmdump instance will connect to.
+
+                  If only an IP or domain is provided,
+                  mitmdump will default to connect using HTTPS.
+                  If this is not wanted, prefix the IP or domain with the 'http://' protocol.
+                '';
+              };
+
+              upstreamPort = mkOption {
+                type = port;
+                description = ''
+                  Port the mitmdump instance will connect to.
+
+                  The port the server is listening on.
+                '';
+              };
+
+              timeout = mkOption {
+                type = types.int;
+                default = 30;
+                description = ''
+                  Time to wait for upstream to start and mitmdump to start.
+                '';
+              };
+
+              after = mkOption {
+                type = listOf str;
+                default = [ ];
+                description = ''
+                  Systemd services that must be started before this mitmdump proxy instance.
+
+                  You are guaranteed the mitmdump is listening on the `listenPort`
+                  when its systemd service has started.
+                '';
+              };
+
+              enabledAddons = mkOption {
+                type = listOf str;
+                default = [ ];
+                description = ''
+                  Addons to enable on this mitmdump instance.
+                '';
+                example = lib.literalExpression "[ config.shb.mitmdump.addons.logger ]";
+              };
+
+              extraArgs = mkOption {
+                type = listOf str;
+                default = [ ];
+                description = ''
+                  Extra arguments to pass to the mitmdump instance.
+
+                  See upstream [manual](https://docs.mitmproxy.org/stable/concepts/options/#flow_detail) for all possible options.
+                '';
+                example = lib.literalExpression ''[ "--set" "verbose_pattern=/api" ]'';
+              };
+            };
+          }
+        )
+      );
+    };
+  };
+
+  config = {
+    systemd.services = mapAttrs' (
+      name: cfg':
+      nameValuePair "mitmdump-${name}" {
+        environment = {
+          "HOME" = "/var/lib/private/mitmdump-${name}";
+          "MITMDUMP_BIN" = "${cfg'.package}/bin/mitmdump";
+        };
+        serviceConfig = {
+          Type = "notify";
+          Restart = "on-failure";
+          StandardOutput = "journal";
+          StandardError = "journal";
+
+          DynamicUser = true;
+          WorkingDirectory = "/var/lib/mitmdump-${name}";
+          StateDirectory = "mitmdump-${name}";
+
+          ExecStart =
+            let
+              addons = lib.concatMapStringsSep " " (addon: "-s ${addon}") cfg'.enabledAddons;
+              extraArgs = lib.concatStringsSep " " cfg'.extraArgs;
+            in
+            "${lib.getExe mitmdumpScript} --listen_host ${cfg'.listenHost} --listen_port ${toString cfg'.listenPort} --upstream_host ${cfg'.upstreamHost} --upstream_port ${toString cfg'.upstreamPort} --timeout ${toString cfg'.timeout} ${addons} ${extraArgs}";
+        };
+        requires = cfg'.after;
+        after = cfg'.after;
+        wantedBy = [ "multi-user.target" ];
+      }
+    ) cfg.instances;
+
+    shb.mitmdump.addons = {
+      inherit logger;
+    };
+  };
+}

modules/blocks/mitmdump/docs/default.md

@@ -0,0 +1,178 @@
+# Mitmdump Block {#blocks-mitmdump}
+
+Defined in [`/modules/blocks/mitmdump.nix`](@REPO@/modules/blocks/mitmdump.nix).
+
+This block sets up an [Mitmdump][] service in [reverse proxy][] mode.
+In other words, you can put this block between a client and a server to inspect all the network traffic.
+
+[Mitmdump]: https://plattner.me/mp-docs/#mitmdump
+[reverse proxy]: https://plattner.me/mp-docs/concepts-modes/#reverse-proxy
+
+Multiple instances of mitmdump all listening on different ports
+and proxying to different upstream servers can be created.
+
+The systemd service is made so it is started only when the mitmdump instance
+has started listening on the expected port.
+
+Also, addons can be enabled with the `enabledAddons` option.
+
+## Usage {#blocks-mitmdump-usage}
+
+Put mitmdump in front of a HTTP server listening on port 8000 on the same machine:
+
+```nix
+shb.mitmdump.instances."my-instance" = {
+  listenPort = 8001;
+  upstreamHost = "http://127.0.0.1";
+  upstreamPort = 8000;
+  after = [ "server.service" ];
+};
+```
+
+`upstreamHost` has its default value here and can be left out.
+
+Put mitmdump in front of a HTTP server listening on port 8000 on another machine:
+
+```nix
+shb.mitmdump.instances."my-instance" = {
+  listenPort = 8001;
+  upstreamHost = "http://otherhost";
+  upstreamPort = 8000;
+  after = [ "server.service" ];
+};
+```
+
+### Handle Upstream TLS {#blocks-mitmdump-usage-https}
+
+Replace `http` with `https` if the server expects an HTTPS connection.
+
+### Accept Connections from Anywhere {#blocks-mitmdump-usage-anywhere}
+
+By default, `mitmdump` is configured to listen only for connections from localhost.
+Add `listenHost=0.0.0.0` to make `mitmdump` accept connections from anywhere.
+
+### Extra Logging {#blocks-mitmdump-usage-logging}
+
+To print request and response bodies and more, increase the logging with:
+
+```nix
+extraArgs = [
+    "--set" "flow_detail=3"
+    "--set" "content_view_lines_cutoff=2000"
+];
+```
+
+The default `flow_details` is 1. See the [manual][] for more explanations on the option.
+
+[manual]: (https://docs.mitmproxy.org/stable/concepts/options/#flow_detail)
+
+This will change the verbosity for all requests and responses.
+If you need more fine grained logging, configure instead the [Logger Addon][].
+
+[Logger Addon]: #blocks-mitmdump-addons-logger
+
+## Addons {#blocks-mitmdump-addons}
+
+All provided addons can be found under the `shb.mitmproxy.addons` option.
+
+To enable one for an instance, add it to the `enabledAddons` option. For example:
+
+```nix
+shb.mitmdump.instances."my-instance" = {
+    enabledAddons = [ config.shb.mitmdump.addons.logger ]
+}
+```
+
+### Fine Grained Logger {#blocks-mitmdump-addons-logger}
+
+The Fine Grained Logger addon is found under `shb.mitmproxy.addons.logger`.
+Enabling this addon will add the `mitmdump` option `verbose_pattern` which takes a regex and if it matches,
+prints the request and response headers and body.
+If it does not match, it will just print the response status.
+
+For example, with the `extraArgs`:
+
+```nix
+extraArgs = [
+  "--set" "verbose_pattern=/verbose"
+];
+```
+
+A `GET` request to `/notverbose` will print something similar to:
+
+```
+mitmdump[972]: 127.0.0.1:53586: GET http://127.0.0.1:8000/notverbose HTTP/1.1
+mitmdump[972]:      << HTTP/1.0 200 OK 16b
+```
+
+While a `GET` request to `/verbose` will print something similar to:
+
+```
+mitmdump[972]: [22:42:58.840]
+mitmdump[972]: RequestHeaders:
+mitmdump[972]:     Host: 127.0.0.1:8000
+mitmdump[972]:     User-Agent: curl/8.14.1
+mitmdump[972]:     Accept: */*
+mitmdump[972]: RequestBody:
+mitmdump[972]: Status:          200
+mitmdump[972]: ResponseHeaders:
+mitmdump[972]:     Server: BaseHTTP/0.6 Python/3.13.4
+mitmdump[972]:     Date: Sun, 03 Aug 2025 22:42:58 GMT
+mitmdump[972]:     Content-Type: text/plain
+mitmdump[972]:     Content-Length: 13
+mitmdump[972]: ResponseBody:    test2/verbose
+mitmdump[972]: 127.0.0.1:53602: GET http://127.0.0.1:8000/verbose HTTP/1.1
+mitmdump[972]:      << HTTP/1.0 200 OK 13b
+```
+
+## Example {#blocks-mitmdump-example}
+
+Let's assume a server is listening on port 8000
+which responds a plain text response `test1`
+and its related systemd service is named `test1.service`.
+Sorry, creative naming is not my forte.
+
+Let's put an mitmdump instance in front of it, like so:
+
+```nix
+shb.mitmdump.instances."test1" = {
+  listenPort = 8001;
+  upstreamPort = 8000;
+  after = [ "test1.service" ];
+  extraArgs = [
+    "--set" "flow_detail=3"
+    "--set" "content_view_lines_cutoff=2000"
+  ];
+};
+```
+
+This creates an `mitmdump-test1.service` systemd service.
+We can then use `journalctl -u mitmdump-test1.service` to see the output.
+
+If we make a `curl` request to it: `curl -v http://127.0.0.1:8001`,
+we will get the following output:
+
+```
+mitmdump-test1[971]: 127.0.0.1:40878: GET http://127.0.0.1:8000/ HTTP/1.1
+mitmdump-test1[971]:     Host: 127.0.0.1:8000
+mitmdump-test1[971]:     User-Agent: curl/8.14.1
+mitmdump-test1[971]:     Accept: */*
+mitmdump-test1[971]:  << HTTP/1.0 200 OK 5b
+mitmdump-test1[971]:     Server: BaseHTTP/0.6 Python/3.13.4
+mitmdump-test1[971]:     Date: Thu, 31 Jul 2025 20:55:16 GMT
+mitmdump-test1[971]:     Content-Type: text/plain
+mitmdump-test1[971]:     Content-Length: 5
+mitmdump-test1[971]:     test1
+```
+
+## Tests {#blocks-mitmdump-tests}
+
+Specific integration tests are defined in [`/test/blocks/mitmdump.nix`](@REPO@/test/blocks/mitmdump.nix).
+
+## Options Reference {#blocks-mitmdump-options}
+
+```{=include=} options
+id-prefix: blocks-mitmdump-options-
+list-id: selfhostblocks-block-mitmdump-options
+source: @OPTIONS_JSON@
+```

modules/blocks/monitoring/dashboards/Errors.json

@@ -20,7 +20,6 @@
   "graphTooltip": 0,
   "id": 8,
   "links": [],
-  "liveNow": false,
   "panels": [
     {
       "datasource": {
@@ -39,6 +38,7 @@
             "axisLabel": "",
             "axisPlacement": "auto",
             "barAlignment": 0,
+            "barWidthFactor": 0.6,
             "drawStyle": "line",
             "fillOpacity": 0,
             "gradientMode": "none",
@@ -110,7 +110,7 @@
           "sort": "none"
         }
       },
-      "pluginVersion": "10.2.0",
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -182,6 +182,7 @@
             "axisPlacement": "auto",
             "axisSoftMin": 0.5,
             "barAlignment": 0,
+            "barWidthFactor": 0.6,
             "drawStyle": "line",
             "fillOpacity": 0,
             "gradientMode": "none",
@@ -254,7 +255,7 @@
           "sort": "none"
         }
       },
-      "pluginVersion": "10.2.0",
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -325,6 +326,7 @@
             "axisLabel": "",
             "axisPlacement": "auto",
             "barAlignment": 0,
+            "barWidthFactor": 0.6,
             "drawStyle": "line",
             "fillOpacity": 0,
             "gradientMode": "none",
@@ -398,7 +400,7 @@
           "sort": "none"
         }
       },
-      "pluginVersion": "10.2.0",
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -457,6 +459,10 @@
         "type": "loki",
         "uid": "cd6cc53e-840c-484d-85f7-96fede324006"
       },
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
       "gridPos": {
         "h": 8,
         "w": 24,
@@ -474,6 +480,7 @@
         "sortOrder": "Descending",
         "wrapLogMessage": false
       },
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -512,8 +519,7 @@
             "mode": "absolute",
             "steps": [
               {
-                "color": "green",
-                "value": null
+                "color": "green"
               },
               {
                 "color": "red",
@@ -743,28 +749,23 @@
       "type": "table"
     }
   ],
+  "preload": false,
   "refresh": "",
-  "schemaVersion": 38,
+  "schemaVersion": 40,
   "tags": [],
   "templating": {
     "list": [
       {
         "allValue": ".+",
         "current": {
-          "selected": true,
-          "text": [
-            "All"
-          ],
-          "value": [
-            "$__all"
-          ]
+          "text": "All",
+          "value": "$__all"
         },
         "datasource": {
           "type": "prometheus",
           "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
         },
         "definition": "query_result(max by (name) (node_systemd_unit_state))",
-        "hide": 0,
         "includeAll": true,
         "multi": true,
         "name": "service",
@@ -776,27 +777,23 @@
         },
         "refresh": 1,
         "regex": "/name=\"(?<value>.*)\\.service\"/",
-        "skipUrlSync": false,
         "sort": 1,
         "type": "query"
       },
       {
         "current": {
-          "selected": false,
-          "text": "",
-          "value": ""
+          "text": ".+",
+          "value": ".+"
         },
-        "hide": 0,
         "name": "server_name",
         "options": [
           {
             "selected": true,
-            "text": "",
-            "value": ""
+            "text": ".+",
+            "value": ".+"
           }
         ],
         "query": ".+",
-        "skipUrlSync": false,
         "type": "textbox"
       }
     ]
@@ -809,6 +806,6 @@
   "timezone": "",
   "title": "Errors",
   "uid": "d66242cf-71e8-417c-8ef7-51b0741545df",
-  "version": 29,
+  "version": 32,
   "weekStart": ""
 }

modules/blocks/monitoring/dashboards/Health.json

@@ -0,0 +1,872 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "links": [],
+  "panels": [
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "id": 4,
+      "panels": [],
+      "repeat": "hostname",
+      "title": "${hostname}",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "custom": {
+            "align": "auto",
+            "cellOptions": {
+              "type": "auto"
+            },
+            "footer": {
+              "reducers": []
+            },
+            "inspect": false
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 12,
+        "x": 0,
+        "y": 1
+      },
+      "id": 3,
+      "maxDataPoints": 400,
+      "options": {
+        "cellHeight": "sm",
+        "showHeader": true
+      },
+      "pluginVersion": "12.4.0",
+      "targets": [
+        {
+          "disableTextWrap": false,
+          "editorMode": "builder",
+          "expr": "node_os_info",
+          "fullMetaSearch": false,
+          "includeNullMetadata": true,
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A",
+          "useBackend": false
+        }
+      ],
+      "title": "OS Versions",
+      "transformations": [
+        {
+          "id": "labelsToFields",
+          "options": {
+            "keepLabels": [
+              "build_id",
+              "domain",
+              "hostname",
+              "id",
+              "instance",
+              "job",
+              "name",
+              "pretty_name",
+              "version",
+              "version_codename",
+              "version_id"
+            ],
+            "mode": "columns"
+          }
+        },
+        {
+          "id": "groupBy",
+          "options": {
+            "fields": {
+              "Time": {
+                "aggregations": [
+                  "firstNotNull"
+                ],
+                "operation": "aggregate"
+              },
+              "pretty_name": {
+                "aggregations": [],
+                "operation": "groupby"
+              }
+            }
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "showValues": false,
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 1
+      },
+      "id": 1,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "12.4.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+          },
+          "editorMode": "code",
+          "expr": "node_hwmon_temp_celsius{hostname=~\"$hostname\"}",
+          "instant": false,
+          "legendFormat": "{{chip}} - {{sensor}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Temperature",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "custom": {
+            "axisPlacement": "auto",
+            "fillOpacity": 70,
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineWidth": 0
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 24,
+        "x": 0,
+        "y": 9
+      },
+      "id": 5,
+      "interval": "1m",
+      "maxDataPoints": 200,
+      "options": {
+        "colWidth": 1,
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": false
+        },
+        "rowHeight": 0.8,
+        "showValue": "never",
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "none",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "12.4.0",
+      "repeat": "hostname",
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "editorMode": "code",
+          "expr": "node_zfs_zpool_state{hostname=~\"$hostname\", state=\"online\"} > 0",
+          "legendFormat": "{{zpool}} - {{state}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "ZFS Pools",
+      "type": "status-history"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "showValues": false,
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "line+area"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": 0
+              },
+              {
+                "color": "transparent",
+                "value": 604808
+              }
+            ]
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 13
+      },
+      "id": 2,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "12.4.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+          },
+          "editorMode": "code",
+          "expr": "ssl_certificate_expiry_seconds",
+          "legendFormat": "{{exported_hostname}}: {{subject}} {{path}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Certificate Remaining Validity",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "axisSoftMin": 0,
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "showValues": false,
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "years"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 13
+      },
+      "id": 7,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "12.4.0",
+      "targets": [
+        {
+          "editorMode": "builder",
+          "expr": "scrutiny_smart_power_on_hours{hostname=~\"$hostname\"} / (24 * 365)",
+          "legendFormat": "{{device_name}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Operating Years",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "continuous-YlRd"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "axisSoftMax": 100,
+            "axisSoftMin": 0,
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineStyle": {
+              "fill": "solid"
+            },
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "showValues": false,
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 21
+      },
+      "id": 6,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true,
+          "width": 400
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "12.4.0",
+      "targets": [
+        {
+          "disableTextWrap": false,
+          "editorMode": "builder",
+          "expr": "sum by(hostname, domain, mountpoint, device) (node_filesystem_free_bytes{hostname=~\"$hostname\"})",
+          "fullMetaSearch": false,
+          "hide": true,
+          "includeNullMetadata": false,
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A",
+          "useBackend": false
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+          },
+          "editorMode": "builder",
+          "expr": "sum by(hostname, domain, mountpoint, device) (node_filesystem_size_bytes{hostname=~\"$hostname\"})",
+          "hide": true,
+          "instant": false,
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "name": "Expression",
+            "type": "__expr__",
+            "uid": "__expr__"
+          },
+          "expression": "(1 - $A / $B) * 100",
+          "refId": "Disk Full",
+          "type": "math"
+        }
+      ],
+      "title": "Filesystem Disk Usage",
+      "transformations": [
+        {
+          "id": "joinByField",
+          "options": {
+            "byField": "Time",
+            "mode": "outer"
+          }
+        }
+      ],
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "axisSoftMin": 0,
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "showValues": false,
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 21
+      },
+      "id": 9,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "12.4.0",
+      "targets": [
+        {
+          "editorMode": "builder",
+          "expr": "scrutiny_smart_power_on_hours{hostname=~\"$hostname\"}",
+          "legendFormat": "{{device_name}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Operating Years",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "custom": {
+            "align": "auto",
+            "cellOptions": {
+              "type": "auto"
+            },
+            "footer": {
+              "reducers": []
+            },
+            "inspect": false
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 29
+      },
+      "id": 8,
+      "options": {
+        "cellHeight": "sm",
+        "showHeader": true
+      },
+      "pluginVersion": "12.4.0",
+      "targets": [
+        {
+          "editorMode": "builder",
+          "exemplar": false,
+          "expr": "scrutiny_device_info{hostname=~\"$hostname\"}",
+          "format": "table",
+          "instant": true,
+          "legendFormat": "{{device_name}}",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Disk Info",
+      "transformations": [
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {
+              "Time": true,
+              "Value": true,
+              "__name__": true,
+              "domain": true,
+              "hostname": true,
+              "instance": true,
+              "job": true,
+              "wwn": false
+            },
+            "includeByName": {},
+            "indexByName": {},
+            "renameByName": {}
+          }
+        }
+      ],
+      "type": "table"
+    }
+  ],
+  "preload": false,
+  "schemaVersion": 42,
+  "tags": [],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "text": "baryum",
+          "value": "baryum"
+        },
+        "definition": "label_values(up,hostname)",
+        "name": "hostname",
+        "options": [],
+        "query": {
+          "qryType": 1,
+          "query": "label_values(up,hostname)",
+          "refId": "PrometheusVariableQueryEditor-VariableQuery"
+        },
+        "refresh": 1,
+        "regex": "",
+        "regexApplyTo": "value",
+        "type": "query"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-30m",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "Node Health",
+  "uid": "edhuvl28vpjwge",
+  "version": 25,
+  "weekStart": ""
+}

modules/blocks/monitoring/dashboards/Performance.json

@@ -20,9 +20,9 @@
   "graphTooltip": 0,
   "id": 6,
   "links": [],
-  "liveNow": false,
   "panels": [
     {
+      "collapsed": false,
       "gridPos": {
         "h": 1,
         "w": 24,
@@ -30,6 +30,7 @@
         "y": 0
       },
       "id": 12,
+      "panels": [],
       "title": "Node",
       "type": "row"
     },
@@ -51,6 +52,7 @@
             "axisLabel": "",
             "axisPlacement": "auto",
             "barAlignment": 0,
+            "barWidthFactor": 0.6,
             "drawStyle": "line",
             "fillOpacity": 0,
             "gradientMode": "none",
@@ -204,6 +206,7 @@
           "sort": "none"
         }
       },
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -211,7 +214,7 @@
             "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
           },
           "editorMode": "code",
-          "expr": "netdata_system_memory_full_pressure_stall_time_ms_average{instance=~\"$instance\"} * -1",
+          "expr": "netdata_system_memory_full_pressure_stall_time_ms_average{hostname=~\"$hostname\"} * -1",
           "hide": false,
           "instant": false,
           "legendFormat": "full stall time",
@@ -269,6 +272,7 @@
             "axisLabel": "",
             "axisPlacement": "auto",
             "barAlignment": 0,
+            "barWidthFactor": 0.6,
             "drawStyle": "line",
             "fillOpacity": 0,
             "gradientMode": "none",
@@ -422,6 +426,7 @@
           "sort": "none"
         }
       },
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -429,7 +434,7 @@
             "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
           },
           "editorMode": "code",
-          "expr": "netdata_system_memory_full_pressure_stall_time_ms_average{instance=~\"$instance\"} * -1",
+          "expr": "netdata_system_memory_full_pressure_stall_time_ms_average{hostname=~\"$hostname\"} * -1",
           "hide": false,
           "instant": false,
           "legendFormat": "full stall time",
@@ -456,7 +461,7 @@
           },
           "disableTextWrap": false,
           "editorMode": "code",
-          "expr": "sum by(dimension, service_name) (netdata_systemd_service_memory_usage_MiB_average{instance=~\"$instance\", service_name=~\"$service\", dimension=\"swap\"})",
+          "expr": "sum by(dimension, service_name) (netdata_systemd_service_memory_usage_MiB_average{hostname=~\"$hostname\", service_name=~\"$service\", dimension=\"swap\"})",
           "fullMetaSearch": false,
           "hide": false,
           "includeNullMetadata": true,
@@ -487,6 +492,7 @@
             "axisLabel": "",
             "axisPlacement": "auto",
             "barAlignment": 0,
+            "barWidthFactor": 0.6,
             "drawStyle": "line",
             "fillOpacity": 0,
             "gradientMode": "none",
@@ -597,6 +603,7 @@
           "sort": "none"
         }
       },
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -604,7 +611,7 @@
             "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
           },
           "editorMode": "code",
-          "expr": "netdata_system_cpu_some_pressure_stall_time_ms_average{instance=~\"$instance\"} * -1",
+          "expr": "netdata_system_cpu_some_pressure_stall_time_ms_average{hostname=~\"$hostname\"} * -1",
           "hide": false,
           "instant": false,
           "legendFormat": "some stall time",
@@ -618,7 +625,7 @@
           },
           "disableTextWrap": false,
           "editorMode": "code",
-          "expr": "sum by(dimension, service_name) (netdata_systemd_service_cpu_utilization_percentage_average{instance=~\"$instance\", service_name=~\"$service\"})",
+          "expr": "sum by(dimension, service_name) (netdata_systemd_service_cpu_utilization_percentage_average{hostname=~\"$hostname\", service_name=~\"$service\"})",
           "fullMetaSearch": false,
           "hide": false,
           "includeNullMetadata": true,
@@ -649,6 +656,7 @@
             "axisLabel": "",
             "axisPlacement": "auto",
             "barAlignment": 0,
+            "barWidthFactor": 0.6,
             "drawStyle": "line",
             "fillOpacity": 0,
             "gradientMode": "none",
@@ -814,6 +822,7 @@
           "sort": "none"
         }
       },
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -821,7 +830,7 @@
             "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
           },
           "editorMode": "code",
-          "expr": "netdata_system_io_full_pressure_stall_time_ms_average{instance=~\"$instance\"} * -1",
+          "expr": "netdata_system_io_full_pressure_stall_time_ms_average{hostname=~\"$hostname\"} * -1",
           "hide": false,
           "instant": false,
           "legendFormat": "full stall time",
@@ -878,6 +887,7 @@
             "axisLabel": "",
             "axisPlacement": "auto",
             "barAlignment": 0,
+            "barWidthFactor": 0.6,
             "drawStyle": "line",
             "fillOpacity": 0,
             "gradientMode": "none",
@@ -943,6 +953,7 @@
           "sort": "none"
         }
       },
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -950,7 +961,7 @@
             "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
           },
           "editorMode": "code",
-          "expr": "netdata_disk_io_KiB_persec_average{instance=~\"$instance\", chart=~\"disk.sd.+\"}",
+          "expr": "netdata_disk_io_KiB_persec_average{hostname=~\"$hostname\", chart=~\"disk.sd.+\"}",
           "instant": false,
           "legendFormat": "{{device}} / {{dimension}}",
           "range": true,
@@ -1031,7 +1042,8 @@
           "layout": "auto"
         },
         "tooltip": {
-          "show": true,
+          "mode": "single",
+          "showColorScale": false,
           "yHistogram": false
         },
         "yAxis": {
@@ -1041,7 +1053,7 @@
           "unit": "s"
         }
       },
-      "pluginVersion": "10.2.0",
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -1049,7 +1061,7 @@
             "uid": "cd6cc53e-840c-484d-85f7-96fede324006"
           },
           "editorMode": "code",
-          "expr": "{unit=\"nginx.service\"} | pattern \"<_> <_> <line>\" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | request_time > 100",
+          "expr": "{hostname=~\"$hostname\",unit=\"nginx.service\"} | pattern \"<_> <_> <line>\" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | request_time > 100",
           "legendFormat": "{{server_name}}",
           "queryType": "range",
           "refId": "A"
@@ -1145,6 +1157,7 @@
             "axisLabel": "",
             "axisPlacement": "auto",
             "barAlignment": 0,
+            "barWidthFactor": 0.6,
             "drawStyle": "points",
             "fillOpacity": 0,
             "gradientMode": "none",
@@ -1175,7 +1188,8 @@
             "mode": "absolute",
             "steps": [
               {
-                "color": "green"
+                "color": "green",
+                "value": null
               },
               {
                 "color": "red",
@@ -1210,7 +1224,7 @@
           "sort": "none"
         }
       },
-      "pluginVersion": "10.2.0",
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -1218,7 +1232,7 @@
             "uid": "cd6cc53e-840c-484d-85f7-96fede324006"
           },
           "editorMode": "code",
-          "expr": "{unit=\"nginx.service\"} | pattern \"<_> <_> <line>\" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | request_time > 100",
+          "expr": "{hostname=~\"$hostname\",unit=\"nginx.service\"} | pattern \"<_> <_> <line>\" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | request_time > 100",
           "legendFormat": "",
           "queryType": "range",
           "refId": "A"
@@ -1317,7 +1331,8 @@
             "mode": "absolute",
             "steps": [
               {
-                "color": "green"
+                "color": "green",
+                "value": null
               },
               {
                 "color": "red",
@@ -1354,7 +1369,7 @@
         },
         "showHeader": true
       },
-      "pluginVersion": "10.2.0",
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -1362,7 +1377,7 @@
             "uid": "cd6cc53e-840c-484d-85f7-96fede324006"
           },
           "editorMode": "code",
-          "expr": "{unit=\"nginx.service\"} | pattern \"<_> <_> <line>\" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | request_time > 1",
+          "expr": "{hostname=~\"$hostname\",unit=\"nginx.service\"} | pattern \"<_> <_> <line>\" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | request_time > 1",
           "queryType": "range",
           "refId": "A"
         }
@@ -1411,7 +1426,8 @@
             "mode": "absolute",
             "steps": [
               {
-                "color": "green"
+                "color": "green",
+                "value": null
               },
               {
                 "color": "red",
@@ -1472,7 +1488,7 @@
         },
         "showHeader": true
       },
-      "pluginVersion": "10.2.0",
+      "pluginVersion": "11.4.0",
       "targets": [
         {
           "datasource": {
@@ -1480,7 +1496,7 @@
             "uid": "cd6cc53e-840c-484d-85f7-96fede324006"
           },
           "editorMode": "code",
-          "expr": "{unit=\"postgresql.service\"} | regexp \".*duration: (?P<duration_ms>[0-9.]+) ms (?P<statement>.*)\" | duration_ms > 500 | __error__ != \"LabelFilterErr\"",
+          "expr": "{hostname=~\"$hostname\",unit=\"postgresql.service\"} | regexp \".*duration: (?P<duration_ms>[0-9.]+) ms (?P<statement>.*)\" | duration_ms > 500 | __error__ != \"LabelFilterErr\"",
           "queryType": "range",
           "refId": "A"
         }
@@ -1513,56 +1529,55 @@
       "type": "table"
     }
   ],
+  "preload": false,
   "refresh": "1m",
-  "schemaVersion": 38,
+  "schemaVersion": 40,
   "tags": [],
   "templating": {
     "list": [
       {
         "allValue": ".*",
         "current": {
-          "selected": true,
           "text": [
-            "127.0.0.1:19999"
+            "baryum"
           ],
           "value": [
-            "127.0.0.1:19999"
+            "baryum"
           ]
         },
         "datasource": {
           "type": "prometheus",
           "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
         },
-        "definition": "label_values(netdata_systemd_service_unit_state_state_average,instance)",
-        "hide": 0,
+        "definition": "label_values(netdata_systemd_service_unit_state_state_average,hostname)",
         "includeAll": false,
         "multi": true,
-        "name": "instance",
+        "name": "hostname",
         "options": [],
         "query": {
           "qryType": 1,
-          "query": "label_values(netdata_systemd_service_unit_state_state_average,instance)",
+          "query": "label_values(netdata_systemd_service_unit_state_state_average,hostname)",
           "refId": "PrometheusVariableQueryEditor-VariableQuery"
         },
         "refresh": 1,
         "regex": "",
-        "skipUrlSync": false,
-        "sort": 0,
         "type": "query"
       },
       {
         "allValue": ".*",
         "current": {
-          "selected": false,
-          "text": "All",
-          "value": "$__all"
+          "text": [
+            "All"
+          ],
+          "value": [
+            "$__all"
+          ]
         },
         "datasource": {
           "type": "prometheus",
           "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
         },
         "definition": "label_values(netdata_systemd_service_unit_state_state_average,unit_name)",
-        "hide": 0,
         "includeAll": true,
         "multi": true,
         "name": "service",
@@ -1574,8 +1589,6 @@
         },
         "refresh": 1,
         "regex": "",
-        "skipUrlSync": false,
-        "sort": 0,
         "type": "query"
       }
     ]
@@ -1588,6 +1601,6 @@
   "timezone": "",
   "title": "Performance",
   "uid": "e01156bf-cdba-42eb-9845-a401dd634d41",
-  "version": 54,
+  "version": 82,
   "weekStart": ""
 }

modules/blocks/monitoring/dashboards/Scraping_Jobs.json

@@ -0,0 +1,334 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "id": 5,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "cd6cc53e-840c-484d-85f7-96fede324006"
+      },
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "id": 5,
+      "options": {
+        "dedupStrategy": "none",
+        "enableInfiniteScrolling": false,
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": false,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": false
+      },
+      "pluginVersion": "12.2.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "cd6cc53e-840c-484d-85f7-96fede324006"
+          },
+          "direction": "backward",
+          "editorMode": "code",
+          "expr": "{unit=~\"prometheus-.*-exporter.service\"}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "title": "Exporter Logs",
+      "type": "logs"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "fixedColor": "red",
+            "mode": "shades"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "showValues": false,
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "id": 4,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "12.2.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+          },
+          "editorMode": "code",
+          "expr": "rate(net_conntrack_dialer_conn_failed_total{hostname=~\"$hostname\"}[2m]) > 0",
+          "instant": false,
+          "legendFormat": "{{dialer_name}} - {{reason}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Errors",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "fixedColor": "red",
+            "mode": "thresholds"
+          },
+          "custom": {
+            "axisPlacement": "auto",
+            "fillOpacity": 70,
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineWidth": 0,
+            "spanNulls": false
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "id": 3,
+      "options": {
+        "alignValue": "center",
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": false
+        },
+        "mergeValues": true,
+        "rowHeight": 0.9,
+        "showValue": "never",
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "12.2.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+          },
+          "editorMode": "code",
+          "expr": "prometheus_sd_discovered_targets{hostname=~\"$hostname\"}",
+          "hide": false,
+          "instant": false,
+          "legendFormat": "{{config}}",
+          "range": true,
+          "refId": "All"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+          },
+          "editorMode": "code",
+          "expr": "label_replace(increase((sum by(dialer_name) (net_conntrack_dialer_conn_failed_total{hostname=~\"$hostname\"}))[15m:1m]), \"config\", \"$1\", \"dialer_name\", \"(.*)\")",
+          "hide": false,
+          "instant": false,
+          "legendFormat": "{{dialer_name}}",
+          "range": true,
+          "refId": "Failed"
+        }
+      ],
+      "title": "Scraping jobs",
+      "transformations": [
+        {
+          "id": "labelsToFields",
+          "options": {
+            "keepLabels": [
+              "config"
+            ],
+            "mode": "columns"
+          }
+        },
+        {
+          "id": "merge",
+          "options": {}
+        },
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {
+              "prometheus_sd_discovered_targets": true
+            },
+            "indexByName": {},
+            "renameByName": {
+              "prometheus_sd_discovered_targets": ""
+            }
+          }
+        },
+        {
+          "id": "partitionByValues",
+          "options": {
+            "fields": [
+              "config"
+            ]
+          }
+        }
+      ],
+      "type": "state-timeline"
+    }
+  ],
+  "preload": false,
+  "refresh": "",
+  "schemaVersion": 42,
+  "tags": [],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "text": "baryum",
+          "value": "baryum"
+        },
+        "definition": "label_values(up,hostname)",
+        "includeAll": false,
+        "name": "hostname",
+        "options": [],
+        "query": {
+          "qryType": 1,
+          "query": "label_values(up,hostname)",
+          "refId": "PrometheusVariableQueryEditor-VariableQuery"
+        },
+        "refresh": 1,
+        "regex": "",
+        "type": "query"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "",
+  "title": "Scraping Jobs",
+  "uid": "debb763d-77aa-47bd-9290-2e02583c8ed2",
+  "version": 24
+}

modules/blocks/monitoring/docs/default.md

@@ -8,38 +8,70 @@ This block sets up the monitoring stack for Self Host Blocks. It is composed of:
 - Prometheus as the database for metrics.
 - Loki as the database for logs.
 
-## Configuration {#blocks-monitoring-configuration}
+## Features {#services-monitoring-features}
+
+- Declarative [LDAP](#blocks-monitoring-options-shb.monitoring.ldap) Configuration.
+  - Needed LDAP groups are created automatically.
+- Declarative [SSO](#blocks-monitoring-options-shb.monitoring.sso) Configuration.
+  - When SSO is enabled, login with user and password is disabled.
+  - Registration is enabled through SSO.
+- Access through [subdomain](#blocks-monitoring-options-shb.monitoring.subdomain) using reverse proxy.
+- Access through [HTTPS](#blocks-monitoring-options-shb.monitoring.ssl) using reverse proxy.
+- Integration with the [dashboard contract](contracts-dashboard.html) for displaying user facing application in a dashboard. [Manual](#blocks-monitoring-usage-applicationdashboard)
+- Out of the box integration with [Scrutiny](https://github.com/AnalogJ/scrutiny) service for Hard Drives monitoring. [Manual](#blocks-monitoring-usage-scrutiny)
+
+## Usage {#blocks-monitoring-usage}
+
+### Initial Configuration {#blocks-monitoring-usage-configuration}
+
+The following snippet assumes a few blocks have been setup already:
+
+- the [secrets block](usage.html#usage-secrets) with SOPS,
+- the [`shb.ssl` block](blocks-ssl.html#usage),
+- the [`shb.lldap` block](blocks-lldap.html#blocks-lldap-global-setup).
+- the [`shb.authelia` block](blocks-authelia.html#blocks-sso-global-setup).
 
 ```nix
-shb.monitoring = {
-  enable = true;
-  subdomain = "grafana";
-  inherit domain;
-  contactPoints = [ "me@example.com" ];
-  adminPasswordFile = config.sops.secrets."monitoring/admin_password".path;
-  secretKeyFile = config.sops.secrets."monitoring/secret_key".path;
-};
+{
+  shb.monitoring = {
+    enable = true;
+    subdomain = "grafana";
+    inherit domain;
+    contactPoints = [ "me@example.com" ];
+    adminPassword.result = config.shb.sops.secret."monitoring/admin_password".result;
+    secretKey.result = config.shb.sops.secret."monitoring/secret_key".result;
 
-sops.secrets."monitoring/admin_password" = {
-  sopsFile = ./secrets.yaml;
-  mode = "0400";
-  owner = "grafana";
-  group = "grafana";
-  restartUnits = [ "grafana.service" ];
-};
-sops.secrets."monitoring/secret_key" = {
-  sopsFile = ./secrets.yaml;
-  mode = "0400";
-  owner = "grafana";
-  group = "grafana";
-  restartUnits = [ "grafana.service" ];
+    sso = {
+      enable = true;
+      authEndpoint = "https://${config.shb.authelia.subdomain}.${config.shb.authelia.domain}";
+
+      sharedSecret.result = config.shb.sops.secret."monitoring/oidcSecret".result;
+      sharedSecretForAuthelia.result = config.shb.sops.secret."monitoring/oidcAutheliaSecret".result;
+    };
+  };
+  
+  shb.sops.secret."monitoring/admin_password".request = config.shb.monitoring.adminPassword.request;
+  shb.sops.secret."monitoring/secret_key".request = config.shb.monitoring.secretKey.request;
+  shb.sops.secret."monitoring/oidcSecret".request = config.shb.monitoring.sso.sharedSecret.request;
+  shb.sops.secret."monitoring/oidcAutheliaSecret" = {
+    request = config.shb.monitoring.sso.sharedSecretForAuthelia.request;
+    settings.key = "monitoring/oidcSecret";
+  };
 };
 ```
 
-With that, Grafana, Prometheus, Loki and Promtail are setup! You can access `Grafana` at
-`grafana.example.com` with user `admin` and password ``.
+Secrets can be randomly generated with `nix run nixpkgs#openssl -- rand -hex 64`.
 
-I recommend adding a STMP server configuration so you receive alerts by email:
+With that, Grafana, Prometheus, Loki and Promtail are setup! You can access `Grafana` at
+`grafana.example.com` with user `admin` and the password from the sops key `monitoring/admin_password`.
+
+The [user](#blocks-monitoring-options-shb.monitoring.ldap.userGroup)
+and [admin](#blocks-monitoring-options-shb.monitoring.ldap.adminGroup)
+LDAP groups are created automatically.
+
+### SMTP {#blocks-monitoring-usage-smtp}
+
+I recommend adding an SMTP server configuration so you receive alerts by email:
 
 ```nix
 shb.monitoring.smtp = {
@@ -60,6 +92,8 @@ sops.secrets."monitoring/secret_key" = {
 };
 ```
 
+### Log Optimization {#blocks-monitoring-usage-log-optimization}
+
 Since all logs are now stored in Loki, you can probably reduce the systemd journal retention
 time with:
 
@@ -73,6 +107,49 @@ MaxFileSec=day
 '';
 ```
 
+Other options are accessible through the upstream services modules.
+You might for example want to update the metrics retention time with:
+
+```nix
+services.prometheus.retentionTime = "60d";
+```
+
+### Application Dashboard {#blocks-monitoring-usage-applicationdashboard}
+
+Integration with the [dashboard contract](contracts-dashboard.html) is provided
+by the [dashboard option](#blocks-monitoring-options-shb.monitoring.dashboard).
+
+For example using the [Homepage](services-homepage.html) service:
+
+```nix
+{
+  shb.homepage.servicesGroups.Admin.services.Grafana = {
+    sortOrder = 10;
+    dashboard.request = config.shb.monitoring.dashboard.request;
+  };
+}
+```
+
+There is also an integration for the scrutiny service, see next section.
+
+### Scrutiny {#blocks-monitoring-usage-scrutiny}
+
+Integration with the [Scrutiny](https://github.com/AnalogJ/scrutiny) service is enabled by default and setup automatically.
+
+The web interface will be served under the [scrutiny.subdomain](#blocks-monitoring-options-shb.monitoring.scrutiny.subdomain) option.
+If you don't want the web interface, set the option to `null`.
+
+For integration with the [dashboard contract](contracts-dashboard.html):
+
+```nix
+{
+  shb.homepage.servicesGroups.Admin.services.Scrutiny = {
+    sortOrder = 11;
+    dashboard.request = config.shb.monitoring.scrutiny.dashboard.request;
+  };
+}
+```
+
 ## Provisioning {#blocks-monitoring-provisioning}
 
 Self Host Blocks will create automatically the following resources:
@@ -101,8 +178,8 @@ This dashboard is meant to be the first stop to understand why a service is misb
 ![](./assets/dashboards_Errors_1.png)
 ![](./assets/dashboards_Errors_2.png)
 
-The yellow and red dashed vertical bars correspond to the [Requests Error Budget
-Alert](#blocks-monitoring-budget-alerts) firing.
+The yellow and red dashed vertical bars correspond to the
+[Requests Error Budget Alert](#blocks-monitoring-budget-alerts) firing.
 
 ## Performance Dashboard {#blocks-monitoring-performance-dashboard}
 
@@ -112,6 +189,54 @@ This dashboard is meant to be the first stop to understand why a service is perf
 ![Performance Dashboard Middle Part](./assets/dashboards_Performance_2.png)
 ![Performance Dashboard Bottom Part](./assets/dashboards_Performance_3.png)
 
+## Nextcloud Dashboard {#blocks-monitoring-nextcloud-dashboard}
+
+See [Nextcloud service](./services-nextcloud.html#services-nextcloudserver-dashboard) manual.
+
+## Deluge Dashboard {#blocks-monitoring-deluge-dashboard}
+
+This dashboard is used to monitor a [deluge](./services-deluge.html) instance.
+
+![Deluge Dashboard Top Part](./assets/dashboards_Deluge_1.png)
+![Deluge Dashboard Bottom Part](./assets/dashboards_Deluge_2.png)
+
+## Backups Dashboard and Alert {#blocks-monitoring-backup}
+
+This dashboard shows Restic and BorgBackup backup jobs, or any job with "backup" in the systemd service name.
+
+### Dashboard {#blocks-monitoring-backup-dashboard}
+
+Variables:
+
+- The "Job" variable allows to select one or more backup jobs. "All" is the default.
+- The "mountpoints" variable allows to select only relevant mountpoints for backup. "All" is the default.
+
+The most important graphs are the first three:
+
+- "Backup Jobs in the Past Week": Shows stats on all backup jobs that ran in the past.
+  It is sorted by the "Failed" column in descending order.
+  This way, one can directly see when a job has failures.
+- "Schedule": Shows when a job will run.
+  The unit is "Datetime from Now" meaning it shows when a job ran or will run relative to the current time.
+  An annotation will show up when the "Late Backups" alert fired or resolved.
+- "Backup jobs": Shows when a backup job ran.
+  Normally, jobs running for less than 15 seconds will not show up in the graph.
+  We crafted a query that still shows them but the length is 15 seconds, even if the backup job
+  took less time to run.
+
+![Backups Dashboard Top Part](./assets/dashboards_Backups_1.png)
+![Backups Dashboard Middle Part](./assets/dashboards_Backups_2.png)
+![Backups Dashboard Bottom Part](./assets/dashboards_Backups_3.png)
+
+### Alerts {#blocks-monitoring-backup-alerts}
+
+- The "Late Backups" alert will fire if a backup job did not run at all in the last 24 hours
+  or if all runs were failures in the last 24 hours.
+  It will show up as annotations in the "Schedule" panel of the dashboard.
+
+![Late Backups Alert Firing](./assets/alert_rules_LateBackups_1.png)
+![Backups Alert Showing Up In Dashboard](./assets/dashboards_Backups_alert.png)
+
 ## Requests Error Budget Alert {#blocks-monitoring-budget-alerts}
 
 This alert will fire when the ratio between number of requests getting a 5XX response from a service
@@ -119,3 +244,60 @@ and the total requests to that service exceeds 1%.
 
 ![Error Dashboard Top Part](./assets/alert_rules_5xx_1.png)
 ![Error Dashboard Bottom Part](./assets/alert_rules_5xx_2.png)
+
+## SSL Certificates Dashboard and Alert {#blocks-monitoring-ssl}
+
+This dashboard shows Let's Encrypt renewal and setup jobs,
+or any job starting with "acme-" in the systemd service name.
+
+### Dashboard {#blocks-monitoring-ssl-dashboard}
+
+Variables:
+
+- The "Job" variable allows to focus on one or more certificate. "All" is the default.
+
+Graphs:
+
+- "Certificate Remaining Validity": Shows in how long will certificates expire.
+  It shows all files under `/var/lib/acme`.
+  An annotation will show up when the "Certificate Did Not Renew" alert fired or resolved.
+- "Schedule": Shows when a job will run.
+  The unit is "Datetime from Now" meaning it shows when a job ran or will run relative to the current time.
+- "Jobs in the Past Week": Shows stats on all renewal jobs that ran in the past.
+  It is sorted by the "Failed" column in descending order.
+  This way, one can directly see when a job has failures.
+  Note, the stats is not accurate because detecting jobs taking taking less than 15 seconds
+  is not supported well.
+- "Job Runs": Shows when a renewal job ran.
+  Normally, jobs running for less than 15 seconds will not show up in the graph.
+  We crafted a query that still shows them but the length is 100 seconds, even if the job
+  took less time to run.
+
+![SSL Dashboard No Filter](./assets/dashboards_SSL_all.png)
+![SSL Dashboard Filter Failing](./assets/dashboards_SSL_fail.png)
+
+### Alerts {#blocks-monitoring-ssl-alerts}
+
+- The "Certificate Did Not Renew" alert will fire if a backup job did not run at all in the last 24 hours
+    or if all runs were failures in the last 24 hours.
+  It will show up as annotations in the "Schedule" panel of the dashboard.
+
+![Late SSL Jobs Alert Firing](./assets/alert_rules_LateSSL_1.png)
+
+## Impermanence {#blocks-monitoring-impermanence}
+
+To save the fluent-bit folder in an impermanence setup, add:
+
+```nix
+{
+  shb.zfs.datasets."safe/monitoring-fluent-bit".path = config.shb.jellyfin.impermanence.fluent-bit;
+}
+```
+
+## Options Reference {#blocks-monitoring-options}
+
+```{=include=} options
+id-prefix: blocks-monitoring-options-
+list-id: selfhostblocks-blocks-monitoring-options
+source: @OPTIONS_JSON@
+```

modules/blocks/monitoring/rules.json

@@ -18,7 +18,7 @@
             "uid": "cd6cc53e-840c-484d-85f7-96fede324006"
           },
           "editorMode": "code",
-          "expr": "(sum by(server_name) (count_over_time({unit=\"nginx.service\"} | pattern \"<_> <_> <line>\" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | status =~ \"[1234]..\" | server_name =~ \".*\" [7d])) / sum by(server_name) (count_over_time({unit=\"nginx.service\"} | pattern \"<_> <_> <line>\" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | server_name =~ \".*\" [7d])))",
+          "expr": "(sum by(server_name) (count_over_time({unit=\"nginx.service\"} | pattern \"<_> <_> <line>\" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | status =~ \"[1234]..\" | server_name =~ \".*\" [1h])) / sum by(server_name) (count_over_time({unit=\"nginx.service\"} | pattern \"<_> <_> <line>\" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | server_name =~ \".*\" [1h])))",
           "intervalMs": 1000,
           "legendFormat": "{{server_name}}",
           "maxDataPoints": 43200,
@@ -120,10 +120,265 @@
       "__panelId__": "9",
       "description": "",
       "runbook_url": "",
-      "summary": "The error budget for a service for the last 7 days is under 99%"
+      "summary": "The error budget for a service for the last 1 hour is under 99%"
+    },
+    "labels": {
+      "role": "sysadmin"
+    },
+    "isPaused": false
+  },
+  {
+    "uid": "ee817l3a88s1sd",
+    "title": "Certificate Did Not Renew",
+    "condition": "C",
+    "data": [
+      {
+        "refId": "A",
+        "relativeTimeRange": {
+          "from": 1800,
+          "to": 0
+        },
+        "datasourceUid": "df80f9f5-97d7-4112-91d8-72f523a02b09",
+        "model": {
+          "adhocFilters": [],
+          "datasource": {
+            "type": "prometheus",
+            "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+          },
+          "editorMode": "code",
+          "expr": "min by(subject) (ssl_certificate_expiry_seconds)",
+          "interval": "",
+          "intervalMs": 15000,
+          "legendFormat": "{{exported_hostname}}: {{subject}} {{path}}",
+          "maxDataPoints": 43200,
+          "range": true,
+          "refId": "A"
+        }
+      },
+      {
+        "refId": "B",
+        "relativeTimeRange": {
+          "from": 0,
+          "to": 0
+        },
+        "datasourceUid": "__expr__",
+        "model": {
+          "conditions": [
+            {
+              "evaluator": {
+                "params": [],
+                "type": "gt"
+              },
+              "operator": {
+                "type": "and"
+              },
+              "query": {
+                "params": [
+                  "B"
+                ]
+              },
+              "reducer": {
+                "params": [],
+                "type": "last"
+              },
+              "type": "query"
+            }
+          ],
+          "datasource": {
+            "type": "__expr__",
+            "uid": "__expr__"
+          },
+          "expression": "A",
+          "intervalMs": 1000,
+          "maxDataPoints": 43200,
+          "reducer": "last",
+          "refId": "B",
+          "type": "reduce"
+        }
+      },
+      {
+        "refId": "C",
+        "relativeTimeRange": {
+          "from": 0,
+          "to": 0
+        },
+        "datasourceUid": "__expr__",
+        "model": {
+          "conditions": [
+            {
+              "evaluator": {
+                "params": [
+                  604800
+                ],
+                "type": "lt"
+              },
+              "operator": {
+                "type": "and"
+              },
+              "query": {
+                "params": [
+                  "C"
+                ]
+              },
+              "reducer": {
+                "params": [],
+                "type": "last"
+              },
+              "type": "query"
+            }
+          ],
+          "datasource": {
+            "type": "__expr__",
+            "uid": "__expr__"
+          },
+          "expression": "B",
+          "intervalMs": 1000,
+          "maxDataPoints": 43200,
+          "refId": "C",
+          "type": "threshold"
+        }
+      }
+    ],
+    "dashboardUid": "ae818js0bvw8wb",
+    "panelId": 3,
+    "noDataState": "NoData",
+    "execErrState": "Error",
+    "for": "20m",
+    "annotations": {
+      "__dashboardUid__": "ae818js0bvw8wb",
+      "__panelId__": "3",
+      "description": "The expiry date of the certificate is 1 week from now.",
+      "summary": "Certificate did not renew on time."
+    },
+    "labels": {
+      "role": "sysadmin"
+    },
+    "isPaused": false
+  },
+  {
+    "uid": "df4doj5pomhvkf",
+    "title": "Late Backups",
+    "condition": "C",
+    "data": [
+      {
+        "refId": "A",
+        "relativeTimeRange": {
+          "from": 10800,
+          "to": 0
+        },
+        "datasourceUid": "df80f9f5-97d7-4112-91d8-72f523a02b09",
+        "model": {
+          "adhocFilters": [],
+          "datasource": {
+            "type": "prometheus",
+            "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+          },
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "(\n  # Timer triggered at least once in the last 24h\n  label_replace((\n    time()\n      -\n    systemd_timer_last_trigger_seconds{name=~\".*backup.*.timer\"}\n  ) < 24*60*60, \"name\", \"$1.service\", \"name\", \"(.*).timer\")\n  AND on(name)\n  # At least one failure in the last 24h\n  (\n    max_over_time(systemd_unit_state{name=~\".*backup.*.service\", state=\"failed\"}[24h]) > 0\n  )\n  AND on(name)\n  # No successes in the last 24h\n  (\n    max_over_time(systemd_unit_state{name=~\".*backup.*.service\", state=\"inactive\"}[24h]) == 0\n  )\n)",
+          "instant": false,
+          "interval": "",
+          "intervalMs": 15000,
+          "legendFormat": "{{name}}",
+          "maxDataPoints": 43200,
+          "range": true,
+          "refId": "A"
+        }
+      },
+      {
+        "refId": "B",
+        "relativeTimeRange": {
+          "from": 0,
+          "to": 0
+        },
+        "datasourceUid": "__expr__",
+        "model": {
+          "conditions": [
+            {
+              "evaluator": {
+                "params": [],
+                "type": "gt"
+              },
+              "operator": {
+                "type": "and"
+              },
+              "query": {
+                "params": [
+                  "B"
+                ]
+              },
+              "reducer": {
+                "params": [],
+                "type": "last"
+              },
+              "type": "query"
+            }
+          ],
+          "datasource": {
+            "type": "__expr__",
+            "uid": "__expr__"
+          },
+          "expression": "A",
+          "intervalMs": 1000,
+          "maxDataPoints": 43200,
+          "reducer": "last",
+          "refId": "B",
+          "type": "reduce"
+        }
+      },
+      {
+        "refId": "C",
+        "relativeTimeRange": {
+          "from": 0,
+          "to": 0
+        },
+        "datasourceUid": "__expr__",
+        "model": {
+          "conditions": [
+            {
+              "evaluator": {
+                "params": [
+                  0
+                ],
+                "type": "gt"
+              },
+              "operator": {
+                "type": "and"
+              },
+              "query": {
+                "params": [
+                  "C"
+                ]
+              },
+              "reducer": {
+                "params": [],
+                "type": "last"
+              },
+              "type": "query"
+            }
+          ],
+          "datasource": {
+            "type": "__expr__",
+            "uid": "__expr__"
+          },
+          "expression": "B",
+          "intervalMs": 1000,
+          "maxDataPoints": 43200,
+          "refId": "C",
+          "type": "threshold"
+        }
+      }
+    ],
+    "dashboardUid": "f05500d0-15ed-4719-b68d-fb898ca13cc8",
+    "panelId": 15,
+    "noDataState": "OK",
+    "execErrState": "Error",
+    "annotations": {
+      "__dashboardUid__": "f05500d0-15ed-4719-b68d-fb898ca13cc8",
+      "__panelId__": "15",
+      "summary": "A backup did not run in the last 24 hours."
     },
     "labels": {
-      "": "",
       "role": "sysadmin"
     },
     "isPaused": false

modules/blocks/nginx.nix

@@ -1,10 +1,13 @@
-{ config, pkgs, lib, ... }:
+{
+  config,
+  lib,
+  shb,
+  ...
+}:
 
 let
   cfg = config.shb.nginx;
 
-  contracts = pkgs.callPackage ../contracts {};
-
   fqdn = c: "${c.subdomain}.${c.domain}";
 
   vhostConfig = lib.types.submodule {
@@ -23,13 +26,14 @@ let
 
       ssl = lib.mkOption {
         description = "Path to SSL files";
-        type = lib.types.nullOr contracts.ssl.certs;
+        type = lib.types.nullOr shb.contracts.ssl.certs;
         default = null;
       };
 
       upstream = lib.mkOption {
-        type = lib.types.str;
+        type = lib.types.nullOr lib.types.str;
         description = "Upstream url to be protected.";
+        default = null;
         example = "http://127.0.0.1:1234";
       };
 
@@ -42,16 +46,58 @@ let
 
       autheliaRules = lib.mkOption {
         type = lib.types.listOf (lib.types.attrsOf lib.types.anything);
+        default = [ ];
         description = "Authelia rule configuration";
-        example = lib.literalExpression ''[{
-        policy = "two_factor";
-        subject = ["group:service_user"];
-        }]'';
+        example = lib.literalExpression ''
+          [
+            # Protect /admin endpoint with 2FA
+            # and only allow access to admin users.
+            {
+              domain = "myapp.example.com";
+              policy = "two_factor";
+              subject = [ "group:service_admin" ];
+              resources = [
+                "^/admin"
+              ];
+            }
+            # Leave /api endpoint open - assumes an API key is used to protect it.
+            {
+              domain = "myapp.example.com";
+              policy = "bypass";
+              resources = [
+                "^/api"
+              ];
+            },
+            # Protect rest of app with 1FA
+            # and allow access to normal and admin users.
+            {
+              domain = "myapp.example.com";
+              policy = "one_factor";
+              subject = ["group:service_user"];
+            },
+          ]
+        '';
+      };
+
+      phpForwardAuth = lib.mkOption {
+        type = lib.types.bool;
+        default = false;
+        description = "Authelia rule configuration";
+      };
+
+      extraConfig = lib.mkOption {
+        type = lib.types.lines;
+        default = "";
+        description = "Extra config to add to the root / location. Strings separated by newlines.";
       };
     };
   };
 in
 {
+  imports = [
+    ./authelia.nix
+  ];
+
   options.shb.nginx = {
     accessLog = lib.mkOption {
       type = lib.types.bool;
@@ -70,41 +116,44 @@ in
     vhosts = lib.mkOption {
       description = "Endpoints to be protected by authelia.";
       type = lib.types.listOf vhostConfig;
-      default = [];
+      default = [ ];
     };
   };
 
   config = {
-    networking.firewall.allowedTCPPorts = [ 80 443 ];
+    networking.firewall.allowedTCPPorts = [
+      80
+      443
+    ];
 
     services.nginx.enable = true;
     services.nginx.logError = lib.mkIf cfg.debugLog "stderr warn";
     services.nginx.appendHttpConfig = lib.mkIf cfg.accessLog ''
-        log_format apm
-          '{'
-          '"remote_addr":"$remote_addr",'
-          '"remote_user":"$remote_user",'
-          '"time_local":"$time_local",'
-          '"request":"$request",'
-          '"request_length":"$request_length",'
-          '"server_name":"$server_name",'
-          '"status":"$status",'
-          '"bytes_sent":"$bytes_sent",'
-          '"body_bytes_sent":"$body_bytes_sent",'
-          '"referrer":"$http_referrer",'
-          '"user_agent":"$http_user_agent",'
-          '"gzip_ration":"$gzip_ratio",'
-          '"post":"$request_body",'
-          '"upstream_addr":"$upstream_addr",'
-          '"upstream_status":"$upstream_status",'
-          '"request_time":"$request_time",'
-          '"upstream_response_time":"$upstream_response_time",'
-          '"upstream_connect_time":"$upstream_connect_time",'
-          '"upstream_header_time":"$upstream_header_time"'
-          '}';
+      log_format apm
+        '{'
+        '"remote_addr":"$remote_addr",'
+        '"remote_user":"$remote_user",'
+        '"time_local":"$time_local",'
+        '"request":"$request",'
+        '"request_length":"$request_length",'
+        '"server_name":"$server_name",'
+        '"status":"$status",'
+        '"bytes_sent":"$bytes_sent",'
+        '"body_bytes_sent":"$body_bytes_sent",'
+        '"referrer":"$http_referrer",'
+        '"user_agent":"$http_user_agent",'
+        '"gzip_ration":"$gzip_ratio",'
+        '"post":"$request_body",'
+        '"upstream_addr":"$upstream_addr",'
+        '"upstream_status":"$upstream_status",'
+        '"request_time":"$request_time",'
+        '"upstream_response_time":"$upstream_response_time",'
+        '"upstream_connect_time":"$upstream_connect_time",'
+        '"upstream_header_time":"$upstream_header_time"'
+        '}';
 
-        access_log syslog:server=unix:/dev/log apm;
-      '';
+      access_log syslog:server=unix:/dev/log apm;
+    '';
 
     services.nginx.virtualHosts =
       let
@@ -115,51 +164,58 @@ in
             sslCertificateKey = lib.mkIf (!(isNull c.ssl)) c.ssl.paths.key;
 
             # Taken from https://github.com/authelia/authelia/issues/178
-            locations."/".extraConfig = ''
-              add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
-              add_header X-Content-Type-Options nosniff;
-              add_header X-Frame-Options "SAMEORIGIN";
-              add_header X-XSS-Protection "1; mode=block";
-              add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";
-              add_header X-Download-Options noopen;
-              add_header X-Permitted-Cross-Domain-Policies none;
+            locations."/".extraConfig =
+              lib.optionalString (c.upstream != null) ''
+                add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
+                add_header X-Content-Type-Options nosniff;
+                add_header X-Frame-Options "SAMEORIGIN";
+                add_header X-XSS-Protection "1; mode=block";
+                add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";
+                add_header X-Download-Options noopen;
+                add_header X-Permitted-Cross-Domain-Policies none;
 
-              proxy_set_header Host $host;
-              proxy_set_header X-Real-IP $remote_addr;
-              proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-              proxy_set_header X-Forwarded-Proto $scheme;
-              proxy_http_version 1.1;
-              proxy_set_header Upgrade $http_upgrade;
-              proxy_set_header Connection "upgrade";
-              proxy_cache_bypass $http_upgrade;
+                proxy_set_header Host $host;
+                proxy_set_header X-Real-IP $remote_addr;
+                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+                proxy_set_header X-Forwarded-Proto $scheme;
+                proxy_http_version 1.1;
+                proxy_set_header Upgrade $http_upgrade;
+                proxy_set_header Connection "upgrade";
+                proxy_cache_bypass $http_upgrade;
 
-              proxy_pass ${c.upstream};
-            ''
-            + lib.optionalString (c.authEndpoint != null) ''
-              auth_request /authelia;
-              auth_request_set $user $upstream_http_remote_user;
-              auth_request_set $groups $upstream_http_remote_groups;
-              proxy_set_header X-Forwarded-User $user;
-              proxy_set_header X-Forwarded-Groups $groups;
-              # TODO: Are those needed?
-              # auth_request_set $name $upstream_http_remote_name;
-              # auth_request_set $email $upstream_http_remote_email;
-              # proxy_set_header Remote-Name $name;
-              # proxy_set_header Remote-Email $email;
-              # TODO: Would be nice to have this working, I think.
-              # set $new_cookie $http_cookie;
-              # if ($http_cookie ~ "(.*)(?:^|;)\s*example\.com\.session\.id=[^;]+(.*)") {
-              #     set $new_cookie $1$2;
-              # }
-              # proxy_set_header Cookie $new_cookie;
+                proxy_pass ${c.upstream};
+              ''
+              + c.extraConfig
+              + lib.optionalString (c.authEndpoint != null) ''
+                auth_request /authelia;
+                auth_request_set $user $upstream_http_remote_user;
+                auth_request_set $groups $upstream_http_remote_groups;
+                proxy_set_header X-Forwarded-User $user;
+                proxy_set_header X-Forwarded-Groups $groups;
+                # TODO: Are those needed?
+                # auth_request_set $name $upstream_http_remote_name;
+                # auth_request_set $email $upstream_http_remote_email;
+                # proxy_set_header Remote-Name $name;
+                # proxy_set_header Remote-Email $email;
+                # TODO: Would be nice to have this working, I think.
+                # set $new_cookie $http_cookie;
+                # if ($http_cookie ~ "(.*)(?:^|;)\s*example\.com\.session\.id=[^;]+(.*)") {
+                #     set $new_cookie $1$2;
+                # }
+                # proxy_set_header Cookie $new_cookie;
 
-              auth_request_set $redirect $scheme://$http_host$request_uri;
-              error_page 401 =302 ${c.authEndpoint}?rd=$redirect;
-              error_page 403 = ${c.authEndpoint}/error/403;
+                auth_request_set $redirect $scheme://$http_host$request_uri;
+                error_page 401 =302 ${c.authEndpoint}?rd=$redirect;
+                error_page 403 = ${c.authEndpoint}/error/403;
+              '';
+
+            locations."~ \\.php$".extraConfig = lib.mkIf (c.phpForwardAuth) ''
+              fastcgi_param HTTP_X_FORWARDED_USER   $user;
+              fastcgi_param HTTP_X_FORWARDED_GROUPS $groups;
             '';
 
             # Virtual endpoint created by nginx to forward auth requests.
-            locations."/authelia".extraConfig = lib.mkIf (!(isNull c.authEndpoint)) ''
+            locations."/authelia".extraConfig = lib.mkIf (c.authEndpoint != null) ''
               internal;
               proxy_pass ${c.authEndpoint}/api/verify;
 
@@ -181,13 +237,13 @@ in
           };
         };
       in
-        lib.mkMerge (map vhostCfg cfg.vhosts);
+      lib.mkMerge (map vhostCfg cfg.vhosts);
 
     shb.authelia.rules =
       let
         authConfig = c: map (r: r // { domain = fqdn c; }) c.autheliaRules;
       in
-        lib.flatten (map authConfig cfg.vhosts);
+      lib.flatten (map authConfig cfg.vhosts);
 
     security.acme.defaults.reloadServices = [
       "nginx.service"

modules/blocks/nginx/docs/default.md

@@ -0,0 +1,211 @@
+# Nginx Block {#blocks-nginx}
+
+Defined in [`/modules/blocks/nginx.nix`](@REPO@/modules/blocks/nginx.nix).
+
+This block sets up a [Nginx](https://nginx.org/) instance.
+
+It complements the upstream nixpkgs with some authentication and debugging improvements as shows in the Usage section.
+
+## Usage {#blocks-nginx-usage}
+
+### Access Logging {#blocks-nginx-usage-accesslog}
+
+JSON access logging is enabled with the [`shb.nginx.accessLog`](#blocks-nginx-options-shb.nginx.accessLog) option:
+
+```nix
+{
+  shb.nginx.accessLog = true;
+}
+```
+
+Looking at the systemd logs (`journalctl -fu nginx`) will show for example:
+
+```json
+nginx[969]: server nginx:
+  {
+    "remote_addr":"192.168.1.1",
+    "remote_user":"-",
+    "time_local":"29/Dec/2025:14:22:41 +0000",
+    "request":"POST /api/firstfactor HTTP/2.0",
+    "request_length":"264",
+    "server_name":"auth_example_com",
+    "status":"200",
+    "bytes_sent":"855",
+    "body_bytes_sent":"60",
+    "referrer":"-",
+    "user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0",
+    "gzip_ration":"-",
+    "post":"{\x22username\x22:\x22charlie\x22,\x22password\x22:\x22CharliePassword\x22,\x22keepMeLoggedIn\x22:false,\x22targetURL\x22:\x22https://f.example.com/\x22,\x22requestMethod\x22:null}",
+    "upstream_addr":"127.0.0.1:9091",
+    "upstream_status":"200",
+    "request_time":"0.873",
+    "upstream_response_time":"0.873",
+    "upstream_connect_time":"0.001",
+    "upstream_header_time":"0.872"
+  }
+```
+
+This _will_ log the body of POST queries so it should only be enabled for debug logging.
+
+### Debug Logging {#blocks-nginx-usage-debuglog}
+
+Debug logging is enabled with the [`shb.nginx.debugLog`](#blocks-nginx-options-shb.nginx.debugLog) option:
+
+```nix
+{
+  shb.nginx.debugLog = true;
+}
+```
+
+If enabled, it sets:
+
+```
+error_log stderr warn;
+```
+
+### Virtual Host Upstream Proxy {#blocks-nginx-usage-upstream}
+
+Easy upstream proxy setup is done with the [`shb.nginx.vhosts.*.upstream`](#blocks-nginx-options-shb.nginx.vhosts._.upstream) option:
+
+```nix
+{
+  shb.nginx.vhosts = [
+    {
+      domain = "example.com";
+      subdomain = "mysubdomain";
+      upstream = "http://127.0.0.1:9090";
+    }
+  ];
+}
+```
+
+This will set also a few headers.
+Some are shown here and others please see in the [nginx](@REPO@/modules/blocks/nginx.nix) module:
+
+- `Host` = `$host`;
+- `X-Real-IP` = `$remote_addr`;
+- `X-Forwarded-For` = `$proxy_add_x_forwarded_for`;
+- `X-Forwarded-Proto` = `$scheme`;
+
+### Virtual Host SSL Generator Contract Integration {#blocks-nginx-usage-ssl}
+
+This module integrates with the [SSL Generator Contract](./contracts-ssl.html)
+to setup HTTPs with the [`shb.nginx.vhosts.*.ssl`](#blocks-nginx-options-shb.nginx.vhosts._.ssl) option:
+
+```nix
+{
+  shb.nginx.vhosts = [
+    {
+      domain = "example.com";
+      subdomain = "mysubdomain";
+      ssl = config.shb.certs.certs.letsencrypt.${domain};;
+    }
+  ];
+
+  shb.certs.certs.letsencrypt.${domain} = {
+    inherit domain;
+  };
+}
+```
+
+### Virtual Host SHB Forward Authentication {#blocks-nginx-usage-shbforwardauth}
+
+For services provided by SelfHostBlocks that do not handle [OIDC integration][OIDC],
+this block can provide [forward authentication][] which still allows the service
+to still be protected by an SSO server.
+
+[OIDC]: blocks-authelia.html#blocks-authelia-shb-oidc
+
+The user could still be required to authenticate to the service itself,
+although some services can automatically users authorized by Authelia.
+
+[forward authentication]: https://doc.traefik.io/traefik/middlewares/http/forwardauth/
+
+Integrating with this block is done with the following code:
+
+```nix
+shb.<services>.authEndpoint = "https://${config.shb.authelia.subdomain}.${config.shb.authelia.domain}";
+```
+
+### Virtual Host Forward Authentication {#blocks-nginx-usage-forwardauth}
+
+Forward authentication is when Nginx talks with the SSO service directly
+and the user is authenticated before reaching the upstream application.
+
+The SSO service responds with the username, group and more information about the user.
+This is then forwarded to the upstream application by Nginx.
+
+Note that _every_ request is authenticated this way with the SSO server
+so it involves more hops than a direct [OIDC integration](blocks-authelia.html#blocks-authelia-shb-oidc).
+
+```nix
+{
+  shb.nginx.vhosts = [
+    {
+      domain = "example.com";
+      subdomain = "mysubdomain";
+      authEndpoint = "authelia.example.com";
+      autheliaRules = [
+        [
+          # Protect /admin endpoint with 2FA
+          # and only allow access to admin users.
+          {
+            domain = "myapp.example.com";
+            policy = "two_factor";
+            subject = [ "group:service_admin" ];
+            resources = [
+              "^/admin"
+            ];
+          }
+          # Leave /api endpoint open - assumes an API key is used to protect it.
+          {
+            domain = "myapp.example.com";
+            policy = "bypass";
+            resources = [
+              "^/api"
+            ];
+          },
+          # Protect rest of app with 1FA
+          # and allow access to normal and admin users.
+          {
+            domain = "myapp.example.com";
+            policy = "one_factor";
+            subject = ["group:service_user"];
+          },
+        ]
+      ];
+    }
+  ];
+}
+```
+
+If PHP is used with fastCGI,
+extra headers must be added by enabling the [`shb.nginx.vhosts.*.phpForwardAuth`](#blocks-nginx-options-shb.nginx.vhosts._.phpForwardAuth) option.
+
+### Virtual Host Extra Config {#blocks-nginx-usage-extraconfig}
+
+To add extra configuration to a virtual host,
+use the [`shb.nginx.vhosts.*.extraConfig`](#blocks-nginx-options-shb.nginx.vhosts._.extraConfig) option.
+This can be used to add headers, for example:
+
+```nix
+{
+  shb.nginx.vhosts = [
+    {
+      domain = "example.com";
+      subdomain = "mysubdomain";
+      extraConfig = ''
+        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
+      '';
+    }
+  ];
+}
+```
+
+## Options Reference {#blocks-nginx-options}
+
+```{=include=} options
+id-prefix: blocks-nginx-options-
+list-id: selfhostblocks-block-nginx-options
+source: @OPTIONS_JSON@
+```

modules/blocks/postgresql.nix

@@ -1,8 +1,15 @@
-{ config, lib, pkgs, ... }:
+{
+  config,
+  lib,
+  pkgs,
+  shb,
+  ...
+}:
 let
   cfg = config.shb.postgresql;
 
-  upgrade-script = old: new:
+  upgrade-script =
+    old: new:
     let
       oldStr = builtins.toString old;
       newStr = builtins.toString new;
@@ -10,37 +17,41 @@ let
       oldPkg = pkgs.${"postgresql_${oldStr}"};
       newPkg = pkgs.${"postgresql_${newStr}"};
     in
-      pkgs.writeScriptBin "upgrade-pg-cluster-${oldStr}-${newStr}" ''
-        set -eux
-        # XXX it's perhaps advisable to stop all services that depend on postgresql
-        systemctl stop postgresql
+    pkgs.writeScriptBin "upgrade-pg-cluster-${oldStr}-${newStr}" ''
+      set -eux
+      # XXX it's perhaps advisable to stop all services that depend on postgresql
+      systemctl stop postgresql
 
-        export NEWDATA="/var/lib/postgresql/${newPkg.psqlSchema}"
-        export NEWBIN="${newPkg}/bin"
+      export NEWDATA="/var/lib/postgresql/${newPkg.psqlSchema}"
+      export NEWBIN="${newPkg}/bin"
 
-        export OLDDATA="/var/lib/postgresql/${oldPkg.psqlSchema}"
-        export OLDBIN="${oldPkg}/bin"
+      export OLDDATA="/var/lib/postgresql/${oldPkg.psqlSchema}"
+      export OLDBIN="${oldPkg}/bin"
 
-        install -d -m 0700 -o postgres -g postgres "$NEWDATA"
-        cd "$NEWDATA"
-        sudo -u postgres $NEWBIN/initdb -D "$NEWDATA"
+      install -d -m 0700 -o postgres -g postgres "$NEWDATA"
+      cd "$NEWDATA"
+      sudo -u postgres $NEWBIN/initdb -D "$NEWDATA"
 
-        sudo -u postgres $NEWBIN/pg_upgrade \
-          --old-datadir "$OLDDATA" --new-datadir "$NEWDATA" \
-          --old-bindir $OLDBIN --new-bindir $NEWBIN \
-          "$@"
-      '';
+      sudo -u postgres $NEWBIN/pg_upgrade \
+        --old-datadir "$OLDDATA" --new-datadir "$NEWDATA" \
+        --old-bindir $OLDBIN --new-bindir $NEWBIN \
+        "$@"
+    '';
 in
 {
+  imports = [
+    ../../lib/module.nix
+  ];
+
   options.shb.postgresql = {
     debug = lib.mkOption {
       type = lib.types.bool;
       description = ''
-      Enable debugging options.
+        Enable debugging options.
 
-      Currently enables shared_preload_libraries = "auto_explain, pg_stat_statements"
+        Currently enables shared_preload_libraries = "auto_explain, pg_stat_statements"
 
-      See https://www.postgresql.org/docs/current/pgstatstatements.html'';
+        See https://www.postgresql.org/docs/current/pgstatstatements.html'';
       default = false;
     };
     enableTCPIP = lib.mkOption {
@@ -49,35 +60,62 @@ in
       default = false;
     };
 
+    databasebackup = lib.mkOption {
+      description = ''
+        Backup configuration.
+      '';
+
+      default = { };
+      type = lib.types.submodule {
+        options = shb.contracts.databasebackup.mkRequester {
+          user = "postgres";
+
+          backupName = "postgres.sql";
+
+          backupCmd = ''
+            ${pkgs.postgresql}/bin/pg_dumpall --clean --if-exists | ${pkgs.gzip}/bin/gzip --rsyncable
+          '';
+
+          restoreCmd = ''
+            ${pkgs.gzip}/bin/gunzip | ${pkgs.postgresql}/bin/psql postgres
+          '';
+        };
+      };
+    };
+
     ensures = lib.mkOption {
       description = "List of username, database and/or passwords that should be created.";
-      type = lib.types.listOf (lib.types.submodule {
-        options = {
-          username = lib.mkOption {
-            type = lib.types.str;
-            description = "Postgres user name.";
-          };
+      type = lib.types.listOf (
+        lib.types.submodule {
+          options = {
+            username = lib.mkOption {
+              type = lib.types.str;
+              description = "Postgres user name.";
+            };
 
-          database = lib.mkOption {
-            type = lib.types.str;
-            description = "Postgres database.";
-          };
+            database = lib.mkOption {
+              type = lib.types.str;
+              description = "Postgres database.";
+            };
 
-          passwordFile = lib.mkOption {
-            type = lib.types.nullOr lib.types.str;
-            description = "Optional password file for the postgres user. If not given, only peer auth is accepted for this user, otherwise password auth is allowed.";
-            default = null;
-            example = "/run/secrets/postgresql/password";
+            passwordFile = lib.mkOption {
+              type = lib.types.nullOr lib.types.str;
+              description = "Optional password file for the postgres user. If not given, only peer auth is accepted for this user, otherwise password auth is allowed.";
+              default = null;
+              example = "/run/secrets/postgresql/password";
+            };
           };
-        };
-      });
-      default = [];
+        }
+      );
+      default = [ ];
     };
   };
 
   config =
     let
       commonConfig = {
+        systemd.services.postgresql.serviceConfig.Restart = "always";
+
         services.postgresql.settings = {
         };
       };
@@ -97,52 +135,61 @@ in
       dbConfig = ensureCfgs: {
         services.postgresql.enable = lib.mkDefault ((builtins.length ensureCfgs) > 0);
         services.postgresql.ensureDatabases = map ({ database, ... }: database) ensureCfgs;
-        services.postgresql.ensureUsers = map ({ username, database, ... }: {
-          name = username;
-          ensureDBOwnership = true;
-          ensureClauses.login = true;
-        }) ensureCfgs;
+        services.postgresql.ensureUsers = map (
+          { username, database, ... }:
+          {
+            name = username;
+            ensureDBOwnership = true;
+            ensureClauses.login = true;
+          }
+        ) ensureCfgs;
       };
 
       pwdConfig = ensureCfgs: {
-        systemd.services.postgresql.postStart =
+        systemd.services.postgresql-setup.script = lib.mkAfter (
           let
             prefix = ''
-            $PSQL -tA <<'EOF'
-              DO $$
-              DECLARE password TEXT;
-              BEGIN
+              psql -tA <<'EOF'
+                DO $$
+                DECLARE password TEXT;
+                BEGIN
             '';
             suffix = ''
-              END $$;
-            EOF
-            '';
-            exec = { username, passwordFile, ... }: ''
-            password := trim(both from replace(pg_read_file('${passwordFile}'), E'\n', '''));
-            EXECUTE format('ALTER ROLE ${username} WITH PASSWORD '''%s''';', password);
+                END $$;
+              EOF
             '';
+            exec =
+              { username, passwordFile, ... }:
+              ''
+                password := trim(both from replace(pg_read_file('${passwordFile}'), E'\n', '''));
+                EXECUTE format('ALTER ROLE "${username}" WITH PASSWORD '''%s''';', password);
+              '';
             cfgsWithPasswords = builtins.filter (cfg: cfg.passwordFile != null) ensureCfgs;
           in
-            if (builtins.length cfgsWithPasswords) == 0 then "" else
-              prefix + (lib.concatStrings (map exec cfgsWithPasswords)) + suffix;
+          if (builtins.length cfgsWithPasswords) == 0 then
+            ""
+          else
+            prefix + (lib.concatStrings (map exec cfgsWithPasswords)) + suffix
+        );
       };
 
-      debugConfig = enableDebug: lib.mkIf enableDebug {
-        services.postgresql.settings.shared_preload_libraries = "auto_explain, pg_stat_statements";
-      };
+      debugConfig =
+        enableDebug:
+        lib.mkIf enableDebug {
+          services.postgresql.settings.shared_preload_libraries = "auto_explain, pg_stat_statements";
+        };
     in
-      lib.mkMerge ([
-        commonConfig
-        (dbConfig cfg.ensures)
-        (pwdConfig cfg.ensures)
-        (lib.mkIf cfg.enableTCPIP tcpConfig)
-        (debugConfig cfg.debug)
-        {
-          environment.systemPackages = lib.mkIf config.services.postgresql.enable [
-            (upgrade-script 13 14)
-            (upgrade-script 14 15)
-            (upgrade-script 15 16)
-          ];
-        }
-      ]);
+    lib.mkMerge ([
+      commonConfig
+      (dbConfig cfg.ensures)
+      (pwdConfig cfg.ensures)
+      (lib.mkIf cfg.enableTCPIP tcpConfig)
+      (debugConfig cfg.debug)
+      {
+        environment.systemPackages = lib.mkIf config.services.postgresql.enable [
+          (upgrade-script 15 16)
+          (upgrade-script 16 17)
+        ];
+      }
+    ]);
 }

modules/blocks/postgresql/docs/default.md

@@ -0,0 +1,72 @@
+# PostgreSQL Block {#blocks-postgresql}
+
+Defined in [`/modules/blocks/postgresql.nix`](@REPO@/modules/blocks/postgresql.nix).
+
+This block sets up a [PostgreSQL][] database.
+
+[postgresql]: https://www.postgresql.org/
+
+Compared to the upstream nixpkgs module, this module also sets up:
+
+- Enabling TCP/IP login and also accepting password authentication from localhost with [`shb.postgresql.enableTCPIP`](#blocks-postgresql-options-shb.postgresql.enableTCPIP).
+- Enhance the `ensure*` upstream option by setting up a database's password from a password file with [`shb.postgresql.ensures`](#blocks-postgresql-options-shb.postgresql.ensures).
+- Debug logging with `auto_explain` and `pg_stat_statements` with [`shb.postgresql.debug`](#blocks-postgresql-options-shb.postgresql.debug).
+
+## Usage {#blocks-postgresql-usage}
+
+### Ensure User and Database {#blocks-postgresql-ensures}
+
+Ensure a database and user exists:
+
+```nix
+shb.postgresql.ensures = [
+  {
+    username = "firefly-iii";
+    database = "firefly-iii";
+  }
+];
+```
+
+Also set up the database password from a file path:
+
+```nix
+shb.postgresql.ensures = [
+  {
+    username = "firefly-iii";
+    database = "firefly-iii";
+    passwordFile = "/run/secrets/firefly-iii_db_password";
+  }
+];
+```
+
+### Database Backup Requester Contracts {#blocks-postgresql-contract-databasebackup}
+
+This block can be backed up using the [database backup](contracts-databasebackup.html) contract.
+
+Contract integration tests are defined in [`/test/contracts/databasebackup.nix`](@REPO@/test/contracts/databasebackup.nix).
+
+#### Backing up All Databases {#blocks-postgresql-contract-databasebackup-all}
+
+```nix
+{
+  my.backup.provider."postgresql" = {
+    request = config.shb.postgresql.databasebackup;
+
+    settings = {
+      // Specific options for the backup provider.
+    };
+  };
+}
+```
+
+## Tests {#blocks-postgresql-tests}
+
+Specific integration tests are defined in [`/test/blocks/postgresql.nix`](@REPO@/test/blocks/postgresql.nix).
+
+## Options Reference {#blocks-postgresql-options}
+
+```{=include=} options
+id-prefix: blocks-postgresql-options-
+list-id: selfhostblocks-block-postgresql-options
+source: @OPTIONS_JSON@
+```

modules/blocks/restic.nix

@@ -0,0 +1,484 @@
+{
+  config,
+  pkgs,
+  lib,
+  shb,
+  utils,
+  ...
+}:
+
+let
+  cfg = config.shb.restic;
+
+  inherit (lib)
+    concatStringsSep
+    filterAttrs
+    flatten
+    literalExpression
+    optionals
+    listToAttrs
+    mapAttrsToList
+    mkEnableOption
+    mkOption
+    mkMerge
+    ;
+  inherit (lib)
+    hasPrefix
+    mkIf
+    nameValuePair
+    optionalAttrs
+    removePrefix
+    ;
+  inherit (lib.types)
+    attrsOf
+    enum
+    int
+    ints
+    oneOf
+    nonEmptyStr
+    nullOr
+    str
+    submodule
+    ;
+
+  commonOptions =
+    {
+      name,
+      prefix,
+      config,
+      ...
+    }:
+    {
+      enable = mkEnableOption ''
+        SelfHostBlocks' Restic block
+
+        A disabled instance will not backup data anymore
+        but still provides the helper tool to restore snapshots
+      '';
+
+      passphrase = lib.mkOption {
+        description = "Encryption key for the backup repository.";
+        type = lib.types.submodule {
+          options = shb.contracts.secret.mkRequester {
+            mode = "0400";
+            owner = config.request.user;
+            ownerText = "[shb.restic.${prefix}.<name>.request.user](#blocks-restic-options-shb.restic.${prefix}._name_.request.user)";
+            restartUnits = [ "${fullName name config.settings.repository}.service" ];
+            restartUnitsText = "[ [shb.restic.${prefix}.<name>.settings.repository](#blocks-restic-options-shb.restic.${prefix}._name_.settings.repository) ]";
+          };
+        };
+      };
+
+      repository = mkOption {
+        description = "Repositories to back this instance to.";
+        type = submodule {
+          options = {
+            path = mkOption {
+              type = str;
+              description = "Repository location";
+            };
+
+            secrets = mkOption {
+              type = attrsOf shb.secretFileType;
+              default = { };
+              description = ''
+                Secrets needed to access the repository where the backups will be stored.
+
+                See [s3 config](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3) for an example
+                and [list](https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables) for the list of all secrets.
+
+              '';
+              example = literalExpression ''
+                {
+                  AWS_ACCESS_KEY_ID.source = <path/to/secret>;
+                  AWS_SECRET_ACCESS_KEY.source = <path/to/secret>;
+                }
+              '';
+            };
+
+            timerConfig = mkOption {
+              type = attrsOf utils.systemdUtils.unitOptions.unitOption;
+              default = {
+                OnCalendar = "daily";
+                Persistent = true;
+              };
+              description = "When to run the backup. See {manpage}`systemd.timer(5)` for details.";
+              example = {
+                OnCalendar = "00:05";
+                RandomizedDelaySec = "5h";
+                Persistent = true;
+              };
+            };
+          };
+        };
+      };
+
+      retention = mkOption {
+        description = "For how long to keep backup files.";
+        type = attrsOf (oneOf [
+          int
+          nonEmptyStr
+        ]);
+        default = {
+          keep_within = "1d";
+          keep_hourly = 24;
+          keep_daily = 7;
+          keep_weekly = 4;
+          keep_monthly = 6;
+        };
+      };
+
+      limitUploadKiBs = mkOption {
+        type = nullOr int;
+        description = "Limit upload bandwidth to the given KiB/s amount.";
+        default = null;
+        example = 8000;
+      };
+
+      limitDownloadKiBs = mkOption {
+        type = nullOr int;
+        description = "Limit download bandwidth to the given KiB/s amount.";
+        default = null;
+        example = 8000;
+      };
+    };
+
+  repoSlugName = name: builtins.replaceStrings [ "/" ":" ] [ "_" "_" ] (removePrefix "/" name);
+  fullName = name: repository: "restic-backups-${name}_${repoSlugName repository.path}";
+in
+{
+  imports = [
+    ../../lib/module.nix
+    ../blocks/monitoring.nix
+  ];
+
+  options.shb.restic = {
+    enableDashboard = lib.mkEnableOption "the Backups SHB dashboard" // {
+      default = true;
+    };
+
+    instances = mkOption {
+      description = "Files to backup following the [backup contract](./shb.contracts-backup.html).";
+      default = { };
+      type = attrsOf (
+        submodule (
+          { name, config, ... }:
+          {
+            options = shb.contracts.backup.mkProvider {
+              settings = mkOption {
+                description = ''
+                  Settings specific to the Restic provider.
+                '';
+
+                type = submodule {
+                  options = commonOptions {
+                    inherit name config;
+                    prefix = "instances";
+                  };
+                };
+              };
+
+              resultCfg = {
+                restoreScript = fullName name config.settings.repository;
+                restoreScriptText = "${fullName "<name>" { path = "path/to/repository"; }}";
+
+                backupService = "${fullName name config.settings.repository}.service";
+                backupServiceText = "${fullName "<name>" { path = "path/to/repository"; }}.service";
+              };
+            };
+          }
+        )
+      );
+    };
+
+    databases = mkOption {
+      description = "Databases to backup following the [database backup contract](./shb.contracts-databasebackup.html).";
+      default = { };
+      type = attrsOf (
+        submodule (
+          { name, config, ... }:
+          {
+            options = shb.contracts.databasebackup.mkProvider {
+              settings = mkOption {
+                description = ''
+                  Settings specific to the Restic provider.
+                '';
+
+                type = submodule {
+                  options = commonOptions {
+                    inherit name config;
+                    prefix = "databases";
+                  };
+                };
+              };
+
+              resultCfg = {
+                restoreScript = fullName name config.settings.repository;
+                restoreScriptText = "${fullName "<name>" { path = "path/to/repository"; }}";
+
+                backupService = "${fullName name config.settings.repository}.service";
+                backupServiceText = "${fullName "<name>" { path = "path/to/repository"; }}.service";
+              };
+            };
+          }
+        )
+      );
+    };
+
+    # Taken from https://github.com/HubbeKing/restic-kubernetes/blob/73bfbdb0ba76939a4c52173fa2dbd52070710008/README.md?plain=1#L23
+    performance = mkOption {
+      description = "Reduce performance impact of backup jobs.";
+      default = { };
+      type = submodule {
+        options = {
+          niceness = mkOption {
+            type = ints.between (-20) 19;
+            description = "nice priority adjustment, defaults to 15 for ~20% CPU time of normal-priority process";
+            default = 15;
+          };
+          ioSchedulingClass = mkOption {
+            type = enum [
+              "idle"
+              "best-effort"
+              "realtime"
+            ];
+            description = "ionice scheduling class, defaults to best-effort IO. Only used for `restic backup`, `restic forget` and `restic check` commands.";
+            default = "best-effort";
+          };
+          ioPriority = mkOption {
+            type = nullOr (ints.between 0 7);
+            description = "ionice priority, defaults to 7 for lowest priority IO. Only used for `restic backup`, `restic forget` and `restic check` commands.";
+            default = 7;
+          };
+        };
+      };
+    };
+  };
+
+  config = mkIf (cfg.instances != { } || cfg.databases != { }) (
+    let
+      enabledInstances = filterAttrs (k: i: i.settings.enable) cfg.instances;
+      enabledDatabases = filterAttrs (k: i: i.settings.enable) cfg.databases;
+    in
+    mkMerge [
+      {
+        environment.systemPackages = optionals (enabledInstances != { } || enabledDatabases != { }) [
+          pkgs.restic
+        ];
+      }
+      {
+        # Create repository if it is a local path.
+        systemd.tmpfiles.rules =
+          let
+            mkSettings =
+              name: instance:
+              optionals (hasPrefix "/" instance.settings.repository.path) [
+                "d '${instance.settings.repository.path}' 0750 ${instance.request.user} root - -"
+              ];
+          in
+          flatten (mapAttrsToList mkSettings (cfg.instances // cfg.databases));
+      }
+      {
+        services.restic.backups =
+          let
+            mkSettings = name: instance: {
+              "${name}_${repoSlugName instance.settings.repository.path}" = {
+                inherit (instance.request) user;
+
+                repository = instance.settings.repository.path;
+
+                paths = instance.request.sourceDirectories;
+
+                passwordFile = toString instance.settings.passphrase.result.path;
+
+                initialize = true;
+
+                inherit (instance.settings.repository) timerConfig;
+
+                pruneOpts = mapAttrsToList (
+                  name: value: "--${builtins.replaceStrings [ "_" ] [ "-" ] name} ${builtins.toString value}"
+                ) instance.settings.retention;
+
+                backupPrepareCommand = concatStringsSep "\n" instance.request.hooks.beforeBackup;
+
+                backupCleanupCommand = concatStringsSep "\n" instance.request.hooks.afterBackup;
+
+                extraBackupArgs =
+                  (optionals (instance.settings.limitUploadKiBs != null) [
+                    "--limit-upload=${toString instance.settings.limitUploadKiBs}"
+                  ])
+                  ++ (optionals (instance.settings.limitDownloadKiBs != null) [
+                    "--limit-download=${toString instance.settings.limitDownloadKiBs}"
+                  ]);
+              }
+              // optionalAttrs (builtins.length instance.request.excludePatterns > 0) {
+                exclude = instance.request.excludePatterns;
+              };
+            };
+          in
+          mkMerge (flatten (mapAttrsToList mkSettings enabledInstances));
+      }
+      {
+        services.restic.backups =
+          let
+            mkSettings = name: instance: {
+              "${name}_${repoSlugName instance.settings.repository.path}" = {
+                inherit (instance.request) user;
+
+                repository = instance.settings.repository.path;
+
+                dynamicFilesFrom = "echo";
+
+                passwordFile = toString instance.settings.passphrase.result.path;
+
+                initialize = true;
+
+                inherit (instance.settings.repository) timerConfig;
+
+                pruneOpts = mapAttrsToList (
+                  name: value: "--${builtins.replaceStrings [ "_" ] [ "-" ] name} ${builtins.toString value}"
+                ) instance.settings.retention;
+
+                extraBackupArgs =
+                  (optionals (instance.settings.limitUploadKiBs != null) [
+                    "--limit-upload=${toString instance.settings.limitUploadKiBs}"
+                  ])
+                  ++ (optionals (instance.settings.limitDownloadKiBs != null) [
+                    "--limit-download=${toString instance.settings.limitDownloadKiBs}"
+                  ])
+                  ++ (
+                    let
+                      cmd = pkgs.writeShellScriptBin "dump.sh" instance.request.backupCmd;
+                    in
+                    [
+                      "--stdin-filename ${instance.request.backupName} --stdin-from-command -- ${cmd}/bin/dump.sh"
+                    ]
+                  );
+              };
+            };
+          in
+          mkMerge (flatten (mapAttrsToList mkSettings enabledDatabases));
+      }
+      {
+        systemd.services =
+          let
+            mkSettings =
+              name: instance:
+              let
+                serviceName = fullName name instance.settings.repository;
+              in
+              {
+                ${serviceName} = mkMerge [
+                  {
+                    serviceConfig = {
+                      Nice = cfg.performance.niceness;
+                      IOSchedulingClass = cfg.performance.ioSchedulingClass;
+                      IOSchedulingPriority = cfg.performance.ioPriority;
+                      # BindReadOnlyPaths = instance.sourceDirectories;
+                    };
+                  }
+                  (optionalAttrs (instance.settings.repository.secrets != { }) {
+                    serviceConfig.EnvironmentFile = [
+                      "/run/secrets_restic/${serviceName}"
+                    ];
+                    after = [ "${serviceName}-pre.service" ];
+                    requires = [ "${serviceName}-pre.service" ];
+                  })
+                ];
+
+                "${serviceName}-pre" = mkIf (instance.settings.repository.secrets != { }) (
+                  let
+                    script = shb.genConfigOutOfBandSystemd {
+                      config = instance.settings.repository.secrets;
+                      configLocation = "/run/secrets_restic/${serviceName}";
+                      generator = shb.toEnvVar;
+                      user = instance.request.user;
+                    };
+                  in
+                  {
+                    script = script.preStart;
+                    serviceConfig.Type = "oneshot";
+                    serviceConfig.LoadCredential = script.loadCredentials;
+                  }
+                );
+              };
+          in
+          mkMerge (flatten (mapAttrsToList mkSettings (enabledInstances // enabledDatabases)));
+      }
+      {
+        systemd.services =
+          let
+            mkEnv =
+              name: instance:
+              nameValuePair "${fullName name instance.settings.repository}_restore_gen" {
+                enable = true;
+                wantedBy = [ "multi-user.target" ];
+                # Purposely not a oneshot systemd service otherwise
+                # the service waits on the completion the backup before deactivating.
+                # This seems like a nice property at first but there is one annoying
+                # edge case when deploying. If a backup job somehow is started when
+                # the deploy happens, the deploy will wait on the service to finish
+                # before considering the deploy done. And worse, it will consider the
+                # deploy as failed if the backup fails, which is not what you want.
+                script = (
+                  shb.replaceSecrets {
+                    userConfig = instance.settings.repository.secrets // {
+                      RESTIC_PASSWORD_FILE = toString instance.settings.passphrase.result.path;
+                      RESTIC_REPOSITORY = instance.settings.repository.path;
+                    };
+                    resultPath = "/run/secrets_restic_env/${fullName name instance.settings.repository}";
+                    generator = shb.toEnvVar;
+                    user = instance.request.user;
+                  }
+                );
+              };
+          in
+          listToAttrs (flatten (mapAttrsToList mkEnv (cfg.instances // cfg.databases)));
+      }
+      {
+        environment.systemPackages =
+          let
+            mkResticBinary =
+              name: instance:
+              shb.contracts.backup.mkRestoreScript {
+                name = fullName name instance.settings.repository;
+                user = instance.request.user;
+                sudoPreserveEnvs = [
+                  "RESTIC_REPOSITORY"
+                  "RESTIC_PASSWORD_FILE"
+                ];
+                secretsFile = "/run/secrets_restic_env/${fullName name instance.settings.repository}";
+                restoreCmd = ''${pkgs.restic}/bin/restic restore \"$snapshot\" --target /'';
+                listCmd = ''if [ -e \"$RESTIC_REPOSITORY/index\" ]; then ${pkgs.restic}/bin/restic snapshots --json | ${pkgs.jq}/bin/jq '.[].id'; fi'';
+              };
+          in
+          flatten (mapAttrsToList mkResticBinary cfg.instances);
+      }
+      {
+        environment.systemPackages =
+          let
+            mkResticBinary =
+              name: instance:
+              shb.contracts.backup.mkRestoreScript {
+                name = fullName name instance.settings.repository;
+                user = instance.request.user;
+                sudoPreserveEnvs = [
+                  "RESTIC_REPOSITORY"
+                  "RESTIC_PASSWORD_FILE"
+                ];
+                secretsFile = "/run/secrets_restic_env/${fullName name instance.settings.repository}";
+                restoreCmd = ''${pkgs.restic}/bin/restic dump \"$snapshot\" ${instance.request.backupName} | ${instance.request.restoreCmd}'';
+                listCmd = ''if [ -e \"$RESTIC_REPOSITORY/index\" ]; then ${pkgs.restic}/bin/restic snapshots --json | ${pkgs.jq}/bin/jq '.[].id'; fi'';
+              };
+          in
+          flatten (mapAttrsToList mkResticBinary cfg.databases);
+      }
+
+      (lib.mkIf (cfg.enableDashboard && (cfg.instances != { } || cfg.databases != { })) {
+        shb.monitoring.dashboards = [
+          ./backup/dashboard/Backups.json
+        ];
+      })
+    ]
+  );
+}

modules/blocks/restic/docs/default.md

@@ -0,0 +1,305 @@
+# Restic Block {#blocks-restic}
+
+Defined in [`/modules/blocks/restic.nix`](@REPO@/modules/blocks/restic.nix).
+
+This block sets up a backup job using [Restic][].
+It is heavily based on the nixpkgs Restic module.
+
+[restic]: https://restic.net/
+
+## Provider Contracts {#blocks-restic-contract-provider}
+
+This block provides the following contracts:
+
+- [backup contract](contracts-backup.html) under the [`shb.restic.instances`][instances] option.
+  It is tested with [contract tests][backup contract tests].
+- [database backup contract](contracts-databasebackup.html) under the [`shb.restic.databases`][databases] option.
+  It is tested with [contract tests][database backup contract tests].
+
+[instances]: #blocks-restic-options-shb.restic.instances
+[databases]: #blocks-restic-options-shb.restic.databases
+[backup contract tests]: @REPO@/test/contracts/backup.nix
+[database backup contract tests]: @REPO@/test/contracts/databasebackup.nix
+
+As requested by those two contracts, when setting up a backup with Restic,
+a backup Systemd service and a [restore script](#blocks-restic-maintenance) are provided.
+
+## Usage {#blocks-restic-usage}
+
+The following examples assume usage of the [sops block][] to provide secrets
+although any blocks providing the [secrets contract][] works too.
+
+[sops block]: ./blocks-sops.html
+[secrets contract]: ./contracts-secrets.html
+
+### One folder backed up manually {#blocks-restic-usage-provider-manual}
+
+The following snippet shows how to configure
+the backup of 1 folder to 1 repository.
+We assume that the folder `/var/lib/myfolder` of the service `myservice` must be backed up.
+
+```nix
+shb.restic.instances."myservice" = {
+  request = {
+    user = "myservice";
+
+    sourceDirectories = [
+      "/var/lib/myfolder"
+    ];
+  };
+
+  settings = {
+    enable = true;
+
+    passphrase.result = config.shb.sops.secret."passphrase".result;
+
+    repository = {
+      path = "/srv/backups/myservice";
+      timerConfig = {
+        OnCalendar = "00:00:00";
+        RandomizedDelaySec = "3h";
+      };
+    };
+
+    retention = {
+      keep_within = "1d";
+      keep_hourly = 24;
+      keep_daily = 7;
+      keep_weekly = 4;
+      keep_monthly = 6;
+    };
+  };
+};
+
+shb.sops.secret."passphrase".request =
+  config.shb.restic.instances."myservice".settings.passphrase.request;
+```
+
+### One folder backed up with contract {#blocks-restic-usage-provider-contract}
+
+With the same example as before but assuming the `myservice` service
+has a `myservice.backup` option that is a requester for the backup contract,
+the snippet above becomes:
+
+```nix
+shb.restic.instances."myservice" = {
+  request = config.myservice.backup.request;
+
+  settings = {
+    enable = true;
+
+    passphrase.result = config.shb.sops.secret."passphrase".result;
+
+    repository = {
+      path = "/srv/backups/myservice";
+      timerConfig = {
+        OnCalendar = "00:00:00";
+        RandomizedDelaySec = "3h";
+      };
+    };
+
+    retention = {
+      keep_within = "1d";
+      keep_hourly = 24;
+      keep_daily = 7;
+      keep_weekly = 4;
+      keep_monthly = 6;
+    };
+  };
+};
+
+shb.sops.secret."passphrase".request =
+  config.shb.restic.instances."myservice".settings.passphrase.request;
+```
+
+### One folder backed up to S3 {#blocks-restic-usage-provider-remote}
+
+Here we will only highlight the differences with the previous configuration.
+
+This assumes you have access to such a remote S3 store, for example by using [Backblaze](https://www.backblaze.com/).
+
+```diff
+  shb.test.backup.instances.myservice = {
+
+    repository = {
+-     path = "/srv/pool1/backups/myfolder";
++     path = "s3:s3.us-west-000.backblazeb2.com/backups/myfolder";
+      timerConfig = {
+        OnCalendar = "00:00:00";
+        RandomizedDelaySec = "3h";
+      };
+
++     extraSecrets = {
++       AWS_ACCESS_KEY_ID.source="<path/to/access_key_id>";
++       AWS_SECRET_ACCESS_KEY.source="<path/to/secret_access_key>";
++     };
+    };
+  }
+```
+
+### Multiple directories to multiple destinations {#blocks-restic-usage-multiple}
+
+The following snippet shows how to configure backup of any number of folders to 3 repositories,
+each happening at different times to avoid I/O contention.
+
+We will also make sure to be able to re-use as much as the configuration as possible.
+
+A few assumptions:
+- 2 hard drive pools used for backup are mounted respectively on `/srv/pool1` and `/srv/pool2`.
+- You have a backblaze account.
+
+First, let's define a variable to hold all the repositories we want to back up to:
+
+```nix
+repos = [
+  {
+    path = "/srv/pool1/backups";
+    timerConfig = {
+      OnCalendar = "00:00:00";
+      RandomizedDelaySec = "3h";
+    };
+  }
+  {
+    path = "/srv/pool2/backups";
+    timerConfig = {
+      OnCalendar = "08:00:00";
+      RandomizedDelaySec = "3h";
+    };
+  }
+  {
+    path = "s3:s3.us-west-000.backblazeb2.com/backups";
+    timerConfig = {
+      OnCalendar = "16:00:00";
+      RandomizedDelaySec = "3h";
+    };
+  }
+];
+```
+
+Compared to the previous examples, we do not include the name of what we will back up in the
+repository paths.
+
+Now, let's define a function to create a backup configuration. It will take a list of repositories,
+a name identifying the backup and a list of folders to back up.
+
+```nix
+backupcfg = repositories: name: sourceDirectories {
+  enable = true;
+
+  backend = "restic";
+
+  keySopsFile = ../secrets/backup.yaml;
+
+  repositories = builtins.map (r: {
+    path = "${r.path}/${name}";
+    inherit (r) timerConfig;
+  }) repositories;
+
+  inherit sourceDirectories;
+
+  retention = {
+    keep_within = "1d";
+    keep_hourly = 24;
+    keep_daily = 7;
+    keep_weekly = 4;
+    keep_monthly = 6;
+  };
+
+  environmentFile = true;
+};
+```
+
+Now, we can define multiple backup jobs to backup different folders:
+
+```nix
+shb.test.backup.instances.myfolder1 = backupcfg repos ["/var/lib/myfolder1"];
+shb.test.backup.instances.myfolder2 = backupcfg repos ["/var/lib/myfolder2"];
+```
+
+The difference between the above snippet and putting all the folders into one configuration (shown
+below) is the former splits the backups into sub-folders on the repositories.
+
+```nix
+shb.test.backup.instances.all = backupcfg repos ["/var/lib/myfolder1" "/var/lib/myfolder2"];
+```
+
+## Monitoring {#blocks-restic-monitoring}
+
+A generic dashboard for all backup solutions is provided.
+See [Backups Dashboard and Alert](blocks-monitoring.html#blocks-monitoring-backup) section in the monitoring chapter.
+
+## Maintenance {#blocks-restic-maintenance}
+
+### Manual Backup {#blocks-restic-maintenance-manuql}
+
+To launch a backup manually, just run:
+
+```bash
+systemctl start <systemd-service-name>
+```
+
+You can easily discover the systemd service name you need by either listing the units:
+
+```bash
+systemctl list-units 'restic*'
+```
+
+Or by autocompleting the unit name with `<TAB>`:
+
+```bash
+systemctl start restic<TAB><TAB>
+```
+
+Note that the systemd services are of `Type=simple` which means the systemd service
+will not wait for the backup completion to terminate.
+If you want instead to wait for the backup to complete, use the `--wait` flag:
+
+```bash
+systemctl start --wait <systemd-service-name>
+```
+
+### Restore {#blocks-restic-maintenance-restore}
+
+One command-line helper is provided per backup instance and repository pair which allows to:
+
+- list snapshots: `<script> snapshots`
+- to restore a snapshot: `<script> restore <snapshot>`
+
+The restore script has all the secrets needed to access the repo,
+it will run `sudo` automatically
+and the user running it needs to have correct permissions for privilege escalation.
+
+In the [multiple directories example](#blocks-restic-usage-multiple) above, the following 6 helpers are provided in the `$PATH`:
+
+```bash
+restic-myfolder1_srv_pool1_backups
+restic-myfolder1_srv_pool2_backups
+restic-myfolder1_s3_s3.us-west-000.backblazeb2.com_backups
+restic-myfolder2_srv_pool1_backups
+restic-myfolder2_srv_pool2_backups
+restic-myfolder2_s3_s3.us-west-000.backblazeb2.com_backups
+```
+
+Discovering those is easy thanks to tab-completion.
+
+One can then restore a backup from a given repository with:
+
+```bash
+restic-myfolder1_srv_pool1_backups restore latest
+```
+
+### Troubleshooting {#blocks-restic-maintenance-troubleshooting}
+
+In case something bad happens with a backup, the [official documentation](https://restic.readthedocs.io/en/stable/077_troubleshooting.html) has a lot of tips.
+
+## Tests {#blocks-restic-tests}
+
+Specific integration tests are defined in [`/test/blocks/restic.nix`](@REPO@/test/blocks/restic.nix).
+
+## Options Reference {#blocks-restic-options}
+
+```{=include=} options
+id-prefix: blocks-restic-options-
+list-id: selfhostblocks-block-restic-options
+source: @OPTIONS_JSON@
+```

modules/blocks/restic/dummyModule.nix

@@ -0,0 +1,3 @@
+{ lib, ... }:
+{
+}

modules/blocks/sanoid.nix

@@ -0,0 +1,164 @@
+{
+  config,
+  lib,
+  pkgs,
+  shb,
+  utils,
+  ...
+}:
+let
+  cfg = config.shb.sanoid;
+
+  restoreScriptName = name: "sanoid-${utils.escapeSystemdPath name}-restore";
+  backupScriptBase = "sanoid";
+
+  restoreScript =
+    name:
+    pkgs.writers.writePython3Bin (restoreScriptName name)
+      {
+        flakeIgnore = [ "E501" ];
+      }
+      ''
+        import argparse
+        import subprocess
+        import sys
+
+
+        dataset = "${name}"
+
+
+        class ZFSError(Exception):
+            pass
+
+
+        def run_command(cmd: list[str]) -> str:
+            try:
+                result = subprocess.run(
+                    cmd,
+                    check=True,
+                    text=True,
+                    capture_output=True,
+                )
+                return result.stdout.strip()
+            except subprocess.CalledProcessError as e:
+                raise ZFSError(
+                    f"Command failed: {' '.join(cmd)}\n"
+                    f"Exit code: {e.returncode}\n"
+                    f"stderr: {e.stderr.strip()}"
+                ) from e
+
+
+        def list_snapshots() -> None:
+            """List all ZFS snapshots."""
+            output = run_command(["zfs", "list", "-H", "-t", "snapshot", dataset])
+            if not output:
+                return []
+            return output.splitlines()
+
+
+        def restore_snapshot(snapshot: str) -> None:
+            """Rollback to a given snapshot."""
+            if not snapshot:
+                raise ValueError("Snapshot name must not be empty")
+
+            print(f"Rolling back to snapshot: {snapshot}")
+            run_command(["zfs", "rollback", "-r", snapshot])
+            print("Rollback completed successfully.")
+
+
+        def build_parser() -> argparse.ArgumentParser:
+            parser = argparse.ArgumentParser(description=f"Restore script for {dataset}")
+
+            subparsers = parser.add_subparsers(dest="command", required=True)
+
+            subparsers.add_parser(
+                "snapshots",
+                help="List all ZFS snapshots",
+            )
+
+            restore_parser = subparsers.add_parser(
+                "restore",
+                help="Rollback to a specific snapshot",
+            )
+            restore_parser.add_argument(
+                "snapshot",
+                help="Snapshot name (e.g. pool/dataset@snapname)",
+            )
+
+            return parser
+
+
+        def main():
+            parser = build_parser()
+            args = parser.parse_args()
+
+            try:
+                if args.command == "snapshots":
+                    snapshots = list_snapshots()
+                    for s in snapshots:
+                        print(s)
+                elif args.command == "restore":
+                    restore_snapshot(args.snapshot)
+                else:
+                    parser.print_help()
+                    sys.exit(1)
+
+            except ZFSError as e:
+                print(f"ERROR: {e}", file=sys.stderr)
+                sys.exit(1)
+            except Exception as e:
+                print(f"Unexpected error: {e}", file=sys.stderr)
+                sys.exit(1)
+
+
+        if __name__ == "__main__":
+            main()
+      '';
+in
+{
+  imports = [
+    ../../lib/module.nix
+  ];
+
+  options.shb.sanoid.backup = lib.mkOption {
+    description = "Sanoid prodiver for file backup contract";
+    default = { };
+    type = lib.types.attrsOf (
+      lib.types.submodule (
+        { name, ... }:
+        {
+          options = shb.contracts.backup.mkProvider {
+            resultCfg = {
+              restoreScript = restoreScriptName name;
+              restoreScriptText = restoreScriptName "<name>";
+
+              backupService = "${backupScriptBase}.service";
+              backupServiceText = "${backupScriptBase}.service";
+            };
+
+            settings = lib.mkOption {
+              description = "Options passed to the `services.sanoid.datasets.<name>` option.";
+              default = { };
+              type = lib.types.attrsOf lib.types.anything;
+            };
+          };
+        }
+      )
+    );
+  };
+
+  config = lib.mkIf (cfg.backup != { }) {
+    services.sanoid.enable = true;
+
+    services.sanoid.datasets =
+      let
+        mkDataset = name: cfg': {
+          inherit name;
+          value = cfg'.settings;
+        };
+      in
+      lib.mapAttrs' mkDataset cfg.backup;
+
+    environment.systemPackages = map restoreScript (lib.attrNames cfg.backup);
+  };
+}

modules/blocks/sanoid/docs/default.md

@@ -0,0 +1,97 @@
+# Sanoid Block {#blocks-sanoid}
+
+Defined in [`/modules/blocks/sanoid.nix`](@REPO@/modules/blocks/sanoid.nix):
+
+```nix
+{
+  imports = [
+    inputs.selfhostblocks.nixosModules.sanoid
+  ];
+}
+```
+
+## Provider Contracts {#blocks-sanoid-contract-provider}
+
+This block provides the following contracts:
+
+- [dataset backup contract](contracts-datasetbackup.html) under the [`shb.sanoid.backup`][backup] option.
+  It is tested with the [generic contract tests][backup contract tests].
+
+[backup]: #blocks-sanoid-options-shb.sanoid.backup
+[backup contract tests]: @REPO@/test/contracts/backup.nix
+
+## Usage {#blocks-sanoid-usage}
+
+Sanoid uses templates to know when snapshots should be kept or pruned.
+
+### Default Template {#blocks-sanoid-usage-default-template}
+
+Backup a dataset using the default Sanoid template:
+
+```nix
+{
+  shb.zfs.pools.root.datasets.home = {
+    path = "/home";
+  };
+
+  shb.sanoid.backup."root/home" = {
+    request = shb.zfs.pools.root.datasets.home.datasetBackup.request;
+  };
+}
+```
+
+This uses the dataset backup contract which is exposed through the ZFS module's [`shb.zfs.pools.<name>.datasets.<name>.datasetBackup`](blocks-zfs.html#blocks-zfs-options-shb.zfs.pools._name_.datasets._name_.datasetbackup) option.
+
+### Custom Template {#blocks-sanoid-usage-custom-template}
+
+Create a custom template and use it:
+
+```nix
+{
+  shb.zfs.pools.root.datasets.home = {
+    path = "/home";
+  };
+
+  services.sanoid.templates."yearly" = {
+    hourly = 10;
+    daily = 3;
+    monthly = 3;
+    yearly = 2;
+  };
+
+  shb.sanoid.backup."root/home" = {
+    request = shb.zfs.pools.root.datasets.home.datasetBackup.request;
+    template = "yearly";
+  };
+}
+```
+
+Note we use the upstream `services.sanoid.templates` option to define the templates.
+
+### Without contract {#blocks-sanoid-usage-without-contract}
+
+To backup a dataset that does not provide the dataset backup contract,
+we can just set the request manually:
+
+```nix
+{
+  shb.zfs.pools.root.datasets.home = {
+    path = "/home";
+  };
+
+  shb.sanoid.backup."root/home" = {
+    request.dataset = "root/home";
+  };
+}
+```
+
+Note the attr name under the `shb.sanoid.backup` option does not
+set the dataset name.
+
+## Options Reference {#blocks-sanoid-options}
+
+```{=include=} options
+id-prefix: blocks-sanoid-options-
+list-id: selfhostblocks-blocks-sanoid-options
+source: @OPTIONS_JSON@
+```

modules/blocks/sops.nix

@@ -0,0 +1,59 @@
+{
+  config,
+  lib,
+  shb,
+  ...
+}:
+let
+  inherit (lib) mapAttrs mkOption;
+  inherit (lib.types) attrsOf anything submodule;
+
+  cfg = config.shb.sops;
+in
+{
+  imports = [
+    ../../lib/module.nix
+  ];
+
+  options.shb.sops = {
+    secret = mkOption {
+      description = "Secret following the [secret contract](./contracts-secret.html).";
+      default = { };
+      type = attrsOf (
+        submodule (
+          { name, options, ... }:
+          {
+            options = shb.contracts.secret.mkProvider {
+              settings = mkOption {
+                description = ''
+                  Settings specific to the Sops provider.
+
+                  This is a passthrough option to set [sops-nix options](https://github.com/Mic92/sops-nix/blob/master/modules/sops/default.nix).
+
+                  Note though that the `mode`, `owner`, `group`, and `restartUnits`
+                  are managed by the [shb.sops.secret.<name>.request](#blocks-sops-options-shb.sops.secret._name_.request) option.
+                '';
+
+                type = attrsOf anything;
+                default = { };
+              };
+
+              resultCfg = {
+                path = "/run/secrets/${name}";
+                pathText = "/run/secrets/<name>";
+              };
+            };
+          }
+        )
+      );
+    };
+  };
+
+  config = {
+    sops.secrets =
+      let
+        mkSecret = n: secretCfg: secretCfg.request // secretCfg.settings;
+      in
+      mapAttrs mkSecret cfg.secret;
+  };
+}

modules/blocks/sops/docs/default.md

@@ -0,0 +1,60 @@
+# SOPS Block {#blocks-sops}
+
+Defined in [`/modules/blocks/sops.nix`](@REPO@/modules/blocks/sops.nix).
+
+This block sets up a [sops-nix][] secret.
+
+It is only a small layer on top of `sops-nix` options
+to adapt it to the [secret contract](./contracts-secret.html).
+
+[sops-nix]: https://github.com/Mic92/sops-nix
+
+## Provider Contracts {#blocks-sops-contract-provider}
+
+This block provides the following contracts:
+
+- [secret contract][] under the [`shb.sops.secret`][secret] option.
+  It is not yet tested with [contract tests][secret contract tests] but it is used extensively on several machines.
+
+[secret]: #blocks-sops-options-shb.sops.secret
+[secret contract]: contracts-secret.html
+[secret contract tests]: @REPO@/test/contracts/secret.nix
+
+As requested by the contract, when asking for a secret with the `shb.sops` module,
+the path where the secret will be located can be found under the [`shb.sops.secret.<name>.result`][result] option.
+
+[result]: #blocks-sops-options-shb.sops.secret._name_.result
+
+## Usage {#blocks-sops-usage}
+
+First, a file with encrypted secrets must be created by following the [secrets setup section](usage.html#usage-secrets).
+
+### With Requester Module {#blocks-sops-usage-requester}
+
+This example shows how to use this sops block
+to fulfill the request of a module using the [secret contract][] under the option `services.mymodule.mysecret`.
+
+```nix
+shb.sops.secret."mymodule/mysecret".request = config.services.mymodule.mysecret.request;
+services.mymodule.mysecret.result = config.shb.sops.secret."mymodule/mysecret".result;
+```
+
+### Manual Module {#blocks-sops-usage-manual}
+
+The provider module can be used on its own, without a requester module:
+
+```nix
+shb.sops.secret."mymodule/mysecret".request = {
+  mode = "0400";
+  owner = "owner";
+};
+services.mymodule.mysecret.path = config.sops.secret."mymodule/mysecret".result.path;
+```
+
+## Options Reference {#blocks-sops-options}
+
+```{=include=} options
+id-prefix: blocks-sops-options-
+list-id: selfhostblocks-block-sops-options
+source: @OPTIONS_JSON@
+```

modules/blocks/ssl/dashboard/SSL.json

@@ -0,0 +1,727 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "id": 16,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "axisSoftMin": 0,
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "showValues": false,
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "line+area"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": 0
+              },
+              {
+                "color": "transparent",
+                "value": 604808
+              }
+            ]
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "id": 3,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true,
+          "sortBy": "Last *",
+          "sortDesc": false
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "12.2.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+          },
+          "editorMode": "code",
+          "expr": "min by(exported_hostname, subject, path) (ssl_certificate_expiry_seconds{subject=~\"CN=$job\"})",
+          "legendFormat": "{{exported_hostname}}: {{subject}} {{path}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Certificate Remaining Validity",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineStyle": {
+              "fill": "solid"
+            },
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "showValues": false,
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "fieldMinMax": false,
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "dateTimeFromNow"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "id": 5,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true,
+          "sortBy": "Last *",
+          "sortDesc": true,
+          "width": 300
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "12.2.0",
+      "targets": [
+        {
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "systemd_timer_next_trigger_seconds{name=~\"acme-renew-$job.timer\"} * 1000",
+          "format": "time_series",
+          "instant": false,
+          "legendFormat": "{{name}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Schedule",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "fixedColor": "green",
+            "mode": "fixed"
+          },
+          "custom": {
+            "align": "auto",
+            "cellOptions": {
+              "type": "auto"
+            },
+            "footer": {
+              "reducers": []
+            },
+            "inspect": false
+          },
+          "decimals": 0,
+          "mappings": [],
+          "noValue": "0",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "green",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": [
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "% failed"
+            },
+            "properties": [
+              {
+                "id": "unit",
+                "value": "percentunit"
+              },
+              {
+                "id": "custom.cellOptions",
+                "value": {
+                  "mode": "gradient",
+                  "type": "color-background"
+                }
+              },
+              {
+                "id": "color",
+                "value": {
+                  "mode": "continuous-GrYlRd"
+                }
+              },
+              {
+                "id": "max",
+                "value": 1
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "total"
+            },
+            "properties": [
+              {
+                "id": "custom.cellOptions",
+                "value": {
+                  "mode": "gradient",
+                  "type": "color-background"
+                }
+              },
+              {
+                "id": "color",
+                "value": {
+                  "mode": "thresholds"
+                }
+              },
+              {
+                "id": "thresholds",
+                "value": {
+                  "mode": "absolute",
+                  "steps": [
+                    {
+                      "color": "red",
+                      "value": 0
+                    },
+                    {
+                      "color": "transparent",
+                      "value": 1
+                    }
+                  ]
+                }
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byType",
+              "options": "string"
+            },
+            "properties": [
+              {
+                "id": "custom.minWidth",
+                "value": 150
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byType",
+              "options": "number"
+            },
+            "properties": [
+              {
+                "id": "custom.width",
+                "value": 100
+              }
+            ]
+          }
+        ]
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "id": 4,
+      "options": {
+        "cellHeight": "sm",
+        "enablePagination": true,
+        "frozenColumns": {},
+        "showHeader": true,
+        "sortBy": []
+      },
+      "pluginVersion": "12.2.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+          },
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "increase(systemd_unit_state{name=~\"acme-(order-renew-)?[[job]].service\", state=\"activating\"}[7d])",
+          "instant": true,
+          "legendFormat": "__auto",
+          "range": false,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+          },
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "increase(systemd_unit_state{name=~\"acme-(order-renew-)?[[job]].service\", state=\"failed\"}[7d])",
+          "hide": false,
+          "instant": true,
+          "legendFormat": "__auto",
+          "range": false,
+          "refId": "B"
+        }
+      ],
+      "title": "Jobs in the Past Week",
+      "transformations": [
+        {
+          "id": "labelsToFields",
+          "options": {
+            "mode": "columns"
+          }
+        },
+        {
+          "id": "merge",
+          "options": {}
+        },
+        {
+          "id": "groupingToMatrix",
+          "options": {
+            "columnField": "state",
+            "rowField": "name",
+            "valueField": "Value"
+          }
+        },
+        {
+          "id": "calculateField",
+          "options": {
+            "alias": "total",
+            "binary": {
+              "left": {
+                "matcher": {
+                  "id": "byName",
+                  "options": "activating"
+                }
+              },
+              "operator": "+",
+              "right": {
+                "matcher": {
+                  "id": "byName",
+                  "options": "failed"
+                }
+              }
+            },
+            "mode": "binary",
+            "reduce": {
+              "include": [
+                "activating",
+                "failed"
+              ],
+              "reducer": "sum"
+            }
+          }
+        },
+        {
+          "id": "calculateField",
+          "options": {
+            "alias": "% failed",
+            "binary": {
+              "left": {
+                "matcher": {
+                  "id": "byName",
+                  "options": "failed"
+                }
+              },
+              "operator": "/",
+              "right": {
+                "matcher": {
+                  "id": "byName",
+                  "options": "total"
+                }
+              }
+            },
+            "mode": "binary",
+            "reduce": {
+              "reducer": "sum"
+            }
+          }
+        },
+        {
+          "id": "sortBy",
+          "options": {
+            "fields": {},
+            "sort": [
+              {
+                "desc": true,
+                "field": "total"
+              }
+            ]
+          }
+        },
+        {
+          "id": "sortBy",
+          "options": {
+            "fields": {},
+            "sort": [
+              {
+                "desc": true,
+                "field": "failed"
+              }
+            ]
+          }
+        },
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {},
+            "includeByName": {},
+            "indexByName": {},
+            "renameByName": {
+              "activating": "success",
+              "name\\state": "Job"
+            }
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "description": "",
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 2,
+        "w": 12,
+        "x": 12,
+        "y": 8
+      },
+      "id": 7,
+      "options": {
+        "code": {
+          "language": "plaintext",
+          "showLineNumbers": false,
+          "showMiniMap": false
+        },
+        "content": "If the log panel is empty, it may be because the amount of lines is too high. Try filtering a few jobs first.",
+        "mode": "markdown"
+      },
+      "pluginVersion": "12.2.0",
+      "title": "",
+      "type": "text"
+    },
+    {
+      "datasource": {
+        "default": false,
+        "type": "loki",
+        "uid": "cd6cc53e-840c-484d-85f7-96fede324006"
+      },
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 14,
+        "w": 12,
+        "x": 12,
+        "y": 10
+      },
+      "id": 8,
+      "options": {
+        "dedupStrategy": "none",
+        "enableInfiniteScrolling": false,
+        "enableLogDetails": false,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "pluginVersion": "12.2.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "cd6cc53e-840c-484d-85f7-96fede324006"
+          },
+          "direction": "backward",
+          "editorMode": "code",
+          "expr": "{unit=~\"acme-(order-renew-)?($job).service\"}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "title": "Logs - $job",
+      "type": "logs"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+      },
+      "description": "The job duration is not accurate. Jobs taking less than 15s to run will sometimes appear as taking 100s.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "fixed"
+          },
+          "custom": {
+            "axisPlacement": "auto",
+            "fillOpacity": 70,
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineWidth": 0,
+            "spanNulls": false
+          },
+          "mappings": [
+            {
+              "options": {
+                "1": {
+                  "color": "green",
+                  "index": 0,
+                  "text": "Running"
+                }
+              },
+              "type": "value"
+            }
+          ],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "#EAB839",
+                "value": 0
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 16
+      },
+      "id": 6,
+      "options": {
+        "alignValue": "left",
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": false
+        },
+        "mergeValues": true,
+        "rowHeight": 0.9,
+        "showValue": "never",
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "12.2.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09"
+          },
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "max(label_replace(systemd_unit_state{name=~\"acme-(order-renew-)?$job.service\", state=\"activating\"}, \"name\", \"$1\", \"name\", \"acme-order-renew-(.*).service\") > 0 or on(name) label_replace(clamp(systemd_timer_last_trigger_seconds{name=~\"acme-renew-$job.timer\"} - (systemd_timer_last_trigger_seconds{name=~\"acme-renew-$job.timer\"} offset 100s) > 0, 0, 1), \"name\", \"$1\", \"name\", \"acme-renew-(.*).timer\")) by (name)",
+          "format": "time_series",
+          "hide": false,
+          "instant": false,
+          "key": "Q-e1d5c07a-8dcc-4f34-aa5c-cdebcbdda322-0",
+          "legendFormat": "{{name}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Job Runs",
+      "type": "state-timeline"
+    }
+  ],
+  "preload": false,
+  "schemaVersion": 42,
+  "tags": [],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "text": [
+            "All"
+          ],
+          "value": [
+            "$__all"
+          ]
+        },
+        "definition": "label_values(systemd_unit_state{name=~\"acme-renew-.*.timer\"},name)",
+        "includeAll": true,
+        "label": "Job",
+        "multi": true,
+        "name": "job",
+        "options": [],
+        "query": {
+          "qryType": 1,
+          "query": "label_values(systemd_unit_state{name=~\"acme-renew-.*.timer\"},name)",
+          "refId": "PrometheusVariableQueryEditor-VariableQuery"
+        },
+        "refresh": 1,
+        "regex": "/acme-renew-(?<value>.*).timer/",
+        "type": "query"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "SSL Certificates",
+  "uid": "ae818js0bvw8wb",
+  "version": 25
+}

modules/blocks/ssl/docs/default.md

@@ -1,4 +1,4 @@
-# SSL Generator Block {#ssl-block}
+# SSL Generator Block {#block-ssl}
 
 This NixOS module is a block that implements the [SSL certificate generator](contracts-ssl.html) contract.
 
@@ -14,7 +14,7 @@ It is implemented by:
 [11]: blocks-ssl.html#blocks-ssl-options-shb.certs.certs.selfsigned
 [12]: blocks-ssl.html#blocks-ssl-options-shb.certs.certs.letsencrypt
 
-## Self-Signed Certificates {#ssl-block-impl-self-signed}
+## Self-Signed Certificates {#block-ssl-impl-self-signed}
 
 Defined in [`/modules/blocks/ssl.nix`](@REPO@/modules/blocks/ssl.nix).
 
@@ -52,7 +52,7 @@ shb.certs.certs.selfsigned = {
 The group has been chosen to be `nginx` to be consistent with the examples further down in this
 document.
 
-## Let's Encrypt {#ssl-block-impl-lets-encrypt}
+## Let's Encrypt {#block-ssl-impl-lets-encrypt}
 
 Defined in [`/modules/blocks/ssl.nix`](@REPO@/modules/blocks/ssl.nix).
 
@@ -62,6 +62,7 @@ We can ask Let's Encrypt to generate a certificate with:
 shb.certs.certs.letsencrypt."example.com" = {
   domain = "example.com";
   group = "nginx";
+  reloadServices = [ "nginx.service" ];
   dnsProvider = "linode";
   adminEmail = "admin@example.com";
   credentialsFile = /path/to/secret/file;
@@ -79,9 +80,16 @@ The credential file's content would be a key-value pair:
 LINODE_TOKEN=XYZ...
 ```
 
+If you use one subdomain per service,
+asking for certificates for a subdomain is done with:
+
+```nix
+shb.certs.certs.letsencrypt."example.com".extraDomains = [ "nextcloud.${domain}" ];
+```
+
 For other providers, see the [official instruction](https://go-acme.github.io/lego/dns/).
 
-## Usage {#ssl-block-usage}
+## Usage {#block-ssl-usage}
 
 To use either a self-signed certificates or a Let's Encrypt generated one, we can reference the path
 where the certificate and the private key are located:
@@ -110,16 +118,40 @@ config.shb.certs.systemdService
 See also the [SSL certificate generator usage](contracts-ssl.html#ssl-contract-usage) for a more detailed usage
 example.
 
-## Debug {#ssl-block-debug}
+## Monitoring {#blocks-ssl-monitoring}
+
+A dashboard for SSL certificates is provided.
+See [SSL Certificates Dashboard and Alert](blocks-monitoring.html#blocks-monitoring-ssl) section in the monitoring chapter.
+
+## Debug {#block-ssl-debug}
 
 Each CA and Cert is generated by a systemd service whose name can be seen in the `systemdService`
 option. You can then see the latest errors messages using `journalctl`.
 
-## Tests {#ssl-block-tests}
+### Let's Encrypt debug {#blocks-ssl-debug-lets-encrypt}
+
+Since the SHB SSL block uses the [`security.acme`][] module under the hood,
+knowing how that one works can become required if something goes wrong.
+
+For each domain and subdomain, noted as `fqdn` hereunder,
+the following systemd timers and services are created:
+
+- `acme-renew-${fqdn}.timer` triggers the `acme-order-renew-${fqdn}.service` service every day.
+- `acme-${fqdn}.service` (re)generate the initial self-signed certificate,
+   only if the following job never succeeded at least once yet.
+- `acme-order-renew-${fqdn}.service` asks for a new certificate
+   only if the certificate will expire in the next 30 days.
+   Has logic to only renew if the list of domains has not changed.
+
+Also, a global service named `acme-setup.service` is created
+
+[`security.acme`]: https://nixos.org/manual/nixos/stable/#module-security-acme
+
+## Tests {#block-ssl-tests}
 
 The self-signed implementation is tested in [`/tests/vm/ssl.nix`](@REPO@/tests/vm/ssl.nix).
 
-## Options Reference {#ssl-block-options}
+## Options Reference {#block-ssl-options}
 
 ```{=include=} options
 id-prefix: blocks-ssl-options-

modules/blocks/tinyproxy.nix

@@ -1,32 +1,43 @@
 # Inspired from https://github.com/NixOS/nixpkgs/pull/231152 but made it so we can have multiple instances.
-{ config, lib, pkgs, ... }:
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
 
 with lib;
 
 let
   cfg = config.shb.tinyproxy;
 
-  mkValueStringTinyproxy = with lib; v:
-        if true  ==         v then "yes"
-        else if false ==    v then "no"
-        else generators.mkValueStringDefault {} v;
+  mkValueStringTinyproxy =
+    with lib;
+    v:
+    if true == v then
+      "yes"
+    else if false == v then
+      "no"
+    else
+      generators.mkValueStringDefault { } v;
 
-  mkKeyValueTinyproxy = {
-    mkValueString ? mkValueStringDefault {}
-  }: sep: k: v:
-    if null     ==  v then ""
-    else "${lib.strings.escape [sep] k}${sep}${mkValueString v}";
+  mkKeyValueTinyproxy =
+    {
+      mkValueString ? mkValueStringDefault { },
+    }:
+    sep: k: v:
+    if null == v then "" else "${lib.strings.escape [ sep ] k}${sep}${mkValueString v}";
 
-  settingsFormat = (pkgs.formats.keyValue {
+  settingsFormat = (
+    pkgs.formats.keyValue {
       mkKeyValue = mkKeyValueTinyproxy {
         mkValueString = mkValueStringTinyproxy;
       } " ";
-      listsAsDuplicateKeys= true;
-  });
+      listsAsDuplicateKeys = true;
+    }
+  );
 
   configFile = name: cfg: settingsFormat.generate "tinyproxy-${name}.conf" cfg.settings;
-
-  someEnabled = any (mapAttrsToList (name: c: c.enable) cfg);
 in
 {
   options =
@@ -35,115 +46,124 @@ in
         options = {
           enable = mkEnableOption "Tinyproxy daemon";
 
-          package = mkPackageOptionMD pkgs "tinyproxy" {};
+          package = mkPackageOption pkgs "tinyproxy" { };
 
           dynamicBindFile = mkOption {
             description = ''
-            File holding the IP to bind to.
+              File holding the IP to bind to.
             '';
             default = "";
           };
 
           settings = mkOption {
             description = ''
-            Configuration for [tinyproxy](https://tinyproxy.github.io/).
+              Configuration for [tinyproxy](https://tinyproxy.github.io/).
             '';
             default = { };
-            example = literalExpression ''{
-            Port 8888;
-            Listen 127.0.0.1;
-            Timeout 600;
-            Allow 127.0.0.1;
-            Anonymous = ['"Host"' '"Authorization"'];
-            ReversePath = '"/example/" "http://www.example.com/"';
-            }'';
-            type = types.submodule ({name, ...}: {
-              freeformType = settingsFormat.type;
-              options = {
-                Listen = mkOption {
-                  type = types.str;
-                  default = "127.0.0.1";
-                  description = ''
-                  Specify which address to listen to.
-                  '';
+            example = literalExpression ''
+              {
+                          Port 8888;
+                          Listen 127.0.0.1;
+                          Timeout 600;
+                          Allow 127.0.0.1;
+                          Anonymous = ['"Host"' '"Authorization"'];
+                          ReversePath = '"/example/" "http://www.example.com/"';
+                          }'';
+            type = types.submodule (
+              { name, ... }:
+              {
+                freeformType = settingsFormat.type;
+                options = {
+                  Listen = mkOption {
+                    type = types.str;
+                    default = "127.0.0.1";
+                    description = ''
+                      Specify which address to listen to.
+                    '';
+                  };
+                  Port = mkOption {
+                    type = types.int;
+                    default = 8888;
+                    description = ''
+                      Specify which port to listen to.
+                    '';
+                  };
+                  Anonymous = mkOption {
+                    type = types.listOf types.str;
+                    default = [ ];
+                    description = ''
+                      If an `Anonymous` keyword is present, then anonymous proxying is enabled. The
+                      headers listed with `Anonymous` are allowed through, while all others are denied.
+                      If no Anonymous keyword is present, then all headers are allowed through. You must
+                      include quotes around the headers.
+                    '';
+                  };
+                  Filter = mkOption {
+                    type = types.nullOr types.path;
+                    default = null;
+                    description = ''
+                      Tinyproxy supports filtering of web sites based on URLs or domains. This option
+                      specifies the location of the file containing the filter rules, one rule per line.
+                    '';
+                  };
                 };
-                Port = mkOption {
-                  type = types.int;
-                  default = 8888;
-                  description = ''
-                  Specify which port to listen to.
-                  '';
-                };
-                Anonymous = mkOption {
-                  type = types.listOf types.str;
-                  default = [];
-                  description = ''
-                  If an `Anonymous` keyword is present, then anonymous proxying is enabled. The
-                  headers listed with `Anonymous` are allowed through, while all others are denied.
-                  If no Anonymous keyword is present, then all headers are allowed through. You must
-                  include quotes around the headers.
-                  '';
-                };
-                Filter = mkOption {
-                  type = types.nullOr types.path;
-                  default = null;
-                  description = ''
-                  Tinyproxy supports filtering of web sites based on URLs or domains. This option
-                  specifies the location of the file containing the filter rules, one rule per line.
-                  '';
-                };
-              };
-            });
+              }
+            );
           };
         };
       };
     in
-      {
-        shb.tinyproxy = mkOption {
-          description = "Tinyproxy instances.";
-          default = {};
-          type = types.attrsOf instanceOption;
-        };
+    {
+      shb.tinyproxy = mkOption {
+        description = "Tinyproxy instances.";
+        default = { };
+        type = types.attrsOf instanceOption;
       };
+    };
 
   config = {
     systemd.services =
       let
-        instanceConfig = name: c: mkIf c.enable {
-          "tinyproxy-${name}" = {
-            description = "TinyProxy daemon - instance ${name}";
-            after = [ "network.target" ];
-            wantedBy = [ "multi-user.target" ];
-            serviceConfig = {
-              User = "tinyproxy";
-              Group = "tinyproxy";
-              Type = "simple";
-              ExecStart = "${getExe c.package} -d -c /etc/tinyproxy/${name}.conf";
-              ExecReload = "${pkgs.coreutils}/bin/kill -SIGHUP $MAINPID";
-              KillSignal = "SIGINT";
-              TimeoutStopSec = "30s";
-              Restart = "on-failure";
-              RestartSec = "1s";
-              RestartSteps = "3";
-              RestartMaxDelaySec = "10s";
-              ConfigurationDirectory = "tinyproxy";
+        instanceConfig =
+          name: c:
+          mkIf c.enable {
+            "tinyproxy-${name}" = {
+              description = "TinyProxy daemon - instance ${name}";
+              after = [ "network.target" ];
+              wantedBy = [ "multi-user.target" ];
+              serviceConfig = {
+                User = "tinyproxy";
+                Group = "tinyproxy";
+                Type = "simple";
+                ExecStart = "${getExe c.package} -d -c /etc/tinyproxy/${name}.conf";
+                ExecReload = "${pkgs.coreutils}/bin/kill -SIGHUP $MAINPID";
+                KillSignal = "SIGINT";
+                TimeoutStopSec = "30s";
+                Restart = "on-failure";
+                RestartSec = "1s";
+                RestartSteps = "3";
+                RestartMaxDelaySec = "10s";
+                ConfigurationDirectory = "tinyproxy";
+              };
+              preStart = concatStringsSep "\n" (
+                [
+                  "cat ${configFile name c} > /etc/tinyproxy/${name}.conf"
+                ]
+                ++ optionals (c.dynamicBindFile != "") [
+                  "echo -n 'Bind ' >> /etc/tinyproxy/${name}.conf"
+                  "cat ${c.dynamicBindFile} >> /etc/tinyproxy/${name}.conf"
+                ]
+              );
             };
-            preStart = concatStringsSep "\n" ([
-              "cat ${configFile name c} > /etc/tinyproxy/${name}.conf"
-            ] ++ optionals (c.dynamicBindFile != "") [
-              "echo -n 'Bind ' >> /etc/tinyproxy/${name}.conf"
-              "cat ${c.dynamicBindFile} >> /etc/tinyproxy/${name}.conf"
-            ]);
           };
-        };
       in
-        mkMerge (mapAttrsToList instanceConfig cfg);
+      mkMerge (mapAttrsToList instanceConfig cfg);
 
     users.users.tinyproxy = {
       group = "tinyproxy";
       isSystemUser = true;
     };
-    users.groups.tinyproxy = {};
+    users.groups.tinyproxy = { };
   };
 
   meta.maintainers = with maintainers; [ tcheronneau ];

modules/blocks/vpn.nix

@@ -1,4 +1,9 @@
-{ config, pkgs, lib, ... }:
+{
+  config,
+  pkgs,
+  lib,
+  ...
+}:
 
 let
   cfg = config.shb.vpn;
@@ -6,199 +11,212 @@ let
   quoteEach = lib.concatMapStrings (x: ''"${x}"'');
 
   nordvpnConfig =
-    { name
-    , dev
-    , authFile
-    , remoteServerIP
-    , dependentServices ? []
-    }: ''
-    client
-    dev ${dev}
-    proto tcp
-    remote ${remoteServerIP} 443
-    resolv-retry infinite
-    remote-random
-    nobind
-    tun-mtu 1500
-    tun-mtu-extra 32
-    mssfix 1450
-    persist-key
-    persist-tun
-    ping 15
-    ping-restart 0
-    ping-timer-rem
-    reneg-sec 0
-    comp-lzo no
+    {
+      name,
+      dev,
+      authFile,
+      remoteServerIP,
+      dependentServices ? [ ],
+    }:
+    ''
+      client
+      dev ${dev}
+      proto tcp
+      remote ${remoteServerIP} 443
+      resolv-retry infinite
+      remote-random
+      nobind
+      tun-mtu 1500
+      tun-mtu-extra 32
+      mssfix 1450
+      persist-key
+      persist-tun
+      ping 15
+      ping-restart 0
+      ping-timer-rem
+      reneg-sec 0
+      comp-lzo no
 
-    status /tmp/openvpn/${name}.status
+      status /tmp/openvpn/${name}.status
 
-    remote-cert-tls server
+      remote-cert-tls server
 
-    auth-user-pass ${authFile}
-    verb 3
-    pull
-    fast-io
-    cipher AES-256-CBC
-    auth SHA512
+      auth-user-pass ${authFile}
+      verb 3
+      pull
+      fast-io
+      cipher AES-256-CBC
+      auth SHA512
 
-    script-security 2
-    route-noexec
-    route-up ${routeUp name dependentServices}/bin/routeUp.sh
-    down ${routeDown name dependentServices}/bin/routeDown.sh
+      script-security 2
+      route-noexec
+      route-up ${routeUp name dependentServices}/bin/routeUp.sh
+      down ${routeDown name dependentServices}/bin/routeDown.sh
 
-    <ca>
-    -----BEGIN CERTIFICATE-----
-    MIIFCjCCAvKgAwIBAgIBATANBgkqhkiG9w0BAQ0FADA5MQswCQYDVQQGEwJQQTEQ
-    MA4GA1UEChMHTm9yZFZQTjEYMBYGA1UEAxMPTm9yZFZQTiBSb290IENBMB4XDTE2
-    MDEwMTAwMDAwMFoXDTM1MTIzMTIzNTk1OVowOTELMAkGA1UEBhMCUEExEDAOBgNV
-    BAoTB05vcmRWUE4xGDAWBgNVBAMTD05vcmRWUE4gUm9vdCBDQTCCAiIwDQYJKoZI
-    hvcNAQEBBQADggIPADCCAgoCggIBAMkr/BYhyo0F2upsIMXwC6QvkZps3NN2/eQF
-    kfQIS1gql0aejsKsEnmY0Kaon8uZCTXPsRH1gQNgg5D2gixdd1mJUvV3dE3y9FJr
-    XMoDkXdCGBodvKJyU6lcfEVF6/UxHcbBguZK9UtRHS9eJYm3rpL/5huQMCppX7kU
-    eQ8dpCwd3iKITqwd1ZudDqsWaU0vqzC2H55IyaZ/5/TnCk31Q1UP6BksbbuRcwOV
-    skEDsm6YoWDnn/IIzGOYnFJRzQH5jTz3j1QBvRIuQuBuvUkfhx1FEwhwZigrcxXu
-    MP+QgM54kezgziJUaZcOM2zF3lvrwMvXDMfNeIoJABv9ljw969xQ8czQCU5lMVmA
-    37ltv5Ec9U5hZuwk/9QO1Z+d/r6Jx0mlurS8gnCAKJgwa3kyZw6e4FZ8mYL4vpRR
-    hPdvRTWCMJkeB4yBHyhxUmTRgJHm6YR3D6hcFAc9cQcTEl/I60tMdz33G6m0O42s
-    Qt/+AR3YCY/RusWVBJB/qNS94EtNtj8iaebCQW1jHAhvGmFILVR9lzD0EzWKHkvy
-    WEjmUVRgCDd6Ne3eFRNS73gdv/C3l5boYySeu4exkEYVxVRn8DhCxs0MnkMHWFK6
-    MyzXCCn+JnWFDYPfDKHvpff/kLDobtPBf+Lbch5wQy9quY27xaj0XwLyjOltpiST
-    LWae/Q4vAgMBAAGjHTAbMAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgEGMA0GCSqG
-    SIb3DQEBDQUAA4ICAQC9fUL2sZPxIN2mD32VeNySTgZlCEdVmlq471o/bDMP4B8g
-    nQesFRtXY2ZCjs50Jm73B2LViL9qlREmI6vE5IC8IsRBJSV4ce1WYxyXro5rmVg/
-    k6a10rlsbK/eg//GHoJxDdXDOokLUSnxt7gk3QKpX6eCdh67p0PuWm/7WUJQxH2S
-    DxsT9vB/iZriTIEe/ILoOQF0Aqp7AgNCcLcLAmbxXQkXYCCSB35Vp06u+eTWjG0/
-    pyS5V14stGtw+fA0DJp5ZJV4eqJ5LqxMlYvEZ/qKTEdoCeaXv2QEmN6dVqjDoTAo
-    k0t5u4YRXzEVCfXAC3ocplNdtCA72wjFJcSbfif4BSC8bDACTXtnPC7nD0VndZLp
-    +RiNLeiENhk0oTC+UVdSc+n2nJOzkCK0vYu0Ads4JGIB7g8IB3z2t9ICmsWrgnhd
-    NdcOe15BincrGA8avQ1cWXsfIKEjbrnEuEk9b5jel6NfHtPKoHc9mDpRdNPISeVa
-    wDBM1mJChneHt59Nh8Gah74+TM1jBsw4fhJPvoc7Atcg740JErb904mZfkIEmojC
-    VPhBHVQ9LHBAdM8qFI2kRK0IynOmAZhexlP/aT/kpEsEPyaZQlnBn3An1CRz8h0S
-    PApL8PytggYKeQmRhl499+6jLxcZ2IegLfqq41dzIjwHwTMplg+1pKIOVojpWA==
-    -----END CERTIFICATE-----
-    </ca>
-    key-direction 1
-    <tls-auth>
-    #
-    # 2048 bit OpenVPN static key
-    #
-    -----BEGIN OpenVPN Static key V1-----
-    e685bdaf659a25a200e2b9e39e51ff03
-    0fc72cf1ce07232bd8b2be5e6c670143
-    f51e937e670eee09d4f2ea5a6e4e6996
-    5db852c275351b86fc4ca892d78ae002
-    d6f70d029bd79c4d1c26cf14e9588033
-    cf639f8a74809f29f72b9d58f9b8f5fe
-    fc7938eade40e9fed6cb92184abb2cc1
-    0eb1a296df243b251df0643d53724cdb
-    5a92a1d6cb817804c4a9319b57d53be5
-    80815bcfcb2df55018cc83fc43bc7ff8
-    2d51f9b88364776ee9d12fc85cc7ea5b
-    9741c4f598c485316db066d52db4540e
-    212e1518a9bd4828219e24b20d88f598
-    a196c9de96012090e333519ae18d3509
-    9427e7b372d348d352dc4c85e18cd4b9
-    3f8a56ddb2e64eb67adfc9b337157ff4
-    -----END OpenVPN Static key V1-----
-    </tls-auth>
+      <ca>
+      -----BEGIN CERTIFICATE-----
+      MIIFCjCCAvKgAwIBAgIBATANBgkqhkiG9w0BAQ0FADA5MQswCQYDVQQGEwJQQTEQ
+      MA4GA1UEChMHTm9yZFZQTjEYMBYGA1UEAxMPTm9yZFZQTiBSb290IENBMB4XDTE2
+      MDEwMTAwMDAwMFoXDTM1MTIzMTIzNTk1OVowOTELMAkGA1UEBhMCUEExEDAOBgNV
+      BAoTB05vcmRWUE4xGDAWBgNVBAMTD05vcmRWUE4gUm9vdCBDQTCCAiIwDQYJKoZI
+      hvcNAQEBBQADggIPADCCAgoCggIBAMkr/BYhyo0F2upsIMXwC6QvkZps3NN2/eQF
+      kfQIS1gql0aejsKsEnmY0Kaon8uZCTXPsRH1gQNgg5D2gixdd1mJUvV3dE3y9FJr
+      XMoDkXdCGBodvKJyU6lcfEVF6/UxHcbBguZK9UtRHS9eJYm3rpL/5huQMCppX7kU
+      eQ8dpCwd3iKITqwd1ZudDqsWaU0vqzC2H55IyaZ/5/TnCk31Q1UP6BksbbuRcwOV
+      skEDsm6YoWDnn/IIzGOYnFJRzQH5jTz3j1QBvRIuQuBuvUkfhx1FEwhwZigrcxXu
+      MP+QgM54kezgziJUaZcOM2zF3lvrwMvXDMfNeIoJABv9ljw969xQ8czQCU5lMVmA
+      37ltv5Ec9U5hZuwk/9QO1Z+d/r6Jx0mlurS8gnCAKJgwa3kyZw6e4FZ8mYL4vpRR
+      hPdvRTWCMJkeB4yBHyhxUmTRgJHm6YR3D6hcFAc9cQcTEl/I60tMdz33G6m0O42s
+      Qt/+AR3YCY/RusWVBJB/qNS94EtNtj8iaebCQW1jHAhvGmFILVR9lzD0EzWKHkvy
+      WEjmUVRgCDd6Ne3eFRNS73gdv/C3l5boYySeu4exkEYVxVRn8DhCxs0MnkMHWFK6
+      MyzXCCn+JnWFDYPfDKHvpff/kLDobtPBf+Lbch5wQy9quY27xaj0XwLyjOltpiST
+      LWae/Q4vAgMBAAGjHTAbMAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgEGMA0GCSqG
+      SIb3DQEBDQUAA4ICAQC9fUL2sZPxIN2mD32VeNySTgZlCEdVmlq471o/bDMP4B8g
+      nQesFRtXY2ZCjs50Jm73B2LViL9qlREmI6vE5IC8IsRBJSV4ce1WYxyXro5rmVg/
+      k6a10rlsbK/eg//GHoJxDdXDOokLUSnxt7gk3QKpX6eCdh67p0PuWm/7WUJQxH2S
+      DxsT9vB/iZriTIEe/ILoOQF0Aqp7AgNCcLcLAmbxXQkXYCCSB35Vp06u+eTWjG0/
+      pyS5V14stGtw+fA0DJp5ZJV4eqJ5LqxMlYvEZ/qKTEdoCeaXv2QEmN6dVqjDoTAo
+      k0t5u4YRXzEVCfXAC3ocplNdtCA72wjFJcSbfif4BSC8bDACTXtnPC7nD0VndZLp
+      +RiNLeiENhk0oTC+UVdSc+n2nJOzkCK0vYu0Ads4JGIB7g8IB3z2t9ICmsWrgnhd
+      NdcOe15BincrGA8avQ1cWXsfIKEjbrnEuEk9b5jel6NfHtPKoHc9mDpRdNPISeVa
+      wDBM1mJChneHt59Nh8Gah74+TM1jBsw4fhJPvoc7Atcg740JErb904mZfkIEmojC
+      VPhBHVQ9LHBAdM8qFI2kRK0IynOmAZhexlP/aT/kpEsEPyaZQlnBn3An1CRz8h0S
+      PApL8PytggYKeQmRhl499+6jLxcZ2IegLfqq41dzIjwHwTMplg+1pKIOVojpWA==
+      -----END CERTIFICATE-----
+      </ca>
+      key-direction 1
+      <tls-auth>
+      #
+      # 2048 bit OpenVPN static key
+      #
+      -----BEGIN OpenVPN Static key V1-----
+      e685bdaf659a25a200e2b9e39e51ff03
+      0fc72cf1ce07232bd8b2be5e6c670143
+      f51e937e670eee09d4f2ea5a6e4e6996
+      5db852c275351b86fc4ca892d78ae002
+      d6f70d029bd79c4d1c26cf14e9588033
+      cf639f8a74809f29f72b9d58f9b8f5fe
+      fc7938eade40e9fed6cb92184abb2cc1
+      0eb1a296df243b251df0643d53724cdb
+      5a92a1d6cb817804c4a9319b57d53be5
+      80815bcfcb2df55018cc83fc43bc7ff8
+      2d51f9b88364776ee9d12fc85cc7ea5b
+      9741c4f598c485316db066d52db4540e
+      212e1518a9bd4828219e24b20d88f598
+      a196c9de96012090e333519ae18d3509
+      9427e7b372d348d352dc4c85e18cd4b9
+      3f8a56ddb2e64eb67adfc9b337157ff4
+      -----END OpenVPN Static key V1-----
+      </tls-auth>
     '';
 
-  routeUp = name: dependentServices: pkgs.writeShellApplication {
-    name = "routeUp.sh";
+  routeUp =
+    name: dependentServices:
+    pkgs.writeShellApplication {
+      name = "routeUp.sh";
 
-    runtimeInputs = [ pkgs.iproute2 pkgs.systemd pkgs.nettools ];
+      runtimeInputs = [
+        pkgs.iproute2
+        pkgs.systemd
+        pkgs.nettools
+      ];
 
-    text = ''
-    echo "Running route-up..."
+      text = ''
+        echo "Running route-up..."
 
-    echo "dev=''${dev:?}"
-    echo "ifconfig_local=''${ifconfig_local:?}"
-    echo "route_vpn_gateway=''${route_vpn_gateway:?}"
+        echo "dev=''${dev:?}"
+        echo "ifconfig_local=''${ifconfig_local:?}"
+        echo "route_vpn_gateway=''${route_vpn_gateway:?}"
 
-    set -x
+        set -x
 
-    ip rule
-    ip rule add from "''${ifconfig_local:?}/32" table ${name}
-    ip rule add to "''${route_vpn_gateway:?}/32" table ${name}
-    ip rule
+        ip rule
+        ip rule add from "''${ifconfig_local:?}/32" table ${name}
+        ip rule add to "''${route_vpn_gateway:?}/32" table ${name}
+        ip rule
 
-    ip route list table ${name} || :
-    retVal=$?
-    if [ $retVal -eq 2 ]; then
-      echo "table is empty"
-    elif [ $retVal -ne 0 ]; then
-      exit 1
-    fi
-    ip route add default via "''${route_vpn_gateway:?}" dev "''${dev:?}" table ${name}
-    ip route flush cache
-    ip route list table ${name} || :
-    retVal=$?
-    if [ $retVal -eq 2 ]; then
-      echo "table is empty"
-    elif [ $retVal -ne 0 ]; then
-      exit 1
-    fi
+        ip route list table ${name} || :
+        retVal=$?
+        if [ $retVal -eq 2 ]; then
+          echo "table is empty"
+        elif [ $retVal -ne 0 ]; then
+          exit 1
+        fi
+        ip route add default via "''${route_vpn_gateway:?}" dev "''${dev:?}" table ${name}
+        ip route flush cache
+        ip route list table ${name} || :
+        retVal=$?
+        if [ $retVal -eq 2 ]; then
+          echo "table is empty"
+        elif [ $retVal -ne 0 ]; then
+          exit 1
+        fi
 
-    echo "''${ifconfig_local:?}" > /run/openvpn/${name}/ifconfig_local
+        echo "''${ifconfig_local:?}" > /run/openvpn/${name}/ifconfig_local
 
-    dependencies=(${quoteEach dependentServices})
-    for i in "''${dependencies[@]}"; do
-        systemctl restart "$i" || :
-    done
+        dependencies=(${quoteEach dependentServices})
+        for i in "''${dependencies[@]}"; do
+            systemctl restart "$i" || :
+        done
 
-    echo "Running route-up DONE"
-    '';
-  };
+        echo "Running route-up DONE"
+      '';
+    };
 
-  routeDown = name: dependentServices: pkgs.writeShellApplication {
-    name = "routeDown.sh";
+  routeDown =
+    name: dependentServices:
+    pkgs.writeShellApplication {
+      name = "routeDown.sh";
 
-    runtimeInputs = [ pkgs.iproute2 pkgs.systemd pkgs.nettools pkgs.coreutils ];
+      runtimeInputs = [
+        pkgs.iproute2
+        pkgs.systemd
+        pkgs.nettools
+        pkgs.coreutils
+      ];
 
-    text = ''
-    echo "Running route-down..."
+      text = ''
+        echo "Running route-down..."
 
-    echo "dev=''${dev:?}"
-    echo "ifconfig_local=''${ifconfig_local:?}"
-    echo "route_vpn_gateway=''${route_vpn_gateway:?}"
+        echo "dev=''${dev:?}"
+        echo "ifconfig_local=''${ifconfig_local:?}"
+        echo "route_vpn_gateway=''${route_vpn_gateway:?}"
 
-    set -x
+        set -x
 
-    ip rule
-    ip rule del from "''${ifconfig_local:?}/32" table ${name}
-    ip rule del to "''${route_vpn_gateway:?}/32" table ${name}
-    ip rule
+        ip rule
+        ip rule del from "''${ifconfig_local:?}/32" table ${name}
+        ip rule del to "''${route_vpn_gateway:?}/32" table ${name}
+        ip rule
 
-    # This will probably fail because the dev is already gone.
-    ip route list table ${name} || :
-    retVal=$?
-    if [ $retVal -eq 2 ]; then
-      echo "table is empty"
-    elif [ $retVal -ne 0 ]; then
-      exit 1
-    fi
-    ip route del default via "''${route_vpn_gateway:?}" dev "''${dev:?}" table ${name} || :
-    ip route flush cache
-    ip route list table ${name} || :
-    retVal=$?
-    if [ $retVal -eq 2 ]; then
-      echo "table is empty"
-    elif [ $retVal -ne 0 ]; then
-      exit 1
-    fi
+        # This will probably fail because the dev is already gone.
+        ip route list table ${name} || :
+        retVal=$?
+        if [ $retVal -eq 2 ]; then
+          echo "table is empty"
+        elif [ $retVal -ne 0 ]; then
+          exit 1
+        fi
+        ip route del default via "''${route_vpn_gateway:?}" dev "''${dev:?}" table ${name} || :
+        ip route flush cache
+        ip route list table ${name} || :
+        retVal=$?
+        if [ $retVal -eq 2 ]; then
+          echo "table is empty"
+        elif [ $retVal -ne 0 ]; then
+          exit 1
+        fi
 
-    rm /run/openvpn/${name}/ifconfig_local
+        rm /run/openvpn/${name}/ifconfig_local
 
-    dependencies=(${quoteEach dependentServices})
-    for i in "''${dependencies[@]}"; do
-        systemctl stop "$i" || :
-    done
+        dependencies=(${quoteEach dependentServices})
+        for i in "''${dependencies[@]}"; do
+            systemctl stop "$i" || :
+        done
 
-    echo "Running route-down DONE"
-    '';
-  };
-
-  someEnabled = lib.any (lib.mapAttrsToList (name: c: c.enable) cfg);
+        echo "Running route-down DONE"
+      '';
+    };
 in
 {
   options =
@@ -207,7 +225,7 @@ in
         options = {
           enable = lib.mkEnableOption "OpenVPN config";
 
-          package = lib.mkPackageOptionMD pkgs "openvpn" {};
+          package = lib.mkPackageOption pkgs "openvpn" { };
 
           provider = lib.mkOption {
             description = "VPN provider, if given uses ready-made configuration.";
@@ -245,68 +263,76 @@ in
         };
       };
     in
-      {
-        shb.vpn = lib.mkOption {
-          description = "OpenVPN instances.";
-          default = {};
-          type = lib.types.attrsOf instanceOption;
-        };
+    {
+      shb.vpn = lib.mkOption {
+        description = "OpenVPN instances.";
+        default = { };
+        type = lib.types.attrsOf instanceOption;
       };
+    };
 
   config = {
     services.openvpn.servers =
       let
-        instanceConfig = name: c: lib.mkIf c.enable {
-          ${name} = {
-            autoStart = true;
+        instanceConfig =
+          name: c:
+          lib.mkIf c.enable {
+            ${name} = {
+              autoStart = true;
 
-            up = "mkdir -p /run/openvpn/${name}";
+              up = "mkdir -p /run/openvpn/${name}";
 
-            config = nordvpnConfig {
-              inherit name;
-              inherit (c) dev remoteServerIP authFile;
-              dependentServices = lib.optional (c.proxyPort != null) "tinyproxy-${name}.service";
+              config = nordvpnConfig {
+                inherit name;
+                inherit (c) dev remoteServerIP authFile;
+                dependentServices = lib.optional (c.proxyPort != null) "tinyproxy-${name}.service";
+              };
             };
           };
-        };
       in
-        lib.mkMerge (lib.mapAttrsToList instanceConfig cfg);
+      lib.mkMerge (lib.mapAttrsToList instanceConfig cfg);
 
-    systemd.tmpfiles.rules = map (name:
-      "d /tmp/openvpn/${name}.status 0700 root root"
-    ) (lib.attrNames cfg);
+    systemd.tmpfiles.rules = map (name: "d /tmp/openvpn/${name}.status 0700 root root") (
+      lib.attrNames cfg
+    );
 
     networking.iproute2.enable = true;
-    networking.iproute2.rttablesExtraConfig =
-      lib.concatStringsSep "\n" (lib.mapAttrsToList (name: c: "${toString c.routingNumber} ${name}") cfg);
+    networking.iproute2.rttablesExtraConfig = lib.concatStringsSep "\n" (
+      lib.mapAttrsToList (name: c: "${toString c.routingNumber} ${name}") cfg
+    );
 
     shb.tinyproxy =
       let
-        instanceConfig = name: c: lib.mkIf (c.enable && c.proxyPort != null) {
-          ${name} = {
-            enable = true;
-            # package = pkgs.tinyproxy.overrideAttrs (old: {
-            #   withDebug = false;
-            #   patches = old.patches ++ [
-            #     (pkgs.fetchpatch {
-            #       name = "";
-            #       url = "https://github.com/tinyproxy/tinyproxy/pull/494/commits/2532ba09896352b31f3538d7819daa1fc3f829f1.patch";
-            #       sha256 = "sha256-Q0MkHnttW8tH3+hoCt9ACjHjmmZQgF6pC/menIrU0Co=";
-            #     })
-            #   ];
-            # });
-            dynamicBindFile = "/run/openvpn/${name}/ifconfig_local";
-            settings = {
-              Port = c.proxyPort;
-              Listen = "127.0.0.1";
-              Syslog = "On";
-              LogLevel = "Info";
-              Allow = [ "127.0.0.1" "::1" ];
-              ViaProxyName = ''"tinyproxy"'';
+        instanceConfig =
+          name: c:
+          lib.mkIf (c.enable && c.proxyPort != null) {
+            ${name} = {
+              enable = true;
+              # package = pkgs.tinyproxy.overrideAttrs (old: {
+              #   withDebug = false;
+              #   patches = old.patches ++ [
+              #     (pkgs.fetchpatch {
+              #       name = "";
+              #       url = "https://github.com/tinyproxy/tinyproxy/pull/494/commits/2532ba09896352b31f3538d7819daa1fc3f829f1.patch";
+              #       sha256 = "sha256-Q0MkHnttW8tH3+hoCt9ACjHjmmZQgF6pC/menIrU0Co=";
+              #     })
+              #   ];
+              # });
+              dynamicBindFile = "/run/openvpn/${name}/ifconfig_local";
+              settings = {
+                Port = c.proxyPort;
+                Listen = "127.0.0.1";
+                Syslog = "On";
+                LogLevel = "Info";
+                Allow = [
+                  "127.0.0.1"
+                  "::1"
+                ];
+                ViaProxyName = ''"tinyproxy"'';
+              };
             };
           };
-        };
       in
-        lib.mkMerge (lib.mapAttrsToList instanceConfig cfg);
+      lib.mkMerge (lib.mapAttrsToList instanceConfig cfg);
   };
 }

modules/blocks/zfs.nix

@@ -0,0 +1,259 @@
+{
+  config,
+  pkgs,
+  lib,
+  shb,
+  utils,
+  ...
+}:
+
+let
+  cfg = config.shb.zfs;
+
+  datasets =
+    if cfg.snapshotBeforeActivation.datasets != null then
+      cfg.snapshotBeforeActivation.datasets
+    else
+      config.boot.zfs.extraPools;
+in
+{
+  imports = [
+    ../../lib/module.nix
+  ];
+
+  options.shb.zfs = {
+    pools = lib.mkOption {
+      description = ''
+        Attrset of ZFS pools under which datasets will be created.
+
+        The ZFS pools are not managed by this module, they should already exist.
+
+        Each pool named here will be added to the [`boot.zfs.extraPools`](https://search.nixos.org/options?channel=unstable&include_modular_service_options=0&include_nixos_options=1&query=boot.zfs.extrapools&show=option:boot.zfs.extraPools) option.
+      '';
+      default = { };
+      type = lib.types.attrsOf (
+        lib.types.submodule {
+          options = {
+            datasets = lib.mkOption {
+              description = ''
+                ZFS Datasets.
+
+                Each entry in the attrset will be created and mounted in the given path.
+                The attrset name is the dataset name.
+
+                This block implements the following contracts:
+                  - mount
+              '';
+              default = { };
+              example = lib.literalExpression ''
+                shb.zfs."safe/postgresql".path = "/var/lib/postgresql";
+              '';
+              type = lib.types.attrsOf (
+                lib.types.submodule (
+                  { config, name, ... }:
+                  {
+                    options = {
+                      enable = lib.mkEnableOption "shb.zfs.datasets";
+
+                      path = lib.mkOption {
+                        type = lib.types.str;
+                        description = "Path this dataset should be mounted on. If the string 'none' is given, the dataset will not be mounted.";
+                      };
+
+                      mode = lib.mkOption {
+                        type = lib.types.nullOr lib.types.str;
+                        description = "If non null, unix mode to apply to the dataset root folder.";
+                        default = null;
+                        example = "ug=rwx,g+s";
+                      };
+
+                      owner = lib.mkOption {
+                        type = lib.types.nullOr lib.types.str;
+                        description = "If non null, unix user to apply to the dataset root folder.";
+                        default = null;
+                        example = "syncthing";
+                      };
+
+                      group = lib.mkOption {
+                        type = lib.types.nullOr lib.types.str;
+                        description = "If non null, unix group to apply to the dataset root folder.";
+                        default = null;
+                        example = "syncthing";
+                      };
+
+                      defaultACLs = lib.mkOption {
+                        type = lib.types.nullOr lib.types.str;
+                        description = ''
+                          If non null, default ACL to set on the dataset root folder.
+
+                          Executes "setfacl -d -m $acl $path"
+                        '';
+                        default = null;
+                        example = "g:syncthing:rwX";
+                      };
+
+                      after = lib.mkOption {
+                        type = lib.types.listOf lib.types.str;
+                        description = ''
+                          Order creating this dataset after the mentioned ones.
+                          This only works with datasets managed by this module.
+
+                          Use the name of the dataset without the pool name.
+                        '';
+                        default = [ ];
+                        example = lib.literalExpression ''
+                          [
+                            "backup"
+                          ]
+                        '';
+                      };
+
+                      backup = lib.mkOption {
+                        description = ''
+                          Backup contract consumer configuration.
+
+                          This contract will backup the files inside the dataset.
+                        '';
+                        type = lib.types.submodule {
+                          options = shb.contracts.backup.mkRequester {
+                            user = if config.owner == null then "root" else config.owner;
+                            sourceDirectories = [
+                              config.path
+                            ];
+                            sourceDirectoriesText = ''
+                              [
+                                shb.zfs.pools.<name>.datasets.<name>.path
+                              ]
+                            '';
+                          };
+                        };
+                      };
+
+                      datasetbackup = lib.mkOption {
+                        description = ''
+                          ZFS dataset backup contract configuration.
+
+                          This contract will take snaphots of the dataset.
+                        '';
+                        type = lib.types.submodule {
+                          options = shb.contracts.datasetbackup.mkRequester {
+                            dataset = name;
+                          };
+                        };
+                      };
+                    };
+                  }
+                )
+              );
+            };
+          };
+        }
+
+      );
+    };
+
+    snapshotBeforeActivation = {
+      enable = lib.mkOption {
+        type = lib.types.bool;
+        description = "Take a snapshot of all datasets before activation";
+        default = false;
+      };
+
+      template = lib.mkOption {
+        type = lib.types.str;
+        description = "Bash command to generate the snapshot name. $1 is the path to the new generation.";
+        default = ''pre-$(date --utc '+%y%m%dT%H%M%S')-$(basename "$1")'';
+      };
+
+      datasets = lib.mkOption {
+        type = lib.types.nullOr (lib.types.listOf lib.types.str);
+        description = ''
+          Defines all datasets to take a snapshot of.
+
+          If set to `null`, the default, take the list of datasets from `config.boot.zfs.extraPools`.
+
+          The snapshots are not recursive unless specificed in the `recursive` option.
+        '';
+      };
+
+      recursive = lib.mkOption {
+        type = lib.types.bool;
+        description = "If true, take snapshots recurisevly on the given datasets.";
+        default = false;
+      };
+    };
+  };
+
+  # The implementation is greatly inspired by https://discourse.nixos.org/t/configure-zfs-filesystems-after-install/48633/3
+  config = {
+    boot.zfs.extraPools = lib.uniqueStrings (builtins.attrNames cfg.pools);
+
+    systemd.services =
+      let
+        mkPool =
+          poolName: poolCfg: lib.listToAttrs (lib.mapAttrsToList (mkDataset poolName) poolCfg.datasets);
+
+        mkDataset =
+          poolName: name: cfg':
+          let
+            dataset = poolName + "/" + name;
+          in
+          lib.attrsets.nameValuePair "zfs-create-${utils.escapeSystemdPath dataset}" {
+            # oneshot is used to make the systemd service wait on completion of the script.
+            serviceConfig.Type = "oneshot";
+            unitConfig.DefaultDependencies = false;
+            requiredBy = [ "local-fs.target" ];
+            before = [ "local-fs.target" ];
+            after = [
+              "zfs-import-${poolName}.service"
+              "zfs-mount.service"
+            ]
+            ++ map (n: "zfs-create-${poolName}-${n}.service") cfg'.after;
+
+            unitConfig.ConditionPathIsMountPoint = lib.mkIf (cfg'.path != "none") "!${cfg'.path}";
+
+            script = ''
+              ${pkgs.zfs}/bin/zfs list ${dataset} > /dev/null 2>&1 \
+                || ${pkgs.zfs}/bin/zfs create \
+                   -o mountpoint=none \
+                   ${dataset} || :
+
+              [ "$(${pkgs.zfs}/bin/zfs get -H mountpoint -o value ${dataset})" = ${cfg'.path} ] \
+                || ${pkgs.zfs}/bin/zfs set \
+                   mountpoint="${cfg'.path}" \
+                   ${dataset}
+
+            ''
+            + lib.optionalString (cfg'.path != "none" && cfg'.mode != null) ''
+              chmod "${cfg'.mode}" "${cfg'.path}"
+            ''
+            + lib.optionalString (cfg'.path != "none" && cfg'.owner != null) ''
+              chown "${cfg'.owner}" "${cfg'.path}"
+            ''
+            + lib.optionalString (cfg'.path != "none" && cfg'.group != null) ''
+              chown :"${cfg'.group}" "${cfg'.path}"
+            ''
+            + lib.optionalString (cfg'.path != "none" && cfg'.defaultACLs != null) ''
+              ${pkgs.acl}/bin/setfacl -d -m "${cfg'.defaultACLs}" "${cfg'.path}"
+            '';
+          };
+        mergeAttrs = lib.foldl lib.mergeAttrs { };
+      in
+      mergeAttrs (lib.mapAttrsToList mkPool cfg.pools);
+
+    system.preSwitchChecks."shb-zfs-snapshot" = lib.mkIf cfg.snapshotBeforeActivation.enable (
+      ''
+        name="${cfg.snapshotBeforeActivation.template}"
+      ''
+      + (
+        let
+          recursiveFlag = lib.optionalString cfg.snapshotBeforeActivation.recursive "-r";
+        in
+        lib.concatMapStringsSep "\n" (ds: ''
+          echo "Taking ZFS snapshot of ${ds}@$name"
+          zfs snapshot ${recursiveFlag} ${ds}@"$name"
+        '') (lib.uniqueStrings datasets)
+      )
+    );
+  };
+}

modules/blocks/zfs/docs/default.md

@@ -0,0 +1,138 @@
+# ZFS Block {#blocks-zfs}
+
+Defined in [`/modules/blocks/zfs.nix`](@REPO@/modules/blocks/zfs.nix):
+
+```nix
+{
+  inputs,
+  ...
+}:
+{
+  imports = [
+    inputs.selfhostblocks.nixosModules.zfs
+  ];
+}
+```
+
+This block creates ZFS datasets, optionally mounts them and sets permissions on the mount point.
+It also enables taking snapshots on activation.
+
+## Features {#blocks-zfs-features}
+
+- Creates ZFS dataset which is [optionally mounted](#blocks-zfs-options-shb.zfs.pools._name_.datasets._name_.path).
+- Sets permissions, [owner](#blocks-zfs-options-shb.zfs.pools._name_.datasets._name_.owner), [group](#blocks-zfs-options-shb.zfs.pools._name_.datasets._name_.group) and [ACL](#blocks-zfs-options-shb.zfs.pools._name_.datasets._name_.defaultACLs) on the mount point.
+- Backup of the files in the dataset [`shb.zfs.<name>.backup`][backup] through the [backup contract](./contracts-backup.html).
+- Backup of the dataset itself [`shb.zfs.<name>.datasetbackup`][datasetbackup] through the [dataset backup contract](./contracts-datasetbackup.html).
+- Take a snapshot [before activating or deploying][activationsnapshot] allowing a safe rollback.
+
+[backup]: #blocks-zfs-options-shb.zfs.pools._name_.datasets._name_.backup
+[datasetbackup]: #blocks-zfs-options-shb.zfs.pools._name_.datasets._name_.datasetbackup
+[activationsnapshot]: #blocks-zfs-options-shb.zfs.snapshotBeforeActivation.enable
+
+## Usage {#blocks-zfs-usage}
+
+Create a dataset at `root/safe/users` mounted on `/var/lib/nixos`:
+
+```nix
+shb.zfs.pools.root.datasets."safe/users".path = "/var/lib/nixos";
+```
+
+Create a dataset at `backup/syncoid` but do not mount it:
+
+```nix
+shb.zfs.pools.backup.datasets."syncoid".path = "none";
+```
+
+Create a dataset at `root/syncthing` and set custom permissions and ACL.
+Permission and ACL are only enforced for the mount point.
+
+```nix
+shb.zfs.pools.root.datasets."syncthing" = {
+  path = "/srv/syncthing";
+
+  mode = "ug=rwx,g+s,o=";
+  owner = "syncthing";
+  group = "syncthing";
+  defaultACLs = "g:syncthing:rwX";
+};
+```
+
+### Backup dataset {#blocks-zfs-usage-backup-dataset}
+
+To backup the dataset directly, use the [dataset backup contract](contracts-datasetbackup.html).
+For example, with the sanoid module as the dataset backup contract provider:
+
+```nix
+{
+  shb.zfs.pools.root.datasets.home = {
+    path = "/home";
+  };
+
+  shb.sanoid.backup."root/home" = {
+    request = shb.zfs.pools.root.datasets.home.datasetBackup.request;
+    template = "yearly";
+  };
+}
+```
+
+See the [Sanoid block](blocks-sanoid.html) for more examples.
+
+### Backup files {#blocks-zfs-usage-backup-files}
+
+To backup the files in the dataset, use the [file backup contract](contracts-backup.html).
+For example, with the restic module as the file backup contract provider:
+
+```nix
+{
+  shb.zfs.pools.root.datasets.home = {
+    path = "/home";
+  };
+
+  shb.restic.instances."myservice" = {
+    request = shb.zfs.pools.root.datasets.home.backup.request;
+  };
+}
+```
+
+See the [Restic block](blocks-restic.html) for more examples.
+
+### Snapshot before activation {#blocks-zfs-usage-snapshot-before-activation}
+
+To take a snapshot of all datasets before activating a new configuration:
+
+```nix
+{
+  shb.zfs.snapshotBeforeActivation.enable = true;
+}
+```
+
+This does not take a recursive snapshot by default, to do it recursively, do:
+
+```nix
+{
+  shb.zfs.snapshotBeforeActivation = {
+    enable = true;
+    recursive = true;
+  };
+}
+```
+
+We did not specify datasets manually, which means the datasets list comes from `boot.zfs.extraPools`.
+To specify the datasets manually, you can use:
+
+```nix
+{
+  shb.zfs.snapshotBeforeActivation = {
+    enable = true;
+    datasets = [ "root" "root/var" ];
+  };
+}
+```
+
+## Options Reference {#blocks-zfs-options}
+
+```{=include=} options
+id-prefix: blocks-zfs-options-
+list-id: selfhostblocks-block-zfs-options
+source: @OPTIONS_JSON@
+```

modules/contracts/backup.nix

@@ -0,0 +1,284 @@
+{
+  lib,
+  shb,
+  pkgs,
+  ...
+}:
+let
+  inherit (lib)
+    concatStringsSep
+    literalMD
+    mkOption
+    optionalAttrs
+    optionalString
+    ;
+  inherit (lib.types)
+    listOf
+    nonEmptyListOf
+    submodule
+    str
+    ;
+  inherit (shb) anyNotNull;
+in
+{
+  mkRequest =
+    {
+      user ? "",
+      userText ? null,
+      sourceDirectories ? [ "/var/lib/example" ],
+      sourceDirectoriesText ? null,
+      excludePatterns ? [ ],
+      excludePatternsText ? null,
+      beforeBackup ? [ ],
+      beforeBackupText ? null,
+      afterBackup ? [ ],
+      afterBackupText ? null,
+    }:
+    mkOption {
+      description = ''
+        Request part of the backup contract.
+
+        Options set by the requester module
+        enforcing how to backup files.
+      '';
+
+      default = {
+        inherit user sourceDirectories excludePatterns;
+        hooks = {
+          inherit beforeBackup afterBackup;
+        };
+      };
+
+      defaultText =
+        optionalString
+          (anyNotNull [
+            userText
+            sourceDirectoriesText
+            excludePatternsText
+            beforeBackupText
+            afterBackupText
+          ])
+          (literalMD ''
+            {
+              user = ${if userText != null then userText else user};
+              sourceDirectories = ${
+                if sourceDirectoriesText != null then
+                  sourceDirectoriesText
+                else
+                  "[ " + concatStringsSep " " sourceDirectories + " ]"
+              };
+              excludePatterns = ${
+                if excludePatternsText != null then
+                  excludePatternsText
+                else
+                  "[ " + concatStringsSep " " excludePatterns + " ]"
+              };
+              hooks.beforeBackup = ${
+                if beforeBackupText != null then
+                  beforeBackupText
+                else
+                  "[ " + concatStringsSep " " beforeBackup + " ]"
+              };
+              hooks.afterBackup = ${
+                if afterBackupText != null then afterBackupText else "[ " + concatStringsSep " " afterBackup + " ]"
+              };
+            };
+          '');
+
+      type = submodule {
+        options = {
+          user =
+            mkOption {
+              description = ''
+                Unix user doing the backups.
+              '';
+              type = str;
+              example = "vaultwarden";
+              default = user;
+            }
+            // optionalAttrs (userText != null) {
+              defaultText = literalMD userText;
+            };
+
+          sourceDirectories =
+            mkOption {
+              description = "Directories to backup.";
+              type = nonEmptyListOf str;
+              example = "/var/lib/vaultwarden";
+              default = sourceDirectories;
+            }
+            // optionalAttrs (sourceDirectoriesText != null) {
+              defaultText = literalMD sourceDirectoriesText;
+            };
+
+          excludePatterns =
+            mkOption {
+              description = "File patterns to exclude.";
+              type = listOf str;
+              default = excludePatterns;
+            }
+            // optionalAttrs (excludePatternsText != null) {
+              defaultText = literalMD excludePatternsText;
+            };
+
+          hooks = mkOption {
+            description = "Hooks to run around the backup.";
+            default = { };
+            type = submodule {
+              options = {
+                beforeBackup =
+                  mkOption {
+                    description = "Hooks to run before backup.";
+                    type = listOf str;
+                    default = beforeBackup;
+                  }
+                  // optionalAttrs (beforeBackupText != null) {
+                    defaultText = literalMD beforeBackupText;
+                  };
+
+                afterBackup =
+                  mkOption {
+                    description = "Hooks to run after backup.";
+                    type = listOf str;
+                    default = afterBackup;
+                  }
+                  // optionalAttrs (afterBackupText != null) {
+                    defaultText = literalMD afterBackupText;
+                  };
+              };
+            };
+          };
+        };
+      };
+    };
+
+  mkResult =
+    {
+      restoreScript ? "restore",
+      restoreScriptText ? null,
+      backupService ? "backup.service",
+      backupServiceText ? null,
+    }:
+    mkOption {
+      description = ''
+        Result part of the backup contract.
+
+        Options set by the provider module that indicates the name of the backup and restore scripts.
+      '';
+      default = {
+        inherit restoreScript backupService;
+      };
+
+      defaultText =
+        optionalString
+          (anyNotNull [
+            restoreScriptText
+            backupServiceText
+          ])
+          (literalMD ''
+            {
+              restoreScript = ${if restoreScriptText != null then restoreScriptText else restoreScript};
+              backupService = ${if backupServiceText != null then backupServiceText else backupService};
+            }
+          '');
+
+      type = submodule {
+        options = {
+          restoreScript =
+            mkOption {
+              description = ''
+                Name of script that can restore the database.
+                One can then list snapshots with:
+
+                ```bash
+                $ ${if restoreScriptText != null then restoreScriptText else restoreScript} snapshots
+                <snapshot 1> <metadata>
+                <snapshot 2> <metadata>
+                ```
+
+                And restore the database with:
+
+                ```bash
+                $ ${if restoreScriptText != null then restoreScriptText else restoreScript} restore <snapshot 1>
+                ```
+              '';
+              type = str;
+              default = restoreScript;
+            }
+            // optionalAttrs (restoreScriptText != null) {
+              defaultText = literalMD restoreScriptText;
+            };
+
+          backupService =
+            mkOption {
+              description = ''
+                Name of service backing up the database.
+
+                This script can be ran manually to backup the database:
+
+                ```bash
+                $ systemctl start ${if backupServiceText != null then backupServiceText else backupService}
+                ```
+              '';
+              type = str;
+              default = backupService;
+            }
+            // optionalAttrs (backupServiceText != null) {
+              defaultText = literalMD backupServiceText;
+            };
+        };
+      };
+    };
+
+  passthru.mkRestoreScript =
+    {
+      name,
+      user,
+      sudoPreserveEnvs ? [ ],
+      secretsFile,
+      restoreCmd,
+      listCmd,
+    }:
+    pkgs.writeShellApplication {
+      inherit name;
+
+      text = ''
+        usage() {
+          echo "$0 snapshots"
+          echo "$0 restore <snapshot>"
+        }
+
+        sudocmd() {
+          sudo --preserve-env=${lib.concatStringsSep "," sudoPreserveEnvs} -u ${user} "$@"
+        }
+
+        loadenv() {
+          set -a
+          # shellcheck disable=SC1090
+          source <(sudocmd cat "${secretsFile}")
+          set +a
+        }
+
+        if [ "$1" = "restore" ]; then
+          shift
+          snapshot="$1"
+
+          loadenv
+
+          echo "Will restore snapshot '$snapshot'"
+
+          sudocmd sh -c "${restoreCmd}"
+        elif [ "$1" = "snapshots" ]; then
+          shift
+
+          loadenv
+
+          sudocmd sh -c "${listCmd}"
+        else
+          usage
+          exit 1
+        fi
+      '';
+    };
+
+}

modules/contracts/backup/docs/default.md

@@ -0,0 +1,104 @@
+# Backup Contract {#contract-backup}
+
+This NixOS contract represents a backup job
+that will backup one or more files or directories
+on a regular schedule.
+
+It is a contract between a service that has files to be backed up
+and a service that backs up files.
+
+## Contract Reference {#contract-backup-options}
+
+These are all the options that are expected to exist for this contract to be respected.
+
+```{=include=} options
+id-prefix: contracts-backup-options-
+list-id: selfhostblocks-options
+source: @OPTIONS_JSON@
+```
+
+## Usage {#contract-backup-usage}
+
+A service that can be backed up will provide a `backup` option.
+
+Here is an example module defining such a `backup` option,
+which defines what directories to backup (`sourceDirectories`)
+and the user to backup with (`user`).
+
+```nix
+{
+  options = {
+    myservice.backup = mkOption {
+      type = lib.types.submodule {
+        options = shb.contracts.backup.mkRequester {
+          user = "nextcloud";
+          sourceDirectories = [
+            "/var/lib/nextcloud"
+          ];
+        };
+      };
+    };
+  };
+};
+```
+
+Now, on the other side we have a service that uses this `backup` option and actually backs up files.
+This service is a `provider` of this contract and will provide a `result` option.
+
+Let's assume such a module is available under the `backupService` option
+and that one can create multiple backup instances under `backupService.instances`.
+Then, to actually backup the `myservice` service, one would write:
+
+```nix
+backupService.instances.myservice = {
+  request = myservice.backup.request;
+  
+  settings = {
+    enable = true;
+
+    repository = {
+      path = "/srv/backup/myservice";
+    };
+
+    # ... Other options specific to backupService like scheduling.
+  };
+};
+```
+
+It is advised to backup files to different location, to improve redundancy.
+Thanks to using contracts, this can be made easily either with the same `backupService`:
+
+```nix
+backupService.instances.myservice_2 = {
+  request = myservice.backup.request;
+  
+  settings = {
+    enable = true;
+  
+    repository = {
+      path = "<remote path>";
+    };
+  };
+};
+```
+
+Or with another module `backupService_2`!
+
+## Providers of the Backup Contract {#contract-backup-providers}
+
+- [Restic block](blocks-restic.html).
+- [Borgbackup block](blocks-borgbackup.html).
+- [ZFS block](blocks-zfs.html).
+
+## Requester Blocks and Services {#contract-backup-requesters}
+
+- <!-- [ -->Audiobookshelf<!-- ](services-audiobookshelf.html). --> (no manual yet)
+- <!-- [ -->Deluge<!--](services-deluge.html). --> (no manual yet)
+- <!-- [ -->Grocy<!--](services-grocy.html). --> (no manual yet)
+- <!-- [ -->Hledger<!--](services-hledger.html). --> (no manual yet)
+- <!-- [ -->Home Assistant<!--](services-home-assistant.html). --> (no manual yet)
+- <!-- [ -->Jellyfin<!--](services-jellyfin.html). --> (no manual yet)
+- <!-- [ -->LLDAP<!--](blocks-ldap.html). --> (no manual yet)
+- [Nextcloud](services-nextcloud.html#services-nextcloudserver-usage-backup).
+- [Vaultwarden](services-vaultwarden.html#services-vaultwarden-backup).
+- <!-- [ -->*arr<!--](services-arr.html). --> (no manual yet)