diff --git a/README.md b/README.md index bcb6efa..b4c6fdb 100644 --- a/README.md +++ b/README.md @@ -3,34 +3,19 @@ *Building blocks for self-hosting with battery included.* SHB's (Self Host Blocks) goal is to provide a lower entry-bar for self-hosting. SHB provides -opinionated [building blocks](#building-blocks) fitting together to self-host any service you'd -want. Some [common services](#provided-services) are provided out of the box. +opinionated building blocks fitting together to self-host any service you want. Some common services +are provided out of the box. -To achieve this, SHB is using the full power of NixOS modules. Indeed, each building block and each -service is a NixOS module and uses the modules defined in -[Nixpkgs](https://github.com/NixOS/nixpkgs/). +- You are new to self hosting and want pre-configured services to deploy easily. Look at the + [services section](https://shb.skarabox.com/services.html). +- You are a seasoned self-hoster but want to enhance some services you deploy already. Go to the + [blocks section](https://shb.skarabox.com/blocks.html). +- You are a user of Self Host Blocks but would like to use your own implementation for a block. Head + over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org) to talk about it + (this is WIP). -Each building block defines a part of what a self-hosted app should provide. For example, HTTPS -access through a subdomain or Single Sign-On. The goal of SHB is to make sure those blocks all fit -together, whatever the actual implementation you choose. For example, the subdomain access could be -done using Caddy or Nginx. This is achieved by providing an explicit contract for each block and validating that contract using NixOS VM integration tests. - -One important goal of SHB is to be the smallest amount of code above what is available in -[nixpkgs](https://github.com/NixOS/nixpkgs). It should be the minimum necessary to make packages -available there conform with the contracts. This way, there are less chance of breakage when nixpkgs -gets updated. - -SHB provides some out of the box implementation of those blocks: -- Access through a subdomain ([Nginx](https://www.nginx.com/)). -- HTTPS access ([Nginx](https://www.nginx.com/) + [Letsencrypt](https://letsencrypt.org/)). -- Backup ([Borgmatic](https://torsion.org/borgmatic/) and/or [Restic](https://restic.net/)). -- Single sign-on ([Authelia](https://www.authelia.com/)). -- LDAP user management ([LLDAP](https://github.com/lldap/lldap)). -- Metrics, logs and alerting ([Grafana](https://grafana.com/) + [Prometheus](https://prometheus.io/) + [Loki](https://grafana.com/oss/loki/) + [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/) + [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)). -- Database setup (Only [Postgresql](https://www.postgresql.org/) so far). -- VPN tunnels with optional proxys ([OpenVPN](https://openvpn.net/) with [Tinyproxy](http://tinyproxy.github.io/)). - -SHB provides also services that integrate with those blocks out of the box. Progress is detailed in the [Supported Features](#supported-features) section. +SHB is using the full power of NixOS. Indeed, each building block and each service is a NixOS module +and uses the modules defined in [Nixpkgs](https://github.com/NixOS/nixpkgs/). > **Caution:** You should know that although I am using everything in this repo for my personal > production server, this is really just a one person effort for now and there are most certainly @@ -39,6 +24,8 @@ SHB provides also services that integrate with those blocks out of the box. Prog ## TOC +- [Core Principles](#core-principles) +- [Provided Blocks](#provided-blocks) - [Supported Features](#supported-features) - [Usage](#usage) - [Manual](#manual) @@ -51,6 +38,113 @@ SHB provides also services that integrate with those blocks out of the box. Prog - [License](#license) +## Why Should You Use Self Host Blocks? + +Self Host Blocks is an opinionated layer on top of nixpkgs that makes sure blocks and services fit +nicely together and it does that by having an extensive test suite. (To be honest, the test suite +exists but has very poor coverage yet). + +By using Self Host Blocks, you get all the benefits of NixOS which are, for self hosted applications +specifically: + +- declarative configuration; +- atomic configuration rollbacks; +- real programming language to define configurations; +- user-defined abstractions (create your own functions or NixOS modules on top of SHB!); +- integration with the rest of nixpkgs. + +Also, SHB intends to be a library, not a framework, so you can make it fit in your existing +deployment, slowly transitioning to using SHB one block at a time. + +That leaves one big issue though, application state management. For example, not all applications +can be downgraded as easily as configuration files can be. There's no silver bullet for that and +each application must be managed differently. NixOS at least allows us to handle those issues in a +principle manner. + +## Core Principles + +### Best practices by default + +Backups, SSL, monitoring, etc. should be enabled by default and should be easier to configure than +not having those. + +### Contracts With Tests + +Each building block defines a part of what a self-hosted app should provide. For example, HTTPS +access through a subdomain or Single Sign-On. + +The goal of SHB is to make sure those blocks all fit together, whatever the actual implementation. +For example, the subdomain access could be done using Caddy or Nginx. This is achieved by providing +an explicit contract for each block and validating that contract using NixOS VM integration tests. + +Ensuring the blocks respect their respective contracts is done through module and NixOS VM tests. + +### Nixpkgs First + +SHB uses as many packages already defined in [nixpkgs](https://github.com/NixOS/nixpkgs) as +possible. Not doing so would be at minimum a terrible waste of time and efficiency. + +SHB should then be the smallest amount of code above what is available in nixpkgs. It should be the +minimum necessary to make packages available there conform with the contracts. This way, there are +less chance of breakage when nixpkgs gets updated. Related, SHB should contribute as much upstream +as it makes sense. + +### Be a library, not a framework + +Users should be able to pick one block from SHB and add it to their pre-existing system. + +Deploying SHB to a machine is done through any of the [supported +methods](https://nixos.wiki/wiki/Applications#Deployment) for NixOS. + +### Perennial Software + +Thanks to the principles above, I believe Self Host Blocks can be useful even if the project one day +does not get updates anymore. You can always copy parts you like and incorporate that in your own repository. + +## Why Not? + +Self Host Block is still in alpha version. + +I'm still making progress and making breaking changes with no intent to be backwards compatible. I +will consider this project to be v1 when I'll have a contract fully defined for each block, with a +default implementation for each, a way for you to add your own implementation easily and with tests +to ensure the contracts and to be able to test your own implementation. + +That being said, what's already in this repo works and I use it personally. You can probably read +through the source code and find useful things you can copy over in your project. + +## Provided Blocks + +SHB provides some out of the box implementation of these blocks: +- Access through a subdomain with [Nginx](https://www.nginx.com/). +- HTTPS access with [Nginx](https://www.nginx.com/) and [Letsencrypt](https://letsencrypt.org/). +- Backup with [Borgmatic](https://torsion.org/borgmatic/) and/or [Restic](https://restic.net/). +- Single sign-on with [Authelia](https://www.authelia.com/). +- LDAP user management with [LLDAP](https://github.com/lldap/lldap). +- Metrics, logs and alerting with the [Grafana](https://grafana.com/) / + [Prometheus](https://prometheus.io/) / [Loki](https://grafana.com/oss/loki/) / + [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/) / + [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) suite with out of the box + dashboards and alerts. +- Database setup with [Postgresql](https://www.postgresql.org/). +- VPN tunnels and optional proxys with [OpenVPN](https://openvpn.net/) and + [Tinyproxy](http://tinyproxy.github.io/). + +SHB provides also services that integrate with those blocks out of the box. Progress is detailed in +the [Supported Features](#supported-features) section. + +## Provided Services + +- [`arr.nix`](./modules/services/arr.nix) for finding media https://wiki.servarr.com/. +- [`deluge.nix`](./modules/services/deluge.nix) for downloading linux isos https://deluge-torrent.org/. +- [`hledger.nix`](./modules/services/hledger.nix) for managing finances https://hledger.org/. +- [`home-assistant.nix`](./modules/services/home-assistant.nix) for private IoT https://www.home-assistant.io/. +- [`jellyfin.nix`](./modules/services/jellyfin.nix) for watching media https://jellyfin.org/. +- [Nextcloud Server](https://shb.skarabox.com/services-nextcloud.html) for private documents, contacts, calendar, etc. https://nextcloud.com. +- [`vaultwarden.nix`](./modules/services/vaultwarden.nix) for passwords https://github.com/dani-garcia/vaultwarden. + +The services above are those I am using myself. I intend to add more. + ## Supported Features Currently supported services and features are: @@ -70,13 +164,16 @@ Currently supported services and features are: - [X] Backup support. - [X] Monitoring through Prometheus and Grafana. - [X] Export systemd services status. - - [ ] Provide out of the box dashboards and alerts for common tasks. + - [X] Out of the box dashboards and alerts for common tasks. - [ ] LDAP auth. - [ ] SSO auth. - [X] Vaultwarden - [X] UI only accessible for `vaultwarden_user` LDAP group. - [X] `/admin` only accessible for `vaultwarden_admin` LDAP group. - - [WIP] True SSO support, see [dani-garcia/vaultwarden/issues/246](https://github.com/dani-garcia/vaultwarden/issues/246). For now, Authelia protects access to the UI but you need to login afterwards to Vaultwarden. So there are two login required. + - [WIP] True SSO support, see + [dani-garcia/vaultwarden/issues/246](https://github.com/dani-garcia/vaultwarden/issues/246). For + now, Authelia protects access to the UI but you need to login afterwards to Vaultwarden. So + there are two login required. - [X] Nextcloud - [X] LDAP auth, unfortunately we need to configure this manually. - [ ] Declarative setup. @@ -86,23 +183,27 @@ Currently supported services and features are: - [X] Optional tracing debug. - [ ] Export traces to Prometheus. - [ ] Export metrics to Prometheus. + - [X] Export requests to Prometheus (through Nginx). - [X] Home Assistant. - - [ ] Export metrics to Prometheus. - [X] LDAP auth through `homeassistant_user` LDAP group. - [ ] SSO auth. - [X] Backup support. -- [X] Jellyfin - [ ] Export metrics to Prometheus. + - [X] Export requests to Prometheus (through Nginx). +- [X] Jellyfin - [X] LDAP auth through `jellyfin_user` and `jellyfin_admin` LDAP groups. - [X] SSO auth. - [X] Backup support. -- [X] Hledger - [ ] Export metrics to Prometheus. + - [X] Export requests to Prometheus (through Nginx). +- [X] Hledger - [X] LDAP auth through `hledger_user` LDAP group. - [X] SSO auth. - [ ] Backup support. + - [ ] Export metrics to Prometheus. + - [X] Export requests to Prometheus (through Nginx). - [X] Database Postgres - - [ ] Slow log monitoring. + - [X] Slow log monitoring. - [ ] Export metrics to Prometheus. - [X] VPN tunnel - [X] Arr suite @@ -112,11 +213,11 @@ Currently supported services and features are: - [ ] Gitea to deploy - [ ] Scrutiny to monitor hard drives health - [ ] Export metrics to Prometheus. -- [x] QoL - - [x] Unit tests for modules. - - [x] Running in CI. - - [ ] Integration tests with real nodes. - - [ ] Self published documentation for options. +- [X] QoL + - [X] Unit tests for modules. + - [X] Running in CI. + - [X] Integration tests with real nodes. + - [X] Self published documentation for options. - [ ] Examples for all building blocks. ## Usage @@ -180,6 +281,8 @@ The best way for now to understand how to use those modules is to read the code how they are used in the [provided services](#provided-services) and in the [demos](#demos). Also, here are a few examples taken from my personal usage of selfhostblocks. +## Examples + ### Add SSL configuration This is pretty much a prerequisite for all services. @@ -363,21 +466,6 @@ $ curl --proxy 127.0.0.1:12000 'https://api.ipify.org?format=json' {"ip":"46.12.123.113"} ``` -## Provided Services - -- [`arr.nix`](./modules/services/arr.nix) for finding media https://wiki.servarr.com/. -- [`deluge.nix`](./modules/services/deluge.nix) for downloading linux isos https://deluge-torrent.org/. -- [`hledger.nix`](./modules/services/hledger.nix) for managing finances https://hledger.org/. -- [`home-assistant.nix`](./modules/services/home-assistant.nix) for private IoT https://www.home-assistant.io/. -- [`jellyfin.nix`](./modules/services/jellyfin.nix) for watching media https://jellyfin.org/. -- [Nextcloud Server](https://shb.skarabox.com/services-nextcloud.html) for private documents, contacts, calendar, etc https://nextcloud.com. -- [`vaultwarden.nix`](./modules/services/vaultwarden.nix) for passwords https://github.com/dani-garcia/vaultwarden. - -The services above are those I am using myself. I intend to add more. - -The best way for now to understand how to use those modules is to read the code linked above and see -how they are used in the demos. Also, here are a -few examples taken from my personal usage of selfhostblocks. ### Common Options @@ -531,24 +619,32 @@ Along the way, I made quite a few changes to the ubderlying nixpkgs module I'm u Run all tests: ```bash -$ nix build .#checks.${system}.all -# or -$ nix flake check -# or -$ nix run github:Mic92/nix-fast-build -- --skip-cached --flake ".#checks.$(nix eval --raw --impure --expr builtins.currentSystem)" +nix build .#checks.${system}.all ``` -Run one group of tests: +or ```bash -$ nix build .#checks.${system}.modules -$ nix build .#checks.${system}.vm_postgresql_peerAuth +nix flake check +``` + +or + +```bash +nix run github:Mic92/nix-fast-build -- --skip-cached --flake ".#checks.$(nix eval --raw --impure --expr builtins.currentSystem)" +``` + +Run one group of tests (this has bash autocompletion!): + +```bash +nix build .#checks.${system}.modules +nix build .#checks.${system}.vm_postgresql_peerAuth ``` Run one VM test interactively: ```bash -$ nix run .#checks.${system}.vm_postgresql_peerAuth.driverInteractive +nix run .#checks.${system}.vm_postgresql_peerAuth.driverInteractive ``` When you get to the shell, run either `start_all()` or `test_script()`. The former just starts all diff --git a/docs/coreprinciples.md b/docs/coreprinciples.md new file mode 100644 index 0000000..7bee9ec --- /dev/null +++ b/docs/coreprinciples.md @@ -0,0 +1,41 @@ +# Core Principles {#core-principles} + +## Best practices by default {#core-principles-best-practices} + +Backups, SSL, monitoring, etc. should be enabled by default and should be easier to configure than +not having those. + +## Contracts With Tests {#core-principles-contracts} + +Each building block defines a part of what a self-hosted app should provide. For example, HTTPS +access through a subdomain or Single Sign-On. + +The goal of SHB is to make sure those blocks all fit together, whatever the actual implementation. +For example, the subdomain access could be done using Caddy or Nginx. This is achieved by providing +an explicit contract for each block and validating that contract using NixOS VM integration tests. + +Ensuring the blocks respect their respective contracts is done through module and NixOS VM tests. + +## Nixpkgs First {#core-principles-nixpkgs} + +SHB uses as many packages already defined in [nixpkgs](https://github.com/NixOS/nixpkgs) as +possible. Not doing so would be at minimum a terrible waste of time and efficiency. + +SHB should then be the smallest amount of code above what is available in nixpkgs. It should be the +minimum necessary to make packages available there conform with the contracts. This way, there are +less chance of breakage when nixpkgs gets updated. Related, SHB should contribute as much upstream +as it makes sense. + +## Be a library, not a framework {#core-principles-library} + +Users should be able to pick one block from SHB and add it to their pre-existing system. + +Deploying SHB to a machine is done through any of the [supported +methods](https://nixos.wiki/wiki/Applications#Deployment) for NixOS. + +As few abstractions should be created as possible. Stick to NixOS modules. + +## Perennial Software {#core-principles-perennial} + +Thanks to the principles above, I believe Self Host Blocks can be useful even if the project one day +does not get updates anymore. You can always copy parts you like and incorporate that in your own repository. diff --git a/docs/manual.md b/docs/manual.md index 6cbe2f2..d20e31a 100644 --- a/docs/manual.md +++ b/docs/manual.md @@ -11,6 +11,10 @@ preface.md usage.md ``` +```{=include=} chapters html:into-file=//core-principles.html +coreprinciples.md +``` + ```{=include=} chapters html:into-file=//services.html services.md ``` diff --git a/docs/preface.md b/docs/preface.md index bb8b84f..ebe5dbf 100644 --- a/docs/preface.md +++ b/docs/preface.md @@ -1,5 +1,8 @@ # Preface {#preface} +Self Host Blocks intends to help you self host any service you would like with best practices out of +the box. + ::: {.note} Self Host Blocks is hosted on [GitHub](https://github.com/ibizaman/selfhostblocks). If you encounter problems or bugs then please report them on the [issue @@ -9,20 +12,16 @@ Feel free to join the dedicated Matrix room [matrix.org#selfhostblocks](https://matrix.to/#/#selfhostblocks:matrix.org). ::: -Self Host Blocks intends to help you self host any service you would like with best practices out of -the box. - -To achieve this, Self Host Blocks provides building blocks which each provide part of what a self -hosted app should do (SSO, HTTPS, etc.). It also provides some services that are already integrated -with all those building blocks. +Self Host Blocks provides building blocks, each providing part of what a self hosted app should do +(SSO, HTTPS, backup, etc.). It also provides some services that are already integrated with all +those building blocks (Nextcloud, Home Assistant, etc.). - You are new to self hosting and want pre-configured services to deploy easily. Look at the [services section](services.html). - You are a seasoned self-hoster but want to enhance some services you deploy already. Go to the [blocks section](blocks.html). - You are a user of Self Host Blocks but would like to use your own implementation for a block. Head - over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org) (the manual for this - is WIP). + over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org) (this is WIP). Self Host Blocks uses the full power of NixOS modules to achieve these goals. Blocks and service are both NixOS modules.