diff --git a/docs/redirects.json b/docs/redirects.json index 56f5935..485cee2 100644 --- a/docs/redirects.json +++ b/docs/redirects.json @@ -1121,6 +1121,15 @@ "blocks-monitoring-provisioning": [ "blocks-monitoring.html#blocks-monitoring-provisioning" ], + "blocks-monitoring-ssl": [ + "blocks-monitoring.html#blocks-monitoring-ssl" + ], + "blocks-monitoring-ssl-alerts": [ + "blocks-monitoring.html#blocks-monitoring-ssl-alerts" + ], + "blocks-monitoring-ssl-dashboard": [ + "blocks-monitoring.html#blocks-monitoring-ssl-dashboard" + ], "blocks-monitoring-usage": [ "blocks-monitoring.html#blocks-monitoring-usage" ], @@ -1457,6 +1466,12 @@ "blocks-sops-usage-requester": [ "blocks-sops.html#blocks-sops-usage-requester" ], + "blocks-ssl-debug-lets-encrypt": [ + "blocks-ssl.html#blocks-ssl-debug-lets-encrypt" + ], + "blocks-ssl-monitoring": [ + "blocks-ssl.html#blocks-ssl-monitoring" + ], "blocks-ssl-options-shb.certs.cas.selfsigned": [ "blocks-ssl.html#blocks-ssl-options-shb.certs.cas.selfsigned" ], diff --git a/modules/blocks/monitoring/docs/assets/alert_rules_LateSSL_1.png b/modules/blocks/monitoring/docs/assets/alert_rules_LateSSL_1.png new file mode 100644 index 0000000..66cf312 Binary files /dev/null and b/modules/blocks/monitoring/docs/assets/alert_rules_LateSSL_1.png differ diff --git a/modules/blocks/monitoring/docs/assets/dashboards_SSL_all.png b/modules/blocks/monitoring/docs/assets/dashboards_SSL_all.png new file mode 100644 index 0000000..d69fcbc Binary files /dev/null and b/modules/blocks/monitoring/docs/assets/dashboards_SSL_all.png differ diff --git a/modules/blocks/monitoring/docs/assets/dashboards_SSL_fail.png b/modules/blocks/monitoring/docs/assets/dashboards_SSL_fail.png new file mode 100644 index 0000000..5268df4 Binary files /dev/null and b/modules/blocks/monitoring/docs/assets/dashboards_SSL_fail.png differ diff --git a/modules/blocks/monitoring/docs/default.md b/modules/blocks/monitoring/docs/default.md index 7d55c51..86e325c 100644 --- a/modules/blocks/monitoring/docs/default.md +++ b/modules/blocks/monitoring/docs/default.md @@ -205,6 +205,45 @@ and the total requests to that service exceeds 1%. ![Error Dashboard Top Part](./assets/alert_rules_5xx_1.png) ![Error Dashboard Bottom Part](./assets/alert_rules_5xx_2.png) +## SSL Certificates Dashboard and Alert {#blocks-monitoring-ssl} + +This dashboard shows Let's Encrypt renewal and setup jobs, +or any job starting with "acme-" in the systemd service name. + +### Dashboard {#blocks-monitoring-ssl-dashboard} + +Variables: + +- The "Job" variable allows to focus on one or more certificate. "All" is the default. + +Graphs: + +- "Certificate Remaining Validity": Shows in how long will certificates expire. + It shows all files under `/var/lib/acme`. + An annotation will show up when the "Certificate Did Not Renew" alert fired or resolved. +- "Schedule": Shows when a job will run. + The unit is "Datetime from Now" meaning it shows when a job ran or will run relative to the current time. +- "Jobs in the Past Week": Shows stats on all renewal jobs that ran in the past. + It is sorted by the "Failed" column in descending order. + This way, one can directly see when a job has failures. + Note, the stats is not accurate because detecting jobs taking taking less than 15 seconds + is not supported well. +- "Job Runs": Shows when a renewal job ran. + Normally, jobs running for less than 15 seconds will not show up in the graph. + We crafted a query that still shows them but the length is 100 seconds, even if the job + took less time to run. + +![SSL Dashboard No Filter](./assets/dashboards_SSL_all.png) +![SSL Dashboard Filter Failing](./assets/dashboards_SSL_fail.png) + +### Alerts {#blocks-monitoring-ssl-alerts} + +- The "Certificate Did Not Renew" alert will fire if a backup job did not run at all in the last 24 hours + or if all runs were failures in the last 24 hours. + It will show up as annotations in the "Schedule" panel of the dashboard. + +![Late SSL Jobs Alert Firing](./assets/alert_rules_LateSSL_1.png) + ## Options Reference {#blocks-monitoring-options} ```{=include=} options diff --git a/modules/blocks/ssl/docs/default.md b/modules/blocks/ssl/docs/default.md index 9e237fe..a860e4a 100644 --- a/modules/blocks/ssl/docs/default.md +++ b/modules/blocks/ssl/docs/default.md @@ -62,6 +62,7 @@ We can ask Let's Encrypt to generate a certificate with: shb.certs.certs.letsencrypt."example.com" = { domain = "example.com"; group = "nginx"; + reloadServices = [ "nginx.service" ]; dnsProvider = "linode"; adminEmail = "admin@example.com"; credentialsFile = /path/to/secret/file; @@ -79,6 +80,13 @@ The credential file's content would be a key-value pair: LINODE_TOKEN=XYZ... ``` +If you use one subdomain per service, +asking for certificates for a subdomain is done with: + +```nix +shb.certs.certs.letsencrypt."example.com".extraDomains = [ "nextcloud.${domain}" ]; +``` + For other providers, see the [official instruction](https://go-acme.github.io/lego/dns/). ## Usage {#block-ssl-usage} @@ -110,11 +118,35 @@ config.shb.certs.systemdService See also the [SSL certificate generator usage](contracts-ssl.html#ssl-contract-usage) for a more detailed usage example. +## Monitoring {#blocks-ssl-monitoring} + +A dashboard for SSL certificates is provided. +See [SSL Certificates Dashboard and Alert](blocks-monitoring.html#blocks-monitoring-ssl) section in the monitoring chapter. + ## Debug {#block-ssl-debug} Each CA and Cert is generated by a systemd service whose name can be seen in the `systemdService` option. You can then see the latest errors messages using `journalctl`. +### Let's Encrypt debug {#blocks-ssl-debug-lets-encrypt} + +Since the SHB SSL block uses the [`security.acme`][] module under the hood, +knowing how that one works can become required if something goes wrong. + +For each domain and subdomain, noted as `fqdn` hereunder, +the following systemd timers and services are created: + +- `acme-renew-${fqdn}.timer` triggers the `acme-order-renew-${fqdn}.service` service every day. +- `acme-${fqdn}.service` (re)generate the initial self-signed certificate, + only if the following job never succeeded at least once yet. +- `acme-order-renew-${fqdn}.service` asks for a new certificate + only if the certificate will expire in the next 30 days. + Has logic to only renew if the list of domains has not changed. + +Also, a global service named `acme-setup.service` is created + +[`security.acme`]: https://nixos.org/manual/nixos/stable/#module-security-acme + ## Tests {#block-ssl-tests} The self-signed implementation is tested in [`/tests/vm/ssl.nix`](@REPO@/tests/vm/ssl.nix).