ssl: update documentation

docs/redirects.json

@@ -1121,6 +1121,15 @@
   "blocks-monitoring-provisioning": [
     "blocks-monitoring.html#blocks-monitoring-provisioning"
   ],
+  "blocks-monitoring-ssl": [
+    "blocks-monitoring.html#blocks-monitoring-ssl"
+  ],
+  "blocks-monitoring-ssl-alerts": [
+    "blocks-monitoring.html#blocks-monitoring-ssl-alerts"
+  ],
+  "blocks-monitoring-ssl-dashboard": [
+    "blocks-monitoring.html#blocks-monitoring-ssl-dashboard"
+  ],
   "blocks-monitoring-usage": [
     "blocks-monitoring.html#blocks-monitoring-usage"
   ],
@@ -1457,6 +1466,12 @@
   "blocks-sops-usage-requester": [
     "blocks-sops.html#blocks-sops-usage-requester"
   ],
+  "blocks-ssl-debug-lets-encrypt": [
+    "blocks-ssl.html#blocks-ssl-debug-lets-encrypt"
+  ],
+  "blocks-ssl-monitoring": [
+    "blocks-ssl.html#blocks-ssl-monitoring"
+  ],
   "blocks-ssl-options-shb.certs.cas.selfsigned": [
     "blocks-ssl.html#blocks-ssl-options-shb.certs.cas.selfsigned"
   ],

modules/blocks/monitoring/docs/default.md

@@ -205,6 +205,45 @@ and the total requests to that service exceeds 1%.
 ![Error Dashboard Top Part](./assets/alert_rules_5xx_1.png)
 ![Error Dashboard Bottom Part](./assets/alert_rules_5xx_2.png)
 
+## SSL Certificates Dashboard and Alert {#blocks-monitoring-ssl}
+
+This dashboard shows Let's Encrypt renewal and setup jobs,
+or any job starting with "acme-" in the systemd service name.
+
+### Dashboard {#blocks-monitoring-ssl-dashboard}
+
+Variables:
+
+- The "Job" variable allows to focus on one or more certificate. "All" is the default.
+
+Graphs:
+
+- "Certificate Remaining Validity": Shows in how long will certificates expire.
+  It shows all files under `/var/lib/acme`.
+  An annotation will show up when the "Certificate Did Not Renew" alert fired or resolved.
+- "Schedule": Shows when a job will run.
+  The unit is "Datetime from Now" meaning it shows when a job ran or will run relative to the current time.
+- "Jobs in the Past Week": Shows stats on all renewal jobs that ran in the past.
+  It is sorted by the "Failed" column in descending order.
+  This way, one can directly see when a job has failures.
+  Note, the stats is not accurate because detecting jobs taking taking less than 15 seconds
+  is not supported well.
+- "Job Runs": Shows when a renewal job ran.
+  Normally, jobs running for less than 15 seconds will not show up in the graph.
+  We crafted a query that still shows them but the length is 100 seconds, even if the job
+  took less time to run.
+
+![SSL Dashboard No Filter](./assets/dashboards_SSL_all.png)
+![SSL Dashboard Filter Failing](./assets/dashboards_SSL_fail.png)
+
+### Alerts {#blocks-monitoring-ssl-alerts}
+
+- The "Certificate Did Not Renew" alert will fire if a backup job did not run at all in the last 24 hours
+    or if all runs were failures in the last 24 hours.
+  It will show up as annotations in the "Schedule" panel of the dashboard.
+
+![Late SSL Jobs Alert Firing](./assets/alert_rules_LateSSL_1.png)
+
 ## Options Reference {#blocks-monitoring-options}
 
 ```{=include=} options

modules/blocks/ssl/docs/default.md

@@ -62,6 +62,7 @@ We can ask Let's Encrypt to generate a certificate with:
 shb.certs.certs.letsencrypt."example.com" = {
   domain = "example.com";
   group = "nginx";
+  reloadServices = [ "nginx.service" ];
   dnsProvider = "linode";
   adminEmail = "admin@example.com";
   credentialsFile = /path/to/secret/file;
@@ -79,6 +80,13 @@ The credential file's content would be a key-value pair:
 LINODE_TOKEN=XYZ...
 ```
 
+If you use one subdomain per service,
+asking for certificates for a subdomain is done with:
+
+```nix
+shb.certs.certs.letsencrypt."example.com".extraDomains = [ "nextcloud.${domain}" ];
+```
+
 For other providers, see the [official instruction](https://go-acme.github.io/lego/dns/).
 
 ## Usage {#block-ssl-usage}
@@ -110,11 +118,35 @@ config.shb.certs.systemdService
 See also the [SSL certificate generator usage](contracts-ssl.html#ssl-contract-usage) for a more detailed usage
 example.
 
+## Monitoring {#blocks-ssl-monitoring}
+
+A dashboard for SSL certificates is provided.
+See [SSL Certificates Dashboard and Alert](blocks-monitoring.html#blocks-monitoring-ssl) section in the monitoring chapter.
+
 ## Debug {#block-ssl-debug}
 
 Each CA and Cert is generated by a systemd service whose name can be seen in the `systemdService`
 option. You can then see the latest errors messages using `journalctl`.
 
+### Let's Encrypt debug {#blocks-ssl-debug-lets-encrypt}
+
+Since the SHB SSL block uses the [`security.acme`][] module under the hood,
+knowing how that one works can become required if something goes wrong.
+
+For each domain and subdomain, noted as `fqdn` hereunder,
+the following systemd timers and services are created:
+
+- `acme-renew-${fqdn}.timer` triggers the `acme-order-renew-${fqdn}.service` service every day.
+- `acme-${fqdn}.service` (re)generate the initial self-signed certificate,
+   only if the following job never succeeded at least once yet.
+- `acme-order-renew-${fqdn}.service` asks for a new certificate
+   only if the certificate will expire in the next 30 days.
+   Has logic to only renew if the list of domains has not changed.
+
+Also, a global service named `acme-setup.service` is created
+
+[`security.acme`]: https://nixos.org/manual/nixos/stable/#module-security-acme
+
 ## Tests {#block-ssl-tests}
 
 The self-signed implementation is tested in [`/tests/vm/ssl.nix`](@REPO@/tests/vm/ssl.nix).