diff --git a/README.md b/README.md index 67c3fd7..40b9a7c 100644 --- a/README.md +++ b/README.md @@ -98,11 +98,67 @@ The goal is to help spread adoption of self-hosting by providing an opinionated Self Host Blocks takes care of common self-hosting needs: - Backup for all services. +- Automatic creation of ZFS datasets per service. - LDAP and SSO integration for most services. - Monitoring with Grafana and Prometheus stack with provided dashboards. - Automatic reverse proxy and certificate management for HTTPS. - VPN and proxy tunneling services. +### Services + +[Provided services](https://shb.skarabox.com/services.html) are: + +- Nextcloud +- Audiobookshelf +- Deluge + *arr stack +- Forgejo +- Grocy +- Hledger +- Home-Assistant +- Jellyfin +- Nextcloud +- Vaultwarden + +Like explained above, those services all benefit from +out of the box backup, +LDAP and SSO integration, +monitoring with Grafana, +reverse proxy and certificate management +and VPN integration for the *arr suite. + +Some services do not have an entry yet in the manual. +To know options for those, the only way for now +is to go to the [All Options][] section of the manual. + +[All Options]: https://shb.skarabox.com/options.html + +### Blocks + +To provided out of the box common functionality, +the services above use the following [common blocks][]: + +[common blocks]: https://shb.skarabox.com/blocks.html + +- Authelia +- BorgBackup +- Davfs +- LDAP +- Monitoring (Grafana - Prometheus - Loki stack) +- Nginx +- PostgreSQL +- Restic +- Sops +- SSL +- Tinyproxy +- VPN +- ZFS + +Those blocks can be used outside of Self Host Blocks too. + +Some blocks do not have an entry yet in the manual. +To know options for those, the only way for now +is to go to the [All Options][] section of the manual. + ### Unified Interfaces SHB's first goal is to provide unified [building blocks](#available-blocks) diff --git a/docs/default.nix b/docs/default.nix index f66962a..2516c0e 100644 --- a/docs/default.nix +++ b/docs/default.nix @@ -137,6 +137,11 @@ in stdenv.mkDerivation { '@OPTIONS_JSON@' \ ${individualModuleOptionsDocs [ ../modules/blocks/ssl.nix ]}/share/doc/nixos/options.json + substituteInPlace ./modules/blocks/monitoring/docs/default.md \ + --replace \ + '@OPTIONS_JSON@' \ + ${individualModuleOptionsDocs [ ../modules/blocks/monitoring.nix ]}/share/doc/nixos/options.json + substituteInPlace ./modules/blocks/postgresql/docs/default.md \ --replace \ '@OPTIONS_JSON@' \ diff --git a/docs/redirects.json b/docs/redirects.json index 03a843c..8c7b54d 100644 --- a/docs/redirects.json +++ b/docs/redirects.json @@ -14,6 +14,9 @@ "blocks-monitoring-configuration": [ "blocks-monitoring.html#blocks-monitoring-configuration" ], + "blocks-monitoring-nextcloud-dashboard": [ + "blocks-monitoring.html#blocks-monitoring-nextcloud-dashboard" + ], "blocks-monitoring-deluge-dashboard": [ "blocks-monitoring.html#blocks-monitoring-deluge-dashboard" ], @@ -26,6 +29,126 @@ "blocks-monitoring-provisioning": [ "blocks-monitoring.html#blocks-monitoring-provisioning" ], + "blocks-monitoring-options": [ + "blocks-monitoring.html#blocks-monitoring-options" + ], + "blocks-monitoring-options-shb.monitoring.adminPassword": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.adminPassword" + ], + "blocks-monitoring-options-shb.monitoring.adminPassword.request": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.adminPassword.request" + ], + "blocks-monitoring-options-shb.monitoring.adminPassword.request.group": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.adminPassword.request.group" + ], + "blocks-monitoring-options-shb.monitoring.adminPassword.request.mode": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.adminPassword.request.mode" + ], + "blocks-monitoring-options-shb.monitoring.adminPassword.request.owner": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.adminPassword.request.owner" + ], + "blocks-monitoring-options-shb.monitoring.adminPassword.request.restartUnits": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.adminPassword.request.restartUnits" + ], + "blocks-monitoring-options-shb.monitoring.adminPassword.result": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.adminPassword.result" + ], + "blocks-monitoring-options-shb.monitoring.adminPassword.result.path": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.adminPassword.result.path" + ], + "blocks-monitoring-options-shb.monitoring.contactPoints": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.contactPoints" + ], + "blocks-monitoring-options-shb.monitoring.debugLog": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.debugLog" + ], + "blocks-monitoring-options-shb.monitoring.domain": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.domain" + ], + "blocks-monitoring-options-shb.monitoring.enable": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.enable" + ], + "blocks-monitoring-options-shb.monitoring.grafanaPort": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.grafanaPort" + ], + "blocks-monitoring-options-shb.monitoring.lokiMajorVersion": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.lokiMajorVersion" + ], + "blocks-monitoring-options-shb.monitoring.lokiPort": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.lokiPort" + ], + "blocks-monitoring-options-shb.monitoring.orgId": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.orgId" + ], + "blocks-monitoring-options-shb.monitoring.prometheusPort": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.prometheusPort" + ], + "blocks-monitoring-options-shb.monitoring.provisionDashboards": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.provisionDashboards" + ], + "blocks-monitoring-options-shb.monitoring.secretKey": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.secretKey" + ], + "blocks-monitoring-options-shb.monitoring.secretKey.request": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.secretKey.request" + ], + "blocks-monitoring-options-shb.monitoring.secretKey.request.group": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.secretKey.request.group" + ], + "blocks-monitoring-options-shb.monitoring.secretKey.request.mode": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.secretKey.request.mode" + ], + "blocks-monitoring-options-shb.monitoring.secretKey.request.owner": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.secretKey.request.owner" + ], + "blocks-monitoring-options-shb.monitoring.secretKey.request.restartUnits": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.secretKey.request.restartUnits" + ], + "blocks-monitoring-options-shb.monitoring.secretKey.result": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.secretKey.result" + ], + "blocks-monitoring-options-shb.monitoring.secretKey.result.path": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.secretKey.result.path" + ], + "blocks-monitoring-options-shb.monitoring.smtp": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.smtp" + ], + "blocks-monitoring-options-shb.monitoring.smtp.from_address": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.smtp.from_address" + ], + "blocks-monitoring-options-shb.monitoring.smtp.from_name": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.smtp.from_name" + ], + "blocks-monitoring-options-shb.monitoring.smtp.host": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.smtp.host" + ], + "blocks-monitoring-options-shb.monitoring.smtp.passwordFile": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.smtp.passwordFile" + ], + "blocks-monitoring-options-shb.monitoring.smtp.port": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.smtp.port" + ], + "blocks-monitoring-options-shb.monitoring.smtp.username": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.smtp.username" + ], + "blocks-monitoring-options-shb.monitoring.ssl": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.ssl" + ], + "blocks-monitoring-options-shb.monitoring.ssl.paths": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.ssl.paths" + ], + "blocks-monitoring-options-shb.monitoring.ssl.paths.cert": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.ssl.paths.cert" + ], + "blocks-monitoring-options-shb.monitoring.ssl.paths.key": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.ssl.paths.key" + ], + "blocks-monitoring-options-shb.monitoring.ssl.systemdService": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.ssl.systemdService" + ], + "blocks-monitoring-options-shb.monitoring.subdomain": [ + "blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.subdomain" + ], "blocks-postgresql": [ "blocks-postgresql.html#blocks-postgresql" ], @@ -1136,8 +1259,8 @@ "services-nextcloudserver-features": [ "services-nextcloud.html#services-nextcloudserver-features" ], - "services-nextcloudserver-maintenance": [ - "services-nextcloud.html#services-nextcloudserver-maintenance" + "services-nextcloudserver-dashboard": [ + "services-nextcloud.html#services-nextcloudserver-dashboard" ], "services-nextcloudserver-options": [ "services-nextcloud.html#services-nextcloudserver-options" diff --git a/modules/blocks/monitoring/dashboards/Nextcloud.json b/modules/blocks/monitoring/dashboards/Nextcloud.json new file mode 100644 index 0000000..515691a --- /dev/null +++ b/modules/blocks/monitoring/dashboards/Nextcloud.json @@ -0,0 +1,2178 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": 13, + "links": [], + "panels": [ + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 0, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 19, + "options": { + "dedupStrategy": "none", + "enableLogDetails": true, + "prettifyLogMessage": false, + "showCommonLabels": false, + "showLabels": false, + "showTime": false, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "11.3.0+security-01", + "repeat": "other_service", + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "editorMode": "code", + "expr": "{unit=\"$other_service.service\"} | json | line_format \"{{.message}}\" | json | drop message | line_format \"[{{.app}} - {{.url}}] {{.Message}}: {{.exception_details}}{{.Previous_Message}}\"", + "queryType": "range", + "refId": "A" + } + ], + "title": "Panel Title", + "type": "logs" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 12, + "panels": [], + "title": "General", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "description": "Some stall time means that 100% of the CPU is used. It's not an issue if this happens occasionally but can mean the CPU is underpowered for the current use case if this happens most of the time.\nTo fix this, the \"nice\" property of processes can be adjusted.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 2, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "transparent", + "value": 0.05 + } + ] + }, + "unit": "percent" + }, + "overrides": [ + { + "matcher": { + "id": "byFrameRefID", + "options": "B" + }, + "properties": [ + { + "id": "custom.axisPlacement", + "value": "right" + }, + { + "id": "unit", + "value": "ms" + }, + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + }, + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 0, + 10 + ], + "fill": "dot" + } + }, + { + "id": "custom.lineWidth", + "value": 2 + }, + { + "id": "custom.fillOpacity", + "value": 34 + }, + { + "id": "min", + "value": -40 + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 1 + }, + "id": 8, + "options": { + "legend": { + "calcs": [ + "max", + "lastNotNull" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "maxHeight": 600, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "editorMode": "code", + "expr": "netdata_system_cpu_some_pressure_stall_time_ms_average{hostname=~\"$hostname\"} * -1", + "hide": false, + "instant": false, + "legendFormat": "some stall time", + "range": true, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "disableTextWrap": false, + "editorMode": "code", + "expr": "sum by(dimension, service_name) (netdata_systemd_service_cpu_utilization_percentage_average{hostname=~\"$hostname\", service_name=~\".*$service.*\"})", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "legendFormat": "{{service_name}} / {{dimension}}", + "range": true, + "refId": "used", + "useBackend": false + } + ], + "title": "CPU", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "axisSoftMin": -100, + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 2, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "transparent", + "value": 0.05 + } + ] + }, + "unit": "mbytes" + }, + "overrides": [ + { + "matcher": { + "id": "byFrameRefID", + "options": "A" + }, + "properties": [ + { + "id": "custom.axisPlacement", + "value": "right" + }, + { + "id": "unit", + "value": "ms" + }, + { + "id": "decimals" + }, + { + "id": "color", + "value": { + "fixedColor": "dark-red", + "mode": "fixed" + } + }, + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + }, + { + "id": "custom.lineWidth", + "value": 2 + } + ] + }, + { + "matcher": { + "id": "byFrameRefID", + "options": "B" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + }, + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 0, + 10 + ], + "fill": "dot" + } + }, + { + "id": "custom.lineWidth", + "value": 2 + }, + { + "id": "custom.fillOpacity", + "value": 10 + }, + { + "id": "custom.axisPlacement", + "value": "auto" + }, + { + "id": "custom.stacking", + "value": { + "group": "A", + "mode": "none" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 1 + }, + "id": 7, + "options": { + "legend": { + "calcs": [ + "max", + "lastNotNull" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "maxHeight": 600, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "editorMode": "code", + "expr": "netdata_system_memory_full_pressure_stall_time_ms_average{hostname=~\"$hostname\"} * -1", + "hide": false, + "instant": false, + "legendFormat": "full stall time", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "editorMode": "code", + "expr": "sum(netdata_mem_available_MiB_average{hostname=~\"$hostname\"})", + "hide": false, + "instant": false, + "legendFormat": "total available", + "range": true, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "disableTextWrap": false, + "editorMode": "code", + "expr": "sum by(dimension, service_name) (netdata_systemd_service_memory_usage_MiB_average{hostname=~\"$hostname\", service_name=~\".*$service.*\", dimension=\"ram\"})", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "legendFormat": "{{service_name}}", + "range": true, + "refId": "used", + "useBackend": false + } + ], + "title": "Memory", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 2, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "transparent", + "value": 0.05 + } + ] + }, + "unit": "KBs" + }, + "overrides": [ + { + "matcher": { + "id": "byFrameRefID", + "options": "B" + }, + "properties": [ + { + "id": "custom.axisPlacement", + "value": "right" + }, + { + "id": "unit", + "value": "ms" + }, + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + }, + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 0, + 10 + ], + "fill": "dot" + } + }, + { + "id": "custom.lineWidth", + "value": 2 + }, + { + "id": "custom.fillOpacity", + "value": 34 + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 9 + }, + "id": 4, + "options": { + "legend": { + "calcs": [ + "max", + "mean" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "maxHeight": 600, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "disableTextWrap": false, + "editorMode": "code", + "expr": "sum by(dimension) (netdata_system_net_kilobits_persec_average{hostname=~\"$hostname\"})", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "legendFormat": "{{dimension}}", + "range": true, + "refId": "used", + "useBackend": false + } + ], + "title": "Network I/O", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 2, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "transparent", + "value": 0.05 + } + ] + }, + "unit": "Kibits" + }, + "overrides": [ + { + "matcher": { + "id": "byFrameRefID", + "options": "A" + }, + "properties": [ + { + "id": "custom.axisPlacement", + "value": "right" + }, + { + "id": "unit", + "value": "ms" + }, + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + }, + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 0, + 10 + ], + "fill": "dot" + } + }, + { + "id": "custom.lineWidth", + "value": 2 + }, + { + "id": "custom.fillOpacity", + "value": 12 + }, + { + "id": "custom.stacking", + "value": { + "group": "A", + "mode": "none" + } + }, + { + "id": "min", + "value": -200 + } + ] + }, + { + "matcher": { + "id": "byFrameRefID", + "options": "B" + }, + "properties": [ + { + "id": "custom.axisPlacement", + "value": "right" + }, + { + "id": "unit", + "value": "ms" + }, + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + }, + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 0, + 10 + ], + "fill": "dot" + } + }, + { + "id": "custom.lineWidth", + "value": 2 + }, + { + "id": "custom.fillOpacity", + "value": 17 + }, + { + "id": "custom.stacking", + "value": { + "group": "A", + "mode": "none" + } + }, + { + "id": "min", + "value": -200 + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 9 + }, + "id": 6, + "options": { + "legend": { + "calcs": [ + "max", + "sum" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "maxHeight": 600, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "editorMode": "code", + "expr": "netdata_system_io_full_pressure_stall_time_ms_average{hostname=~\"$hostname\"} * -1", + "hide": false, + "instant": false, + "legendFormat": "full stall time", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "editorMode": "code", + "expr": "netdata_system_io_some_pressure_stall_time_ms_average{hostname=~\"$hostname\"} * -1", + "hide": false, + "instant": false, + "legendFormat": "some stall time", + "range": true, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "disableTextWrap": false, + "editorMode": "code", + "expr": "sum by(dimension, service_name) (netdata_systemd_service_disk_io_KiB_persec_average{hostname=~\"$hostname\", service_name=~\".*$service.*\", dimension=\"read\"})", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "legendFormat": "{{service_name}} / {{dimension}}", + "range": true, + "refId": "read", + "useBackend": false + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "editorMode": "code", + "expr": "sum by(dimension, service_name) (netdata_systemd_service_disk_io_KiB_persec_average{hostname=~\"$hostname\", service_name=~\".*$service.*\", dimension=\"write\"}) * -1", + "hide": false, + "instant": false, + "legendFormat": "{{service_name}} / {{dimension}}", + "range": true, + "refId": "write" + } + ], + "title": "Disk I/O", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "area" + } + }, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "percentage", + "steps": [ + { + "color": "transparent", + "value": null + }, + { + "color": "orange", + "value": 80 + }, + { + "color": "red", + "value": 90 + }, + { + "color": "transparent", + "value": 100 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "total" + }, + "properties": [ + { + "id": "custom.hideFrom", + "value": { + "legend": true, + "tooltip": false, + "viz": false + } + }, + { + "id": "custom.lineWidth", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 17 + }, + "id": 22, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "editorMode": "code", + "expr": "sum (phpfpm_active_processes{hostname=~\"$hostname\",pool=\"nextcloud\"})", + "hide": false, + "legendFormat": "active", + "range": true, + "refId": "active" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "editorMode": "code", + "expr": "sum (phpfpm_total_processes{hostname=~\"$hostname\",pool=\"nextcloud\"})", + "hide": false, + "instant": false, + "legendFormat": "total", + "range": true, + "refId": "total" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "editorMode": "code", + "expr": "sum (phpfpm_max_active_processes{hostname=~\"$hostname\",pool=\"nextcloud\"})", + "hide": false, + "instant": false, + "legendFormat": "max active", + "range": true, + "refId": "max active" + } + ], + "title": "PHP-FPM Processes", + "transformations": [ + { + "disabled": true, + "id": "calculateField", + "options": { + "binary": { + "left": { + "matcher": { + "id": "byName", + "options": "total" + } + }, + "operator": "*", + "right": { + "fixed": "0.8" + } + }, + "mode": "binary", + "reduce": { + "reducer": "sum" + } + } + } + ], + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 25 + }, + "id": 14, + "panels": [], + "title": "Network", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "description": "If requests occasionally take longer than the threshold time, that's fine. If instead most of the queries take longer than the threshold, performance issue should be investigated.", + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "purple", + "mode": "fixed", + "seriesBy": "max" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed+area" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "transparent", + "value": null + }, + { + "color": "red", + "value": 700000 + } + ] + }, + "unit": "µs" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 26 + }, + "id": 23, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "disableTextWrap": false, + "editorMode": "code", + "expr": "max by (hostname,child) (phpfpm_process_request_duration and (abs(phpfpm_process_request_duration - phpfpm_process_request_duration offset $__interval) > 1))", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "legendFormat": "__auto", + "range": true, + "refId": "A", + "useBackend": false + } + ], + "title": "PHP-FPM Request Duration", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 26 + }, + "id": 24, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "editorMode": "code", + "expr": "rate(phpfpm_max_listen_queue[2m])", + "hide": false, + "instant": false, + "legendFormat": "{{hostname}} - max queue", + "range": true, + "refId": "B" + }, + { + "editorMode": "code", + "expr": "phpfpm_listen_queue", + "legendFormat": "{{hostname}} - queue", + "range": true, + "refId": "A" + } + ], + "title": "PHP-FPM Requests Queue Length", + "type": "timeseries" + }, + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "points", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "ms" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 34 + }, + "id": 9, + "options": { + "legend": { + "calcs": [ + "max", + "mean", + "variance" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "maxHeight": 600, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "editorMode": "code", + "expr": "{hostname=~\"$hostname\",unit=\"nginx.service\"} | pattern \"<_> <_> \" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | server_name =~ \"^$subdomain.*\"", + "legendFormat": "", + "queryType": "range", + "refId": "A" + } + ], + "title": "Requests", + "transformations": [ + { + "id": "extractFields", + "options": { + "keepTime": true, + "replace": true, + "source": "labels" + } + }, + { + "id": "organize", + "options": { + "excludeByName": { + "body_bytes_sent": true, + "bytes_sent": true, + "gzip_ration": true, + "job": true, + "line": true, + "post": true, + "referrer": true, + "remote_addr": false, + "remote_user": true, + "request": true, + "request_length": true, + "status": true, + "time_local": true, + "unit": true, + "upstream_addr": true, + "upstream_connect_time": true, + "upstream_header_time": true, + "upstream_response_time": true, + "upstream_status": false, + "user_agent": true + }, + "includeByName": {}, + "indexByName": {}, + "renameByName": {} + } + }, + { + "id": "convertFieldType", + "options": { + "conversions": [ + { + "dateFormat": "", + "destinationType": "number", + "targetField": "request_time" + } + ], + "fields": {} + } + }, + { + "id": "partitionByValues", + "options": { + "fields": [ + "server_name", + "remote_addr" + ], + "keepFields": false + } + }, + { + "id": "renameByRegex", + "options": { + "regex": "request_time (.*)", + "renamePattern": "$1" + } + } + ], + "type": "timeseries" + }, + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "status" + }, + "properties": [ + { + "id": "custom.width", + "value": 70 + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 42 + }, + "id": 3, + "options": { + "cellHeight": "sm", + "footer": { + "countRows": false, + "fields": "", + "reducer": [ + "sum" + ], + "show": false + }, + "showHeader": true, + "sortBy": [ + { + "desc": true, + "displayName": "Time" + } + ] + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "editorMode": "code", + "expr": "{hostname=~\"$hostname\",unit=\"nginx.service\"} | pattern \"<_> <_> \" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | upstream_addr =~ \"$upstream_addr\"", + "queryType": "range", + "refId": "A" + } + ], + "title": "Requests Details", + "transformations": [ + { + "id": "extractFields", + "options": { + "source": "Line" + } + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Line": true, + "id": true, + "labels": true, + "server_name": true, + "time_local": true, + "tsNs": true, + "upstream_addr": true + }, + "includeByName": {}, + "indexByName": { + "Line": 2, + "Time": 1, + "body_bytes_sent": 13, + "bytes_sent": 12, + "gzip_ration": 16, + "id": 4, + "labels": 0, + "post": 17, + "referrer": 14, + "remote_addr": 7, + "remote_user": 8, + "request": 6, + "request_length": 10, + "request_time": 20, + "server_name": 11, + "status": 5, + "time_local": 9, + "tsNs": 3, + "upstream_addr": 18, + "upstream_connect_time": 22, + "upstream_header_time": 23, + "upstream_response_time": 21, + "upstream_status": 19, + "user_agent": 15 + }, + "renameByName": {} + } + } + ], + "type": "table" + }, + { + "datasource": { + "default": false, + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "status" + }, + "properties": [ + { + "id": "custom.width", + "value": 70 + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 50 + }, + "id": 11, + "options": { + "cellHeight": "sm", + "footer": { + "countRows": false, + "fields": "", + "reducer": [ + "sum" + ], + "show": false + }, + "showHeader": true, + "sortBy": [] + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "editorMode": "code", + "expr": "{hostname=~\"$hostname\",unit=\"nginx.service\"} | pattern \"<_> <_> \" | line_format \"{{.line}}\" | json | __error__ != \"JSONParserErr\" | upstream_addr = \"$upstream_addr\" | status =~\"5..\"", + "queryType": "range", + "refId": "A" + } + ], + "title": "5XX Requests Details", + "transformations": [ + { + "id": "extractFields", + "options": { + "source": "Line" + } + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Line": true, + "id": true, + "labels": true, + "server_name": true, + "time_local": true, + "tsNs": true, + "upstream_addr": true + }, + "includeByName": {}, + "indexByName": { + "Line": 2, + "Time": 1, + "body_bytes_sent": 13, + "bytes_sent": 12, + "gzip_ration": 16, + "id": 4, + "labels": 0, + "post": 17, + "referrer": 14, + "remote_addr": 7, + "remote_user": 8, + "request": 6, + "request_length": 10, + "request_time": 20, + "server_name": 11, + "status": 5, + "time_local": 9, + "tsNs": 3, + "upstream_addr": 18, + "upstream_connect_time": 22, + "upstream_header_time": 23, + "upstream_response_time": 21, + "upstream_status": 19, + "user_agent": 15 + }, + "renameByName": {} + } + } + ], + "type": "table" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 58 + }, + "id": 18, + "panels": [], + "title": "Logs", + "type": "row" + }, + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 59 + }, + "id": 20, + "maxPerRow": 2, + "options": { + "dedupStrategy": "none", + "enableLogDetails": true, + "prettifyLogMessage": false, + "showCommonLabels": false, + "showLabels": false, + "showTime": true, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "11.4.0", + "repeat": "other_service", + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "editorMode": "code", + "expr": "{hostname=~\"$hostname\",unit=\"$other_service.service\"} | line_format \"{{ if hasPrefix \\\"{\\\" __line__ }}{{ with $parsed := fromJson __line__ }}[{{.app}} - {{.url}}] {{ if hasPrefix \\\"{\\\" .message }}{{ with $mParsed := fromJson .message }}{{ $mParsed.Message }} @ {{ $mParsed.File }}:{{ $mParsed.Line }}{{ end }}{{ else }}{{ .message }}{{ end }}{{ end }}{{ else }}{{ __line__ }}{{ end }}\"", + "queryType": "range", + "refId": "A" + } + ], + "title": "Log: $other_service", + "type": "logs" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 89 + }, + "id": 15, + "panels": [], + "title": "Backup", + "type": "row" + }, + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 90 + }, + "id": 16, + "maxPerRow": 2, + "options": { + "dedupStrategy": "none", + "enableLogDetails": true, + "prettifyLogMessage": false, + "showCommonLabels": false, + "showLabels": false, + "showTime": true, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "11.4.0", + "repeat": "service_backup", + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "editorMode": "code", + "expr": "{hostname=~\"$hostname\",unit=\"$service_backup.service\"}", + "legendFormat": "", + "queryType": "range", + "refId": "A" + } + ], + "title": "Log: $service_backup", + "transformations": [ + { + "disabled": true, + "id": "extractFields", + "options": { + "source": "Line" + } + }, + { + "disabled": true, + "id": "organize", + "options": { + "excludeByName": { + "Line": true, + "id": true, + "labels": true, + "tsNs": true + }, + "includeByName": {}, + "indexByName": {}, + "renameByName": {} + } + } + ], + "type": "logs" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 100 + }, + "id": 13, + "panels": [], + "title": "Supporting Services", + "type": "row" + }, + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "fieldConfig": { + "defaults": { + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "duration_ms" + }, + "properties": [ + { + "id": "custom.width", + "value": 100 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "unit" + }, + "properties": [ + { + "id": "custom.width", + "value": 150 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "statement" + }, + "properties": [ + { + "id": "custom.width", + "value": 505 + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 101 + }, + "id": 10, + "links": [ + { + "title": "explore", + "url": "https://grafana.tiserbox.com/explore?panes=%7B%22HWt%22:%7B%22datasource%22:%22cd6cc53e-840c-484d-85f7-96fede324006%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bunit%3D%5C%22nginx.service%5C%22%7D%20%7C%20pattern%20%5C%22%3C_%3E%20%3C_%3E%20%3Cline%3E%5C%22%20%7C%20line_format%20%5C%22%7B%7B.line%7D%7D%5C%22%20%7C%20json%20%7C%20status%20%21~%20%5C%222..%5C%22%20%7C%20__error__%20%21%3D%20%5C%22JSONParserErr%5C%22%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22cd6cc53e-840c-484d-85f7-96fede324006%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%22now-6h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1&orgId=1" + } + ], + "options": { + "cellHeight": "sm", + "footer": { + "countRows": false, + "fields": "", + "reducer": [ + "sum" + ], + "show": false + }, + "showHeader": true, + "sortBy": [ + { + "desc": true, + "displayName": "Time" + } + ] + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "editorMode": "code", + "expr": "{hostname=~\"$hostname\",unit=\"postgresql.service\"} | regexp \".*duration: (?P[0-9.]+) ms (?P.*)\" | duration_ms > 1000", + "queryType": "range", + "refId": "A" + } + ], + "title": "Slow PostgreSQL Queries", + "transformations": [ + { + "id": "extractFields", + "options": { + "keepTime": false, + "replace": false, + "source": "labels" + } + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Line": true, + "Time": false, + "id": true, + "job": true, + "labels": true, + "tsNs": true, + "unit": true + }, + "includeByName": {}, + "indexByName": { + "Line": 6, + "Time": 0, + "duration_ms": 1, + "id": 8, + "job": 2, + "labels": 5, + "statement": 4, + "tsNs": 7, + "unit": 3 + }, + "renameByName": {} + } + } + ], + "type": "table" + }, + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 109 + }, + "id": 2, + "options": { + "dedupStrategy": "none", + "enableLogDetails": false, + "prettifyLogMessage": false, + "showCommonLabels": false, + "showLabels": false, + "showTime": true, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "cd6cc53e-840c-484d-85f7-96fede324006" + }, + "editorMode": "code", + "expr": "{unit=\"redis-$service.service\"} |= ``", + "queryType": "range", + "refId": "A" + } + ], + "title": "Redis", + "type": "logs" + } + ], + "preload": false, + "schemaVersion": 40, + "tags": [], + "templating": { + "list": [ + { + "current": { + "text": [ + "baryum" + ], + "value": [ + "baryum" + ] + }, + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "definition": "label_values(up,hostname)", + "includeAll": false, + "multi": true, + "name": "hostname", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(up,hostname)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": "", + "type": "query" + }, + { + "current": { + "text": "nextcloud", + "value": "nextcloud" + }, + "description": "", + "hide": 2, + "name": "service", + "query": "nextcloud", + "skipUrlSync": true, + "type": "constant" + }, + { + "current": { + "text": "n", + "value": "n" + }, + "hide": 2, + "name": "subdomain", + "query": "n", + "skipUrlSync": true, + "type": "constant" + }, + { + "current": { + "text": "unix:/run/phpfpm/nextcloud.sock", + "value": "unix:/run/phpfpm/nextcloud.sock" + }, + "hide": 2, + "name": "upstream_addr", + "query": "unix:/run/phpfpm/nextcloud.sock", + "skipUrlSync": true, + "type": "constant" + }, + { + "current": { + "text": "All", + "value": "$__all" + }, + "datasource": { + "type": "prometheus", + "uid": "df80f9f5-97d7-4112-91d8-72f523a02b09" + }, + "definition": "label_values({unit_name=~\".*$service.*\", unit_name=~\".*backups.*\", unit_name!~\".*restore_gen\"},unit_name)", + "description": "", + "hide": 2, + "includeAll": true, + "name": "service_backup", + "options": [], + "query": { + "qryType": 1, + "query": "label_values({unit_name=~\".*$service.*\", unit_name=~\".*backups.*\", unit_name!~\".*restore_gen\"},unit_name)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": "", + "sort": 1, + "type": "query" + }, + { + "current": { + "text": "All", + "value": "$__all" + }, + "definition": "label_values(netdata_systemd_service_unit_state_state_average{unit_name=~\".*nextcloud.*\", unit_name!~\".*backup.*\", unit_name!~\".*redis.*\"},unit_name)", + "hide": 2, + "includeAll": true, + "name": "other_service", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(netdata_systemd_service_unit_state_state_average{unit_name=~\".*nextcloud.*\", unit_name!~\".*backup.*\", unit_name!~\".*redis.*\"},unit_name)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": "", + "type": "query" + } + ] + }, + "time": { + "from": "now-2d", + "to": "now" + }, + "timepicker": {}, + "timezone": "browser", + "title": "Nextcloud", + "uid": "cdsszybv2gow0d", + "version": 49, + "weekStart": "" +} diff --git a/modules/blocks/monitoring/docs/default.md b/modules/blocks/monitoring/docs/default.md index 721a615..c78462f 100644 --- a/modules/blocks/monitoring/docs/default.md +++ b/modules/blocks/monitoring/docs/default.md @@ -101,8 +101,8 @@ This dashboard is meant to be the first stop to understand why a service is misb ![](./assets/dashboards_Errors_1.png) ![](./assets/dashboards_Errors_2.png) -The yellow and red dashed vertical bars correspond to the [Requests Error Budget -Alert](#blocks-monitoring-budget-alerts) firing. +The yellow and red dashed vertical bars correspond to the +[Requests Error Budget Alert](#blocks-monitoring-budget-alerts) firing. ## Performance Dashboard {#blocks-monitoring-performance-dashboard} @@ -112,6 +112,10 @@ This dashboard is meant to be the first stop to understand why a service is perf ![Performance Dashboard Middle Part](./assets/dashboards_Performance_2.png) ![Performance Dashboard Bottom Part](./assets/dashboards_Performance_3.png) +## Nextcloud Dashboard {#blocks-monitoring-nextcloud-dashboard} + +See [Nextcloud service](./services-nextcloud.html#services-nextcloudserver-dashboard) manual. + ## Deluge Dashboard {#blocks-monitoring-deluge-dashboard} This dashboard is used to monitor a [deluge](./services-deluge.html) instance. @@ -126,3 +130,11 @@ and the total requests to that service exceeds 1%. ![Error Dashboard Top Part](./assets/alert_rules_5xx_1.png) ![Error Dashboard Bottom Part](./assets/alert_rules_5xx_2.png) + +## Options Reference {#blocks-monitoring-options} + +```{=include=} options +id-prefix: blocks-monitoring-options- +list-id: selfhostblocks-blocks-monitoring-options +source: @OPTIONS_JSON@ +``` diff --git a/modules/services/nextcloud-server.nix b/modules/services/nextcloud-server.nix index 241c1bd..a78d175 100644 --- a/modules/services/nextcloud-server.nix +++ b/modules/services/nextcloud-server.nix @@ -168,7 +168,11 @@ in phpFpmPoolSettings = lib.mkOption { type = lib.types.nullOr (lib.types.attrsOf lib.types.anything); description = "Settings for PHPFPM."; - default = null; + default = { + "pm" = "static"; + "pm.max_children" = 5; + "pm.start_servers" = 5; + }; example = lib.literalExpression '' { "pm" = "dynamic"; diff --git a/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_1.png b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_1.png new file mode 100644 index 0000000..3e3b3c1 Binary files /dev/null and b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_1.png differ diff --git a/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_2.png b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_2.png new file mode 100644 index 0000000..f7c3548 Binary files /dev/null and b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_2.png differ diff --git a/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_3.png b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_3.png new file mode 100644 index 0000000..698e67a Binary files /dev/null and b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_3.png differ diff --git a/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_4.png b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_4.png new file mode 100644 index 0000000..565273e Binary files /dev/null and b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_4.png differ diff --git a/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_5.png b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_5.png new file mode 100644 index 0000000..01de3f0 Binary files /dev/null and b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_5.png differ diff --git a/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_6.png b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_6.png new file mode 100644 index 0000000..debb0c2 Binary files /dev/null and b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_6.png differ diff --git a/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_7.png b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_7.png new file mode 100644 index 0000000..6c0a48f Binary files /dev/null and b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_7.png differ diff --git a/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_error_parsing.png b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_error_parsing.png new file mode 100644 index 0000000..2610e12 Binary files /dev/null and b/modules/services/nextcloud-server/docs/assets/dashboards_Nextcloud_error_parsing.png differ diff --git a/modules/services/nextcloud-server/docs/default.md b/modules/services/nextcloud-server/docs/default.md index 1e22c3e..04b20d1 100644 --- a/modules/services/nextcloud-server/docs/default.md +++ b/modules/services/nextcloud-server/docs/default.md @@ -29,7 +29,7 @@ It is based on the nixpkgs Nextcloud server and provides opinionated defaults. - Forces PostgreSQL as the database. - Forces Redis as the cache and sets good defaults. - Backup of the [`shb.nextcloud.dataDir`][dataDir] through the [backup block](./blocks-backup.html). -- Monitoring of reverse proxy, PHP-FPM, and database backups through the [monitoring +- [Dashboard](#services-nextcloudserver-dashboard) for monitoring of reverse proxy, PHP-FPM, and database backups through the [monitoring block](./blocks-monitoring.html). - [Integration Tests](@REPO@/test/services/nextcloud.nix) - Tests system cron job is setup correctly. @@ -311,6 +311,9 @@ I don't have a good heuristic for what are good values here but what I found is that you don't want too high of a `max_children` value to avoid I/O strain on the hard drives, especially if you use spinning drives. +To see the effect of your settings, +go to the provided [Grafana dashboard](#services-nextcloudserver-dashboard). + ### Tweak PostgreSQL Settings {#services-nextcloudserver-usage-postgres} These settings will impact all databases since the NixOS Postgres module @@ -353,6 +356,9 @@ shb.nextcloud.postgresSettings = { }; ``` +To see the effect of your settings, +go to the provided [Grafana dashboard](#services-nextcloudserver-dashboard). + ### Backup {#services-nextcloudserver-usage-backup} Backing up Nextcloud data files using the [Restic block](blocks-restic.html) is done like so: @@ -526,12 +532,80 @@ by issuing the command `nextcloud-occ config:system:delete instanceid`. Head over to the [Nextcloud demo](demo-nextcloud-server.html) for a demo that installs Nextcloud with or without LDAP integration on a VM with minimal manual steps. -## Maintenance {#services-nextcloudserver-maintenance} +## Dashboard {#services-nextcloudserver-dashboard} -On the command line, the `occ` tool is called `nextcloud-occ`. +The dashboard is added to Grafana automatically under "Self Host Blocks > Nextcloud" +as long as the Nextcloud service is [enabled][] +as well as the [monitoring block][]. + +[enabled]: #services-nextcloudserver-options-shb.nextcloud.enable +[monitoring block]: ./blocks-monitoring.html#blocks-monitoring-options-shb.monitoring.enable + +- The *General* section shows Nextcloud related services. + This includes cronjobs, Redis and backup jobs. +- *CPU* shows stall time which means CPU is maxed out. + This graph is inverted so having a small area at the top means the stall time is low. +- *Memory* shows stall time which means some job is waiting on memory to be allocated. + This graph is inverted so having a small area at the top means the stall time is low. + Some stall time will always be present. Under 10% is fine + but having constantly over 50% usually means available memory is low and SWAP is being used. + *Memory* also shows available memory which is the remaining allocatable memory. +- Caveat: *Network I/O* shows the network input and output for + all services running, not only those related to Nextcloud. +- *Disk I/O* shows "some" stall time which means some jobs were waiting on disk I/O. + Disk is usually the slowest bottleneck so having "some" stall time is not surprising. + Fixing this can be done by using disks allowing higher speeds or switching to SSDs. + If the "full" stall time is shown, this means _all_ jobs were waiting on disk i/o which + can be more worrying. This could indicate a failing disk if "full" stall time appeared recently. + These graphs are inverted so having a small area at the top means the stall time is low. + *Memory* also shows available memory which is the remaining allocatable memory. +![Nextcloud Dashboard First Part](./assets/dashboards_Nextcloud_1.png) +- *PHP-FPM Processes* shows how many processes are used by PHP-FPM. + The orange area goes from 80% to 90% of the maximum allowed processes. + The read area goes from 90% to 100% of the maximum allowed processes. + If the number of active processes reaches those areas once in a while, that's fine + but if it happens most of the time, the maximum allowed processes should be increased. +- *PHP-FPM Request Duration* shows one dot per request and how long it took. + Request time is fine if it is under 400ms. + If most requests take longer than that, some [tracing](#services-nextcloudserver-server-usage-tracing) + is required to understand which subsystem is taking some time. + That being said, maybe another graph in this dashboard will show + why the requests are slow - like disk + or other processes hoarding some resources running at the same time. +- *PHP-FPM Requests Queue Length* shows how many requests are waiting + to be picked up by a PHP-FPM process. Usually, this graph won't show + anything as long as the *PHP-FPM Processes* graph is not in the red area. + Fixing this requires also increasing the maximum allowed processes. +![Nextcloud Dashboard Second Part](./assets/dashboards_Nextcloud_2.png) +- *Requests Details* shows all requests to the Nextcloud service and the related headers. +- *5XX Requests Details* shows only the requests having a 500 to 599 http status. + Having any requests appearing here should be investigated as soon as possible. +![Nextcloud Dashboard Third Part](./assets/dashboards_Nextcloud_3.png) +- *Log: \* shows all logs from related systemd `.service` job. + Having no line here most often means the job ran + at a time not currently included in the time range of the dashboard. +![Nextcloud Dashboard Fourth Part](./assets/dashboards_Nextcloud_4.png) +![Nextcloud Dashboard Fifth Part](./assets/dashboards_Nextcloud_5.png) +- A lot of care has been taken to parse error messages correctly. + Nextcloud mixes json and non-json messages so extracting errors + from json messages was not that easy. + Also, the stacktrace is reduced. + The result though is IMO pretty nice as can be seen by the following screenshot. + The top line is the original json message and the bottom one is the parsed error. +![Nextcloud Dashboard Error Parsing](./assets/dashboards_Nextcloud_error_parsing.png) +- *Backup logs* show the output of the backup jobs. + Here, there are two backup jobs, one for the core files of Nextcloud + stored on an SSD which includes the appdata folder. + The other backup job is for the external data stored on HDDs which contain all user files. +![Nextcloud Dashboard Sixth Part](./assets/dashboards_Nextcloud_6.png) +- *Slow PostgreSQL queries* shows all database queries taking longer than 1s to run. +- *Redis* shows all Redis log output. +![Nextcloud Dashboard Seventh Part](./assets/dashboards_Nextcloud_7.png) ## Debug {#services-nextcloudserver-debug} +On the command line, the `occ` tool is called `nextcloud-occ`. + In case of an issue, check the logs for any systemd service mentioned in this section. On startup, the oneshot systemd service `nextcloud-setup.service` starts. After it finishes, the