From 6bb11270a1fa108b95d0418e2eb17ab74d12897c Mon Sep 17 00:00:00 2001 From: "courtmanr@gmail.com" Date: Tue, 25 Nov 2025 00:26:00 +0000 Subject: [PATCH] Refactor auth and monitoring docs to be concise --- docs/OIDC.md | 222 +++------ docs/PROXY_AUTH.md | 268 ++--------- docs/TEMPERATURE_MONITORING.md | 831 +++------------------------------ docs/VM_DISK_MONITORING.md | 344 ++------------ 4 files changed, 178 insertions(+), 1487 deletions(-) diff --git a/docs/OIDC.md b/docs/OIDC.md index d1ecf71..080dc1f 100644 --- a/docs/OIDC.md +++ b/docs/OIDC.md @@ -1,185 +1,67 @@ -# OpenID Connect (OIDC) Single Sign-On +# πŸ” OIDC Single Sign-On -Pulse ships with first-class OIDC support so you can authenticate through identity providers such as Authentik, Keycloak, Okta, Azure AD, and others. +Enable Single Sign-On (SSO) with providers like Authentik, Keycloak, Okta, and Azure AD. -## Requirements +## πŸš€ Quick Start -- Pulse v4.16.0 or later -- A reachable Pulse public URL (used for redirect callbacks) -- An OIDC client registration on your IdP with the redirect URI: - ``` - https:///api/oidc/callback - ``` -- Scopes that include at least `openid` +1. **Configure Provider**: Create an OIDC application in your IdP. + * **Redirect URI**: `https:///api/oidc/callback` + * **Scopes**: `openid`, `profile`, `email` +2. **Enable in Pulse**: Go to **Settings β†’ Security β†’ Single sign-on (OIDC)**. +3. **Enter Details**: + * **Issuer URL**: The base URL of your IdP (e.g., `https://auth.example.com/application/o/pulse/`). + * **Client ID & Secret**: From your IdP. +4. **Save**: The login page will now show a "Continue with Single Sign-On" button. -## Quick Start +## βš™οΈ Configuration -1. Open **Settings β†’ Security β†’ Single sign-on (OIDC)**. -2. Toggle **Enable** and fill in the following fields: - - **Issuer URL** – Base issuer endpoint from your IdP metadata document. - - **Client ID** and **Client secret** – From your IdP application registration. - - **Redirect URL** – Optional. Pulse auto-populates this based on the public URL; override only when necessary. -3. Use the **Advanced options** section to customise scopes, claim names, or access restrictions by group, domain, or email. -4. Click **Save changes**. After a successful save the OIDC login button appears on the Pulse sign-in page. +| Setting | Description | +| :--- | :--- | +| **Issuer URL** | The OIDC provider's issuer URL. Must match the `iss` claim in tokens. | +| **Client ID** | The application ID from your provider. | +| **Client Secret** | The application secret. | +| **Redirect URL** | Auto-detected. Override only if running behind a complex proxy setup. | +| **Scopes** | Space-separated scopes. Default: `openid profile email`. | +| **Claim Mapping** | Map `email`, `username`, and `groups` to specific token claims. | -### Using the bundled mock IdP +### Access Control +Restrict access to specific users or groups: +* **Allowed Groups**: Only users in these groups can login. Requires the `groups` scope/claim. +* **Allowed Domains**: Restrict to specific email domains (e.g., `example.com`). +* **Allowed Emails**: Allow specific email addresses. -For local end-to-end testing Pulse ships with a Dex-based mock server configuration. With Docker running: - -```bash -./scripts/dev/start-oidc-mock.sh -``` - -This exposes an issuer at `http://127.0.0.1:5556/dex` with: - -- Client ID: `pulse-dev` -- Client secret: `pulse-secret` -- Redirect URIs: `http://127.0.0.1:5173/api/oidc/callback`, `http://127.0.0.1:7655/api/oidc/callback` -- Test user: `admin@example.com` / `password` - -Point the OIDC settings screen at that issuer, save, and use the SSO button to exercise the full login flow. - -## Classic password login stays - -OIDC is optional. Pulse continues to ship with the familiar username/password flow: - -- First-run setup still prompts you to create an admin credential or you can pre-seed it via `PULSE_AUTH_USER` / `PULSE_AUTH_PASS`. -- If OIDC is **enabled**, the login page shows both the password form and the **Continue with Single Sign-On** button. Either path issues the same session cookie (`pulse_session`). -- To run **password-only**, leave OIDC disabled (the default). For an **OIDC-first** experience, enable OIDC and rotate the local admin password to a randomly generated value; Pulse always keeps the password form available for emergency access. -- The `allowedGroups`, `allowedDomains`, and `allowedEmails` settings only affect OIDC logins; password authentication continues to honour the account you created locally. - -## Provider Cheat-Sheet - -You do not need to ship per-provider templates. Pulse speaks standard OIDC, so administrators bring their own identity provider and supply the issuer URL, client ID, and client secret they created for Pulse. Below are the high-level steps we tested against three common providersβ€”share these with users who ask β€œwhat do I enter?” +## πŸ“š Provider Examples ### Authentik +* **Type**: OAuth2/OpenID (Confidential) +* **Redirect URI**: `https://pulse.example.com/api/oidc/callback` +* **Signing Key**: Must use **RS256** (create a certificate/key pair if needed). +* **Issuer URL**: `https://auth.example.com/application/o/pulse/` -1. In Authentik, create a new **Provider** of type **OAuth2/OpenID**. - - **Name**: Pulse - - **Client type**: Confidential - - **Redirect URIs**: `https://pulse.example.com/api/oidc/callback` (replace with your Pulse URL) - - **Scopes**: Include `openid`, `profile`, and `email` - - Note the generated **Client ID** and **Client Secret** +### Keycloak +* **Client ID**: `pulse` +* **Access Type**: Confidential +* **Valid Redirect URIs**: `https://pulse.example.com/api/oidc/callback` +* **Issuer URL**: `https://keycloak.example.com/realms/myrealm` -2. Create an **Application** and link it to the provider you just created. +### Azure AD +* **Redirect URI**: `https://pulse.example.com/api/oidc/callback` (Web) +* **Issuer URL**: `https://login.microsoftonline.com//v2.0` +* **Note**: Enable "ID tokens" in Authentication settings. -3. In Pulse, configure OIDC with: - - **Issuer URL**: `https://auth.example.com/application/o/pulse/` (the full path to your application) - - **Client ID**: The client ID from your Authentik provider - - **Client Secret**: The client secret from your Authentik provider - - **Scopes**: `openid profile email` +## πŸ”§ Troubleshooting -4. In Authentik, open **Applications β†’ [your Pulse app] β†’ Advanced** and set a **Signing Key** that advertises the `RS256` algorithm (generate or assign an RSA key). Authentik defaults to `HS256` when no signing key is configured, which Pulse rejects with the error `unexpected signature algorithm "HS256"; expected ["RS256"]`. +| Issue | Solution | +| :--- | :--- | +| **`invalid_id_token`** | Issuer URL mismatch. Check logs (`LOG_LEVEL=debug`) to see the expected vs. received issuer. | +| **`unexpected signature algorithm "HS256"`** | Your IdP is signing with HS256. Configure it to use **RS256**. | +| **Redirect Loop** | Check `X-Forwarded-Proto` header (must be `https`) and cookie settings. | +| **Self-Signed Certs** | Mount your CA bundle to `/etc/ssl/certs/oidc-ca.pem` and set `OIDC_CA_BUNDLE`. | -5. For group-based access control: - - Set **Groups claim** to `groups` (Authentik's default) - - Add your allowed group names to **Allowed groups** (e.g., `admin`) - -**Important notes**: -- If you see "invalid_id_token" errors, the issuer URL might not match what Authentik puts in tokens. Check your Pulse logs with `LOG_LEVEL=debug` to see the exact error. The issuer claim in the token must match your configured `OIDC_ISSUER_URL` exactly. -- When using OIDC behind a reverse proxy with HTTPS, ensure `PUBLIC_URL` or `OIDC_REDIRECT_URL` uses `https://` (not `http://`). If these are not set, Pulse will auto-detect the protocol from `X-Forwarded-Proto` headers. - -### Dex / other self-hosted issuers - -This matches the bundled Dex mock server: - -1. Create a new OAuth 2.0 / OIDC application, mark it *confidential*, and note the generated `client_id` and `client_secret`. -2. Add every Pulse hostname you expose (for example `https://pulse.example.com/api/oidc/callback`) to the list of redirect URIs. -3. Ensure the application scopes include at least `openid profile email`. -4. Paste the issuer URL (for Dex that is `https:///dex`) plus the client credentials into Pulse. - -### Azure Active Directory - -1. Register a **Web** app in Azure AD and capture the **Application (client) ID**. -2. Create a **Client secret** under *Certificates & secrets*. -3. Add a redirect URL `https:///api/oidc/callback` (type **Web**). -4. Under *Token configuration*, add optional claims for `email` and `preferred_username`. If you plan to restrict by groups enable the *Groups* claim. -5. In Pulse, use `https://login.microsoftonline.com//v2.0` as the issuer and paste the client credentials. - -### Okta - -1. Create an **OIDC Web App** integration. -2. Trusted redirect URIs: `https:///api/oidc/callback`. -3. Assign the integration to the users or groups who need access. -4. Copy the **Client ID**, **Client secret**, and **Okta domain**. Use `https:///oauth2/default` as the issuer within Pulse. - -### Group and domain restrictions - -- Set `OIDC_GROUPS_CLAIM` to the claim that carries group names (default `groups`). -- Combine `allowedGroups`, `allowedDomains`, or `allowedEmails` in the UI to fence access without editing your IdP. -- Azure AD group names appear as GUIDs unless you enable *Security groups* in token configuration; Okta and Authentik emit the literal group name. - -## Environment Overrides - -All configuration can be provided via environment variables (see [`docs/CONFIGURATION.md`](./CONFIGURATION.md#oidc-variables-optional-overrides)). When any `OIDC_*` variable is present the UI is placed in read-only mode and values must be changed from the deployment configuration instead. - -## Self-signed / private CA issuers - -Pulse validates OIDC discovery, JWKS fetches, and token exchanges with the container trust store. For self-signed or private PKI issuers, mount your IdP root CA into the container and point `OIDC_CA_BUNDLE` at that PEM file. - -Example Helm override: - -```yaml -server: - extraVolumes: - - name: oidc-ca - secret: - secretName: oidc-ca - extraVolumeMounts: - - name: oidc-ca - mountPath: /etc/ssl/certs/oidc-ca.pem - subPath: oidc-ca.pem - readOnly: true - extraEnv: - - name: OIDC_CA_BUNDLE - value: /etc/ssl/certs/oidc-ca.pem +### Debugging +Enable debug logs to trace the OIDC flow: +```bash +export LOG_LEVEL=debug +# Restart Pulse ``` - -This keeps TLS verification enabled while trusting your internal CA. Avoid disabling verification; use a CA bundle instead. - -## Login Flow - -- The login screen shows a **Continue with Single Sign-On** button when OIDC is enabled. -- Users are redirected to the configured issuer for authentication and returned to `/api/oidc/callback`. -- Pulse validates the ID token, enforces optional group/domain/email restrictions, then creates the usual session cookie (`pulse_session`). -- Existing username/password login remains available unless explicitly disabled in the environment. - -## Troubleshooting - -| Symptom | Resolution | -| --- | --- | -| `invalid_id_token` error | The issuer URL configured in Pulse doesn't match the `iss` claim in the ID token from your provider. Enable `LOG_LEVEL=debug` to see the exact verification error. For Authentik, try both `https://auth.domain.com` (base URL) and `https://auth.domain.com/application/o/pulse/` (application URL) to see which matches your provider's token issuer. | -| `unexpected signature algorithm "HS256"; expected ["RS256"]` in logs | Authentik falls back to HS256 if no signing key is configured. Assign an RSA signing key to the application (token settings β†’ Signing key) so ID tokens are issued with RS256. | -| Redirect loops back to login | After successful OIDC login, if you're redirected back to the login page, check that: (1) cookies are enabled in your browser, (2) if behind a proxy, ensure `X-Forwarded-Proto` header is set correctly, (3) check browser console for cookie errors. | -| Users see `single sign-on failed` | Check `journalctl -u pulse.service` for detailed OIDC audit logs. Common causes include mismatched client IDs, incorrect redirect URLs, or group/domain restrictions. | -| UI shows "OIDC settings are managed by environment variables" | Remove the relevant `OIDC_*` environment variables or update them directly in your deployment. | -| Provider discovery fails | Verify the issuer URL is reachable from the Pulse server and returns valid OIDC discovery metadata at `/.well-known/openid-configuration`. | -| `tls: failed to verify certificate: x509: certificate signed by unknown authority` | Mount your IdP CA into the container and set `OIDC_CA_BUNDLE` to that PEM path (see self-signed section above). | -| Group restrictions not working | Enable debug logging to see which groups the IdP is sending and verify the `groups_claim` setting matches your IdP's claim name. | -| Auto-redirect to OIDC when password auth still enabled | This is expected behavior when OIDC is enabled. Users can still use password auth by clicking "Use your admin credentials to sign in below" on the login page. To disable auto-redirect, comment out the auto-redirect code in the frontend. | - -### Debug Logging - -For detailed troubleshooting, temporarily raise the log level to `debug` (via **Settings β†’ System β†’ Logging** or by exporting `LOG_LEVEL=debug` and restarting). Debug logs include: - -- OIDC provider initialization (issuer URL, endpoints discovered) -- Authorization flow start (client ID, scopes requested) -- Token exchange details (success/failure with specific errors) -- ID token verification (subject extracted) -- Claims extraction (username, email, groups found) -- Access control checks (which emails/domains/groups were checked and why they passed or failed) - -Example debug log output: -``` -DBG Initializing OIDC provider issuer=https://auth.example.com redirect_url=https://pulse.example.com/api/oidc/callback scopes=[openid,profile,email] -DBG OIDC provider discovery successful issuer=https://auth.example.com auth_endpoint=https://auth.example.com/authorize token_endpoint=https://auth.example.com/token -DBG Starting OIDC login flow issuer=https://auth.example.com client_id=pulse-client -DBG Processing OIDC callback issuer=https://auth.example.com -DBG OIDC code exchange successful -DBG ID token verified successfully subject=user@example.com -DBG Extracted user identity from claims username=user@example.com email=user@example.com email_claim=email username_claim=preferred_username -DBG Checking group membership user_groups=[admins,users] allowed_groups=[admins] groups_claim=groups -DBG User group membership verified -``` - -After reviewing the logs, set `LOG_LEVEL=info` to reduce log volume. +Logs will show discovery, token exchange, and claim parsing details. diff --git a/docs/PROXY_AUTH.md b/docs/PROXY_AUTH.md index c3381cf..aaaa748 100644 --- a/docs/PROXY_AUTH.md +++ b/docs/PROXY_AUTH.md @@ -1,256 +1,72 @@ -# Proxy Authentication +# πŸ›‘οΈ Proxy Authentication -Pulse supports proxy-based authentication for integration with SSO providers like Authentik, Authelia, Caddy, and others. This allows you to authenticate users via your existing reverse proxy authentication system while maintaining security. +Authenticate users via your existing reverse proxy (Authentik, Authelia, Cloudflare Zero Trust, etc.). -> **When to use this**: If you already have an authentication proxy (Authentik, Authelia, etc.) protecting your services and want Pulse to trust that authentication instead of requiring its own login. +## πŸš€ Quick Start -## Quick Start +1. **Generate Secret**: Create a strong random string. +2. **Configure Pulse**: + ```bash + PROXY_AUTH_SECRET=your-random-secret + PROXY_AUTH_USER_HEADER=X-Authentik-Username + ``` +3. **Configure Proxy**: Set the proxy to send `X-Proxy-Secret` and the user header. -1. Set `PROXY_AUTH_SECRET` to a random secret string -2. Configure your proxy to send this secret in the `X-Proxy-Secret` header -3. Set `PROXY_AUTH_USER_HEADER` to match your proxy's username header -4. (Optional) Configure role-based access control with `PROXY_AUTH_ROLE_HEADER` +## βš™οΈ Configuration -## Configuration +| Variable | Description | Default | +| :--- | :--- | :--- | +| `PROXY_AUTH_SECRET` | **Required**. Shared secret to verify requests. | - | +| `PROXY_AUTH_USER_HEADER` | **Required**. Header containing the username. | - | +| `PROXY_AUTH_ROLE_HEADER` | Header containing user groups/roles. | - | +| `PROXY_AUTH_ADMIN_ROLE` | Role name that grants admin access. | `admin` | +| `PROXY_AUTH_LOGOUT_URL` | URL to redirect to after logout. | - | -Set the following environment variables to enable proxy authentication: - -### Required Settings - -```bash -# Shared secret between proxy and Pulse (required) -PROXY_AUTH_SECRET=your-secure-secret-here - -# Header containing the authenticated username (optional but recommended) -PROXY_AUTH_USER_HEADER=X-Authentik-Username -``` - -### Optional Settings - -```bash -# Header containing user roles/groups -PROXY_AUTH_ROLE_HEADER=X-Authentik-Groups - -# Separator for multiple roles (default: |) -PROXY_AUTH_ROLE_SEPARATOR=| - -# Role name that grants admin access (default: admin) -PROXY_AUTH_ADMIN_ROLE=admin - -# URL to redirect users to for logout -PROXY_AUTH_LOGOUT_URL=/outpost.goauthentik.io/sign_out -``` - -## How It Works - -1. **User visits Pulse** β†’ Your proxy intercepts the request -2. **Proxy authenticates user** β†’ Via its own login page/SSO -3. **Proxy adds headers** to the request: - - `X-Proxy-Secret`: Shared secret (prevents spoofing) - - Username header (e.g., `X-Authentik-Username`) - - Roles header (e.g., `X-Authentik-Groups`) -4. **Pulse validates** the secret and trusts the user identity -5. **No Pulse login required** β†’ User sees the dashboard immediately - -## Example Configurations - -### Authentik with Traefik +## πŸ“¦ Examples +### Authentik (with Traefik) +**docker-compose.yml**: ```yaml -# docker-compose.yml environment variables environment: - - PROXY_AUTH_SECRET=your-secure-secret-here + - PROXY_AUTH_SECRET=secure-secret - PROXY_AUTH_USER_HEADER=X-Authentik-Username - - PROXY_AUTH_ROLE_HEADER=X-Authentik-Groups - - PROXY_AUTH_ROLE_SEPARATOR=| - - PROXY_AUTH_ADMIN_ROLE=admin - - PROXY_AUTH_LOGOUT_URL=/outpost.goauthentik.io/sign_out ``` -Traefik middleware configuration: - +**Traefik Middleware**: ```yaml -http: - middlewares: - proxy-header-secret: - headers: - customRequestHeaders: - X-Proxy-Secret: "your-secure-secret-here" - - authentik-auth: - forwardAuth: - address: http://authentik:9000/outpost.goauthentik.io/auth/traefik - trustForwardHeader: true - authResponseHeaders: - - X-Authentik-Username - - X-Authentik-Groups - - X-Authentik-Email - - routers: - pulse: - rule: Host(`pulse.example.com`) - entryPoints: - - websecure - middlewares: - - authentik-auth - - proxy-header-secret - service: pulse-service - - pulse-auth: - rule: Host(`pulse.example.com`) && PathPrefix(`/outpost.goauthentik.io/`) - entryPoints: - - websecure - service: authentik-outpost +headers: + customRequestHeaders: + X-Proxy-Secret: "secure-secret" ``` -### Authelia Example - -```yaml -# docker-compose.yml environment variables -environment: - - PROXY_AUTH_SECRET=your-secure-secret-here - - PROXY_AUTH_USER_HEADER=Remote-User - - PROXY_AUTH_ROLE_HEADER=Remote-Groups - - PROXY_AUTH_ROLE_SEPARATOR=, - - PROXY_AUTH_ADMIN_ROLE=admins - - PROXY_AUTH_LOGOUT_URL=/logout -``` - -Nginx configuration: +### Authelia (Nginx) ```nginx location / { - # Authelia authorization auth_request /authelia; - auth_request_set $user $upstream_http_remote_user; - auth_request_set $groups $upstream_http_remote_groups; - - # Pass headers to Pulse - proxy_set_header X-Proxy-Secret "your-secure-secret-here"; - proxy_set_header Remote-User $user; - proxy_set_header Remote-Groups $groups; - proxy_pass_header X-RateLimit-Limit; - proxy_pass_header X-RateLimit-Remaining; - proxy_pass_header X-RateLimit-Reset; - proxy_pass_header Retry-After; - + proxy_set_header X-Proxy-Secret "secure-secret"; + proxy_set_header Remote-User $upstream_http_remote_user; proxy_pass http://pulse:7655; } ``` -### Caddy with Forward Auth +### Cloudflare Tunnel +1. **Zero Trust Dashboard**: Applications β†’ Add Application. +2. **Settings**: HTTP Settings β†’ HTTP Headers. +3. **Add Header**: `X-Proxy-Secret` = `your-secret`. +4. **Pulse Config**: `PROXY_AUTH_USER_HEADER=Cf-Access-Authenticated-User-Email`. -```caddyfile -pulse.example.com { - forward_auth authelia:9091 { - uri /api/verify?rd=https://auth.example.com - copy_headers Remote-User Remote-Groups - } - - header_downstream X-Proxy-Secret "your-secure-secret-here" - - reverse_proxy pulse:7655 -} -``` +## πŸ”§ Troubleshooting -### Nginx Proxy Manager +| Issue | Check | +| :--- | :--- | +| **401 Unauthorized** | Verify `X-Proxy-Secret` matches `PROXY_AUTH_SECRET`. Check if headers are being stripped by intermediate proxies. | +| **Not Admin** | Verify `PROXY_AUTH_ROLE_HEADER` is set and contains `PROXY_AUTH_ADMIN_ROLE`. | +| **Logout Fails** | Ensure `PROXY_AUTH_LOGOUT_URL` is set to your IdP's logout endpoint. | -In NPM's Advanced tab for your Pulse proxy host: - -```nginx -# Custom Nginx Configuration -proxy_set_header X-Proxy-Secret "your-secure-secret-here"; -proxy_set_header X-Authentik-Username $http_x_authentik_username; -proxy_set_header X-Authentik-Groups $http_x_authentik_groups; -``` - -## Security Considerations - -1. **Use a strong secret**: Generate a secure random string for `PROXY_AUTH_SECRET` -2. **HTTPS only**: Always use HTTPS between the proxy and Pulse in production -3. **Network isolation**: Ensure Pulse is not directly accessible, only through the proxy -4. **Header validation**: Pulse validates all headers and the proxy secret on every request -5. **Preserve rate-limit headers**: Do not strip `X-RateLimit-*` or `Retry-After`. Clients rely on them when Pulse throttles requests. - -## Combining with Other Auth Methods - -Proxy authentication can work alongside other authentication methods: - -- If `PROXY_AUTH_SECRET` is set, proxy auth takes precedence -- API tokens (`API_TOKENS` or legacy `API_TOKEN`) still work for programmatic access -- Basic auth (`PULSE_AUTH_USER`/`PULSE_AUTH_PASS`) can be used as fallback - -## Troubleshooting - -### Users can't access Pulse (401 Unauthorized) - -1. **Check the secret header**: - ```bash - # Test with curl - curl -H "X-Proxy-Secret: your-secret" \ - -H "X-Authentik-Username: testuser" \ - http://pulse:7655/api/state - ``` - -2. **Verify headers are being sent**: - - Temporarily raise logging to debug via **Settings β†’ System β†’ Logging** (or set `LOG_LEVEL=debug` and restart). Remember to return to `info` when finished. - - Check Pulse logs: `docker logs pulse` or `journalctl -u pulse` - - Look for "Invalid proxy secret" or "Proxy auth user header not found" - -3. **Common issues**: - - Typo in `PROXY_AUTH_SECRET` - - Header names are case-sensitive in configuration - - Proxy not forwarding headers correctly - -### Admin features not available - -Check if user is recognized as admin: +### Verify Headers +Use `curl` to simulate a proxy request: ```bash curl -H "X-Proxy-Secret: your-secret" \ -H "X-Authentik-Username: admin" \ - -H "X-Authentik-Groups: users|admin" \ - http://pulse:7655/api/security/status | jq '.proxyAuthIsAdmin' -``` - -- Ensure the roles header contains the admin role -- Verify `PROXY_AUTH_ADMIN_ROLE` matches your configuration -- Check the role separator matches your proxy's format (default: `|`) - -### Logout doesn't work - -- Verify `PROXY_AUTH_LOGOUT_URL` points to your proxy's logout endpoint -- Ensure the logout URL is accessible from the user's browser -- For Authentik: `/outpost.goauthentik.io/sign_out` -- For Authelia: `/logout` or custom path - -### Testing your configuration - -Test proxy auth without a reverse proxy: -```bash -# Should return 401 -curl http://localhost:7655/api/state - -# Should return 200 with state data -curl -H "X-Proxy-Secret: your-secret-here" \ - -H "X-Your-User-Header: testuser" \ http://localhost:7655/api/state ``` - -## FAQ - -**Q: Do I still need to set up Pulse authentication?** -A: No, when proxy auth is configured, Pulse trusts your proxy's authentication. Users won't see Pulse's login page. - -**Q: Can I use this with Cloudflare Access or Tailscale?** -A: Yes, any service that can add custom headers after authentication will work. - -**Q: What happens if someone bypasses my proxy?** -A: They can't authenticate. Without the correct `X-Proxy-Secret` header, all requests are rejected with 401. - -**Q: Can I have some users with read-only access?** -A: Currently, Pulse has admin and non-admin roles. Non-admin users have read-only access to monitoring data. - -**Q: Is the username displayed in Pulse?** -A: Yes, the authenticated username appears in the top-right corner of the UI. - -**Q: Can I use both proxy auth and API tokens?** -A: Yes! API tokens still work for automation/scripts. Proxy auth is for human users via the web UI. diff --git a/docs/TEMPERATURE_MONITORING.md b/docs/TEMPERATURE_MONITORING.md index 4f2f4ff..ecb605c 100644 --- a/docs/TEMPERATURE_MONITORING.md +++ b/docs/TEMPERATURE_MONITORING.md @@ -1,804 +1,79 @@ -# Temperature Monitoring +# 🌑️ Temperature Monitoring -Pulse can display real-time CPU and NVMe temperatures directly in your dashboard, giving you instant visibility into your hardware health. +Monitor real-time CPU and NVMe temperatures for your Proxmox nodes. -## Features +## πŸš€ Quick Start -- **CPU Package Temperature**: Shows the overall CPU temperature when available -- **Individual Core Temperatures**: Tracks each CPU core -- **NVMe Drive Temperatures**: Monitors NVMe SSD temperatures (visible in the Storage tab's disk list) -- **Color-Coded Display**: - - Green: < 60Β°C (normal) - - Yellow: 60-80Β°C (warm) - - Red: > 80Β°C (hot) +### 1. Enable in Pulse +Go to **Settings β†’ Nodes β†’ [Node] β†’ Advanced Monitoring** and enable "Temperature Monitoring". -## Transport Architecture +### 2. Install Sensor Proxy +The setup depends on your deployment: -Pulse attempts temperature collection in a fixed order so you can reason about which path is active for any node: +| Deployment | Recommended Method | +| :--- | :--- | +| **LXC (Pulse)** | Run the **Setup Script** in Pulse UI. It auto-installs the proxy on the host. | +| **Docker (Pulse)** | Install proxy on host + bind mount socket. (See below) | +| **Remote Node** | Install proxy in **HTTP Mode** on the remote node. | -1. **HTTPS proxy (`pulse-sensor-proxy --http-mode`)** – each Proxmox host can expose its own TLS endpoint on port 8443. Pulse stores the proxy URL and bearer token in `nodes.enc` and talks to that proxy first. -2. **Unix socket proxy** – when Pulse runs on the same machine, it mounts `/run/pulse-sensor-proxy` from the host and speaks to the proxy over a Unix socket with SO\_PEERCRED authentication. -3. **Direct SSH** – final fallback for bare-metal installs. Container deployments keep this disabled unless `PULSE_DEV_ALLOW_CONTAINER_SSH=true`. +## πŸ“¦ Docker Setup (Manual) -If a node has an HTTPS proxy configured, Pulse does **not** fall back to socket or SSH. Instead it marks the node unavailable and surfaces the HTTP error in diagnostics so you can fix the underlying issue rather than masking it. +If running Pulse in Docker, you must install the proxy on the host and share the socket. -## Deployment-Specific Setup +1. **Install Proxy on Host**: + ```bash + curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh | \ + sudo bash -s -- --standalone --pulse-server http://:7655 + ``` -> **Important:** Pick the transport that matches your deployment: -> - **Pulse running inside a container (Docker/LXC):** Use the Unix-socket path (bind mount `/run/pulse-sensor-proxy`) so SSH keys never leave the host. Details in [Quick Start for Docker Deployments](#quick-start-for-docker-deployments). -> - **Pulse talking to remote Proxmox hosts / standalone nodes:** Install `pulse-sensor-proxy` directly on each host with `--standalone --http-mode` so Pulse reaches it over HTTPS 8443. See [HTTP Mode for Remote Hosts](#http-mode-for-remote-hosts). -> - **Docker-in-VM / β€œPulse can’t see the host sensors”:** Use `pulse-host-agent`. -> - **Native installs:** Direct SSH works, but the proxy/host-agent options are preferred for key isolation. -> -> **Automation users:** both installers (`install-sensor-proxy.sh` and `install-host-agent.sh`) accept non-interactive flags; jump to [Automation-Friendly Installation](#automation-friendly-installation) for samples. - -### Transport decision matrix - -| Pulse Deployment | Recommended Transport | Why | -|------------------|----------------------|-----| -| Pulse in Docker/LXC on the Proxmox host | Unix socket via `/run/pulse-sensor-proxy` bind mount | Keeps SSH keys on the host, enforces SO\_PEERCRED auth, no network exposure | -| Pulse in Docker inside a VM | `pulse-host-agent` on the Proxmox host | VM can’t mount the host socket, host agent reports over HTTPS instead | -| Pulse (any host) monitoring additional Proxmox nodes on the LAN | `install-sensor-proxy.sh --standalone --http-mode` on each node | Lets each node host its own proxy so Pulse reaches it over HTTPS 8443 | -| Bare-metal Pulse install on the same Proxmox host | Either socket or HTTP works; socket is simpler | You already have direct filesystem access; the installer auto-configures the socket | - -Use the socket path wherever Pulse is containerised. Use HTTP mode when the sensors live on machines Pulse cannot mount directly. - -### Monitoring proxy health - -Pulse surfaces the current transport status under **Settings β†’ Diagnostics β†’ Temperature proxy**. - -- The **Control plane sync** table lists every proxy registered with the new control-plane channel (`install-sensor-proxy.sh` now configures this automatically). Each entry shows the last time the proxy fetched its authorized node list, the expected refresh interval, and whether it is healthy, stale, or offline. -- If a proxy falls behind more than one refresh interval you will see a yellow β€œBehind” badge; Pulse also adds a diagnostic note explaining which host is lagging. After four consecutive missed polls the badge turns red (β€œOffline”). -- HTTPS-mode proxies still appear under the **HTTPS proxies** section with reachability/error information, so you can see socket/HTTP transport issues side-by-side. - -If a proxy never completes its first sync the diagnostics card will call that out explicitly (status β€œPending”). Rerun the host installer or check the proxy journal (`journalctl -u pulse-sensor-proxy`) to resolve any startup problems, then refresh Diagnostics to confirm the sync is healthy. - -## Docker in VM Setup - -**Running Pulse in Docker inside a VM on Proxmox?** The proxy socket cannot cross VM boundaries, so use pulse-host-agent instead. - -pulse-host-agent runs natively on your Proxmox host and reports temperatures back to Pulse over HTTPS. This works across VM boundaries without requiring socket mounts or SSH configuration. - -**Setup steps:** - -1. Install lm-sensors on your Proxmox host (if not already installed): - ```bash - apt-get update && apt-get install -y lm-sensors - sensors-detect --auto - ``` - -2. Install pulse-host-agent on your Proxmox host: - ```bash - # Generate an API token in Pulse (Settings β†’ Security β†’ API Tokens) with host-agent:report scope - curl -fsSL http://your-pulse-vm:7655/install-host-agent.sh | \ - bash -s -- --url http://your-pulse-vm:7655 --token YOUR_API_TOKEN - ``` - -3. Verify temperatures appear in Pulse UI under the Hosts tab - -The host agent will report CPU, NVMe, and GPU temperatures alongside other system metrics. No proxy installation or socket mounting needed. - -## Quick Start for Docker Deployments - -**Running Pulse in Docker directly on Proxmox?** Temperature monitoring requires installing a small service on your Proxmox host that reads hardware sensors. The Pulse container connects to this service through a shared socket. - -**Why this is needed:** Docker containers cannot directly access hardware sensors. The proxy runs on your Proxmox host where it has access to sensor data, then shares that data with the Pulse container through a secure connection. - -Follow these steps to set it up: - -### 1. Install the proxy on your Proxmox host - -SSH to your **Proxmox host** (not the Docker container) and run as root: - -```bash -sudo curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh | \ - sudo bash -s -- --standalone --pulse-server http://192.168.1.100:7655 -``` - -Replace `192.168.1.100:7655` with your Pulse server's actual IP address and port. - -The script will install and start the `pulse-sensor-proxy` service. You should see output confirming the installation succeeded. - -### 2. Add bind mount to docker-compose.yml - -Edit your `docker-compose.yml` file and add the highlighted line to your Pulse service volumes: - -```yaml -services: - pulse: - image: rcourtman/pulse:latest - ports: - - "7655:7655" +2. **Update `docker-compose.yml`**: + Add the socket volume to your Pulse service: + ```yaml volumes: - - pulse-data:/data - - /run/pulse-sensor-proxy:/run/pulse-sensor-proxy:ro # Add this line (read-only) + - /run/pulse-sensor-proxy:/run/pulse-sensor-proxy:ro + ``` -volumes: - pulse-data: -``` +3. **Restart Pulse**: `docker compose up -d` -This connects the proxy socket from your host into the container so Pulse can communicate with it. +## 🌐 Remote Nodes (HTTP Mode) -> **Security Note:** The socket mount is read-only (`:ro`) to prevent compromised containers from tampering with the socket directory. The proxy enforces access control via SO_PEERCRED, so write access is not needed. +For nodes *other* than the one running Pulse, install the proxy in HTTP mode. -### 3. Restart Pulse container +1. **Run Installer on Remote Node**: + ```bash + curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh | \ + sudo bash -s -- --standalone --http-mode --pulse-server http://:7655 + ``` +2. **Verify**: Pulse will automatically detect the proxy and start collecting data. +## πŸ”§ Troubleshooting + +| Issue | Solution | +| :--- | :--- | +| **No Data** | Check **Settings β†’ Diagnostics β†’ Temperature Proxy**. | +| **Proxy Unreachable** | Ensure port `8443` is open on the remote node. | +| **"Permission Denied"** | Re-run the installer to fix permissions or SSH keys. | +| **LXC Issues** | Ensure the container has the bind mount: `lxc.mount.entry: /run/pulse-sensor-proxy ...` | + +### Check Proxy Status +On the Proxmox host: ```bash -docker compose down -docker compose up -d -``` - -Note: If you're using older Docker Compose v1, use `docker-compose` (with hyphen) instead. - -### 4. Verify the setup - -**Check proxy is running on your Proxmox host:** -```bash -sudo systemctl status pulse-sensor-proxy -``` - -You should see `active (running)` in green. - -**Check Pulse detected the proxy:** -```bash -docker logs pulse 2>&1 | grep -i "temperature.*proxy" -``` - -Replace `pulse` with your container name if different (check with `docker ps`). - -You should see: `Temperature proxy detected - using secure host-side bridge` - -**Check temperatures appear in the UI:** - -Open Pulse in your browser and check the node dashboard. CPU and drive temperatures should now be visible. If you still see blank temperature fields, proceed to troubleshooting below. - -**Having issues?** See [Troubleshooting](#troubleshooting) below. - -## HTTP Mode for Remote Hosts - -When Pulse cannot share the `/run/pulse-sensor-proxy` socket (for example, you run Pulse on one host but want temperatures from other Proxmox nodes), install the proxy directly on each target host and expose it over HTTPS 8443. - -1. **Install the proxy in HTTP mode** on each Proxmox node: - ```bash - curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh | \ - sudo bash -s -- --standalone --http-mode --pulse-server http://192.168.0.123:7655 - ``` - Replace the Pulse URL with the server that should receive temperatures. The installer: - - Generates a TLS certificate (`/etc/pulse-sensor-proxy/tls/`) - - Registers with Pulse and writes the proxy URL/token into `nodes.enc` - - Starts `pulse-sensor-proxy.service` listening on `https://:8443` - -2. **Allow Pulse to reach port 8443** (host firewall, VLAN ACLs, etc.). Only Pulse needs access; the installer’s service file restricts the listener to HTTPS with bearer auth. - -3. **Verify the endpoint manually** (optional but recommended): - ```bash - TOKEN=$(sudo cat /etc/pulse-sensor-proxy/.http-auth-token) - curl -k -H "Authorization: Bearer ${TOKEN}" "https://node.example:8443/temps?node=shortname" - ``` - You should receive JSON with the `sensors` payload. - -4. **Restart Pulse** (or wait for config reload) so it notices the new proxy URL/token. Pulse will automatically try HTTP first for nodes with `TemperatureProxyURL` configured, then fall back to the Unix socket (if mounted) and finally SSH. - -This HTTP path complements the socket pathβ€”you can run both simultaneously. Containerised Pulse stacks still need the socket for their own host, while HTTP mode covers every additional Proxmox node on the LAN or across sites. - -Pulse now isolates transport failures per node: when a proxy reports that a node is invalid or unreachable, Pulse cools down polling for that node only instead of tearing down the shared socket. You will see a cooldown note in the diagnostics card if a node keeps failing; fix the proxy or disable temperature monitoring for that node to resume collection. - -> **Tip:** When Pulse is running inside a container and temperatures are blocked, open **Settings β†’ Nodes β†’ Edit node β†’ Temperature monitoring**. The UI now offers a one-click β€œGenerate HTTPS proxy command” button that produces the exact `install-sensor-proxy.sh --standalone --http-mode --pulse-server …` command for that node, so you can copy it straight to the host shell without rebuilding the instructions manually. - ---- - -## Disable Temperature Monitoring - -Don't need the sensor data? Open **Settings β†’ Proxmox**, edit any node, and scroll to the **Advanced monitoring** section. The temperature toggle there controls collection for all nodes: - -- When disabled, Pulse skips every SSH/proxy request for temperature data. -- CPU and NVMe readings disappear from dashboards and node tables. -- You can re-enable it later without re-running the setup scripts. - -For scripted environments, set either: - -- `temperatureMonitoringEnabled: false` in `/etc/pulse/system.json`, or -- `ENABLE_TEMPERATURE_MONITORING=false` in the environment (locks the UI toggle until removed). - -## How It Works - -### Proxy-first architecture - -Pulse keeps temperature collection outside the Pulse process whenever possible: - -1. **pulse-sensor-proxy** runs on each Proxmox host, owns the SSH keys, and reads sensor data locally. -2. Pulse talks to the proxy over HTTPS (`--http-mode`) when the host is remote, or over the Unix socket (`/run/pulse-sensor-proxy`) when Pulse runs on the same machine. -3. The proxy fan-outs to cluster members using its own SSH configuration, so the Pulse container never needs direct SSH access. - -**Benefits:** -- SSH keys never enter the container -- HTTPS mode lets Pulse collect temperatures from any node reachable over TCP/8443 without sharing sockets across hosts -- Container compromise doesn't expose infrastructure credentials -- **LXC:** Automatically configured during installation (fully turnkey) -- **Docker:** Requires manual proxy installation and volume mount (see Quick Start above) - -#### Manual installation (host-side) - -When you need to provision the proxy yourself (for example via your own automation), run these steps on the host that runs your Pulse container: - -1. **Install the binary** - ```bash - curl -L https://github.com/rcourtman/Pulse/releases/download//pulse-sensor-proxy-linux-amd64 \ - -o /tmp/pulse-sensor-proxy - install -D -m 0755 /tmp/pulse-sensor-proxy /opt/pulse/sensor-proxy/bin/pulse-sensor-proxy - ``` - Use the arm64/armv7 artefact if required. - -2. **Create the service account if missing** - ```bash - id pulse-sensor-proxy >/dev/null 2>&1 || \ - useradd --system --user-group --no-create-home --shell /usr/sbin/nologin pulse-sensor-proxy - ``` - -3. **Provision the data directories** - ```bash - install -d -o pulse-sensor-proxy -g pulse-sensor-proxy -m 0750 /var/lib/pulse-sensor-proxy - install -d -o pulse-sensor-proxy -g pulse-sensor-proxy -m 0700 /var/lib/pulse-sensor-proxy/ssh - ``` - -4. **(Optional) Add `/etc/pulse-sensor-proxy/config.yaml`** - Only needed if you want explicit subnet/metrics settings; otherwise the proxy auto-detects host CIDRs and registers with Pulse automatically. - ```yaml - allowed_source_subnets: - - 192.168.1.0/24 - metrics_address: 0.0.0.0:9127 # use "disabled" to switch metrics off - allowed_nodes_file: /etc/pulse-sensor-proxy/allowed_nodes.yaml - http_enabled: true # only when polling remote hosts over HTTPS - http_listen_addr: ":8443" - http_tls_cert: /etc/pulse-sensor-proxy/tls/server.crt - http_tls_key: /etc/pulse-sensor-proxy/tls/server.key - pulse_control_plane: - url: https://pulse.example.com:7655 - token_file: /etc/pulse-sensor-proxy/.pulse-control-token - refresh_interval: 60 - ``` - - Provide `http_auth_token` (32+ bytes of random data) only when you cannot register through Pulse, and ensure the TLS files exist. Allowed nodes live in `/etc/pulse-sensor-proxy/allowed_nodes.yaml`β€”use `pulse-sensor-proxy config set-allowed-nodes` instead of editing it by hand. - -5. **Install the hardened systemd unit** - Copy the unit from `scripts/install-sensor-proxy.sh` or create `/etc/systemd/system/pulse-sensor-proxy.service` with: - ```ini - [Unit] - Description=Pulse Temperature Proxy - After=network.target - - [Service] - Type=simple - User=pulse-sensor-proxy - Group=pulse-sensor-proxy - WorkingDirectory=/var/lib/pulse-sensor-proxy - ExecStartPre=/opt/pulse/sensor-proxy/bin/pulse-sensor-proxy config validate --config /etc/pulse-sensor-proxy/config.yaml - ExecStart=/opt/pulse/sensor-proxy/bin/pulse-sensor-proxy --config /etc/pulse-sensor-proxy/config.yaml - Restart=on-failure - RestartSec=5s - RuntimeDirectory=pulse-sensor-proxy - RuntimeDirectoryMode=0775 - RuntimeDirectoryPreserve=yes - LogsDirectory=pulse/sensor-proxy - LogsDirectoryMode=0750 - UMask=0007 - NoNewPrivileges=true - ProtectSystem=strict - ProtectHome=read-only - ReadWritePaths=/var/lib/pulse-sensor-proxy - ReadWritePaths=-/run/corosync - ProtectKernelTunables=true - ProtectKernelModules=true - ProtectControlGroups=true - ProtectClock=true - PrivateTmp=true - PrivateDevices=true - ProtectProc=invisible - ProcSubset=pid - LockPersonality=true - RemoveIPC=true - RestrictSUIDSGID=true - RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK - RestrictNamespaces=true - SystemCallFilter=@system-service - SystemCallErrorNumber=EPERM - CapabilityBoundingSet= - AmbientCapabilities= - KeyringMode=private - LimitNOFILE=1024 - StandardOutput=journal - StandardError=journal - SyslogIdentifier=pulse-sensor-proxy - - [Install] - WantedBy=multi-user.target - ``` - -6. **Enable the service** - ```bash - systemctl daemon-reload - systemctl enable --now pulse-sensor-proxy.service - ``` - Confirm the socket appears at `/run/pulse-sensor-proxy/pulse-sensor-proxy.sock`. - -7. **Expose the socket to Pulse** - - **Proxmox LXC:** append `lxc.mount.entry: /run/pulse-sensor-proxy run/pulse-sensor-proxy none bind,create=dir 0 0` to `/etc/pve/lxc/.conf` and restart the container. - - **Docker:** bind mount `/run/pulse-sensor-proxy` into the container (`- /run/pulse-sensor-proxy:/run/pulse-sensor-proxy:ro`). - -After the container restarts, the backend will automatically use the proxy. To refresh SSH keys on cluster nodes (e.g., after adding a new node), SSH to your Proxmox host and re-run the setup script: `curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh | bash -s -- --ctid ` - -> **Tip:** Re-running the installer now rewrites the `pulse_control_plane` block with the latest Pulse URL and token file automatically. If you rebuild the Pulse container or change its address, just rerun `install-sensor-proxy.sh` and the control-plane config will follow without manual edits. - -### Post-install Verification - -1. **Confirm proxy metrics** - ```bash - curl -s http://127.0.0.1:9127/metrics | grep pulse_proxy_build_info - ``` -2. **Ensure adaptive polling sees the proxy** - ```bash - curl -s http://localhost:7655/api/monitoring/scheduler/health \ - | jq '.instances[] | select(.key | contains("temperature")) | {key, pollStatus}' - ``` - - Expect recent `lastSuccess` timestamps, `breaker.state == "closed"`, and `deadLetter.present == false`. -3. **Check update history** – Any future proxy restarts/rollbacks are logged under **Settings β†’ System β†’ Updates**; include the associated `event_id` in post-change notes. -4. **Measure queue depth/staleness** – Grafana panels `pulse_monitor_poll_queue_depth` and `pulse_monitor_poll_staleness_seconds` should return to baseline within a few polling cycles. - -### Legacy SSH Architecture (native installs) - -For native (non-containerized) installations, Pulse connects directly via SSH: - -1. Pulse uses SSH key authentication (like Ansible, Terraform, etc.) -2. Runs `sensors -j` command to read hardware temperatures -3. SSH key stored in Pulse's home directory - -> **Important for native installs:** Run every setup command as the same user account that executes the Pulse service (typically `pulse`). The backend reads the SSH key from that user's home directory. - -## Requirements - -1. **SSH Key Authentication**: Your Pulse server needs SSH key access to nodes (no passwords) -2. **lm-sensors Package**: Installed on nodes to read hardware sensors -3. **Passwordless root SSH** (Proxmox clusters only): For proxy architecture, the Proxmox host running Pulse must have passwordless root SSH access to all cluster nodes. This is standard for Proxmox clusters but hardened environments may need to create an alternate service account. - -## Setup (Automatic) - -The auto-setup script (Settings β†’ Nodes β†’ Setup Script) provides different experiences based on deployment type: - -### For LXC Deployments (Fully Automatic) - -When run on a Proxmox host with Pulse in an LXC container: - -1. Run the auto-setup script on your Proxmox node -2. The script automatically detects your Pulse LXC container -3. Installs `pulse-sensor-proxy` on the host -4. Configures the container bind mount automatically -5. Sets up SSH keys and cluster discovery -6. **Fully turnkey - no manual steps required!** - -> **Note:** The main `install.sh` already installs the host-side proxy when you opt-in during bootstrap, so the Quick Setup script simply verifies it and moves onβ€”you won’t be prompted a second time. Remote/standalone nodes still prompt to deploy their own HTTPS proxy. - -### For Docker Deployments (Manual Steps Required) - -When Pulse runs in Docker, the setup script will show you manual steps: - -1. Create the Proxmox API token (manual) -2. Add the node in Pulse UI -3. **For temperature monitoring**: Follow the [Quick Start for Docker](#quick-start-for-docker-deployments) above - -### For Node Configuration (All Deployments) - -When prompted for SSH setup on Proxmox nodes: - -1. Choose "y" when asked about SSH configuration -2. The script will: - - Install `lm-sensors` - - Run `sensors-detect --auto` - - Configure SSH keys (for standalone nodes) - -If the node is part of a Proxmox cluster, the script will detect other members and offer to configure the same SSH/lm-sensors setup on each of them automatically. - -### Host-side responsibilities (Docker only) - -> **Note:** For LXC deployments, the setup script handles all of this automatically. This section applies to **Docker deployments only**. - -- Run the host installer (`install-sensor-proxy.sh --standalone`) on the Proxmox machine that hosts Pulse to install and maintain the `pulse-sensor-proxy` service -- Add the bind mount to your docker-compose.yml: `- /run/pulse-sensor-proxy:/run/pulse-sensor-proxy:ro` -- Re-run the host installer if the service or socket disappears after a host upgrade or configuration cleanup; the installer is idempotent -- The installer will also refresh the `pulse_control_plane` stanza every run, so changing the Pulse URL is as simple as re-running `install-sensor-proxy.sh --standalone --pulse-server ` -- The installer ships a self-heal timer (`pulse-sensor-proxy-selfheal.timer`) that restarts or reinstalls the proxy if it ever goes missing; leave it enabled for automatic recovery -- Hot dev builds warn when only a container-local proxy socket is present, signaling that the host proxy needs to be reinstalled before temperatures will flow back into Pulse - -### Turnkey Setup for Standalone Nodes (v4.25.0+) - -**For standalone nodes** (not in a Proxmox cluster) running **containerized Pulse**, the setup script now automatically configures temperature monitoring with zero manual steps: - -1. The script detects the node is standalone (not in a cluster) -2. Automatically fetches the temperature proxy's SSH public key from your Pulse server via `/api/system/proxy-public-key` -3. Installs it with forced commands (`command="sensors -j"`) automatically -4. Temperature monitoring "just works" - no manual SSH key management needed! - -**Example output:** -``` -━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -Standalone Node Temperature Setup -━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -Detected: This is a standalone node (not in a Proxmox cluster) - -Fetching temperature proxy public key... - βœ“ Retrieved proxy public key - βœ“ Temperature proxy key installed (restricted to sensors -j) - -βœ“ Standalone node temperature monitoring configured - The Pulse temperature proxy can now collect temperature data - from this node using secure SSH with forced commands. -``` - -**Security:** -- Public keys are safe to expose (it's in the name!) -- Forced commands restrict the key to only `command="sensors -j"` -- All other SSH features disabled (no-port-forwarding, no-pty, etc.) -- Works exactly like cluster setups, but fully automated - -**Note:** This only works for containerized Pulse deployments where the temperature proxy is running. For native (non-containerized) installs, you'll still need to provide your Pulse server's public key manually as described in step 3 above. - -## Setup (Manual) - -If you skipped SSH setup during auto-setup, you can configure it manually: - -### 1. Generate SSH Key (on Pulse server) - -```bash -# Run as the user running Pulse (usually the pulse service account) -ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa -``` - -### 2. Copy Public Key to Proxmox Nodes - -```bash -# Get your public key -cat ~/.ssh/id_rsa.pub - -# Add it to each Proxmox node -ssh root@your-proxmox-node -mkdir -p /root/.ssh -chmod 700 /root/.ssh -echo "YOUR_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys -chmod 600 /root/.ssh/authorized_keys -``` - -### 3. Install lm-sensors (on each Proxmox node) - -```bash -apt-get update -apt-get install -y lm-sensors -sensors-detect --auto -``` - -### 4. Test SSH Connection - -From your Pulse server: -```bash -ssh root@your-proxmox-node "sensors -j" -``` - -You should see JSON output with temperature data. - -## Temperature Collection Pipeline - -1. **HTTPS proxy** – If a node has `TemperatureProxyURL` + `TemperatureProxyToken` configured (set automatically during `--http-mode` installs), Pulse calls `https://node:8443/temps?node=` with the bearer token. The proxy enforces TLS, CIDR allowlists, and rate limits before handing Pulse the JSON payload from `sensors -j`. -2. **Unix socket proxy** – If no HTTPS proxy is configured but the `/run/pulse-sensor-proxy` socket is mounted, Pulse talks to the proxy locally using SO\_PEERCRED authentication. -3. **SSH fallback** – Bare-metal installs can still let Pulse SSH directly into the node, run `sensors -j`, and parse the output. Containerized Pulse keeps this disabled unless explicitly overridden for development. - -Regardless of transport, Pulse parses CPU package/core temperatures plus NVMe sensor data and surfaces it on the dashboard, Host details, and the Storage tab. - -## Troubleshooting - -### HTTPS proxy not responding - -**Symptom:** Settings β†’ Diagnostics β†’ Temperature Proxy shows `proxy_unreachable`, `invalid_token`, or HTTP timeout errors for a node. - -**Verify connectivity:** -```bash -# On the Pulse host/container -curl -vk https://node.example:8443/health \ - -H "Authorization: Bearer $(sudo cat /etc/pulse-sensor-proxy/.http-auth-token)" -``` - -If that fails, confirm: -- Port 8443/TCP is reachable from the Pulse host. -- `/etc/pulse-sensor-proxy/config.yaml` lists the Pulse source CIDR in `allowed_source_subnets`. -- `/etc/pulse-sensor-proxy/.http-auth-token` and `/etc/pulse-sensor-proxy/tls/*` exist with the correct permissions. - -**Resetting the proxy:** When you need to rebuild the proxy configuration, run the uninstall path first to clear sockets, TLS keys, and state: -```bash -curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh | \ - sudo bash -s -- --uninstall --purge -``` -Then reinstall with the desired flags (for example, `--standalone --http-mode --pulse-server https://pulse:7655`). - -### Config Validation Failure on Startup - -**Symptom:** Proxy service fails to start with "Config validation failed" or "duplicate allowed_nodes blocks detected" - -**Cause:** Config file corruption from earlier versions that had dual code paths for managing the allowed nodes list. This was the root cause of 99% of temperature monitoring failures. - -**Fix (Automatic):** -Version 4.31.1+ automatically migrates to file-based config management during installation. Simply reinstall: - -```bash -curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh | \ - sudo bash -s -- --standalone --pulse-server http://your-pulse:7655 -``` - -The installer will: -- Detect and repair duplicate `allowed_nodes:` blocks in config.yaml -- Migrate to separate `/etc/pulse-sensor-proxy/allowed_nodes.yaml` file -- Use atomic Go CLI for all future config updates - -**Verify the fix:** -```bash -# Check for duplicates (should only appear once, in allowed_nodes.yaml) -grep -n "allowed_nodes:" /etc/pulse-sensor-proxy/*.yaml - -# Validate configuration -pulse-sensor-proxy config validate - -# Check service status systemctl status pulse-sensor-proxy ``` -**Manual recovery (if needed):** -See troubleshooting section in `/opt/pulse/cmd/pulse-sensor-proxy/README.md` - -### SSH Connection Attempts from Container ([preauth] Logs) - -**Symptom:** Proxmox host logs (`/var/log/auth.log`) show repeated SSH connection attempts from your Pulse container: -``` -Connection closed by authenticating user root port [preauth] -``` - -**This indicates a misconfiguration.** Containerized Pulse should communicate via the sensor proxy, not direct SSH. - -**Common causes:** -- Dev mode enabled (`PULSE_DEV_ALLOW_CONTAINER_SSH=true` environment variable) -- Sensor proxy not installed or socket not accessible -- Leftover SSH keys from legacy installations - -**Fix:** -- **Docker:** Follow [Quick Start for Docker Deployments](#quick-start-for-docker-deployments) to install the proxy and add the bind mount -- **LXC:** Run the setup script on your Proxmox host (see [Setup (Automatic)](#setup-automatic)) -- **Dev mode:** Remove `PULSE_DEV_ALLOW_CONTAINER_SSH=true` from your environment/docker-compose -- **Verify:** Check Pulse logs for `Temperature proxy detected - using secure host-side bridge` - -Once the proxy is properly configured, these log entries will stop immediately. See [Container Security Considerations](#container-security-considerations) for why direct container SSH is blocked. - -### No Temperature Data Shown - -Check Settings β†’ Diagnostics β†’ Temperature Proxy first; it usually reports the precise HTTPS or socket error. If diagnostics are clear but temperatures are still blank, validate the legacy SSH path: - -**Check SSH access**: +### View Logs ```bash -# From Pulse server -ssh root@your-proxmox-node "echo test" +journalctl -u pulse-sensor-proxy -f ``` -**Check lm-sensors**: -```bash -# On Proxmox node -sensors -j -``` +## 🧠 How It Works -**Check Pulse logs**: -```bash -journalctl -u pulse -f | grep -i temp -``` - -### Temperature Shows as Unavailable - -- lm-sensors may not be installed -- Node may not have temperature sensors -- SSH key authentication may not be working - -### ARM Devices (Raspberry Pi, etc.) - -ARM devices typically don't have the same sensor interfaces. Temperature monitoring may not work or may show different sensors (like `thermal_zone0` instead of `coretemp`). - -## Security & Architecture - -### How Temperature Collection Works - -Temperature monitoring uses **SSH key authentication** - the same trusted method used by automation tools like Ansible, Terraform, and Saltstack for managing infrastructure at scale. - -**What Happens**: -1. Pulse connects to your node via SSH using a key (no passwords) -2. Runs `sensors -j` to get temperature readings in JSON format -3. Parses the data and displays it in the dashboard -4. Disconnects (entire operation takes <1 second) - -**Security Design**: -- βœ… **Key-based authentication** - More secure than passwords, industry standard -- βœ… **Read-only operation** - `sensors` command only reads hardware data -- βœ… **Private key stays on Pulse server** - Never transmitted or exposed -- βœ… **Public key on nodes** - Safe to store, can't be used to gain access -- βœ… **Instantly revocable** - Remove key from authorized_keys to disable -- βœ… **Logged and auditable** - All connections logged in `/var/log/auth.log` - -### What Pulse Uses SSH For - -Pulse reuses the SSH access only for the actions already described in [Setup (Automatic)](#setup-automatic) and [How It Works](#how-it-works): adding the public key during setup (if you opt in) and polling `sensors -j` each cycle. It does nothing elseβ€”no extra commands, file changes, or config editsβ€”and revoking the key stops temperature collection immediately. - -This is the same security model used by thousands of organizations for infrastructure automation. - -### Best Practices - -1. **Dedicated key**: Generate a separate SSH key just for Pulse (recommended) -2. **Firewall rules**: Optionally restrict SSH to your Pulse server's IP -3. **Regular monitoring**: Review auth logs if you want extra visibility -4. **Secure your Pulse server**: Keep it updated and behind proper access controls - -### Command Restrictions (Default) - -Pulse now writes the temperature key with a forced command so the connection can only execute `sensors -j`. Port/X11/agent forwarding and PTY allocation are all disabled automatically when you opt in through the setup script. Re-running the script upgrades older installs to the restricted entry without touching any of your other SSH keys. - -```bash -# Example entry in /root/.ssh/authorized_keys installed by Pulse -command="sensors -j",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2E... -``` - -You can still manage the entry manually if you prefer, but no extra steps are required for new installations. - -## Performance Impact - -- Minimal: SSH connection is made once per polling cycle -- Timeout: 5 seconds (non-blocking) -- Falls back gracefully if SSH fails -- No impact if SSH is not configured - -## Container Security Considerations - -βœ… **Resolved in v4.24.0** - -### Secure Proxy Architecture (Current) - -As of v4.24.0, containerized deployments use **pulse-sensor-proxy** which eliminates the security concerns: - -- **SSH keys stored on host** - Not accessible from container -- **Unix socket communication** - Pulse never touches SSH keys -- **Automatic during installation** - No manual configuration needed -- **Container compromise = No credential exposure** - Attacker gains nothing - -**For new installations:** The proxy is installed automatically during LXC setup. No action required. - -**Installed from inside an existing LXC?** The container-only installer cannot create the host bind mount. Run the host-side script below on your Proxmox node to enable temperature monitoring. When Pulse is running in that container, append the server URL so the proxy script can fall back to downloading the binary from Pulse itself if GitHub isn’t available. - -**For existing installations (pre-v4.24.0):** Upgrade your deployment to use the proxy: - -```bash -# On your Proxmox host -curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh | \ - bash -s -- --ctid --pulse-server http://:7655 -``` - -> **Heads up for v4.23.x:** Those builds don't ship a standalone `pulse-sensor-proxy` binary yet and the HTTP fallback still requires authentication. Either upgrade to a newer release, install Pulse from source (`install.sh --source main`), or pass a locally built binary with `--local-binary /path/to/pulse-sensor-proxy`. - -### Automation-Friendly Installation - -For infrastructure-as-code tools (Ansible, Terraform, Salt, Puppet), the installer script is fully scriptable. - -#### Installation Script Flags - -```bash -install-sensor-proxy.sh [OPTIONS] -``` - -**Required (choose one):** -- `--ctid ` - For LXC containers (auto-configures bind mount) -- `--standalone` - For Docker or standalone deployments - -**Optional:** -- `--pulse-server ` - Pulse server URL (for binary fallback if GitHub unavailable) -- `--version ` - Specific version to install (default: latest) -- `--local-binary ` - Use local binary instead of downloading -- `--quiet` - Non-interactive mode (suppress progress output) -- `--skip-restart` - Don't restart LXC container after installation -- `--uninstall` - Remove the proxy service -- `--purge` - Remove data directories (use with --uninstall) - -**Behavior:** -- βœ… **Idempotent** - Safe to re-run, won't break existing installations -- βœ… **Non-interactive** - Use `--quiet` for automated deployments -- βœ… **Verifiable** - Returns exit code 0 on success, non-zero on failure - -#### Ansible Playbook Example - -**For LXC deployments:** - -```yaml ---- -- name: Install Pulse sensor proxy for LXC - hosts: proxmox_hosts - become: yes - tasks: - - name: Download installer script - get_url: - url: https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh - dest: /tmp/install-sensor-proxy.sh - mode: '0755' - - - name: Install sensor proxy - command: > - /tmp/install-sensor-proxy.sh - --ctid {{ pulse_container_id }} - --pulse-server {{ pulse_server_url }} - --quiet - register: install_result - changed_when: "'already exists' not in install_result.stdout" - failed_when: install_result.rc != 0 - - - name: Verify proxy is running - systemd: - name: pulse-sensor-proxy - state: started - enabled: yes - register: service_status -``` - -**For Docker deployments:** - -```yaml ---- -- name: Install Pulse sensor proxy for Docker - hosts: proxmox_hosts - become: yes - tasks: - - name: Download installer script - get_url: - url: https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh - dest: /tmp/install-sensor-proxy.sh - mode: '0755' - - - name: Install sensor proxy (standalone mode) - command: > - /tmp/install-sensor-proxy.sh - --standalone - --pulse-server {{ pulse_server_url }} - --quiet - register: install_result - failed_when: install_result.rc != 0 - - - name: Verify proxy is running - systemd: - name: pulse-sensor-proxy - state: started - enabled: yes - - - name: Ensure docker-compose includes sensor proxy bind mount - blockinfile: - path: /opt/pulse/docker-compose.yml - marker: "# {mark} ANSIBLE MANAGED - Sensor Proxy" - insertafter: "volumes:" - block: | - - /run/pulse-sensor-proxy:/run/pulse-sensor-proxy:ro - notify: restart pulse container - - handlers: - - name: restart pulse container - community.docker.docker_compose: - project_src: /opt/pulse - state: restarted -``` - -#### Terraform Example - -```hcl -resource "null_resource" "pulse_sensor_proxy" { - for_each = var.proxmox_hosts - - connection { - type = "ssh" - host = each.value.host - user = "root" +1. **Pulse Sensor Proxy**: A lightweight service runs on the Proxmox host. +2. **Secure Access**: It reads sensors (via `lm-sensors`) and exposes them securely. +3. **Transport**: + * **Local**: Uses a Unix socket (`/run/pulse-sensor-proxy`) for zero-latency, secure access. + * **Remote**: Uses mutual TLS over HTTPS (port 8443). +4. **No SSH Keys**: Pulse containers no longer need SSH keys to read temperatures. user = "root" private_key = file(var.ssh_private_key) } diff --git a/docs/VM_DISK_MONITORING.md b/docs/VM_DISK_MONITORING.md index b2239e6..dc24d97 100644 --- a/docs/VM_DISK_MONITORING.md +++ b/docs/VM_DISK_MONITORING.md @@ -1,326 +1,44 @@ -# VM Disk Usage Monitoring +# πŸ’Ύ VM Disk Monitoring -Pulse can show actual disk usage for VMs (just like containers) when the QEMU Guest Agent is installed and configured properly. +Monitor actual disk usage inside your VMs using the QEMU Guest Agent. -## Quick Summary +## πŸš€ Quick Start -**Without QEMU Guest Agent:** -- VMs show "-" for disk usage (no data available) -- Cannot monitor actual disk usage inside the VM +1. **Install Guest Agent**: + * **Linux**: `apt install qemu-guest-agent` (Debian/Ubuntu) or `yum install qemu-guest-agent` (RHEL). + * **Windows**: Install **virtio-win** drivers. +2. **Enable in Proxmox**: + * VM Options β†’ **QEMU Guest Agent** β†’ Enabled. + * Restart the VM. +3. **Verify**: + * Run `qm agent ping` on the Proxmox host. + * Check Pulse dashboard for disk usage (e.g., "5.2GB used of 32GB"). -**With QEMU Guest Agent:** -- VMs show real disk usage like containers do (e.g., "5.2GB used of 32GB / 16%") -- Accurate threshold alerts based on actual usage -- Better capacity planning with real data +## βš™οΈ Requirements -## How It Works +* **QEMU Guest Agent**: Must be installed and running inside the VM. +* **Proxmox Permissions**: + * **Proxmox 8**: `VM.Monitor` + * **Proxmox 9+**: `VM.GuestAgent.Audit` + * *Note: The Pulse setup script handles these permissions automatically.* -Proxmox doesn't track VM disk usage natively (unlike containers which share the host kernel). To get real disk usage from VMs: +## πŸ”§ Troubleshooting -1. Proxmox API returns `disk=0` and `maxdisk=` (this is normal) -2. Pulse automatically queries the QEMU Guest Agent API to get filesystem info -3. Guest agent reports all mounted filesystems from inside the VM -4. Pulse aggregates the data (filtering out special filesystems) and displays it +| Issue | Solution | +| :--- | :--- | +| **Disk shows "-"** | Hover over the dash for details. Common causes: Agent not running, disabled in config, or permission denied. | +| **Permission Denied** | Re-run the Pulse setup script to update permissions, or manually add `VM.GuestAgent.Audit`. | +| **Agent Timeout** | Increase timeouts via env vars if network is slow: `GUEST_AGENT_FSINFO_TIMEOUT=10s`. | +| **Windows VMs** | Ensure the **QEMU Guest Agent** service is running in Windows Services. | -**Important**: This works with both API tokens and password authentication. API tokens work fine for guest agent queries when permissions are set correctly. - -## Requirements - -### 1. Install QEMU Guest Agent in Your VMs - -**Linux VMs:** +### Diagnostic Script +Run this on your Proxmox host to debug specific VMs: ```bash -# Debian/Ubuntu -apt-get install qemu-guest-agent -systemctl enable --now qemu-guest-agent - -# RHEL/Rocky/AlmaLinux -yum install qemu-guest-agent -systemctl enable --now qemu-guest-agent - -# Alpine -apk add qemu-guest-agent -rc-update add qemu-guest-agent -rc-service qemu-guest-agent start -``` - -**Windows VMs:** -- Download virtio-win guest tools from: https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/ -- Install the guest tools package which includes the QEMU Guest Agent -- The service starts automatically after installation - -### 2. Enable Guest Agent in VM Options - -In Proxmox web UI: -1. Select your VM -2. Go to **Options** β†’ **QEMU Guest Agent** -3. Check **Enabled** -4. Start/restart the VM - -Or via CLI: -```bash -qm set --agent enabled=1 -``` - -### 3. Verify Guest Agent is Working - -Check if the agent is responding: -```bash -qm agent ping -``` - -Get filesystem info (what Pulse uses): -```bash -qm agent get-fsinfo -``` - -### 4. Pulse Permissions - -Pulse needs the right permissions to query the guest agent: - -**Proxmox VE 8 and below:** -- Requires `VM.Monitor` for guest agent access -- `Sys.Audit` adds Ceph/cluster metrics and is applied when available -- Pulse setup script creates a `PulseMonitor` role with these privileges automatically - -**Proxmox VE 9+:** -- Requires `VM.GuestAgent.Audit` for guest agent access -- `Sys.Audit` remains recommended for Ceph/cluster metrics -- Pulse setup script applies both via the `PulseMonitor` role (even if `PVEAuditor` lacks them) - -**Both API tokens and passwords work** - tokens do NOT have any limitation accessing guest agent data. - -When you run the Pulse setup script, it automatically detects your Proxmox version and sets the correct permissions. If setting up manually: - -```bash -# Shared read-only access -pveum aclmod / -user pulse-monitor@pam -role PVEAuditor - -# Extra privileges for guest metrics and Ceph -EXTRA_PRIVS=() - -# Sys.Audit (Ceph, cluster status) -if pveum role list 2>/dev/null | grep -q "Sys.Audit"; then - EXTRA_PRIVS+=(Sys.Audit) -else - if pveum role add PulseTmpSysAudit -privs Sys.Audit 2>/dev/null; then - EXTRA_PRIVS+=(Sys.Audit) - pveum role delete PulseTmpSysAudit 2>/dev/null - fi -fi - -# VM guest agent / monitor privileges -VM_PRIV="" -if pveum role list 2>/dev/null | grep -q "VM.Monitor"; then - VM_PRIV="VM.Monitor" -elif pveum role list 2>/dev/null | grep -q "VM.GuestAgent.Audit"; then - VM_PRIV="VM.GuestAgent.Audit" -else - if pveum role add PulseTmpVMMonitor -privs VM.Monitor 2>/dev/null; then - VM_PRIV="VM.Monitor" - pveum role delete PulseTmpVMMonitor 2>/dev/null - elif pveum role add PulseTmpGuestAudit -privs VM.GuestAgent.Audit 2>/dev/null; then - VM_PRIV="VM.GuestAgent.Audit" - pveum role delete PulseTmpGuestAudit 2>/dev/null - fi -fi - -if [ -n "$VM_PRIV" ]; then - EXTRA_PRIVS+=("$VM_PRIV") -fi - -if [ ${#EXTRA_PRIVS[@]} -gt 0 ]; then - PRIV_STRING="${EXTRA_PRIVS[*]}" - pveum role delete PulseMonitor 2>/dev/null - pveum role add PulseMonitor -privs "$PRIV_STRING" - pveum aclmod / -user pulse-monitor@pam -role PulseMonitor -fi -``` - -## Troubleshooting - -### Quick Diagnostic Tool - -Pulse includes a diagnostic script that can identify why a VM isn't showing disk usage: - -```bash -# Run on your Proxmox host (latest version from GitHub) curl -sSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/test-vm-disk.sh | bash - -# Or use the bundled copy installed with Pulse -/opt/pulse/scripts/test-vm-disk.sh ``` -Enter the VM ID when prompted. The script will check: -- VM running status -- Guest agent configuration -- Guest agent runtime status -- Filesystem information -- API permissions +## πŸ“ Notes -### Understanding Disk Display States - -**Shows percentage** (e.g., "45%") -- Everything working correctly -- Guest agent installed and accessible - -**Shows "-" with hover tooltip** -- Hover to see the specific reason -- Common reasons: - - "Guest agent not running" - Agent not installed or service not started - - "Guest agent disabled" - Not enabled in VM config - - "Permission denied" - Token/user lacks required permissions - - "Agent timeout" - Agent installed but not responding - - "No filesystems" - Agent returned no usable filesystem data - -### Guest Agent Not Responding - -**Check if agent is running inside VM:** -```bash -# Linux -systemctl status qemu-guest-agent - -# Windows -Get-Service QEMU-GA -``` - -**Check VM configuration:** -```bash -# Should show "agent: 1" -qm config | grep agent -``` - -**Check agent communication:** -```bash -# Should return without error -qm agent ping -``` - -### Configuring Guest Agent Timeouts - -**New in v4.27:** Guest agent timeouts and retry behavior can be configured via environment variables to handle high-load environments or slow networks (refs #592). - -**Available Environment Variables:** - -```bash -# Timeout for filesystem info queries (default: 15s, previously 5s) -GUEST_AGENT_FSINFO_TIMEOUT=15s - -# Timeout for network interface queries (default: 10s, previously 5s) -GUEST_AGENT_NETWORK_TIMEOUT=10s - -# Timeout for OS info queries (default: 10s, previously 3s) -GUEST_AGENT_OSINFO_TIMEOUT=10s - -# Timeout for agent version queries (default: 10s, previously 3s) -GUEST_AGENT_VERSION_TIMEOUT=10s - -# Number of retries for timeout failures (default: 1, meaning one retry after initial failure) -GUEST_AGENT_RETRIES=1 -``` - -**When to Adjust:** -- **Large environments (50+ VMs):** Increase timeouts to 20-30s if you see frequent timeout errors -- **Slow networks/WAN:** Increase timeouts proportionally to network latency -- **High load periods:** Consider increasing retries to 2 for better resilience -- **Fast local network:** Can reduce timeouts to 5-8s for quicker feedback - -**How to Apply:** - -```bash -# Docker deployment - add to docker run or compose -docker run -e GUEST_AGENT_FSINFO_TIMEOUT=20s -e GUEST_AGENT_RETRIES=2 ... - -# Systemd deployment - add to /etc/systemd/system/pulse.service -[Service] -Environment="GUEST_AGENT_FSINFO_TIMEOUT=20s" -Environment="GUEST_AGENT_RETRIES=2" -``` - -After changing environment variables, restart Pulse for the changes to take effect. - -### Permission Denied Errors - -If you see "permission denied" in Pulse logs when querying guest agent: - -1. **Verify token/user permissions:** - ```bash - pveum user permissions pulse-monitor@pam - ``` - -2. **For Proxmox 9+:** Ensure user has the `VM.GuestAgent.Audit` privilege (PulseMonitor role handles this) - -3. **For Proxmox 8:** Ensure user has the `VM.Monitor` privilege (PulseMonitor role handles this) - -4. **All versions:** Confirm `Sys.Audit` is present for Ceph metrics when applicable - -5. **Re-run setup script** if you added the node before Pulse v4.7 (old scripts didn't add VM.Monitor/guest agent privileges) - -### Disk Usage Still Not Showing - -If the agent is working but Pulse still shows "-": - -1. **Check Pulse logs** for specific error messages: - ```bash - # Docker - docker logs pulse | grep -i "guest agent\|fsinfo" - - # Systemd - journalctl -u pulse -f | grep -i "guest agent\|fsinfo" - ``` - -2. **Test guest agent manually** from Proxmox host: - ```bash - qm agent get-fsinfo - ``` - If this works but Pulse doesn't show data, check Pulse permissions and logs - (v4.24.0: adjust **Settings β†’ System β†’ Logging** to `debug` temporarily if you need more detail, then revert to `info`). - -3. **Check agent version** - Older agents might not support filesystem info - -4. **Windows VMs** - Ensure virtio-win drivers are up to date - -### Network Filesystems - -The agent reports all mounted filesystems. Pulse automatically filters out: -- Network mounts (NFS, CIFS, SMB) -- Special filesystems (proc, sys, tmpfs, devtmpfs, etc.) -- Special Windows partitions ("System Reserved") -- Bind mounts and overlays -- Read-only appliance or optical images (squashfs, erofs, iso9660, CDFS, UDF, cramfs, romfs, fuse.cdfs) - -Only local disk usage is counted toward the VM's total. - -## Best Practices - -1. **Install guest agent in VM templates** - New VMs will have it ready -2. **Monitor agent status** - Set up alerts if critical VMs lose agent connectivity -3. **Keep agents updated** - Update guest agents when updating VM operating systems -4. **Test after VM migrations** - Verify agent still works after moving VMs between nodes -5. **Check logs regularly** - Monitor Pulse logs for guest agent errors - -## Platform-Specific Notes - -### Cloud-Init Images -Most cloud images include qemu-guest-agent pre-installed but may need to be enabled: -```bash -systemctl enable --now qemu-guest-agent -``` - -### Docker/Kubernetes VMs -Container workloads can show high disk usage due to container layers. Consider: -- Using separate disks for container storage -- Monitoring container disk usage separately -- Setting appropriate thresholds for container hosts - -### Database VMs -Databases often pre-allocate space. The guest agent shows actual usage, which might be less than what the database reports internally. - -## Benefits - -With QEMU Guest Agent disk monitoring: -- **Accurate alerts** - Alert on real usage, not allocated space -- **Better planning** - See actual growth trends -- **Prevent surprises** - Know when VMs are actually running out of space -- **Optimize storage** - Identify over-provisioned VMs -- **Consistent monitoring** - VMs and containers use the same metrics +* **Network Mounts**: NFS/SMB mounts are automatically excluded. +* **Databases**: Usage reflects filesystem usage, which may differ from database-internal metrics. +* **Containers**: LXC containers are monitored natively without the guest agent.