Refactor agent and troubleshooting docs to be modern and concise
This commit is contained in:
courtmanr@gmail.com
parent
f5c3955f3f
commit
25eaa2ca18
3 changed files with 172 additions and 1141 deletions
|
|
@ -1,317 +1,73 @@
|
|||
# Docker & Podman Monitoring Agent
|
||||
# 🐳 Docker Monitoring Agent
|
||||
|
||||
Pulse is focused on Proxmox VE and PBS, but many homelabs also run application stacks in container runtimes such as Docker and Podman. The optional Pulse container agent turns runtime health and resource usage into first-class metrics that show up alongside your hypervisor data. The recommended deployment is the bundled, least-privilege systemd service that runs the static `pulse-docker-agent` binary directly on the host. That path lets the installer lock down permissions, manage upgrades automatically, and integrate with the native init system. Containerising the agent is still available for orchestrated environments, but it trades away some of those controls (and still needs the runtime socket) so treat that option as advanced.
|
||||
Monitor Docker and Podman hosts alongside your Proxmox infrastructure.
|
||||
|
||||
## What the agent reports
|
||||
## 🚀 Quick Start
|
||||
|
||||
Every check interval (30s by default) the agent collects:
|
||||
|
||||
- Host metadata (hostname, Docker version, CPU count, total memory, uptime)
|
||||
- Container status (`running`, `exited`, `paused`) and health probe state
|
||||
- Restart counters and exit codes
|
||||
- CPU usage, memory consumption and limits
|
||||
- Images, port mappings, network addresses, and start times
|
||||
- Writable layer size, root filesystem size, block I/O totals, and mount metadata (shown in the Containers table drawer)
|
||||
- Read/write throughput derived from Docker block I/O counters so you can spot noisy workloads at a glance
|
||||
- Health-check failures, restart-loop windows, and recent exit codes (displayed in the UI under each container drawer)
|
||||
|
||||
Data is pushed to Pulse over HTTPS using your existing API token – no inbound firewall rules required.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Pulse v4.22.0 or newer with an API token enabled (`Settings → Security`)
|
||||
- API token with the `docker:report` scope (add `docker:manage` if you use remote lifecycle commands)
|
||||
- Docker 20.10+ **or** Podman 4.7+ on Linux (the agent talks to the runtime API socket)
|
||||
- Access to the runtime socket (`/var/run/docker.sock`, `/run/podman/podman.sock`, or a `unix://` URI)
|
||||
- Go 1.24+ if you plan to build the binary from source
|
||||
|
||||
## Installation
|
||||
|
||||
Grab the `pulse-docker-agent` binary from the release assets (or build it yourself):
|
||||
Generate an installation command in the UI:
|
||||
**Settings → Agents → Docker Agents → "Install New Agent"**
|
||||
|
||||
### Standard Install
|
||||
```bash
|
||||
# Build from source
|
||||
cd /opt/pulse
|
||||
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o pulse-docker-agent ./cmd/pulse-docker-agent
|
||||
curl -fsSL http://<pulse-ip>:7655/install-docker-agent.sh | \
|
||||
sudo bash -s -- --url http://<pulse-ip>:7655 --token <api-token>
|
||||
```
|
||||
*Creates a `pulse-docker-agent` systemd service.*
|
||||
|
||||
Copy the binary to your Docker host (e.g. `/usr/local/bin/pulse-docker-agent`) and make it executable.
|
||||
|
||||
> **Why `CGO_ENABLED=0`?** Building a fully static binary ensures the agent runs on hosts still using older glibc releases (for example Debian 11 with glibc 2.31).
|
||||
|
||||
### Quick install from your Pulse server (recommended)
|
||||
|
||||
Use the bundled installation script (ships with Pulse v4.22.0+) to deploy and manage the agent. Replace the token placeholder with an API token generated in **Settings → Security**. Create a dedicated token for each Docker host so you can revoke individual credentials without touching others—sharing one token across many hosts makes incident response much harder. Tokens used here should include the `docker:report` scope so the agent can submit telemetry (add `docker:manage` only if you plan to issue lifecycle commands remotely).
|
||||
|
||||
### Podman (Rootless)
|
||||
```bash
|
||||
curl -fSL http://pulse.example.com/install-docker-agent.sh -o /tmp/pulse-install-docker-agent.sh && \
|
||||
sudo bash /tmp/pulse-install-docker-agent.sh --url http://pulse.example.com --token <api-token> && \
|
||||
rm -f /tmp/pulse-install-docker-agent.sh
|
||||
curl -fsSL http://<pulse-ip>:7655/install-container-agent.sh | \
|
||||
bash -s -- --runtime podman --rootless --url http://<pulse-ip>:7655 --token <api-token>
|
||||
```
|
||||
|
||||
> **Why sudo?** The installer needs to drop binaries under `/usr/local/bin`, create a systemd service, and start it—actions that require root privileges. Piping to `sudo bash …` saves you from retrying if you run the command as an unprivileged user.
|
||||
---
|
||||
|
||||
The script stores credentials in `/etc/pulse/pulse-docker-agent.env` (mode `600`) and creates a locked-down `pulse-docker` service account that only needs access to the Docker socket. Rotate tokens by editing that env file and running `sudo systemctl restart pulse-docker-agent`.
|
||||
## 📊 Features
|
||||
|
||||
To keep remote stop/remove commands working from Pulse, the installer also drops a small polkit rule that lets the `pulse-docker` service account run `systemctl stop/disable pulse-docker-agent` without password prompts. If you remove that rule, expect to acknowledge stop requests manually with `sudo systemctl disable --now pulse-docker-agent`.
|
||||
- **Container Metrics**: CPU, Memory, Network, Disk I/O.
|
||||
- **Health Checks**: Tracks container health status and restart loops.
|
||||
- **Swarm Support**: Auto-detects Swarm mode and reports service/task data.
|
||||
- **Multi-Target**: Can report to multiple Pulse servers for HA.
|
||||
|
||||
Running the one-liner again from another Pulse server (with its own URL/token) will merge that server into the same agent automatically—no extra flags required.
|
||||
---
|
||||
|
||||
To report to more than one Pulse instance from the same Docker host, repeat the `--target` flag (format: `https://pulse.example.com|<api-token>`) or export `PULSE_TARGETS` before running the script:
|
||||
## ⚙️ Configuration
|
||||
|
||||
```bash
|
||||
curl -fSL http://pulse.example.com/install-docker-agent.sh -o /tmp/pulse-install-docker-agent.sh && \
|
||||
sudo bash /tmp/pulse-install-docker-agent.sh -- \
|
||||
--target https://pulse.example.com|<primary-token> \
|
||||
--target https://pulse-dr.example.com|<dr-token> && \
|
||||
rm -f /tmp/pulse-install-docker-agent.sh
|
||||
```
|
||||
The agent is configured via flags or environment variables (in `/etc/pulse/pulse-docker-agent.env`).
|
||||
|
||||
### Removing the agent
|
||||
| Flag | Env Var | Description | Default |
|
||||
|------|---------|-------------|---------|
|
||||
| `--url` | `PULSE_URL` | Pulse Server URL | `http://localhost:7655` |
|
||||
| `--token` | `PULSE_TOKEN` | API Token (scope: `docker:report`) | *(required)* |
|
||||
| `--interval` | `PULSE_INTERVAL` | Polling Interval | `30s` |
|
||||
| `--runtime` | `PULSE_RUNTIME` | `docker` or `podman` | `docker` |
|
||||
| `--collect-disk` | `PULSE_COLLECT_DISK` | Monitor container disk usage | `true` |
|
||||
|
||||
Re-run the installer with `--uninstall` to stop the service, delete the systemd unit, and keep the logs for later review:
|
||||
<details>
|
||||
<summary><strong>Advanced: Run as Container</strong></summary>
|
||||
|
||||
```bash
|
||||
curl -fSL http://pulse.example.com/install-docker-agent.sh -o /tmp/pulse-install-docker-agent.sh && \
|
||||
sudo bash /tmp/pulse-install-docker-agent.sh --uninstall && \
|
||||
rm -f /tmp/pulse-install-docker-agent.sh
|
||||
```
|
||||
|
||||
Add `--purge` when you also want the installer to remove `/var/log/pulse-docker-agent`, the env/config files under `/etc/pulse`, and the `pulse-docker` service account. Purging cleans up `/etc/passwd` so container processes that run as UID 999 stop showing up as the `pulse-docker` user in `ps` output.
|
||||
|
||||
### Quick install for Podman (system service)
|
||||
|
||||
Use the multi-runtime installer when you want the agent to run against Podman as a systemd service. The script takes care of enabling `podman.socket`, creating a dedicated service account, and wiring the correct runtime socket automatically:
|
||||
|
||||
```bash
|
||||
curl -fSL http://pulse.example.com/install-container-agent.sh -o /tmp/pulse-install-container-agent.sh && \
|
||||
sudo bash /tmp/pulse-install-container-agent.sh --runtime podman --url http://pulse.example.com --token <api-token> && \
|
||||
rm -f /tmp/pulse-install-container-agent.sh
|
||||
```
|
||||
|
||||
The environment file lives at `/etc/pulse/pulse-docker-agent.env` and the unit is still named `pulse-docker-agent.service` for backwards compatibility. The agent exports `PULSE_RUNTIME=podman` and points both `CONTAINER_HOST` and `DOCKER_HOST` at the Podman socket (`/run/podman/podman.sock` by default). Restart the service after editing the env file with `sudo systemctl restart pulse-docker-agent`.
|
||||
|
||||
> **What's new for Podman?** The agent now sends pod- and compose-aware metadata for Podman hosts. Pulse surfaces pod names, infra-container markers, compose project/service identifiers, auto-update policies, and user namespace hints so you can see how containers relate without leaving the UI.
|
||||
|
||||
### Quick install for Podman (rootless user service)
|
||||
|
||||
Podman’s rootless mode works too. Run the installer as the target user and add the `--rootless` flag — no sudo required:
|
||||
|
||||
```bash
|
||||
curl -fSL http://pulse.example.com/install-container-agent.sh -o /tmp/pulse-install-container-agent.sh && \
|
||||
bash /tmp/pulse-install-container-agent.sh --runtime podman --rootless --url http://pulse.example.com --token <api-token> && \
|
||||
rm -f /tmp/pulse-install-container-agent.sh
|
||||
```
|
||||
|
||||
The agent binary is dropped into `~/.local/bin`, configuration lives under `~/.config/pulse`, and a user-level service (`~/.config/systemd/user/pulse-docker-agent.service`) is created. Enable lingering so the agent keeps running after you log out:
|
||||
|
||||
```bash
|
||||
sudo loginctl enable-linger "$USER"
|
||||
```
|
||||
|
||||
If `systemctl --user` is unavailable, the installer will print the exact command you can place in a cron job or another init system.
|
||||
|
||||
## Running the agent
|
||||
|
||||
The agent needs to know where Pulse lives and which API token to use.
|
||||
|
||||
**Single instance:**
|
||||
|
||||
```bash
|
||||
export PULSE_URL="http://pulse.lan:7655"
|
||||
export PULSE_TOKEN="<your-api-token>"
|
||||
|
||||
sudo /usr/local/bin/pulse-docker-agent --interval 30s
|
||||
```
|
||||
|
||||
**Multiple instances (one agent fan-out):**
|
||||
|
||||
```bash
|
||||
export PULSE_TARGETS="https://pulse-primary.lan:7655|<token-primary>;https://pulse-dr.lan:7655|<token-dr>"
|
||||
|
||||
sudo /usr/local/bin/pulse-docker-agent --interval 30s
|
||||
```
|
||||
|
||||
You can also repeat `--target https://pulse.example.com|<token>` on the command line instead of using `PULSE_TARGETS`; the agent will broadcast each heartbeat to every configured URL.
|
||||
|
||||
The binary reads standard Docker environment variables. If you already use TLS-secured remote sockets set `DOCKER_HOST`, `DOCKER_TLS_VERIFY`, etc. as normal. To skip TLS verification for Pulse (not recommended) add `--insecure` or `PULSE_INSECURE_SKIP_VERIFY=true`.
|
||||
|
||||
### Filtering container states
|
||||
|
||||
High churn environments can flood Pulse with noise from short-lived tasks. Restrict the agent to the container states you care about by repeating `--container-state` (for example, `--container-state running --container-state paused`) or by exporting `PULSE_CONTAINER_STATES=running,paused`. Allowed values match Docker’s status filter: `created`, `running`, `restarting`, `removing`, `paused`, `exited`, and `dead`. If no values are provided the agent reports every container, mirroring the previous behaviour.
|
||||
|
||||
### Swarm-aware reporting
|
||||
|
||||
The agent now recognises Docker Swarm roles. Managers query the Swarm control plane for service and task metadata, while workers fall back to the labels present on local containers. The **Settings → Docker Agents** view surfaces role, scope, service counts, and updates per host so you can spot noisy stacks or unhealthy rollouts at a glance.
|
||||
|
||||
Use the new flags to tune the payload:
|
||||
|
||||
- `--swarm-scope` / `PULSE_SWARM_SCOPE` chooses between node-only and cluster-wide aggregation (`auto` switches based on the node’s role).
|
||||
- `--swarm-services` and `--swarm-tasks` disable service or task blocks if you only need a subset of data.
|
||||
- `--include-containers` removes per-container metrics when service-level reporting is sufficient (note that workers need container data to derive task info).
|
||||
|
||||
If a manager cannot reach the Swarm API the agent automatically falls back to node scope so updates keep flowing.
|
||||
|
||||
Adjust warning and critical replica gaps (or disable service alerts entirely) under **Alerts → Thresholds → Containers** in the Pulse UI.
|
||||
|
||||
### Multiple Pulse instances
|
||||
|
||||
A single `pulse-docker-agent` process can now serve any number of Pulse backends. Each target entry keeps its own API token and TLS preference, and Pulse de-duplicates reports using the shared agent ID / machine ID. This avoids running duplicate agents on busy Docker hosts.
|
||||
|
||||
### Systemd unit example
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Pulse Docker Agent
|
||||
After=network-online.target docker.socket docker.service
|
||||
Wants=network-online.target docker.socket
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
EnvironmentFile=-/etc/pulse/pulse-docker-agent.env
|
||||
ExecStart=/usr/local/bin/pulse-docker-agent --interval 30s
|
||||
Restart=on-failure
|
||||
RestartSec=5s
|
||||
StartLimitIntervalSec=120
|
||||
StartLimitBurst=5
|
||||
User=pulse-docker
|
||||
Group=pulse-docker
|
||||
SupplementaryGroups=docker
|
||||
UMask=0077
|
||||
NoNewPrivileges=yes
|
||||
RestrictSUIDSGID=yes
|
||||
RestrictRealtime=yes
|
||||
PrivateTmp=yes
|
||||
ProtectSystem=full
|
||||
ProtectHome=read-only
|
||||
ProtectControlGroups=yes
|
||||
ProtectKernelModules=yes
|
||||
ProtectKernelTunables=yes
|
||||
ProtectKernelLogs=yes
|
||||
LockPersonality=yes
|
||||
MemoryDenyWriteExecute=yes
|
||||
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
|
||||
ReadWritePaths=/var/run/docker.sock
|
||||
ProtectHostname=yes
|
||||
ProtectClock=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Rotate credentials or add additional Pulse targets by editing `/etc/pulse/pulse-docker-agent.env` and reloading the service with `sudo systemctl restart pulse-docker-agent`.
|
||||
|
||||
### Containerised agent (advanced / optional)
|
||||
|
||||
If you prefer to run the agent inside a container, mount the Docker socket and supply the same environment variables:
|
||||
You can run the agent as a container instead of a system service.
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name pulse-docker-agent \
|
||||
--pid=host \
|
||||
--uts=host \
|
||||
-e PULSE_URL="https://pulse.example.com" \
|
||||
-e PULSE_TOKEN="<token>" \
|
||||
-e PULSE_TARGETS="https://pulse.example.com|<token>;https://pulse-dr.example.com|<token-dr>" \
|
||||
-e PULSE_NO_AUTO_UPDATE=true \
|
||||
-v /etc/machine-id:/etc/machine-id:ro \
|
||||
--name pulse-agent \
|
||||
--pid=host --uts=host \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
--restart unless-stopped \
|
||||
-e PULSE_URL="http://pulse:7655" \
|
||||
-e PULSE_TOKEN="<token>" \
|
||||
ghcr.io/rcourtman/pulse-docker-agent:latest
|
||||
```
|
||||
</details>
|
||||
|
||||
Add `-e LOG_LEVEL=warn` when your container logs feed into journald or another centralized collector and you want quieter steady-state logging.
|
||||
---
|
||||
|
||||
> **Note**: Official images for `linux/amd64` and `linux/arm64` are published to `ghcr.io/rcourtman/pulse-docker-agent`. To test local changes, run `docker build --target agent_runtime -t pulse-docker-agent:test .` from the repository root.
|
||||
## ⚠️ Troubleshooting
|
||||
|
||||
`--pid=host`, `--uts=host`, and the `/etc/machine-id` bind keep host metadata stable so Pulse doesn’t think the container itself is the Docker host. Auto-update is disabled in the image by default; rebuild or override `PULSE_NO_AUTO_UPDATE=false` only if you manage upgrades outside of your orchestrator. Expect to grant the container the same level of Docker socket access as the systemd service—running inside Docker doesn’t sandbox the agent from the host.
|
||||
- **Agent Rejected?**
|
||||
If a host was previously removed, you must "Allow re-enroll" in **Settings → Docker → Removed Hosts**.
|
||||
|
||||
## Configuration reference
|
||||
- **Permission Denied (Socket)?**
|
||||
Ensure the `pulse-docker` user is in the `docker` group (`sudo usermod -aG docker pulse-docker`).
|
||||
|
||||
| Flag / Env var | Description | Default |
|
||||
| ----------------------- | --------------------------------------------------------- | --------------- |
|
||||
| `--url`, `PULSE_URL` | Pulse base URL (http/https). | `http://localhost:7655` |
|
||||
| `--token`, `PULSE_TOKEN`| Pulse API token with `docker:report` scope (required). | — |
|
||||
| `--target`, `PULSE_TARGETS` | One or more `url|token[|insecure]` entries to fan-out reports to multiple Pulse servers. Separate entries with `;` or repeat the flag. | — |
|
||||
| `--interval`, `PULSE_INTERVAL` | Reporting cadence (supports `30s`, `1m`, etc.). | `30s` |
|
||||
| `--log-level`, `LOG_LEVEL` | Log verbosity (`debug`, `info`, `warn`, `error`). Use `debug` only while actively troubleshooting to avoid noisy journals. | `info` |
|
||||
| `--runtime`, `PULSE_RUNTIME` | Container runtime to target (`docker`, `podman`, `auto`). | `docker` |
|
||||
| `--container-socket`, `PULSE_CONTAINER_SOCKET` / `CONTAINER_HOST` | Explicit runtime socket path or `unix://` URI. | Runtime default |
|
||||
| `--rootless`, `PULSE_RUNTIME_ROOTLESS` | Install/manage the agent as a user service (Podman). | Auto (rootful) |
|
||||
| `--container-state`, `PULSE_CONTAINER_STATES` | Limit reports to specific Docker statuses (`created`, `running`, `restarting`, `removing`, `paused`, `exited`, `dead`). Separate multiple values with commas/semicolons or repeat the flag. | — |
|
||||
| `--swarm-scope`, `PULSE_SWARM_SCOPE` | Swarm data scope: `node`, `cluster`, or `auto` (auto picks cluster on managers, node on workers). | `node` |
|
||||
| `--swarm-services`, `PULSE_SWARM_SERVICES` | Include Swarm service summaries in reports. | `true` |
|
||||
| `--swarm-tasks`, `PULSE_SWARM_TASKS` | Include individual Swarm tasks in reports. | `true` |
|
||||
| `--include-containers`, `PULSE_INCLUDE_CONTAINERS` | Include per-container metrics (disable when only Swarm data is needed). | `true` |
|
||||
| `--collect-disk`, `PULSE_COLLECT_DISK` | Collect per-container disk usage, block I/O, and mount metadata. Disable to skip Docker size queries on extremely large fleets. | `true` |
|
||||
| `--hostname`, `PULSE_HOSTNAME` | Override host name reported to Pulse. | Docker info / OS hostname |
|
||||
| `--agent-id`, `PULSE_AGENT_ID` | Stable ID for the agent (useful for clustering). | Docker engine ID / machine-id |
|
||||
| `--insecure`, `PULSE_INSECURE_SKIP_VERIFY` | Skip TLS cert validation (unsafe). | `false` |
|
||||
|
||||
The agent automatically discovers the Docker socket via the usual environment variables. To use SSH tunnels or TCP sockets, export `DOCKER_HOST` as you would for the Docker CLI.
|
||||
|
||||
### Disk usage monitoring & alerts
|
||||
|
||||
When `--collect-disk` is enabled (the default), Pulse records each container’s writable layer and root filesystem sizes. The Alerts engine treats the proportion of writable data to total filesystem as the disk usage percentage for that container. A fleet-wide threshold lives under **Alerts → Thresholds → Containers** and defaults to 85% trigger / 80% clear; adjust or disable it per host/container when your workload makes heavy use of copy-on-write layers. Containers that stop reporting disk metrics (for example when size queries are disabled) automatically skip the disk alert evaluation.
|
||||
|
||||
### Suppressing ephemeral containers
|
||||
|
||||
CI runners and short-lived build containers can generate noisy state alerts when they exit on schedule. In Pulse v4.24.0 and later you can provide a list of prefixes to ignore under **Alerts → Thresholds → Containers → Ignored container prefixes**. Any container whose name *or* ID begins with a configured prefix is skipped for state, health, metric, restart-loop, and OOM alerts. Matching is case-insensitive and the list is saved as `dockerIgnoredContainerPrefixes` inside `alerts.json`. Use one entry per family of ephemeral containers (for example, `runner-` or `gitlab-job-`).
|
||||
|
||||
Need the alerts but at a different tone? The same Containers tab exposes global controls for the container state detector. Flip **Disable container state alerts** (`stateDisableConnectivity`) to mute powered-off/offline warnings across the fleet, or change **Default severity** (`statePoweredOffSeverity`) to `critical` so unexpected exits page immediately. Individual host/container overrides still win when you need exceptions.
|
||||
|
||||
## Testing and troubleshooting
|
||||
|
||||
- Run with `--interval 15s --insecure` in a terminal to see log output while testing.
|
||||
- Adjust verbosity with `--log-level` or `LOG_LEVEL` (`info` by default). Drop to `warn` for quiet steady-state logs; use `debug` only when actively debugging.
|
||||
- Ensure the Pulse API token has not expired or been regenerated.
|
||||
- If `pulse-docker-agent` reports `Cannot connect to the Docker daemon`, verify the socket path and permissions.
|
||||
- Check Pulse (`/containers` tab) for the latest heartbeat time. Hosts are marked offline if they stop reporting for >4× the configured interval.
|
||||
- Use the search box above the host grid to filter by host name, stack label, or container name. Restart loops surface in the "Issues" column and display the last five exit codes.
|
||||
|
||||
### Docker hosts cycling or appearing to replace each other
|
||||
|
||||
**Symptom:** The Docker tab switches between hosts instead of showing all of them simultaneously.
|
||||
|
||||
**Cause:** Multiple agents using the same API token or identical agent IDs. Pulse matches incoming reports by agent ID first and falls back to the API token when IDs are missing or identical. Cloning a VM or LXC that already runs `pulse-docker-agent` copies the agent ID, so the two hosts replace each other even with different tokens.
|
||||
|
||||
**Fix:**
|
||||
- If agents share the same API token: Create a dedicated API token for each Docker host in **Settings → API Tokens** and update the agents with their unique tokens. See the [Quick install section](#quick-install-from-your-pulse-server-recommended) for token setup details.
|
||||
- If a cloned VM/LXC has a duplicate agent ID: Regenerate the machine ID on the clone by running `sudo rm /etc/machine-id /var/lib/dbus/machine-id && sudo systemd-machine-id-setup`, then restart `pulse-docker-agent`. Alternatively, set a unique `--agent-id` or `PULSE_AGENT_ID` environment variable when starting the agent.
|
||||
|
||||
### Agent rejected after host removal
|
||||
|
||||
**Symptom:** Agent logs show `docker host was removed at <timestamp> and cannot report again`.
|
||||
|
||||
**Cause:** The host was previously deleted from Pulse and remains on the removal blocklist.
|
||||
|
||||
**Fix:** Open **Settings → Docker → Removed Hosts**, locate the entry, and click **Allow re-enroll**. Then restart the agent or rerun the install script. If the install script runs with a token that includes the `docker:manage` scope it will automatically clear the removal block.
|
||||
|
||||
### Snap-installed Docker permission issues
|
||||
|
||||
**Symptom:** Install script completes successfully but the agent service fails with `permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock`. The install script may have shown `[WARN] docker group not found`.
|
||||
|
||||
**Cause:** Docker installed via Snap does not automatically create a system `docker` group, and snapd refuses to run confined binaries for users whose home directory is outside `/home`. The legacy installer created `/var/lib/pulse-docker-agent`, so the `pulse-docker` user could not talk to the Snap-managed daemon.
|
||||
|
||||
**Fix:** The install script now automatically detects Snap Docker, creates the `docker` group when needed, and relocates the `pulse-docker` home to `/home/pulse-docker-agent` so snapd allows socket access. If you previously installed with an older version of the script:
|
||||
|
||||
1. Create the docker group: `sudo addgroup --system docker`
|
||||
2. Add the service user to the group: `sudo adduser pulse-docker docker`
|
||||
3. Move the service home under `/home`: `sudo usermod -d /home/pulse-docker-agent -m pulse-docker`
|
||||
4. Restart Snap Docker to refresh socket ACLs: `sudo snap restart docker`
|
||||
5. Restart the agent service: `sudo systemctl restart pulse-docker-agent`
|
||||
|
||||
If your distribution intentionally keeps system users outside `/home`, configure snapd to trust the legacy location instead:
|
||||
|
||||
```
|
||||
sudo snap set system homedirs=/var/lib
|
||||
sudo snap get system homedirs # verify
|
||||
```
|
||||
|
||||
The snapcraft documentation at [Home directories outside of `/home`](https://snapcraft.io/docs/home-outside-home) covers additional options (bind mounts, multiple custom roots, etc.).
|
||||
|
||||
Verify socket access with: `sudo -u pulse-docker docker version`
|
||||
|
||||
## Removing the agent
|
||||
|
||||
Stop the systemd service or container and remove the binary. Pulse retains the last reported state until it ages out after a few minutes of inactivity.
|
||||
- **Duplicate Hosts?**
|
||||
If agents flip-flop in the UI, they share a machine-id. Set a unique `--agent-id` flag.
|
||||
|
|
|
|||
|
|
@ -1,246 +1,83 @@
|
|||
# Pulse Host Agent
|
||||
# 🖥️ Host Agent
|
||||
|
||||
The Pulse host agent extends monitoring to standalone servers that do not expose
|
||||
Proxmox or Docker APIs. With it you can surface uptime, OS metadata, CPU load,
|
||||
memory/disk utilisation, temperature sensors, and connection health for any Linux,
|
||||
macOS, or Windows machine alongside the rest of your infrastructure. Starting in
|
||||
v4.26.0 the installer handshakes with Pulse in real time so you can confirm
|
||||
registration from the UI and receive host-agent alerts alongside your existing
|
||||
Docker/Proxmox notifications.
|
||||
Monitor standalone Linux, macOS, and Windows servers that don't run Proxmox or Docker.
|
||||
|
||||
## Temperature Monitoring
|
||||
## 🚀 Quick Start
|
||||
|
||||
The host agent automatically collects temperature data on Linux systems with lm-sensors installed:
|
||||
|
||||
- **CPU Package Temperature**: Overall CPU temperature
|
||||
- **Per-Core Temperatures**: Individual CPU core readings
|
||||
- **NVMe Drive Temperatures**: SSD thermal data
|
||||
- **GPU Temperatures**: AMD and NVIDIA GPU sensors
|
||||
|
||||
Temperature data appears in the **Hosts** tab alongside other host metrics. This is particularly useful for monitoring Proxmox hosts when running Pulse in a VM (where the sensor proxy socket cannot cross VM boundaries).
|
||||
|
||||
**Requirements:**
|
||||
- Linux operating system
|
||||
- lm-sensors package installed (`apt-get install lm-sensors`)
|
||||
- Sensors configured (`sensors-detect --auto`)
|
||||
|
||||
Temperature collection is automatic and best-effort. If lm-sensors is not installed or sensors are unavailable, the agent continues reporting other metrics normally.
|
||||
|
||||
## RAID Monitoring (mdadm)
|
||||
|
||||
The host agent automatically collects mdadm RAID array information on Linux systems:
|
||||
|
||||
- **Array Status**: Displays array state (clean, degraded, recovering, resyncing)
|
||||
- **Device Health**: Shows active, failed, and spare device counts
|
||||
- **Rebuild Progress**: Real-time rebuild or resync percentage and speed
|
||||
- **Automatic Alerting**: Critical alerts for degraded arrays, warnings during rebuilds
|
||||
|
||||
RAID data appears in the **Hosts** tab when you expand a host's details. Each array shows its RAID level, current state, and device status. Color-coded indicators highlight:
|
||||
|
||||
- **Green**: Healthy arrays (clean state, no failed devices)
|
||||
- **Amber**: Rebuilding or resyncing arrays
|
||||
- **Red**: Degraded arrays or arrays with failed devices
|
||||
|
||||
**Requirements:**
|
||||
- Linux operating system
|
||||
- mdadm installed and configured
|
||||
- Root or sudo access for the host agent
|
||||
|
||||
RAID monitoring is automatic and best-effort. If mdadm is not installed or no arrays are present, the agent continues reporting other metrics normally. Only Linux software RAID (mdadm) is supported; hardware RAID controllers are not monitored.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Pulse v4.26.0 or newer (host agent reporting shipped with `/api/agents/host/report`)
|
||||
- An API token with the `host-agent:report` scope (create under **Settings → Security**)
|
||||
- Outbound HTTP/HTTPS connectivity from the host back to Pulse
|
||||
|
||||
> ℹ️ The agent only initiates outbound connections; no inbound firewall rules are required.
|
||||
|
||||
If your Pulse instance does not require API tokens (e.g. during an on-premises
|
||||
lab install) you can still generate commands without embedding a credential.
|
||||
Confirm the warning in **Settings → Agents → Host agents** and the script will
|
||||
prompt for a token instead of hard-coding one.
|
||||
|
||||
## Quick Start
|
||||
|
||||
> Replace `<api-token>` with a Pulse API token limited to the `host-agent:report` scope. Tokens generated from **Settings → Agents → Host agents** already apply this scope.
|
||||
|
||||
### Linux
|
||||
|
||||
The hosted installer handles systemd, rc.local environments, and Unraid automatically.
|
||||
Generate an installation command in the UI:
|
||||
**Settings → Agents → Host Agents → "Install New Agent"**
|
||||
|
||||
### Linux (Universal)
|
||||
```bash
|
||||
curl -fsSL http://pulse.example.local:7655/install-host-agent.sh | \
|
||||
bash -s -- --url http://pulse.example.local:7655 --token <api-token>
|
||||
curl -fsSL http://<pulse-ip>:7655/install-host-agent.sh | \
|
||||
bash -s -- --url http://<pulse-ip>:7655 --token <api-token>
|
||||
```
|
||||
|
||||
- **TrueNAS SCALE**: Use `--platform truenas` (or rely on auto-detect) so the installer writes everything to `/data/pulse-host-agent`, installs a systemd unit from that location, and registers a POSTINIT Init/Shutdown task that re-links and restarts the service on boot.
|
||||
|
||||
```bash
|
||||
curl -fsSL http://pulse.example.local:7655/install-host-agent.sh | \
|
||||
bash -s -- --platform truenas --url http://pulse.example.local:7655 --token <api-token>
|
||||
```
|
||||
|
||||
The TrueNAS flow stores the binary, service unit, and logs under `/data/pulse-host-agent` and creates a POSTINIT entry in **System Settings → Advanced → Init/Shutdown Scripts** pointing at `/data/pulse-host-agent/bootstrap-pulse-host-agent.sh`. Uninstall with `/uninstall-host-agent.sh`; the script removes the init task and the persistent directory. SATA HDD temperatures are not collected yet; CPU/NVMe/GPU sensors continue to report when lm-sensors is available.
|
||||
|
||||
- TrueNAS datasets often default to `exec=off`; the installer now fails fast if `/data` (or the chosen target) is noexec. If you hit a `203/EXEC` error or the installer reports noexec, enable exec on the dataset backing `/data` or choose an exec-capable path before retrying.
|
||||
|
||||
- On systemd machines the script installs the binary, wires up `/etc/systemd/system/pulse-host-agent.service`, enables it, and tails the registration status.
|
||||
- On Unraid hosts it starts the agent under `nohup`, creates `/var/log/pulse`, and (optionally) inserts the auto-start line into `/boot/config/go`.
|
||||
- On minimalist distros without systemd (e.g. Alpine) it creates/updates `/etc/rc.local`, adds the background runner, and verifies it launches.
|
||||
|
||||
Use `--force` to skip interactive prompts or `--interval 1m` to change the polling cadence.
|
||||
*Supports systemd, OpenRC, and Unraid.*
|
||||
|
||||
### macOS
|
||||
|
||||
```bash
|
||||
curl -fsSL http://pulse.example.local:7655/install-host-agent.sh | \
|
||||
bash -s -- --url http://pulse.example.local:7655 --token <api-token>
|
||||
curl -fsSL http://<pulse-ip>:7655/install-host-agent.sh | \
|
||||
bash -s -- --url http://<pulse-ip>:7655 --token <api-token>
|
||||
```
|
||||
*Installs as a LaunchAgent.*
|
||||
|
||||
On macOS the installer stores the token in the Keychain when possible, generates a launchd plist inside `~/Library/LaunchAgents`, and restarts the job so the agent survives logouts and reboots.
|
||||
|
||||
### Windows
|
||||
|
||||
Run the PowerShell bootstrapper as an administrator:
|
||||
|
||||
### Windows (PowerShell)
|
||||
```powershell
|
||||
irm http://pulse.example.local:7655/install-host-agent.ps1 | iex
|
||||
$env:PULSE_URL = "http://<pulse-ip>:7655"
|
||||
$env:PULSE_TOKEN = "<api-token>"
|
||||
irm http://<pulse-ip>:7655/install-host-agent.ps1 | iex
|
||||
```
|
||||
*Installs as a Windows Service.*
|
||||
|
||||
Set `PULSE_URL` and `PULSE_TOKEN` in the environment first for a non-interactive flow:
|
||||
---
|
||||
|
||||
```powershell
|
||||
$env:PULSE_URL = "http://pulse.example.local:7655"
|
||||
$env:PULSE_TOKEN = "<api-token>"
|
||||
irm http://pulse.example.local:7655/install-host-agent.ps1 | iex
|
||||
```
|
||||
## 📊 Features
|
||||
|
||||
If PowerShell is running under emulation or reports an unexpected architecture, force the download target with `-Arch amd64|arm64|386` (or set `PULSE_ARCH` before running the script).
|
||||
- **System Metrics**: CPU, Memory, Disk, Network I/O.
|
||||
- **Temperature**: Auto-detects sensors via `lm-sensors` (Linux).
|
||||
- **RAID Monitoring**: Auto-detects `mdadm` arrays (Linux).
|
||||
- **Smart Alerts**: Integrated with Pulse's alerting engine.
|
||||
|
||||
The script installs the service under `PulseHostAgent`, registers Windows Event Log messages, configures automatic recovery on failure, and waits for Pulse to acknowledge the new host.
|
||||
---
|
||||
|
||||
### Manual installation (advanced)
|
||||
## ⚙️ Configuration
|
||||
|
||||
Prefer to take full control or working in air-gapped environments? You can still download the static binaries and wire them up manually. The commands below mirror what the installer scripts perform for their respective platforms.
|
||||
The agent is a single binary configured via flags.
|
||||
|
||||
#### Linux (systemd)
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--url` | Pulse Server URL | `http://localhost:7655` |
|
||||
| `--token` | API Token (scope: `host-agent:report`) | *(required)* |
|
||||
| `--interval` | Polling Interval | `30s` |
|
||||
| `--hostname` | Override Hostname | *(OS hostname)* |
|
||||
| `--agent-id` | Unique Agent ID | *(machine-id)* |
|
||||
|
||||
<details>
|
||||
<summary><strong>Advanced: Manual Installation</strong></summary>
|
||||
|
||||
Download the binary from [Releases](https://github.com/rcourtman/Pulse/releases) and run it manually:
|
||||
|
||||
```bash
|
||||
# Download the binary for your architecture
|
||||
sudo curl -fsSL https://github.com/rcourtman/Pulse/releases/latest/download/pulse-host-agent-linux-amd64 \
|
||||
-o /usr/local/bin/pulse-host-agent
|
||||
sudo chmod +x /usr/local/bin/pulse-host-agent
|
||||
# Linux / macOS
|
||||
sudo ./pulse-host-agent --url http://pulse:7655 --token <token> --interval 30s
|
||||
|
||||
# Run the agent
|
||||
sudo /usr/local/bin/pulse-host-agent \
|
||||
--url http://pulse.example.local:7655 \
|
||||
--token <api-token> \
|
||||
--interval 30s
|
||||
# Windows
|
||||
.\pulse-host-agent.exe --url http://pulse:7655 --token <token>
|
||||
```
|
||||
|
||||
Available Linux architectures: `pulse-host-agent-linux-amd64`, `pulse-host-agent-linux-arm64`, `pulse-host-agent-linux-armv7`
|
||||
To run as a service, create a systemd unit or use `sc.exe` on Windows.
|
||||
</details>
|
||||
|
||||
For persistence, drop a systemd unit (e.g. `/etc/systemd/system/pulse-host-agent.service`) referencing the same command and enable it with `systemctl enable --now pulse-host-agent`.
|
||||
---
|
||||
|
||||
#### macOS (launchd)
|
||||
## ⚠️ Troubleshooting
|
||||
|
||||
```bash
|
||||
sudo curl -fsSL https://github.com/rcourtman/Pulse/releases/latest/download/pulse-host-agent-darwin-arm64 \
|
||||
-o /usr/local/bin/pulse-host-agent
|
||||
sudo chmod +x /usr/local/bin/pulse-host-agent
|
||||
sudo /usr/local/bin/pulse-host-agent \
|
||||
--url http://pulse.example.local:7655 \
|
||||
--token <api-token> \
|
||||
--interval 30s
|
||||
```
|
||||
- **Duplicate Hosts?**
|
||||
If cloned VMs show up as the same host, they likely share a `/etc/machine-id`.
|
||||
Run `sudo rm /etc/machine-id && sudo systemd-machine-id-setup` to fix.
|
||||
|
||||
Create `~/Library/LaunchAgents/com.pulse.host-agent.plist` to keep the agent running between logins:
|
||||
- **No Temperature Data?**
|
||||
Ensure `lm-sensors` is installed and configured (`sudo sensors-detect`).
|
||||
|
||||
```xml
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key>
|
||||
<string>com.pulse.host-agent</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>/usr/local/bin/pulse-host-agent</string>
|
||||
<string>--url</string>
|
||||
<string>http://pulse.example.local:7655</string>
|
||||
<string>--token</string>
|
||||
<string><api-token></string>
|
||||
<string>--interval</string>
|
||||
<string>30s</string>
|
||||
</array>
|
||||
<key>RunAtLoad</key><true/>
|
||||
<key>KeepAlive</key><true/>
|
||||
<key>StandardOutPath</key><string>/Users/your-user/Library/Logs/pulse-host-agent.log</string>
|
||||
<key>StandardErrorPath</key><string>/Users/your-user/Library/Logs/pulse-host-agent.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
```
|
||||
|
||||
Load it with `launchctl load ~/Library/LaunchAgents/com.pulse.host-agent.plist`.
|
||||
|
||||
#### Windows (manual)
|
||||
|
||||
Compile from source (`GOOS=windows GOARCH=amd64`) or download the latest release, then install the Windows service yourself:
|
||||
|
||||
```powershell
|
||||
New-Service -Name PulseHostAgent `
|
||||
-BinaryPathName '"C:\Program Files\Pulse\pulse-host-agent.exe" --url http://pulse.example.local:7655 --token <api-token> --interval 30s' `
|
||||
-DisplayName "Pulse Host Agent" `
|
||||
-Description "Monitors system metrics and reports to Pulse monitoring server" `
|
||||
-StartupType Automatic
|
||||
Start-Service -Name PulseHostAgent
|
||||
```
|
||||
|
||||
## Command Flags
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--url` | Pulse base URL (defaults to `http://localhost:7655`) |
|
||||
| `--token` | API token with the `host-agent:report` scope |
|
||||
| `--interval` | Polling interval (`30s` default) |
|
||||
| `--log-level` | Log verbosity (`debug`, `info`, `warn`, `error`; defaults to `info`). Also respects `LOG_LEVEL`. Use `warn` on noisy journald setups. |
|
||||
| `--hostname` | Override reported hostname |
|
||||
| `--agent-id` | Override agent identifier (used as dedupe key) |
|
||||
| `--tag` | Optional tag(s) to annotate the host (repeatable) |
|
||||
| `--insecure` | Skip TLS verification (development/testing only) |
|
||||
| `--once` | Send a single report and exit |
|
||||
|
||||
Run `pulse-host-agent --help` for the full list.
|
||||
|
||||
## Avoiding ID collisions
|
||||
|
||||
Pulse keys hosts on the identifier supplied by the agent (machine-id by default). Cloned systems frequently share `/etc/machine-id`, which makes Pulse alternate a single row between multiple machines. Keep the ID unique by regenerating the machine-id on each clone or by providing an explicit `--agent-id`/`PULSE_AGENT_ID` when installing:
|
||||
|
||||
```bash
|
||||
sudo rm /etc/machine-id /var/lib/dbus/machine-id
|
||||
sudo systemd-machine-id-setup
|
||||
sudo systemctl restart pulse-host-agent
|
||||
```
|
||||
|
||||
Re-running the installer with `--agent-id <unique-name>` achieves the same result if regenerating the machine-id is undesirable.
|
||||
|
||||
## Viewing Hosts
|
||||
|
||||
- **Settings → Agents → Host agents** lists every reporting host and provides ready-made install commands.
|
||||
- The **Hosts** tab surfaces host telemetry alongside Proxmox/Docker data in the main dashboard.
|
||||
|
||||
### Checking installation status
|
||||
|
||||
- Click **Check status** under **Settings → Agents → Host agents** and enter the host ID or hostname you just installed.
|
||||
- Pulse hits `/api/agents/host/lookup`, highlights the matching row for 10 seconds, and refreshes the connection badge, last-seen timestamp, and agent version in-line.
|
||||
- If the host has not checked in yet, the UI returns a friendly "Host has not registered" message so you can retry without re-running the script.
|
||||
|
||||
### Alerts and notifications
|
||||
|
||||
- Host agents now participate in the main alert engine. Offline detection, metric thresholds, and override scopes (global or per-host) live in **Settings → Alerts → Thresholds** beside your Docker and Proxmox rules.
|
||||
- Alert notifications, webhooks, and quiet-hours behaviour reuse the existing pipelines—no extra setup is required once you enable host-agent monitoring.
|
||||
|
||||
## Updating
|
||||
|
||||
Since the agent is a single static binary, updates are as simple as replacing the file and restarting your launchd/systemd unit. The Settings pane always links to the latest release artefacts.
|
||||
- **Check Status**
|
||||
Go to **Settings → Agents → Host Agents** to verify connection status.
|
||||
|
|
|
|||
|
|
@ -1,663 +1,101 @@
|
|||
# Pulse Troubleshooting Guide
|
||||
# 🔧 Troubleshooting Guide
|
||||
|
||||
## Common Issues and Solutions
|
||||
|
||||
### Correlate API Calls with Logs
|
||||
|
||||
Every API response includes an `X-Request-ID` header. When escalating issues, capture that value and use it to search the backend logs or log file. The same identifier is emitted as `request_id` in structured logs.
|
||||
## ⚡ Quick Fixes
|
||||
|
||||
### I forgot my password
|
||||
**Docker**:
|
||||
```bash
|
||||
# Capture a request ID
|
||||
curl -i https://pulse.example.com/api/state | grep X-Request-ID
|
||||
|
||||
# Search the rotating log file
|
||||
grep 'request_id=abc123' /var/log/pulse/pulse.log
|
||||
|
||||
# Docker / kubectl example
|
||||
docker logs pulse | grep 'request_id=abc123'
|
||||
```
|
||||
|
||||
Include the `X-Request-ID` in support tickets or incident notes so responders can jump straight to the relevant log lines.
|
||||
|
||||
### Authentication Problems
|
||||
|
||||
#### Forgot Password / Lost Access
|
||||
|
||||
**Solution: Use the built-in recovery endpoint**
|
||||
|
||||
Pulse ships with a guarded recovery API that lets you regain access without wiping configuration.
|
||||
|
||||
1. **From the Pulse host (localhost only)**
|
||||
Generate a short-lived recovery token or temporarily disable auth:
|
||||
```bash
|
||||
# Create a 30 minute recovery token (returns JSON with the token value)
|
||||
curl -s -X POST http://localhost:7655/api/security/recovery \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"action":"generate_token","duration":30}'
|
||||
|
||||
# OR force local-only recovery access (writes .auth_recovery in the data dir)
|
||||
curl -s -X POST http://localhost:7655/api/security/recovery \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"action":"disable_auth"}'
|
||||
```
|
||||
|
||||
2. **If you generated a token**, use it from a trusted workstation:
|
||||
```bash
|
||||
curl -s -X POST https://pulse.example.com/api/security/recovery \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'X-Recovery-Token: YOUR_TOKEN' \
|
||||
-d '{"action":"disable_auth"}'
|
||||
```
|
||||
The token is single-use and expires automatically.
|
||||
|
||||
3. **Log in and reset credentials** using Settings → Security, then re-enable auth:
|
||||
```bash
|
||||
curl -s -X POST http://localhost:7655/api/security/recovery \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"action":"enable_auth"}'
|
||||
```
|
||||
Alternatively, delete `/etc/pulse/.auth_recovery` (or `/data/.auth_recovery` for Docker) and restart Pulse.
|
||||
|
||||
Only fall back to nuking `/etc/pulse` if the recovery endpoint is unreachable.
|
||||
|
||||
**Prevention:**
|
||||
- Use a password manager
|
||||
- Store exported configuration backups securely
|
||||
- Generate API tokens for automation instead of sharing passwords
|
||||
|
||||
#### Cannot login after setting up security
|
||||
**Symptoms**: "Invalid username or password" error despite correct credentials
|
||||
|
||||
**Common causes and solutions:**
|
||||
|
||||
1. **Truncated bcrypt hash** (most common)
|
||||
- Check hash is exactly 60 characters: `echo -n "$PULSE_AUTH_PASS" | wc -c`
|
||||
- Look for error in logs: `Bcrypt hash appears truncated!`
|
||||
- Solution: Use full 60-character hash or Quick Security Setup
|
||||
|
||||
2. **Docker Compose $ character issue**
|
||||
- Docker Compose interprets `$` as variable expansion
|
||||
- **Wrong**: `PULSE_AUTH_PASS='$2a$12$hash...'`
|
||||
- **Right**: `PULSE_AUTH_PASS='$$2a$$12$$hash...'` (escape with $$)
|
||||
- Alternative: Use a .env file where no escaping is needed
|
||||
|
||||
3. **Environment variable not loaded**
|
||||
- Check if variable is set: `docker exec pulse env | grep PULSE_AUTH`
|
||||
- Verify quotes around hash: Must use single quotes
|
||||
- Restart container after changes
|
||||
|
||||
#### Password change fails
|
||||
**Error**: `exec: "sudo": executable file not found`
|
||||
|
||||
**Solution**: Update to v4.3.8+ which removes sudo requirement. For older versions:
|
||||
```bash
|
||||
# Manually update .env file
|
||||
docker exec pulse sh -c "echo \"PULSE_AUTH_PASS='new-hash'\" >> /data/.env"
|
||||
docker exec pulse rm /data/.env
|
||||
docker restart pulse
|
||||
# Access UI to run setup wizard again
|
||||
```
|
||||
|
||||
#### Can't access Pulse - stuck at login
|
||||
**Symptoms**: Can't access Pulse after upgrade, no credentials work
|
||||
|
||||
**Solution**:
|
||||
- If upgrading from pre-v4.5.0, you need to complete security setup first
|
||||
- Clear browser cache and cookies
|
||||
- Access http://your-ip:7655 to see setup wizard
|
||||
- Complete setup, then restart container
|
||||
|
||||
### Docker-Specific Issues
|
||||
|
||||
#### No .env file in /data
|
||||
**This is expected behavior** when using environment variables. The .env file is only created by:
|
||||
- Quick Security Setup wizard
|
||||
- Password change through UI
|
||||
- Manual creation
|
||||
|
||||
If you provide auth via `-e` flags or docker-compose environment section, no .env is created.
|
||||
|
||||
#### Container won't start
|
||||
Check logs: `docker logs pulse`
|
||||
|
||||
Common issues:
|
||||
- Port already in use: Change port mapping
|
||||
- Volume permissions: Ensure volume is writable
|
||||
- Invalid environment variables: Check syntax
|
||||
**Systemd**:
|
||||
Delete `/etc/pulse/.env` and restart the service.
|
||||
|
||||
### Port change didn't take effect
|
||||
1. **Confirm which service name is active**
|
||||
```bash
|
||||
systemctl status pulse 2>/dev/null \\
|
||||
|| systemctl status pulse-backend 2>/dev/null \\
|
||||
|| systemctl status pulse-hot-dev
|
||||
```
|
||||
- Docker: the container port mapping controls the public port (`-p host:7655`).
|
||||
- Kubernetes (Helm chart): service is `svc/pulse`; update `service.port` in your values file and run `helm upgrade`.
|
||||
1. Check which service is running: `systemctl status pulse` (or `pulse-backend`).
|
||||
2. Verify environment override: `systemctl show pulse --property=Environment`.
|
||||
3. Docker: Ensure you updated the `-p` flag (e.g., `-p 8080:7655`).
|
||||
|
||||
2. **Verify configuration/environment overrides**
|
||||
```bash
|
||||
sudo systemctl show pulse --property=Environment
|
||||
```
|
||||
Helm users can run `kubectl get svc pulse -n <namespace> -o yaml` to confirm the current port.
|
||||
### "Connection Refused"
|
||||
- Check if Pulse is running.
|
||||
- Verify the port is open on your firewall.
|
||||
- **PBS**: Remember PBS uses port **8007** and requires **HTTPS**.
|
||||
|
||||
3. **Check for port conflicts**
|
||||
```bash
|
||||
sudo lsof -i :8080
|
||||
```
|
||||
---
|
||||
|
||||
4. **Post-change validation**
|
||||
- Restart the service (`systemctl restart`, `docker restart`, or `helm upgrade`).
|
||||
- v4.24.0 logs these restarts/upgrades in **Settings → System → Updates** and `/api/updates/history`; capture the `event_id` for your change notes.
|
||||
## 🔍 Common Issues
|
||||
|
||||
### Installation Issues
|
||||
### Authentication
|
||||
|
||||
#### Binary not found (v4.3.7)
|
||||
**Error**: `/opt/pulse/pulse: No such file or directory`
|
||||
**"Invalid username or password" after setup**
|
||||
- **Docker Compose**: Did you escape the `$` signs in your hash? Use `$$2a$$...`.
|
||||
- **Truncated Hash**: Ensure your bcrypt hash is exactly 60 characters.
|
||||
|
||||
**Cause**: v4.3.7 install script bug
|
||||
**Cannot login / 401 Unauthorized**
|
||||
- Clear browser cookies.
|
||||
- Check if your IP is banned (wait 15 mins or restart Pulse).
|
||||
|
||||
**Solution**: Update to v4.3.8 or manually fix:
|
||||
### Monitoring Data
|
||||
|
||||
**VMs show "-" for disk usage**
|
||||
- Install **QEMU Guest Agent** in the VM.
|
||||
- Enable "QEMU Guest Agent" in Proxmox VM Options.
|
||||
- Restart the VM.
|
||||
- See [VM Disk Monitoring](VM_DISK_MONITORING.md).
|
||||
|
||||
**Temperature data missing**
|
||||
- Install `lm-sensors` on the host.
|
||||
- Run `sensors-detect`.
|
||||
- Re-run the Pulse setup script to install the sensor proxy.
|
||||
- See [Temperature Monitoring](TEMPERATURE_MONITORING.md).
|
||||
|
||||
**Docker hosts appearing/disappearing**
|
||||
- **Duplicate IDs**: Cloned VMs often share `/etc/machine-id`.
|
||||
- **Fix**: Run `rm /etc/machine-id && systemd-machine-id-setup` on the clone.
|
||||
|
||||
### Notifications
|
||||
|
||||
**Emails not sending**
|
||||
- Check SMTP settings in **Settings → Alerts**.
|
||||
- Check logs: `docker logs pulse | grep email`.
|
||||
- Ensure your SMTP provider allows the connection (e.g., Gmail App Passwords).
|
||||
|
||||
**Webhooks failing**
|
||||
- Verify the URL is reachable from the Pulse server.
|
||||
- Check `Allowed Origins` if you are getting CORS errors.
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Advanced Diagnostics
|
||||
|
||||
### Correlate Logs with Requests
|
||||
Every API response has an `X-Request-ID` header. Use it to find the exact log entry:
|
||||
```bash
|
||||
sudo mkdir -p /opt/pulse/bin
|
||||
sudo mv /opt/pulse/pulse /opt/pulse/bin/pulse
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart pulse
|
||||
grep "request_id=abc123" /var/log/pulse/pulse.log
|
||||
```
|
||||
|
||||
#### Service name confusion
|
||||
Pulse uses different service names depending on installation method:
|
||||
- **Default systemd install**: `pulse`
|
||||
- **Legacy installs (pre-v4.7)**: `pulse-backend`
|
||||
- **Hot dev environment**: `pulse-hot-dev`
|
||||
- **Docker**: N/A (container name)
|
||||
|
||||
To check which you have:
|
||||
### Check Permissions (Proxmox)
|
||||
If Pulse can't see VMs or storage, check the user permissions on Proxmox:
|
||||
```bash
|
||||
systemctl status pulse 2>/dev/null \
|
||||
|| systemctl status pulse-backend 2>/dev/null \
|
||||
|| systemctl status pulse-hot-dev
|
||||
```
|
||||
|
||||
### Automatic Update Failures
|
||||
|
||||
- **Timer never runs** – Ensure both the UI toggle (**Settings → System → Updates → Automatic Updates**) and `pulse-update.timer` are enabled. The timer exits immediately if `autoUpdateEnabled:false` in `/var/lib/pulse/system.json`. Run:
|
||||
```bash
|
||||
systemctl status pulse-update.timer --no-pager
|
||||
jq '.autoUpdateEnabled' /var/lib/pulse/system.json
|
||||
```
|
||||
- **Update stuck or failed** – Tail the journal and inspect the most recent history entry:
|
||||
```bash
|
||||
journalctl -u pulse-update -n 50
|
||||
curl -s http://localhost:7655/api/updates/history?limit=1 | jq '.entries[0]'
|
||||
```
|
||||
Grab the `log_path` from the entry (defaults to `/var/log/pulse/update-*.log`) for full stack traces.
|
||||
- **Need to retry or roll back** – Start `pulse-update.service` manually after fixing the root cause, or redeploy the previous release/restore the recorded `backup_path`.
|
||||
|
||||
See the full runbook in [operations/auto-update.md](operations/auto-update.md) for manual trigger steps, observability checklist, and rollback procedures.
|
||||
|
||||
### Notification Issues
|
||||
|
||||
#### Emails not sending
|
||||
1. Check email configuration in Settings → Alerts
|
||||
2. Verify SMTP settings and credentials
|
||||
3. Check logs for errors: `docker logs pulse | grep -i email`
|
||||
4. Test with a simple webhook first
|
||||
|
||||
#### Webhook not working
|
||||
- Verify URL is accessible from Pulse server
|
||||
- Check for SSL certificate issues
|
||||
- Try a test service like webhook.site
|
||||
- Check logs for response codes (temporarily set `LOG_LEVEL=debug` via **Settings → System → Logging** or export `LOG_LEVEL=debug` and restart; review `webhook.delivery` entries, then revert to `info`)
|
||||
|
||||
### HTTPS/TLS Configuration Issues
|
||||
|
||||
#### Pulse fails to start after enabling HTTPS
|
||||
**Symptoms**: Service exits with status code 1, continuous restart attempts, or permission denied errors in logs
|
||||
|
||||
**Common causes and solutions:**
|
||||
|
||||
1. **Certificate files not readable by pulse user** (most common)
|
||||
- Pulse runs as the `pulse` user and needs read access to certificate files
|
||||
- Check file ownership: `ls -l /etc/pulse/*.pem`
|
||||
- Solution:
|
||||
```bash
|
||||
sudo chown pulse:pulse /etc/pulse/cert.pem /etc/pulse/key.pem
|
||||
sudo chmod 644 /etc/pulse/cert.pem # Certificate
|
||||
sudo chmod 600 /etc/pulse/key.pem # Private key
|
||||
```
|
||||
|
||||
2. **Invalid certificate or key file**
|
||||
- Verify certificate format: `openssl x509 -in /etc/pulse/cert.pem -text -noout`
|
||||
- Verify private key: `openssl rsa -in /etc/pulse/key.pem -check -noout`
|
||||
- Ensure certificate and key match:
|
||||
```bash
|
||||
openssl x509 -noout -modulus -in /etc/pulse/cert.pem | openssl md5
|
||||
openssl rsa -noout -modulus -in /etc/pulse/key.pem | openssl md5
|
||||
# Both should output the same hash
|
||||
```
|
||||
|
||||
3. **File paths incorrect**
|
||||
- Verify paths in environment variables match actual file locations
|
||||
- Check for typos in `TLS_CERT_FILE` and `TLS_KEY_FILE`
|
||||
- Use absolute paths, not relative paths
|
||||
|
||||
4. **Check startup logs**
|
||||
```bash
|
||||
# View recent service logs
|
||||
journalctl -u pulse -n 50
|
||||
|
||||
# Follow logs in real-time during restart
|
||||
journalctl -u pulse -f
|
||||
```
|
||||
|
||||
**Complete HTTPS setup example:**
|
||||
```bash
|
||||
# 1. Place certificate files
|
||||
sudo cp mycert.pem /etc/pulse/cert.pem
|
||||
sudo cp mykey.pem /etc/pulse/key.pem
|
||||
|
||||
# 2. Set ownership and permissions
|
||||
sudo chown pulse:pulse /etc/pulse/cert.pem /etc/pulse/key.pem
|
||||
sudo chmod 644 /etc/pulse/cert.pem
|
||||
sudo chmod 600 /etc/pulse/key.pem
|
||||
|
||||
# 3. Configure systemd service
|
||||
sudo systemctl edit pulse
|
||||
# Add:
|
||||
# [Service]
|
||||
# Environment="HTTPS_ENABLED=true"
|
||||
# Environment="TLS_CERT_FILE=/etc/pulse/cert.pem"
|
||||
# Environment="TLS_KEY_FILE=/etc/pulse/key.pem"
|
||||
|
||||
# 4. Reload and restart
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart pulse
|
||||
|
||||
# 5. Verify service started successfully
|
||||
sudo systemctl status pulse
|
||||
```
|
||||
|
||||
See [Configuration Guide](CONFIGURATION.md#tlshttps-configuration) for complete HTTPS setup documentation.
|
||||
|
||||
### Temperature Monitoring Issues
|
||||
|
||||
#### Sensor proxy fails to start (config validation error)
|
||||
|
||||
**Symptoms:** Service won't start, logs show "Config validation failed" or "duplicate allowed_nodes blocks detected"
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Check service status
|
||||
sudo systemctl status pulse-sensor-proxy
|
||||
|
||||
# Validate config manually
|
||||
pulse-sensor-proxy config validate
|
||||
|
||||
# Look for duplicate blocks
|
||||
grep -n "allowed_nodes:" /etc/pulse-sensor-proxy/config.yaml
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
The issue is config corruption from earlier versions. Version 4.31.1+ fixes this automatically:
|
||||
|
||||
```bash
|
||||
# Reinstall to migrate to new config system
|
||||
curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh | \
|
||||
sudo bash -s -- --standalone --pulse-server http://your-pulse:7655
|
||||
|
||||
# Verify the fix
|
||||
pulse-sensor-proxy config validate
|
||||
sudo systemctl status pulse-sensor-proxy
|
||||
```
|
||||
|
||||
The new config system:
|
||||
- Separates allowed nodes into `/etc/pulse-sensor-proxy/allowed_nodes.yaml`
|
||||
- Uses atomic writes with file locking
|
||||
- Validates config before service startup
|
||||
- Includes CLI for safe config management
|
||||
|
||||
**Manual config management (advanced):**
|
||||
```bash
|
||||
# Add nodes to allowed list
|
||||
pulse-sensor-proxy config set-allowed-nodes --merge 192.168.0.1 --merge node1.local
|
||||
|
||||
# Replace entire list
|
||||
pulse-sensor-proxy config set-allowed-nodes --replace --merge 192.168.0.1
|
||||
|
||||
# Validate before restarting
|
||||
pulse-sensor-proxy config validate
|
||||
sudo systemctl restart pulse-sensor-proxy
|
||||
```
|
||||
|
||||
See `/opt/pulse/cmd/pulse-sensor-proxy/README.md` for complete CLI documentation.
|
||||
|
||||
#### Temperature data flickers after adding nodes
|
||||
|
||||
**Symptoms:** Dashboard temperatures alternate between values and `--`, or new nodes never show readings. Proxy logs contain `limiter.rejection` messages.
|
||||
|
||||
**Diagnosis:**
|
||||
1. Confirm you are running a build with commit 46b8b8d or later (defaults are 1 rps, burst 5). Older binaries throttle multi-node clusters aggressively.
|
||||
2. Check limiter metrics:
|
||||
```bash
|
||||
curl -s http://127.0.0.1:9127/metrics \
|
||||
| grep -E 'pulse_proxy_limiter_(rejects|penalties)_total'
|
||||
```
|
||||
Any recent increment indicates rate-limit saturation.
|
||||
3. Inspect scheduler health for temperature pollers (`breaker.state` should be `closed` and `deadLetter.present` must be `false`).
|
||||
|
||||
**Fix:** Increase the proxy burst/interval in `/etc/pulse-sensor-proxy/config.yaml`:
|
||||
```yaml
|
||||
rate_limit:
|
||||
per_peer_interval_ms: 500 # medium cluster (≈10 nodes)
|
||||
per_peer_burst: 10
|
||||
```
|
||||
Restart `pulse-sensor-proxy`, verify limiter counters stop increasing, and confirm the dashboard stabilises. Document the change in your operations log.
|
||||
|
||||
### VM Disk Monitoring Issues
|
||||
|
||||
#### VMs show "-" for disk usage
|
||||
|
||||
**This is normal and expected** - VMs require QEMU Guest Agent to report disk usage.
|
||||
|
||||
**Quick fix:**
|
||||
1. Install guest agent in VM: `apt install qemu-guest-agent` (Linux) or virtio-win tools (Windows)
|
||||
2. Enable in Proxmox: VM → Options → QEMU Guest Agent → Enable
|
||||
3. Restart the VM
|
||||
4. Wait 10 seconds for Pulse to poll again
|
||||
|
||||
**Detailed troubleshooting:**
|
||||
|
||||
See [VM Disk Monitoring Guide](VM_DISK_MONITORING.md) for full setup instructions.
|
||||
|
||||
#### How to diagnose VM disk issues
|
||||
|
||||
**Step 1: Check if guest agent is running**
|
||||
|
||||
On Proxmox host:
|
||||
```bash
|
||||
# Check if agent is enabled in VM config
|
||||
qm config <VMID> | grep agent
|
||||
|
||||
# Test if agent responds
|
||||
qm agent <VMID> ping
|
||||
|
||||
# Get filesystem info (what Pulse uses)
|
||||
qm agent <VMID> get-fsinfo
|
||||
```
|
||||
|
||||
Inside the VM:
|
||||
```bash
|
||||
# Linux
|
||||
systemctl status qemu-guest-agent
|
||||
|
||||
# Windows (PowerShell)
|
||||
Get-Service QEMU-GA
|
||||
```
|
||||
|
||||
**Step 2: Run diagnostic script**
|
||||
|
||||
```bash
|
||||
# On Proxmox host
|
||||
curl -sSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/test-vm-disk.sh | bash
|
||||
```
|
||||
|
||||
Or if Pulse is installed:
|
||||
```bash
|
||||
/opt/pulse/scripts/test-vm-disk.sh
|
||||
```
|
||||
|
||||
### Ceph Cluster Data Missing
|
||||
|
||||
**Symptoms**: Ceph pools or health section missing in Storage view even though the cluster uses Ceph.
|
||||
|
||||
**Checklist:**
|
||||
1. Confirm the Proxmox node exposes Ceph-backed storage (`Datacenter → Storage`). Types must be `rbd`, `cephfs`, or `ceph`.
|
||||
2. Ensure Pulse has permission to call `/cluster/ceph/status` (Pulse’s Proxmox account needs `Sys.Audit` as part of `PVEAuditor`, provided by the setup script).
|
||||
3. Check the backend logs for `Ceph status unavailable – preserving previous Ceph state`. Intermittent errors are usually network timeouts; steady errors point to permissions.
|
||||
4. Run from the Pulse host:
|
||||
```bash
|
||||
curl -sk https://pve-node:8006/api2/json/cluster/ceph/status \
|
||||
-H "Authorization: PVEAPIToken=pulse-monitor@pam!token=<value>"
|
||||
```
|
||||
If this fails, verify firewall / token scope.
|
||||
|
||||
**Tip**: Pulse polls Ceph after storage refresh. If you recently added Ceph storage, wait one poll cycle or restart the backend to force detection.
|
||||
|
||||
### Backup View Filters Not Working
|
||||
|
||||
**Symptoms**: Backup chart does not highlight the selected time range or the grid ignores the picker.
|
||||
|
||||
**Checklist:**
|
||||
1. Make sure you are running Pulse v4.29.0 or newer (the interactive picker was introduced alongside the new timeline). Check **Settings → System → About**.
|
||||
2. Verify your browser is not forcing Legacy mode – if the top-right toggle shows “Lightweight UI”, switch back to default.
|
||||
3. When filters appear stuck:
|
||||
- Click **Reset Filters** in the toolbar.
|
||||
- Clear any search chips under the chart.
|
||||
- Pick a preset (24h / 7d / 30d) to re-seed the view, then move back to Custom.
|
||||
4. If the grid still shows stale data, open DevTools console and ensure no errors mentioning `chartsSelection` appear. Any error here usually means a stale service worker; hard refresh (Ctrl+Shift+R) clears it.
|
||||
|
||||
**Tip**: Selecting bars in the chart cross-highlights matching rows. If that does not happen, confirm you do not have browser extensions that block pointer events on canvas elements.
|
||||
|
||||
### Container Agent Shows Hosts Offline
|
||||
|
||||
**Symptoms**: `/containers` tab marks hosts as offline or missing container metrics.
|
||||
|
||||
**Checklist:**
|
||||
1. Run the agent manually with verbose logs:
|
||||
```bash
|
||||
sudo /usr/local/bin/pulse-docker-agent --interval 15s --debug
|
||||
```
|
||||
Look for HTTP 401 (token mismatch) or socket errors.
|
||||
2. Confirm the host sees Docker:
|
||||
```bash
|
||||
sudo docker info | head -n 20
|
||||
```
|
||||
3. Make sure the agent ID is stable. If running inside transient containers, set `--agent-id` explicitly so Pulse does not treat each restart as a new host.
|
||||
4. Verify Pulse shows a recent heartbeat (`lastSeen`) in `/api/state` → `dockerHosts`. Hosts are marked offline after 4× the configured interval with no update.
|
||||
5. For reverse proxies/TLS issues, append `--insecure` temporarily to confirm whether certificate validation is the culprit.
|
||||
|
||||
**Restart loops**: The Containers workspace Issues column lists the last exit codes. Investigate recurring non-zero codes in `docker logs <container>` and adjust restart policy if needed.
|
||||
|
||||
**Step 3: Check Pulse logs**
|
||||
|
||||
```bash
|
||||
# Docker
|
||||
docker logs pulse | grep -i "guest agent\|fsinfo"
|
||||
|
||||
# Systemd
|
||||
journalctl -u pulse -f | grep -i "guest agent\|fsinfo"
|
||||
```
|
||||
|
||||
Look for specific error reasons:
|
||||
- `agent-not-running` - Agent service not started in VM
|
||||
- `agent-disabled` - Not enabled in VM config
|
||||
- `agent-timeout` - Agent not responding (may need restart)
|
||||
- `permission-denied` - Check permissions (see below)
|
||||
- `no-filesystems` - Agent returned no usable filesystem data
|
||||
|
||||
#### Permission denied errors
|
||||
|
||||
If Pulse logs show permission denied when querying guest agent:
|
||||
|
||||
**Check permissions:**
|
||||
```bash
|
||||
# On Proxmox host
|
||||
pveum user permissions pulse-monitor@pam
|
||||
```
|
||||
Required: `PVEAuditor` role.
|
||||
Recommended: `VM.Audit`, `Sys.Audit` (added by setup script).
|
||||
|
||||
**Required permissions:**
|
||||
- **Proxmox 9:** `VM.GuestAgent.Audit` privilege (Pulse setup adds this via the `PulseMonitor` role)
|
||||
- **Proxmox 8:** `VM.Monitor` privilege (Pulse setup adds this via the `PulseMonitor` role)
|
||||
- **All versions:** `Sys.Audit` is recommended for Ceph metrics and applied when available
|
||||
|
||||
**Fix permissions:**
|
||||
|
||||
Re-run the Pulse setup script on the Proxmox node:
|
||||
### Recovery Mode
|
||||
If you are completely locked out, you can trigger a recovery token from the localhost CLI:
|
||||
```bash
|
||||
curl -sSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/setup-pve.sh | bash
|
||||
curl -X POST http://localhost:7655/api/security/recovery \
|
||||
-d '{"action":"generate_token","duration":30}'
|
||||
```
|
||||
Use the returned token to bypass auth temporarily.
|
||||
|
||||
Or manually:
|
||||
```bash
|
||||
# Shared read-only access
|
||||
pveum aclmod / -user pulse-monitor@pam -role PVEAuditor
|
||||
---
|
||||
|
||||
# Extra privileges for guest metrics and Ceph
|
||||
EXTRA_PRIVS=()
|
||||
## 🆘 Getting Help
|
||||
|
||||
# Sys.Audit (Ceph, cluster status)
|
||||
if pveum role list 2>/dev/null | grep -q "Sys.Audit"; then
|
||||
EXTRA_PRIVS+=(Sys.Audit)
|
||||
else
|
||||
if pveum role add PulseTmpSysAudit -privs Sys.Audit 2>/dev/null; then
|
||||
EXTRA_PRIVS+=(Sys.Audit)
|
||||
pveum role delete PulseTmpSysAudit 2>/dev/null
|
||||
fi
|
||||
fi
|
||||
|
||||
# VM guest agent / monitor privileges
|
||||
VM_PRIV=""
|
||||
if pveum role list 2>/dev/null | grep -q "VM.Monitor"; then
|
||||
VM_PRIV="VM.Monitor"
|
||||
elif pveum role list 2>/dev/null | grep -q "VM.GuestAgent.Audit"; then
|
||||
VM_PRIV="VM.GuestAgent.Audit"
|
||||
else
|
||||
if pveum role add PulseTmpVMMonitor -privs VM.Monitor 2>/dev/null; then
|
||||
VM_PRIV="VM.Monitor"
|
||||
pveum role delete PulseTmpVMMonitor 2>/dev/null
|
||||
elif pveum role add PulseTmpGuestAudit -privs VM.GuestAgent.Audit 2>/dev/null; then
|
||||
VM_PRIV="VM.GuestAgent.Audit"
|
||||
pveum role delete PulseTmpGuestAudit 2>/dev/null
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -n "$VM_PRIV" ]; then
|
||||
EXTRA_PRIVS+=("$VM_PRIV")
|
||||
fi
|
||||
|
||||
if [ ${#EXTRA_PRIVS[@]} -gt 0 ]; then
|
||||
PRIV_STRING="${EXTRA_PRIVS[*]}"
|
||||
pveum role delete PulseMonitor 2>/dev/null
|
||||
pveum role add PulseMonitor -privs "$PRIV_STRING"
|
||||
pveum aclmod / -user pulse-monitor@pam -role PulseMonitor
|
||||
fi
|
||||
```
|
||||
|
||||
**Important:** Both API tokens and passwords work fine for guest agent access. If you see permission errors, it's a permission configuration issue, not an authentication method limitation.
|
||||
|
||||
#### Guest agent installed but no disk data
|
||||
|
||||
If agent responds to ping but returns no filesystem info:
|
||||
|
||||
1. **Check agent version** - Update to latest:
|
||||
```bash
|
||||
# Linux
|
||||
apt update && apt install --only-upgrade qemu-guest-agent
|
||||
systemctl restart qemu-guest-agent
|
||||
```
|
||||
|
||||
2. **Check filesystem permissions** - Agent needs read access to filesystem data
|
||||
|
||||
3. **Windows VMs** - Ensure VirtIO drivers are up to date from latest virtio-win ISO
|
||||
|
||||
4. **Special filesystems only** - If VM only has special filesystems (tmpfs, ISO mounts), this is normal for Live systems
|
||||
|
||||
#### Specific VM types
|
||||
|
||||
**Cloud images:**
|
||||
- Most have guest agent pre-installed but disabled
|
||||
- Enable with: `systemctl enable --now qemu-guest-agent`
|
||||
|
||||
**Windows VMs:**
|
||||
- Must install VirtIO guest tools
|
||||
- Ensure "QEMU Guest Agent" service is running
|
||||
- May need "QEMU Guest Agent VSS Provider" for full functionality
|
||||
|
||||
**Container-based VMs (Docker/Kubernetes hosts):**
|
||||
- Will show high disk usage due to container layers
|
||||
- This is accurate - containers consume real disk space
|
||||
- Consider monitoring container disk separately
|
||||
|
||||
### Performance Issues
|
||||
|
||||
#### High CPU usage
|
||||
- Polling interval is fixed at 10 seconds (matches Proxmox update cycle)
|
||||
- Check number of monitored nodes
|
||||
- Disable unused features (snapshots, backups monitoring)
|
||||
|
||||
#### High memory usage
|
||||
- Normal for monitoring many nodes
|
||||
- Check metrics retention settings
|
||||
- Restart container to clear any memory leaks
|
||||
|
||||
### Network Issues
|
||||
|
||||
#### Cannot connect to Proxmox nodes
|
||||
1. Verify Proxmox API is accessible:
|
||||
```bash
|
||||
curl -k https://proxmox-ip:8006
|
||||
```
|
||||
2. Check credentials have proper permissions (PVEAuditor minimum)
|
||||
3. Verify network connectivity between Pulse and Proxmox
|
||||
4. Check for firewall rules blocking port 8006
|
||||
|
||||
#### PBS connection issues
|
||||
- Ensure API token has Datastore.Audit permission
|
||||
- Check PBS is accessible on port 8007
|
||||
- Verify token format: `user@realm!tokenid=secret`
|
||||
|
||||
### Update Issues
|
||||
|
||||
#### Updates not showing
|
||||
- Check update channel in Settings → System
|
||||
- Verify internet connectivity
|
||||
- Check GitHub API rate limits
|
||||
- Manual update: Pull latest Docker image or run install script
|
||||
|
||||
#### Update fails to apply
|
||||
**Docker**: Pull new image and recreate container
|
||||
**Native**: Run install script again or check logs
|
||||
|
||||
### Data Recovery
|
||||
|
||||
#### Lost authentication
|
||||
See [Forgot Password / Lost Access](#forgot-password--lost-access) section above.
|
||||
|
||||
**Recommended approach**: Start fresh. Delete your Pulse data and restart.
|
||||
|
||||
#### Corrupt configuration
|
||||
Restore from backup or delete config files to start fresh:
|
||||
```bash
|
||||
# Docker
|
||||
docker exec pulse rm /data/*.json /data/*.enc
|
||||
docker restart pulse
|
||||
|
||||
# Native
|
||||
sudo rm /etc/pulse/*.json /etc/pulse/*.enc
|
||||
sudo systemctl restart pulse
|
||||
```
|
||||
|
||||
## Getting Help
|
||||
|
||||
### Collect diagnostic information
|
||||
```bash
|
||||
# Version
|
||||
curl http://localhost:7655/api/version
|
||||
|
||||
# Logs (last 100 lines)
|
||||
docker logs --tail 100 pulse # Docker
|
||||
journalctl -u pulse -n 100 # Native
|
||||
|
||||
# Environment
|
||||
docker exec pulse env | grep -E "PULSE|API" # Docker
|
||||
systemctl show pulse --property=Environment # Native
|
||||
```
|
||||
|
||||
### Report issues
|
||||
When reporting issues, include:
|
||||
1. Pulse version
|
||||
2. Deployment type (Docker/LXC/Manual)
|
||||
3. Error messages from logs
|
||||
4. Steps to reproduce
|
||||
5. Expected vs actual behavior
|
||||
|
||||
Report at: https://github.com/rcourtman/Pulse/issues
|
||||
If you're still stuck:
|
||||
1. **Check Logs**: `journalctl -u pulse -n 100` or `docker logs --tail 100 pulse`.
|
||||
2. **Check Version**: `curl http://localhost:7655/api/version`.
|
||||
3. **Open Issue**: Report on [GitHub Issues](https://github.com/rcourtman/Pulse/issues) with your logs and version info.
|
||||
|
|
|
|||
Loading…
Reference in a new issue