docs: add operations runbooks and audit fixes

2025-11-14 01:01:21 +00:00 · 2025-11-14 01:01:21 +00:00 · 9864258207
commit 9864258207
parent 4d021f8a50
6 changed files with 380 additions and 1 deletions
--- a/MIGRATION_SCAFFOLDING.md
+++ b/MIGRATION_SCAFFOLDING.md
@ -0,0 +1,58 @@
+# Migration & Rollout Scaffolding
+
+This document tracks temporary code paths, feature flags, and kill switches that
+let us roll out risky changes safely. Keep it current so on-call engineers know
+how to disable new behavior without spelunking through the codebase.
+
+## How to Use This Document
+
+1. **Add an entry** whenever you introduce code that needs a manual cleanup
+   window (feature flags, compatibility layers, installers with rollback paths).
+2. **State the kill switch** (environment variable, UI toggle, config file, or
+   systemd override) and who owns the rollout.
+3. **Define removal criteria**. Once criteria are met, delete the scaffolding
+   and update this file.
+
+### Entry Template
+
+```
+## <Feature / Migration Name>
+- Introduced: vX.Y.Z (date/commit, if known)
+- Owner: <team/person>
+- Kill switch: how to disable + default state
+- Monitoring: metrics/logs to watch
+- Cleanup criteria: what must be true before removing scaffolding
+- Notes: cross-links to docs or runbooks
+```
+
+## Active Scaffolding
+
+### Adaptive Polling Scheduler
+- **Introduced**: v4.24.0 (documented in `docs/monitoring/ADAPTIVE_POLLING.md`)
+- **Owner**: Monitoring subsystem (rcourtman)
+- **Kill switch**: Toggle **Settings → System → Monitoring → Adaptive
+  Polling**, or set `ADAPTIVE_POLLING_ENABLED=false` before starting Pulse
+  (Docker/Kubernetes). Helm users can override `adaptivePollingEnabled: false`.
+- **Monitoring**: `pulse_monitor_poll_queue_depth`,
+  `pulse_monitor_poll_staleness_seconds`, and
+  `/api/monitoring/scheduler/health` for breaker/dead-letter status.
+- **Cleanup criteria**: Remove flag and legacy scheduler code after three stable
+  releases with `queue.depth < 50` during peak hours and no dead-letter growth.
+- **Notes**: Operational guidance lives in
+  `docs/operations/ADAPTIVE_POLLING_ROLLOUT.md`.
+
+### Automatic Updates Service
+- **Introduced**: commit `f46ff1792` (2025-10-11) added `scripts/pulse-auto-update.sh`
+- **Owner**: Installer/Update subsystem (rcourtman)
+- **Kill switch**: Disable **Settings → System → Updates → Automatic Updates** or
+  set `AUTO_UPDATE_ENABLED=false` in `/var/lib/pulse/system.json`. Installers
+  accept `--no-auto-update` to force the flag off.
+- **Monitoring**: `journalctl -u pulse-update.service`, UI update history
+  (Settings → System → Updates), and webhook notifications tagged with
+  `event_id=system.update`.
+- **Cleanup criteria**: Remove legacy opt-out plumbing once the opt-in rate is
+  ≥90 % across supported installers and we have at least two release cycles with
+  zero rollback events triggered by auto-update.
+- **Notes**: Keep release-specific caveats in `docs/RELEASE_NOTES.md`. The
+  update scheduler runs only on systemd installs; container builds ignore the
+  flag.
--- a/cmd/pulse-sensor-proxy/README.md
+++ b/cmd/pulse-sensor-proxy/README.md
@ -0,0 +1,113 @@
+# pulse-sensor-proxy
+
+The sensor proxy keeps SSH identities and temperature polling logic on the
+Proxmox host while presenting a small RPC surface (Unix socket or HTTPS) to the
+Pulse server. It protects SSH keys from container breakouts, enforces per-UID
+capabilities, and produces append-only audit logs.
+
+## Installation Options
+
+| Scenario | Command |
+| --- | --- |
+| **Recommended (automated)** | `curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh \`<br>`  | bash` |
+| **Manual build** | `go build ./cmd/pulse-sensor-proxy` and `sudo install -m 0755 pulse-sensor-proxy /usr/local/bin/` |
+| **Prebuilt artifact** | Copy the binary from `/opt/pulse/bin/pulse-sensor-proxy-*` inside the Pulse Docker image or download via `/download/pulse-sensor-proxy?platform=linux&arch=amd64`. |
+
+The installer script provisions:
+
+- User & group: `pulse-sensor-proxy`
+- Binary: `/usr/local/bin/pulse-sensor-proxy`
+- Config: `/etc/pulse-sensor-proxy/config.yaml`
+- SSH material: `/var/lib/pulse-sensor-proxy/ssh`
+- Socket: `/run/pulse-sensor-proxy/pulse-sensor-proxy.sock`
+- Logs: `/var/log/pulse/sensor-proxy/{proxy.log,audit.log}` (append-only)
+- Systemd units: `pulse-sensor-proxy.service`, cleanup + self-heal timers
+
+Start the service and verify status:
+
+```bash
+systemctl enable --now pulse-sensor-proxy
+systemctl status pulse-sensor-proxy --no-pager
+journalctl -u pulse-sensor-proxy -n 50
+```
+
+## Configuration
+
+The proxy reads `/etc/pulse-sensor-proxy/config.yaml` (see
+`config.example.yaml`). Key fields:
+
+| Key | Purpose | Notes |
+| --- | --- | --- |
+| `allowed_source_subnets` | Restrict peers by CIDR | Empty list = auto-detect host networks |
+| `allowed_peers[].uid/gid` | Capability-scoped authorisation | Prefer over legacy `allowed_peer_uids`|
+| `allowed_peers[].capabilities` | `read`, `write`, `admin` | `read` covers `get_temperature`; `admin` required for `ensure_cluster_keys` |
+| `metrics_address` | Prometheus listener | Default `127.0.0.1:9127`; set `disabled` to turn off |
+| `require_proxmox_hostkeys` | Enforce known-host matches | Protects against SSH MITM |
+| `max_ssh_output_bytes` | Cap command output | Prevents memory exhaustion (default 1 MiB) |
+| `rate_limit.per_peer_interval_ms` / `per_peer_burst` | Token bucket guardrails | Keep interval ≥100 ms in production |
+| `http_*` keys | HTTPS bridge mode | Needs TLS files plus bearer token |
+
+### Environment Overrides
+
+- `PULSE_SENSOR_PROXY_SOCKET`, `PULSE_SENSOR_PROXY_SSH_DIR`,
+  `PULSE_SENSOR_PROXY_CONFIG` – relocate runtime paths
+- `PULSE_SENSOR_PROXY_USER` – run under a different service account (defaults to
+  `pulse-sensor-proxy`)
+- `PULSE_SENSOR_PROXY_ALLOWED_SUBNETS` – comma-separated list appended at boot
+- `PULSE_SENSOR_PROXY_ALLOWED_PEER_UIDS/GIDS` – extend authorisation without
+  editing YAML
+- `PULSE_SENSOR_PROXY_ALLOW_IDMAPPED_ROOT` – explicitly allow/deny ID-mapped root
+- `PULSE_SENSOR_PROXY_READ_TIMEOUT` / `_WRITE_TIMEOUT` – Go duration strings
+- `PULSE_SENSOR_PROXY_AUDIT_LOG` – custom log path (still append-only)
+
+## HTTP Mode
+
+Set `http_enabled: true` when the backend cannot mount the Unix socket (for
+example, Kubernetes). Requirements:
+
+1. Populate `http_listen_addr` (e.g. `0.0.0.0:9443`).
+2. Provide `http_tls_cert`/`http_tls_key`. The installer can place certs under
+   `/etc/pulse-sensor-proxy/tls`.
+3. Set a long `http_auth_token`. Pulse sends it as a bearer token.
+4. Restrict `allowed_source_subnets` to the Pulse control-plane addresses.
+
+The HTTP server exports `/temps` and `/health`, enforces Bearer tokens, and logs
+all HTTP access attempts to the audit log.
+
+## Audit Logging & Rotation
+
+- Location: `/var/log/pulse/sensor-proxy/audit.log`
+- Format: JSON with hash chaining (`prev_hash`, `event_hash`, `seq`)
+- Access: Owned by `pulse-sensor-proxy`, `0640`, `chattr +a`
+
+Follow `docs/operations/audit-log-rotation.md` for rotation (remove `+a`,
+truncate, restart service, reapply `+a`). Also consider forwarding with
+`scripts/setup-log-forwarding.sh` so audit data lands in your SIEM.
+
+## Metrics & Monitoring
+
+| Signal | Command |
+| --- | --- |
+| Prometheus metrics | `curl -s http://127.0.0.1:9127/metrics | head` |
+| Scheduler health (Pulse) | `curl -s http://localhost:7655/api/monitoring/scheduler/health \`<br>`  | jq '.instances[] | select(.key | contains("temperature")) | {key, breaker: .breaker.state, deadLetter: .deadLetter.present}'` |
+| Journal logs | `journalctl -u pulse-sensor-proxy -f` |
+| Rate-limit hits | `journalctl -u pulse-sensor-proxy | grep "rate limit"` |
+
+Set alerts on:
+
+- `pulse_proxy_rate_limit_hits_total` spikes (potential abuse)
+- `pulse_proxy_hostkey_changes_total` increments (SSH MITM)
+- Temperature instances showing `breaker.state != "closed"` for >10 minutes
+
+## Troubleshooting
+
+| Symptom | Guidance |
+| --- | --- |
+| `Cannot open audit log file` | Check permissions on `/var/log/pulse/sensor-proxy`. Remove `chattr +a` only during rotation. |
+| `connection denied` in audit log | UID/GID not listed in `allowed_peers`. Verify Pulse container UID mapping. |
+| `HTTP request from unauthorized source IP` | Update `allowed_source_subnets` or run through a reverse proxy that advertises the client IP via `ProxyProtocol` (not supported yet). |
+| `rate limit exceeded` | Increase `rate_limit.per_peer_burst` or fix noisy hosts before relaxing limits. |
+| `temperature pollers stuck` | Hit `/api/monitoring/scheduler/health`, ensure breakers are `closed`, restart Pulse + proxy if necessary. |
+
+For additional hardening steps, read `docs/PULSE_SENSOR_PROXY_HARDENING.md` and
+`docs/TEMPERATURE_MONITORING_SECURITY.md`.
--- a/docs/DOCKER_HUB_README.md
+++ b/docs/DOCKER_HUB_README.md
@ -2,7 +2,7 @@

 [![GitHub release](https://img.shields.io/github/v/release/rcourtman/Pulse)](https://github.com/rcourtman/Pulse/releases/latest)
 [![Docker Pulls](https://img.shields.io/docker/pulls/rcourtman/pulse)](https://hub.docker.com/r/rcourtman/pulse)
-[![License](https://img.shields.io/github/license/rcourtman/Pulse)](LICENSE)
+[![License](https://img.shields.io/github/license/rcourtman/Pulse)](https://github.com/rcourtman/Pulse/blob/main/LICENSE)

 **Real-time monitoring for Proxmox VE, Proxmox Mail Gateway, PBS, and Docker infrastructure with alerts and webhooks.**

--- a/docs/README.md
+++ b/docs/README.md
@ -37,6 +37,11 @@ section groups related guides so you can jump straight to the material you need.
 - [PROXY_AUTH.md](PROXY_AUTH.md) – Authenticating via Authentik, Authelia, etc.
 - [TEMPERATURE_MONITORING_SECURITY.md](TEMPERATURE_MONITORING_SECURITY.md) – Legacy SSH considerations.

+## Operations Runbooks
+
+- [operations/audit-log-rotation.md](operations/audit-log-rotation.md) – Rotate sensor proxy audit logs safely.
+- [operations/ADAPTIVE_POLLING_ROLLOUT.md](operations/ADAPTIVE_POLLING_ROLLOUT.md) – Step-by-step adaptive polling rollout guide.
+
 ## Reference

 - [API.md](API.md) – REST API overview with examples.
--- a/docs/operations/ADAPTIVE_POLLING_ROLLOUT.md
+++ b/docs/operations/ADAPTIVE_POLLING_ROLLOUT.md
@ -0,0 +1,83 @@
+# Adaptive Polling Rollout Runbook
+
+Adaptive polling (v4.24.0+) lets the scheduler dynamically adjust poll
+intervals per resource. This runbook documents the safe way to enable, monitor,
+and, if needed, disable the feature across environments.
+
+## Scope & Prerequisites
+
+- Pulse **v4.24.0 or newer**
+- Admin access to **Settings → System → Monitoring**
+- Prometheus access to `pulse_monitor_*` metrics
+- Ability to run authenticated `curl` commands against the Pulse API
+
+## Change Windows
+
+Run rollouts during a maintenance window where transient alert jitter is
+acceptable. Adaptive polling touches every monitor queue; give yourself at least
+15 minutes to observe steady state metrics.
+
+## Rollout Steps
+
+1. **Snapshot current health**
+   ```bash
+   curl -s http://localhost:7655/api/monitoring/scheduler/health | jq '.enabled, .queue.depth'
+   ```
+   Record queue depth, breaker count, and dead-letter entries.
+
+2. **Enable adaptive polling**
+   - UI: toggle **Settings → System → Monitoring → Adaptive Polling** → Enable
+   - CLI: `jq '.AdaptivePollingEnabled=true' /var/lib/pulse/system.json > tmp && mv tmp system.json`
+   - Env override: `ADAPTIVE_POLLING_ENABLED=true` before starting Pulse (for
+     containers/k8s)
+
+3. **Watch metrics (first 5 minutes)**
+   ```bash
+   watch -n 5 'curl -s http://localhost:9091/metrics | grep -E "pulse_monitor_(poll_queue_depth|poll_staleness_seconds)" | head'
+   ```
+   Targets:
+   - `pulse_monitor_poll_queue_depth < 50`
+   - `pulse_monitor_poll_staleness_seconds` under your SLA (typically < 60 s)
+   - No spikes in `pulse_monitor_poll_errors_total{category="permanent"}`
+
+4. **Validate scheduler state**
+   ```bash
+   curl -s http://localhost:7655/api/monitoring/scheduler/health \
+     | jq '{enabled, queue: .queue.depth, breakers: [.breakers[]?.instance], deadLetter: .deadLetter.count}'
+   ```
+   Expect `enabled: true`, empty breaker list, and `deadLetter.count == 0`.
+
+5. **Document overrides**
+   - Note any instances moved to manual polling (Settings → Nodes → Polling)
+   - Capture Grafana screenshots for queue depth/staleness widgets
+
+## Rollback
+
+If queue depth climbs uncontrollably or breakers remain open for >10 minutes:
+
+1. Disable the feature the same way you enabled it (UI/environment).
+2. Restart Pulse if environment overrides were used, otherwise hot toggle is
+   immediate.
+3. Continue monitoring until queue depth and staleness return to baseline.
+
+## Canary Strategy Suggestions
+
+| Stage | Action | Acceptance Criteria |
+| --- | --- | --- |
+| Dev | Enable flag in hot-dev (scripts/hot-dev.sh) | No scheduler panics, UI reflects flag instantly |
+| Staging | Enable on one Pulse instance per region | `queue.depth` within ±20 % of baseline after 15 min |
+| Production | Enable per cluster with 30 min soak | No more than 5 breaker openings per hour |
+
+## Instrumentation Checklist
+
+- Grafana dashboard with `queue.depth`, `poll_staleness_seconds`,
+  `poll_errors_total` by type
+- Alert rule: `rate(pulse_monitor_poll_errors_total{category="permanent"}[5m]) > 0`
+- Alert rule: `max_over_time(pulse_monitor_poll_queue_depth[5m]) > 75`
+- JSON log search for `"scheduler":` warnings immediately after enablement
+
+## References
+
+- [Architecture doc](../monitoring/ADAPTIVE_POLLING.md)
+- [Scheduler Health API](../api/SCHEDULER_HEALTH.md)
+- [Kubernetes guidance](../KUBERNETES.md#adaptive-polling-configuration-v4250)
--- a/docs/operations/audit-log-rotation.md
+++ b/docs/operations/audit-log-rotation.md
@ -0,0 +1,120 @@
+# Sensor Proxy Audit Log Rotation
+
+The temperature sensor proxy writes append-only, hash-chained audit events to
+`/var/log/pulse/sensor-proxy/audit.log`. The file is created with `0640`
+permissions, owned by `pulse-sensor-proxy`, and protected with `chattr +a` via
+`scripts/secure-sensor-files.sh`. Because the process keeps the file handle open
+and enforces append-only mode, you **must** follow the steps below to rotate the
+log without losing events.
+
+## When to Rotate
+
+- File exceeds **200 MB** or contains more than 30 days of history
+- Prior to exporting evidence for an incident review
+- Immediately before changing log-forwarding endpoints (rsyslog/RELp)
+
+The proxy falls back to stderr (systemd journal) only when the file cannot be
+opened. Do not rely on the fallback for long-term retention.
+
+## Pre-flight Checklist
+
+1. Confirm the service is healthy:
+   ```bash
+   systemctl status pulse-sensor-proxy --no-pager
+   ```
+2. Make sure `/var/log/pulse/sensor-proxy` is mounted with enough free space:
+   ```bash
+   df -h /var/log/pulse/sensor-proxy
+   ```
+3. Note the current scheduler health inside Pulse for later verification:
+   ```bash
+   curl -s http://localhost:7655/api/monitoring/scheduler/health | jq '.queue.depth, .deadLetter.count'
+   ```
+
+## Manual Rotation Procedure
+
+> Run these steps as **root** on the Proxmox host that runs the proxy.
+
+1. Remove the append-only flag (logrotate needs to truncate the file):
+   ```bash
+   chattr -a /var/log/pulse/sensor-proxy/audit.log
+   ```
+2. Copy the current file to an evidence path, then truncate in place:
+   ```bash
+   ts=$(date +%Y%m%d-%H%M%S)
+   cp -a /var/log/pulse/sensor-proxy/audit.log /var/log/pulse/sensor-proxy/audit.log.$ts
+   : > /var/log/pulse/sensor-proxy/audit.log
+   ```
+3. Restore permissions and the append-only flag:
+   ```bash
+   chown pulse-sensor-proxy:pulse-sensor-proxy /var/log/pulse/sensor-proxy/audit.log
+   chmod 0640 /var/log/pulse/sensor-proxy/audit.log
+   chattr +a /var/log/pulse/sensor-proxy/audit.log
+   ```
+4. Restart the proxy so the file descriptor is reopened:
+   ```bash
+   systemctl restart pulse-sensor-proxy
+   ```
+5. Verify the service recreated the correlation hash chain:
+   ```bash
+   journalctl -u pulse-sensor-proxy -n 20 | grep -i "audit" || true
+   ```
+6. Re-check Pulse adaptive polling health (temperature pollers rely on the
+   proxy):
+   ```bash
+   curl -s http://localhost:7655/api/monitoring/scheduler/health \
+     | jq '.instances[] | select(.key | contains("temperature")) | {key, breaker: .breaker.state, deadLetter: .deadLetter.present}'
+   ```
+   All temperature instances should show `breaker: "closed"` with
+   `deadLetter: false`.
+
+## Logrotate Configuration
+
+Automate rotation with `/etc/logrotate.d/pulse-sensor-proxy`. Copy the snippet
+below and adjust retention to match your compliance needs:
+
+```conf
+/var/log/pulse/sensor-proxy/audit.log {
+    weekly
+    rotate 8
+    compress
+    missingok
+    notifempty
+    create 0640 pulse-sensor-proxy pulse-sensor-proxy
+    sharedscripts
+    prerotate
+        /usr/bin/chattr -a /var/log/pulse/sensor-proxy/audit.log || true
+    endscript
+    postrotate
+        /bin/systemctl restart pulse-sensor-proxy.service || true
+        /usr/bin/chattr +a /var/log/pulse/sensor-proxy/audit.log || true
+    endscript
+}
+```
+
+Keep `copytruncate` disabled—the restart ensures the proxy writes to a fresh
+file with a new hash chain. Always forward rotated files to your SIEM before
+removing them.
+
+## Forwarding Validations
+
+If you forward audit logs over RELP using `scripts/setup-log-forwarding.sh`:
+
+1. Tail the forwarding log:
+   ```bash
+   tail -f /var/log/pulse/sensor-proxy/forwarding.log
+   ```
+2. Ensure queues drain (`action.resumeRetryCount=-1` keeps retrying).
+3. Confirm the remote receiver ingests the new file (look for the `pulse.audit`
+tag).
+
+## Troubleshooting
+
+| Symptom | Action |
+| --- | --- |
+| `Operation not permitted` when truncating | `chattr -a` was not executed or SELinux/AppArmor denies it. Check `auditd`. |
+| Proxy fails to restart | Run `journalctl -u pulse-sensor-proxy -xe` for context. The proxy refuses to start if the audit file cannot be opened. |
+| Temperature polls stop after rotation | Check `/api/monitoring/scheduler/health` for dead-letter entries. Restart the main Pulse service if breakers stay open. |
+
+Once logs are rotated and validated, upload the archived copy to your evidence
+store and record the event in your change log.