diff --git a/MIGRATION_SCAFFOLDING.md b/MIGRATION_SCAFFOLDING.md new file mode 100644 index 0000000..ba7c0f2 --- /dev/null +++ b/MIGRATION_SCAFFOLDING.md @@ -0,0 +1,58 @@ +# Migration & Rollout Scaffolding + +This document tracks temporary code paths, feature flags, and kill switches that +let us roll out risky changes safely. Keep it current so on-call engineers know +how to disable new behavior without spelunking through the codebase. + +## How to Use This Document + +1. **Add an entry** whenever you introduce code that needs a manual cleanup + window (feature flags, compatibility layers, installers with rollback paths). +2. **State the kill switch** (environment variable, UI toggle, config file, or + systemd override) and who owns the rollout. +3. **Define removal criteria**. Once criteria are met, delete the scaffolding + and update this file. + +### Entry Template + +``` +## +- Introduced: vX.Y.Z (date/commit, if known) +- Owner: +- Kill switch: how to disable + default state +- Monitoring: metrics/logs to watch +- Cleanup criteria: what must be true before removing scaffolding +- Notes: cross-links to docs or runbooks +``` + +## Active Scaffolding + +### Adaptive Polling Scheduler +- **Introduced**: v4.24.0 (documented in `docs/monitoring/ADAPTIVE_POLLING.md`) +- **Owner**: Monitoring subsystem (rcourtman) +- **Kill switch**: Toggle **Settings → System → Monitoring → Adaptive + Polling**, or set `ADAPTIVE_POLLING_ENABLED=false` before starting Pulse + (Docker/Kubernetes). Helm users can override `adaptivePollingEnabled: false`. +- **Monitoring**: `pulse_monitor_poll_queue_depth`, + `pulse_monitor_poll_staleness_seconds`, and + `/api/monitoring/scheduler/health` for breaker/dead-letter status. +- **Cleanup criteria**: Remove flag and legacy scheduler code after three stable + releases with `queue.depth < 50` during peak hours and no dead-letter growth. +- **Notes**: Operational guidance lives in + `docs/operations/ADAPTIVE_POLLING_ROLLOUT.md`. + +### Automatic Updates Service +- **Introduced**: commit `f46ff1792` (2025-10-11) added `scripts/pulse-auto-update.sh` +- **Owner**: Installer/Update subsystem (rcourtman) +- **Kill switch**: Disable **Settings → System → Updates → Automatic Updates** or + set `AUTO_UPDATE_ENABLED=false` in `/var/lib/pulse/system.json`. Installers + accept `--no-auto-update` to force the flag off. +- **Monitoring**: `journalctl -u pulse-update.service`, UI update history + (Settings → System → Updates), and webhook notifications tagged with + `event_id=system.update`. +- **Cleanup criteria**: Remove legacy opt-out plumbing once the opt-in rate is + ≥90 % across supported installers and we have at least two release cycles with + zero rollback events triggered by auto-update. +- **Notes**: Keep release-specific caveats in `docs/RELEASE_NOTES.md`. The + update scheduler runs only on systemd installs; container builds ignore the + flag. diff --git a/cmd/pulse-sensor-proxy/README.md b/cmd/pulse-sensor-proxy/README.md new file mode 100644 index 0000000..a3d0c12 --- /dev/null +++ b/cmd/pulse-sensor-proxy/README.md @@ -0,0 +1,113 @@ +# pulse-sensor-proxy + +The sensor proxy keeps SSH identities and temperature polling logic on the +Proxmox host while presenting a small RPC surface (Unix socket or HTTPS) to the +Pulse server. It protects SSH keys from container breakouts, enforces per-UID +capabilities, and produces append-only audit logs. + +## Installation Options + +| Scenario | Command | +| --- | --- | +| **Recommended (automated)** | `curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh \`
` | bash` | +| **Manual build** | `go build ./cmd/pulse-sensor-proxy` and `sudo install -m 0755 pulse-sensor-proxy /usr/local/bin/` | +| **Prebuilt artifact** | Copy the binary from `/opt/pulse/bin/pulse-sensor-proxy-*` inside the Pulse Docker image or download via `/download/pulse-sensor-proxy?platform=linux&arch=amd64`. | + +The installer script provisions: + +- User & group: `pulse-sensor-proxy` +- Binary: `/usr/local/bin/pulse-sensor-proxy` +- Config: `/etc/pulse-sensor-proxy/config.yaml` +- SSH material: `/var/lib/pulse-sensor-proxy/ssh` +- Socket: `/run/pulse-sensor-proxy/pulse-sensor-proxy.sock` +- Logs: `/var/log/pulse/sensor-proxy/{proxy.log,audit.log}` (append-only) +- Systemd units: `pulse-sensor-proxy.service`, cleanup + self-heal timers + +Start the service and verify status: + +```bash +systemctl enable --now pulse-sensor-proxy +systemctl status pulse-sensor-proxy --no-pager +journalctl -u pulse-sensor-proxy -n 50 +``` + +## Configuration + +The proxy reads `/etc/pulse-sensor-proxy/config.yaml` (see +`config.example.yaml`). Key fields: + +| Key | Purpose | Notes | +| --- | --- | --- | +| `allowed_source_subnets` | Restrict peers by CIDR | Empty list = auto-detect host networks | +| `allowed_peers[].uid/gid` | Capability-scoped authorisation | Prefer over legacy `allowed_peer_uids`| +| `allowed_peers[].capabilities` | `read`, `write`, `admin` | `read` covers `get_temperature`; `admin` required for `ensure_cluster_keys` | +| `metrics_address` | Prometheus listener | Default `127.0.0.1:9127`; set `disabled` to turn off | +| `require_proxmox_hostkeys` | Enforce known-host matches | Protects against SSH MITM | +| `max_ssh_output_bytes` | Cap command output | Prevents memory exhaustion (default 1 MiB) | +| `rate_limit.per_peer_interval_ms` / `per_peer_burst` | Token bucket guardrails | Keep interval ≥100 ms in production | +| `http_*` keys | HTTPS bridge mode | Needs TLS files plus bearer token | + +### Environment Overrides + +- `PULSE_SENSOR_PROXY_SOCKET`, `PULSE_SENSOR_PROXY_SSH_DIR`, + `PULSE_SENSOR_PROXY_CONFIG` – relocate runtime paths +- `PULSE_SENSOR_PROXY_USER` – run under a different service account (defaults to + `pulse-sensor-proxy`) +- `PULSE_SENSOR_PROXY_ALLOWED_SUBNETS` – comma-separated list appended at boot +- `PULSE_SENSOR_PROXY_ALLOWED_PEER_UIDS/GIDS` – extend authorisation without + editing YAML +- `PULSE_SENSOR_PROXY_ALLOW_IDMAPPED_ROOT` – explicitly allow/deny ID-mapped root +- `PULSE_SENSOR_PROXY_READ_TIMEOUT` / `_WRITE_TIMEOUT` – Go duration strings +- `PULSE_SENSOR_PROXY_AUDIT_LOG` – custom log path (still append-only) + +## HTTP Mode + +Set `http_enabled: true` when the backend cannot mount the Unix socket (for +example, Kubernetes). Requirements: + +1. Populate `http_listen_addr` (e.g. `0.0.0.0:9443`). +2. Provide `http_tls_cert`/`http_tls_key`. The installer can place certs under + `/etc/pulse-sensor-proxy/tls`. +3. Set a long `http_auth_token`. Pulse sends it as a bearer token. +4. Restrict `allowed_source_subnets` to the Pulse control-plane addresses. + +The HTTP server exports `/temps` and `/health`, enforces Bearer tokens, and logs +all HTTP access attempts to the audit log. + +## Audit Logging & Rotation + +- Location: `/var/log/pulse/sensor-proxy/audit.log` +- Format: JSON with hash chaining (`prev_hash`, `event_hash`, `seq`) +- Access: Owned by `pulse-sensor-proxy`, `0640`, `chattr +a` + +Follow `docs/operations/audit-log-rotation.md` for rotation (remove `+a`, +truncate, restart service, reapply `+a`). Also consider forwarding with +`scripts/setup-log-forwarding.sh` so audit data lands in your SIEM. + +## Metrics & Monitoring + +| Signal | Command | +| --- | --- | +| Prometheus metrics | `curl -s http://127.0.0.1:9127/metrics | head` | +| Scheduler health (Pulse) | `curl -s http://localhost:7655/api/monitoring/scheduler/health \`
` | jq '.instances[] | select(.key | contains("temperature")) | {key, breaker: .breaker.state, deadLetter: .deadLetter.present}'` | +| Journal logs | `journalctl -u pulse-sensor-proxy -f` | +| Rate-limit hits | `journalctl -u pulse-sensor-proxy | grep "rate limit"` | + +Set alerts on: + +- `pulse_proxy_rate_limit_hits_total` spikes (potential abuse) +- `pulse_proxy_hostkey_changes_total` increments (SSH MITM) +- Temperature instances showing `breaker.state != "closed"` for >10 minutes + +## Troubleshooting + +| Symptom | Guidance | +| --- | --- | +| `Cannot open audit log file` | Check permissions on `/var/log/pulse/sensor-proxy`. Remove `chattr +a` only during rotation. | +| `connection denied` in audit log | UID/GID not listed in `allowed_peers`. Verify Pulse container UID mapping. | +| `HTTP request from unauthorized source IP` | Update `allowed_source_subnets` or run through a reverse proxy that advertises the client IP via `ProxyProtocol` (not supported yet). | +| `rate limit exceeded` | Increase `rate_limit.per_peer_burst` or fix noisy hosts before relaxing limits. | +| `temperature pollers stuck` | Hit `/api/monitoring/scheduler/health`, ensure breakers are `closed`, restart Pulse + proxy if necessary. | + +For additional hardening steps, read `docs/PULSE_SENSOR_PROXY_HARDENING.md` and +`docs/TEMPERATURE_MONITORING_SECURITY.md`. diff --git a/docs/DOCKER_HUB_README.md b/docs/DOCKER_HUB_README.md index 2270f1d..a73e9be 100644 --- a/docs/DOCKER_HUB_README.md +++ b/docs/DOCKER_HUB_README.md @@ -2,7 +2,7 @@ [![GitHub release](https://img.shields.io/github/v/release/rcourtman/Pulse)](https://github.com/rcourtman/Pulse/releases/latest) [![Docker Pulls](https://img.shields.io/docker/pulls/rcourtman/pulse)](https://hub.docker.com/r/rcourtman/pulse) -[![License](https://img.shields.io/github/license/rcourtman/Pulse)](LICENSE) +[![License](https://img.shields.io/github/license/rcourtman/Pulse)](https://github.com/rcourtman/Pulse/blob/main/LICENSE) **Real-time monitoring for Proxmox VE, Proxmox Mail Gateway, PBS, and Docker infrastructure with alerts and webhooks.** diff --git a/docs/README.md b/docs/README.md index b36009d..1191437 100644 --- a/docs/README.md +++ b/docs/README.md @@ -37,6 +37,11 @@ section groups related guides so you can jump straight to the material you need. - [PROXY_AUTH.md](PROXY_AUTH.md) – Authenticating via Authentik, Authelia, etc. - [TEMPERATURE_MONITORING_SECURITY.md](TEMPERATURE_MONITORING_SECURITY.md) – Legacy SSH considerations. +## Operations Runbooks + +- [operations/audit-log-rotation.md](operations/audit-log-rotation.md) – Rotate sensor proxy audit logs safely. +- [operations/ADAPTIVE_POLLING_ROLLOUT.md](operations/ADAPTIVE_POLLING_ROLLOUT.md) – Step-by-step adaptive polling rollout guide. + ## Reference - [API.md](API.md) – REST API overview with examples. diff --git a/docs/operations/ADAPTIVE_POLLING_ROLLOUT.md b/docs/operations/ADAPTIVE_POLLING_ROLLOUT.md new file mode 100644 index 0000000..fdcfc59 --- /dev/null +++ b/docs/operations/ADAPTIVE_POLLING_ROLLOUT.md @@ -0,0 +1,83 @@ +# Adaptive Polling Rollout Runbook + +Adaptive polling (v4.24.0+) lets the scheduler dynamically adjust poll +intervals per resource. This runbook documents the safe way to enable, monitor, +and, if needed, disable the feature across environments. + +## Scope & Prerequisites + +- Pulse **v4.24.0 or newer** +- Admin access to **Settings → System → Monitoring** +- Prometheus access to `pulse_monitor_*` metrics +- Ability to run authenticated `curl` commands against the Pulse API + +## Change Windows + +Run rollouts during a maintenance window where transient alert jitter is +acceptable. Adaptive polling touches every monitor queue; give yourself at least +15 minutes to observe steady state metrics. + +## Rollout Steps + +1. **Snapshot current health** + ```bash + curl -s http://localhost:7655/api/monitoring/scheduler/health | jq '.enabled, .queue.depth' + ``` + Record queue depth, breaker count, and dead-letter entries. + +2. **Enable adaptive polling** + - UI: toggle **Settings → System → Monitoring → Adaptive Polling** → Enable + - CLI: `jq '.AdaptivePollingEnabled=true' /var/lib/pulse/system.json > tmp && mv tmp system.json` + - Env override: `ADAPTIVE_POLLING_ENABLED=true` before starting Pulse (for + containers/k8s) + +3. **Watch metrics (first 5 minutes)** + ```bash + watch -n 5 'curl -s http://localhost:9091/metrics | grep -E "pulse_monitor_(poll_queue_depth|poll_staleness_seconds)" | head' + ``` + Targets: + - `pulse_monitor_poll_queue_depth < 50` + - `pulse_monitor_poll_staleness_seconds` under your SLA (typically < 60 s) + - No spikes in `pulse_monitor_poll_errors_total{category="permanent"}` + +4. **Validate scheduler state** + ```bash + curl -s http://localhost:7655/api/monitoring/scheduler/health \ + | jq '{enabled, queue: .queue.depth, breakers: [.breakers[]?.instance], deadLetter: .deadLetter.count}' + ``` + Expect `enabled: true`, empty breaker list, and `deadLetter.count == 0`. + +5. **Document overrides** + - Note any instances moved to manual polling (Settings → Nodes → Polling) + - Capture Grafana screenshots for queue depth/staleness widgets + +## Rollback + +If queue depth climbs uncontrollably or breakers remain open for >10 minutes: + +1. Disable the feature the same way you enabled it (UI/environment). +2. Restart Pulse if environment overrides were used, otherwise hot toggle is + immediate. +3. Continue monitoring until queue depth and staleness return to baseline. + +## Canary Strategy Suggestions + +| Stage | Action | Acceptance Criteria | +| --- | --- | --- | +| Dev | Enable flag in hot-dev (scripts/hot-dev.sh) | No scheduler panics, UI reflects flag instantly | +| Staging | Enable on one Pulse instance per region | `queue.depth` within ±20 % of baseline after 15 min | +| Production | Enable per cluster with 30 min soak | No more than 5 breaker openings per hour | + +## Instrumentation Checklist + +- Grafana dashboard with `queue.depth`, `poll_staleness_seconds`, + `poll_errors_total` by type +- Alert rule: `rate(pulse_monitor_poll_errors_total{category="permanent"}[5m]) > 0` +- Alert rule: `max_over_time(pulse_monitor_poll_queue_depth[5m]) > 75` +- JSON log search for `"scheduler":` warnings immediately after enablement + +## References + +- [Architecture doc](../monitoring/ADAPTIVE_POLLING.md) +- [Scheduler Health API](../api/SCHEDULER_HEALTH.md) +- [Kubernetes guidance](../KUBERNETES.md#adaptive-polling-configuration-v4250) diff --git a/docs/operations/audit-log-rotation.md b/docs/operations/audit-log-rotation.md new file mode 100644 index 0000000..4c09de2 --- /dev/null +++ b/docs/operations/audit-log-rotation.md @@ -0,0 +1,120 @@ +# Sensor Proxy Audit Log Rotation + +The temperature sensor proxy writes append-only, hash-chained audit events to +`/var/log/pulse/sensor-proxy/audit.log`. The file is created with `0640` +permissions, owned by `pulse-sensor-proxy`, and protected with `chattr +a` via +`scripts/secure-sensor-files.sh`. Because the process keeps the file handle open +and enforces append-only mode, you **must** follow the steps below to rotate the +log without losing events. + +## When to Rotate + +- File exceeds **200 MB** or contains more than 30 days of history +- Prior to exporting evidence for an incident review +- Immediately before changing log-forwarding endpoints (rsyslog/RELp) + +The proxy falls back to stderr (systemd journal) only when the file cannot be +opened. Do not rely on the fallback for long-term retention. + +## Pre-flight Checklist + +1. Confirm the service is healthy: + ```bash + systemctl status pulse-sensor-proxy --no-pager + ``` +2. Make sure `/var/log/pulse/sensor-proxy` is mounted with enough free space: + ```bash + df -h /var/log/pulse/sensor-proxy + ``` +3. Note the current scheduler health inside Pulse for later verification: + ```bash + curl -s http://localhost:7655/api/monitoring/scheduler/health | jq '.queue.depth, .deadLetter.count' + ``` + +## Manual Rotation Procedure + +> Run these steps as **root** on the Proxmox host that runs the proxy. + +1. Remove the append-only flag (logrotate needs to truncate the file): + ```bash + chattr -a /var/log/pulse/sensor-proxy/audit.log + ``` +2. Copy the current file to an evidence path, then truncate in place: + ```bash + ts=$(date +%Y%m%d-%H%M%S) + cp -a /var/log/pulse/sensor-proxy/audit.log /var/log/pulse/sensor-proxy/audit.log.$ts + : > /var/log/pulse/sensor-proxy/audit.log + ``` +3. Restore permissions and the append-only flag: + ```bash + chown pulse-sensor-proxy:pulse-sensor-proxy /var/log/pulse/sensor-proxy/audit.log + chmod 0640 /var/log/pulse/sensor-proxy/audit.log + chattr +a /var/log/pulse/sensor-proxy/audit.log + ``` +4. Restart the proxy so the file descriptor is reopened: + ```bash + systemctl restart pulse-sensor-proxy + ``` +5. Verify the service recreated the correlation hash chain: + ```bash + journalctl -u pulse-sensor-proxy -n 20 | grep -i "audit" || true + ``` +6. Re-check Pulse adaptive polling health (temperature pollers rely on the + proxy): + ```bash + curl -s http://localhost:7655/api/monitoring/scheduler/health \ + | jq '.instances[] | select(.key | contains("temperature")) | {key, breaker: .breaker.state, deadLetter: .deadLetter.present}' + ``` + All temperature instances should show `breaker: "closed"` with + `deadLetter: false`. + +## Logrotate Configuration + +Automate rotation with `/etc/logrotate.d/pulse-sensor-proxy`. Copy the snippet +below and adjust retention to match your compliance needs: + +```conf +/var/log/pulse/sensor-proxy/audit.log { + weekly + rotate 8 + compress + missingok + notifempty + create 0640 pulse-sensor-proxy pulse-sensor-proxy + sharedscripts + prerotate + /usr/bin/chattr -a /var/log/pulse/sensor-proxy/audit.log || true + endscript + postrotate + /bin/systemctl restart pulse-sensor-proxy.service || true + /usr/bin/chattr +a /var/log/pulse/sensor-proxy/audit.log || true + endscript +} +``` + +Keep `copytruncate` disabled—the restart ensures the proxy writes to a fresh +file with a new hash chain. Always forward rotated files to your SIEM before +removing them. + +## Forwarding Validations + +If you forward audit logs over RELP using `scripts/setup-log-forwarding.sh`: + +1. Tail the forwarding log: + ```bash + tail -f /var/log/pulse/sensor-proxy/forwarding.log + ``` +2. Ensure queues drain (`action.resumeRetryCount=-1` keeps retrying). +3. Confirm the remote receiver ingests the new file (look for the `pulse.audit` +tag). + +## Troubleshooting + +| Symptom | Action | +| --- | --- | +| `Operation not permitted` when truncating | `chattr -a` was not executed or SELinux/AppArmor denies it. Check `auditd`. | +| Proxy fails to restart | Run `journalctl -u pulse-sensor-proxy -xe` for context. The proxy refuses to start if the audit file cannot be opened. | +| Temperature polls stop after rotation | Check `/api/monitoring/scheduler/health` for dead-letter entries. Restart the main Pulse service if breakers stay open. | + +Once logs are rotated and validated, upload the archived copy to your evidence +store and record the event in your change log.