Pulse/cmd/pulse-sensor-proxy
rcourtman 3dc073a285 feat(sensor-proxy): Phase 2 - atomic config management with CLI
Implements bullet-proof configuration management to completely eliminate
allowed_nodes corruption by design. This builds on Phase 1 (file-only mode)
by replacing all shell/Python config manipulation with proper Go tooling.

**New Features:**
- `pulse-sensor-proxy config validate` - parse and validate config files
- `pulse-sensor-proxy config set-allowed-nodes` - atomic node list updates
- File locking via flock prevents concurrent write races
- Atomic writes (temp file + rename) ensure consistency
- systemd ExecStartPre validation prevents startup with bad config

**Architectural Changes:**
1. Installer now calls config CLI instead of embedded Python/shell scripts
2. All config mutations go through single authoritative writer
3. Deduplication and normalization handled in Go (reuses existing logic)
4. Sanitizer kept as noisy failsafe (warns if corruption still occurs)

**Implementation Details:**
- New cmd/pulse-sensor-proxy/config_cmd.go with cobra commands
- withLockedFile() wrapper ensures exclusive access
- atomicWriteFile() uses temp + rename pattern
- Installer update_allowed_nodes() simplified to CLI calls
- Both systemd service modes include ExecStartPre validation

**Why This Works:**
- Single code path for all writes (no shell/Python divergence)
- File locking serializes self-heal timer + manual installer runs
- Validation gate prevents proxy from starting with corrupt config
- CLI uses same YAML parser as the daemon (guaranteed compatibility)

**Phase 2 Benefits:**
- Corruption impossible by design (not just detected and fixed)
- No more Python dependency for config management
- Atomic operations prevent partial writes
- Clear error messages on validation failures

The defensive sanitizer remains active but now logs loudly if triggered,
allowing us to confirm Phase 2 eliminates corruption in production before
removing the safety net entirely.

This completes the fix for the recurring temperature monitoring outages.

Related to Phase 1 commit 53dec6010
2025-11-19 09:37:49 +00:00
..
audit.go Add HTTP mode to pulse-sensor-proxy for multi-instance temperature monitoring 2025-11-13 16:13:53 +00:00
audit_test.go Make pulse-sensor-proxy resilient to read-only filesystems 2025-11-06 00:18:51 +00:00
auth.go feat(security): Implement range-based rate limiting 2025-11-07 17:08:45 +00:00
auth_test.go feat(security): Implement GID authorization enforcement 2025-11-07 17:09:16 +00:00
capabilities.go Add CI build workflow and tighten proxy diagnostics 2025-11-14 13:32:29 +00:00
cleanup.go feat: add comprehensive node cleanup system 2025-10-17 18:53:45 +00:00
config.example.yaml feat(security): Add node allowlist validation to prevent SSRF attacks 2025-11-07 17:08:28 +00:00
config.go feat(sensor-proxy): Phase 2 - atomic config management with CLI 2025-11-19 09:37:49 +00:00
config_cmd.go feat(sensor-proxy): Phase 2 - atomic config management with CLI 2025-11-19 09:37:49 +00:00
config_test.go fix: sanitize sensor proxy config during self-heal 2025-11-18 22:51:40 +00:00
http_server.go Improve temperature proxy diagnostics and tests 2025-11-13 22:31:53 +00:00
main.go Improve temperature proxy control-plane flow 2025-11-15 21:49:51 +00:00
main_test.go feat(security): Add capability-based authorization 2025-11-07 17:09:32 +00:00
metrics.go feat(security): Add node allowlist validation to prevent SSRF attacks 2025-11-07 17:08:28 +00:00
README.md docs: document sensor proxy log forwarding 2025-11-14 01:12:25 +00:00
ssh.go Add context timeout to local temperature collection 2025-11-13 20:15:05 +00:00
ssh_test.go Add critical safety guards to temperature proxy installation 2025-11-13 10:26:46 +00:00
throttle.go Increase rate limiting for startup bursts 2025-11-13 15:42:26 +00:00
throttle_test.go feat(security): Implement range-based rate limiting 2025-11-07 17:08:45 +00:00
validation.go Improve temperature proxy control-plane flow 2025-11-15 21:49:51 +00:00
validation_fuzz_test.go security: complete Phase 1 sensor proxy hardening 2025-10-20 15:13:37 +00:00
validation_test.go Improve sensor proxy cluster validation (Related to #703) 2025-11-12 19:17:45 +00:00

pulse-sensor-proxy

The sensor proxy keeps SSH identities and temperature polling logic on the Proxmox host while presenting a small RPC surface (Unix socket or HTTPS) to the Pulse server. It protects SSH keys from container breakouts, enforces per-UID capabilities, and produces append-only audit logs.

Installation Options

Scenario Command
Recommended (automated) curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh | bash
Manual build go build ./cmd/pulse-sensor-proxy and sudo install -m 0755 pulse-sensor-proxy /usr/local/bin/
Prebuilt artifact Copy the binary from /opt/pulse/bin/pulse-sensor-proxy-* inside the Pulse Docker image or download via /download/pulse-sensor-proxy?platform=linux&arch=amd64.

The installer script provisions:

  • User & group: pulse-sensor-proxy
  • Binary: /usr/local/bin/pulse-sensor-proxy
  • Config: /etc/pulse-sensor-proxy/config.yaml
  • SSH material: /var/lib/pulse-sensor-proxy/ssh
  • Socket: /run/pulse-sensor-proxy/pulse-sensor-proxy.sock
  • Logs: /var/log/pulse/sensor-proxy/{proxy.log,audit.log} (append-only)
  • Systemd units: pulse-sensor-proxy.service, cleanup + self-heal timers

Start the service and verify status:

systemctl enable --now pulse-sensor-proxy
systemctl status pulse-sensor-proxy --no-pager
journalctl -u pulse-sensor-proxy -n 50

Configuration

The proxy reads /etc/pulse-sensor-proxy/config.yaml (see config.example.yaml). Key fields:

Key Purpose Notes
allowed_source_subnets Restrict peers by CIDR Empty list = auto-detect host networks
allowed_peers[].uid/gid Capability-scoped authorisation Prefer over legacy allowed_peer_uids
allowed_peers[].capabilities read, write, admin read covers get_temperature; admin required for ensure_cluster_keys
metrics_address Prometheus listener Default 127.0.0.1:9127; set disabled to turn off
require_proxmox_hostkeys Enforce known-host matches Protects against SSH MITM
max_ssh_output_bytes Cap command output Prevents memory exhaustion (default 1MiB)
rate_limit.per_peer_interval_ms / per_peer_burst Token bucket guardrails Keep interval ≥100ms in production
http_* keys HTTPS bridge mode Needs TLS files plus bearer token

Environment Overrides

  • PULSE_SENSOR_PROXY_SOCKET, PULSE_SENSOR_PROXY_SSH_DIR, PULSE_SENSOR_PROXY_CONFIG relocate runtime paths
  • PULSE_SENSOR_PROXY_USER run under a different service account (defaults to pulse-sensor-proxy)
  • PULSE_SENSOR_PROXY_ALLOWED_SUBNETS comma-separated list appended at boot
  • PULSE_SENSOR_PROXY_ALLOWED_PEER_UIDS/GIDS extend authorisation without editing YAML
  • PULSE_SENSOR_PROXY_ALLOW_IDMAPPED_ROOT explicitly allow/deny ID-mapped root
  • PULSE_SENSOR_PROXY_READ_TIMEOUT / _WRITE_TIMEOUT Go duration strings
  • PULSE_SENSOR_PROXY_AUDIT_LOG custom log path (still append-only)

HTTP Mode

Set http_enabled: true when the backend cannot mount the Unix socket (for example, Kubernetes). Requirements:

  1. Populate http_listen_addr (e.g. 0.0.0.0:9443).
  2. Provide http_tls_cert/http_tls_key. The installer can place certs under /etc/pulse-sensor-proxy/tls.
  3. Set a long http_auth_token. Pulse sends it as a bearer token.
  4. Restrict allowed_source_subnets to the Pulse control-plane addresses.

The HTTP server exports /temps and /health, enforces Bearer tokens, and logs all HTTP access attempts to the audit log.

Audit Logging & Rotation

  • Location: /var/log/pulse/sensor-proxy/audit.log
  • Format: JSON with hash chaining (prev_hash, event_hash, seq)
  • Access: Owned by pulse-sensor-proxy, 0640, chattr +a

Follow docs/operations/audit-log-rotation.md for rotation (remove +a, truncate, restart service, reapply +a). Also consider forwarding with scripts/setup-log-forwarding.sh; see docs/operations/sensor-proxy-log-forwarding.md for RELP/TLS forwarding instructions and verification steps.

Metrics & Monitoring

Signal Command
Prometheus metrics curl -s http://127.0.0.1:9127/metrics | head
Scheduler health (Pulse) curl -s http://localhost:7655/api/monitoring/scheduler/health | jq '.instances[] | select(.key | contains(\"temperature\")) | {key, breaker: .breaker.state, deadLetter: .deadLetter.present}'
Journal logs journalctl -u pulse-sensor-proxy -f
Rate-limit hits journalctl -u pulse-sensor-proxy | grep "rate limit"

Set alerts on:

  • pulse_proxy_rate_limit_hits_total spikes (potential abuse)
  • pulse_proxy_hostkey_changes_total increments (SSH MITM)
  • Temperature instances showing breaker.state != "closed" for >10minutes

Troubleshooting

Symptom Guidance
Cannot open audit log file Check permissions on /var/log/pulse/sensor-proxy. Remove chattr +a only during rotation.
connection denied in audit log UID/GID not listed in allowed_peers. Verify Pulse container UID mapping.
HTTP request from unauthorized source IP Update allowed_source_subnets or run through a reverse proxy that advertises the client IP via ProxyProtocol (not supported yet).
rate limit exceeded Increase rate_limit.per_peer_burst or fix noisy hosts before relaxing limits.
temperature pollers stuck Hit /api/monitoring/scheduler/health, ensure breakers are closed, restart Pulse + proxy if necessary.

For additional hardening steps, read docs/PULSE_SENSOR_PROXY_HARDENING.md and docs/TEMPERATURE_MONITORING_SECURITY.md.