diff --git a/docs/DEPLOYMENT_CHECKLIST.md b/docs/DEPLOYMENT_CHECKLIST.md deleted file mode 100644 index 2a6f6a4..0000000 --- a/docs/DEPLOYMENT_CHECKLIST.md +++ /dev/null @@ -1,404 +0,0 @@ -# Deployment Checklist - Security Hardening Update -## Version: 2025-11-07 Security Hardening Release - ---- - -## Pre-Deployment Checklist - -### 1. Review Changes -- [ ] Read `docs/SECURITY_AUDIT_2025-11-07.md` for full context -- [ ] Review `docs/SECURITY_CHANGELOG.md` for breaking changes -- [ ] Understand the `:ro` socket mount requirement - -### 2. Backup Current State -```bash -# Backup configuration -cp /etc/pulse-sensor-proxy/config.yaml /etc/pulse-sensor-proxy/config.yaml.backup - -# Backup systemd unit -cp /etc/systemd/system/pulse-sensor-proxy.service \ - /etc/systemd/system/pulse-sensor-proxy.service.backup - -# Note current service status -systemctl status pulse-sensor-proxy > /tmp/proxy-status-before.txt - -# Backup docker-compose (if using Docker) -cp docker-compose.yml docker-compose.yml.backup -``` - -### 3. Verify Test Results -- [ ] All tests pass: `go test ./cmd/pulse-sensor-proxy ./internal/ssh/knownhosts` -- [ ] Binary builds: `go build ./cmd/pulse-sensor-proxy` -- [ ] No compilation errors or warnings - ---- - -## Deployment Steps - -### For Docker Deployments - -#### Step 1: Update docker-compose.yml -```bash -# Change socket mount from :rw to :ro -sed -i 's|/run/pulse-sensor-proxy:rw|/run/pulse-sensor-proxy:ro|g' docker-compose.yml - -# Verify change -grep pulse-sensor-proxy docker-compose.yml -# Should show: - /run/pulse-sensor-proxy:/run/pulse-sensor-proxy:ro -``` - -#### Step 2: Optional Configuration -Create `/etc/pulse-sensor-proxy/config.yaml` (optional but recommended): -```yaml -# Node allowlist (prevents SSRF) -allowed_nodes: - - "10.0.0.0/24" # Replace with your cluster network -strict_node_validation: true - -# Timeouts (prevents DoS) -read_timeout: 5s -write_timeout: 10s - -# SSH output limits -max_ssh_output_bytes: 1048576 # 1MB - -# Host key management -require_proxmox_hostkeys: false # Set true for strict mode - -# Capability-based authorization (optional) -allowed_peers: - - uid: 0 - capabilities: [read, write, admin] - - uid: 1000 - capabilities: [read] # Docker containers: read-only -``` - -#### Step 3: Restart Services -```bash -# Restart proxy on host -sudo systemctl restart pulse-sensor-proxy - -# Check proxy status -sudo systemctl status pulse-sensor-proxy - -# Restart Pulse container -docker compose down -docker compose up -d - -# Check container logs -docker logs pulse -f -# Look for: "Temperature proxy detected - using secure host-side bridge" -``` - ---- - -### For LXC Deployments - -#### Step 1: Verify Mount Configuration -```bash -# Check mount is already correct (install script uses correct format) -pct config | grep mp - -# Should see: mp0: /run/pulse-sensor-proxy,mp=/mnt/pulse-proxy -# If shows old format with explicit :rw, update manually -``` - -#### Step 2: Update Systemd Unit -```bash -# Copy new hardened unit -sudo cp scripts/pulse-sensor-proxy.service /etc/systemd/system/ - -# Reload systemd -sudo systemctl daemon-reload - -# Verify new directives loaded -systemctl cat pulse-sensor-proxy | grep -E "(MemoryDeny|RestrictRealtime)" -``` - -#### Step 3: Optional Configuration -Create `/etc/pulse-sensor-proxy/config.yaml` (same as Docker above) - -#### Step 4: Restart Proxy -```bash -sudo systemctl restart pulse-sensor-proxy -sudo systemctl status pulse-sensor-proxy -``` - ---- - -## Verification Steps - -### 1. Proxy Health Check -```bash -# Check proxy is running -sudo systemctl status pulse-sensor-proxy - -# Check socket exists and is accessible -ls -la /run/pulse-sensor-proxy/pulse-sensor-proxy.sock - -# Test socket from host -echo '{"method":"get_status"}' | \ - sudo socat - UNIX-CONNECT:/run/pulse-sensor-proxy/pulse-sensor-proxy.sock | jq . -# Should return: {"success":true,"data":{"version":"dev","public_key":"...","ssh_dir":"..."}} -``` - -### 2. Container Access Verification - -**Docker:** -```bash -# Check socket is visible in container -docker exec pulse ls -la /run/pulse-sensor-proxy/ - -# Verify container CANNOT write to directory (security fix) -docker exec pulse touch /run/pulse-sensor-proxy/test 2>&1 -# Should fail with: "Read-only file system" (GOOD) - -# Check Pulse logs for proxy detection -docker logs pulse 2>&1 | grep -i "temperature.*proxy" -# Should see: "Temperature proxy detected - using secure host-side bridge" -``` - -**LXC:** -```bash -# Check socket is visible in container -pct exec -- ls -la /mnt/pulse-proxy/ - -# Verify container cannot write -pct exec -- touch /mnt/pulse-proxy/test 2>&1 -# Should fail (GOOD) - -# Check Pulse logs -pct exec -- journalctl -u pulse -n 50 | grep -i temperature -``` - -### 3. Temperature Data Collection -```bash -# Access Pulse UI: http://your-server:7655 -# Navigate to any Proxmox node -# Verify temperature data is shown (if hardware supports it) - -# Or check via API -curl http://localhost:7655/api/nodes | jq '.[].temperature' -``` - -### 4. Security Feature Verification - -**Node Validation (if configured):** -```bash -# Check logs for validation (if you added allowed_nodes) -sudo journalctl -u pulse-sensor-proxy -n 50 | grep -i validation -``` - -**Metrics Check:** -```bash -# Check new security metrics exist -curl -s http://localhost:7655/metrics | grep pulse_proxy_node_validation -curl -s http://localhost:7655/metrics | grep pulse_proxy_read_timeouts -curl -s http://localhost:7655/metrics | grep pulse_proxy_limiter - -# Or check proxy metrics directly (if exposed) -curl -s http://127.0.0.1:9127/metrics | grep pulse_proxy -``` - -### 5. Rate Limiting Verification -```bash -# Check limiter is using range-based identification for containers -sudo journalctl -u pulse-sensor-proxy -n 100 | grep -i "range:" -# Should see logs like: peer="range:100000-165535" if container is accessing - -# Check metrics -curl -s http://127.0.0.1:9127/metrics | grep pulse_proxy_limiter_rejections_total -``` - ---- - -## Post-Deployment Monitoring - -### Immediate (First Hour) -- [ ] Watch proxy logs: `journalctl -u pulse-sensor-proxy -f` -- [ ] Watch Pulse logs: `docker logs pulse -f` or `journalctl -u pulse -f` -- [ ] Check for errors or warnings -- [ ] Verify temperature data is updating in UI - -### First Day -- [ ] Monitor for security events in logs -- [ ] Check metrics for anomalies -- [ ] Verify no legitimate requests being blocked - -### First Week -- [ ] Review security metrics trends -- [ ] Check for any false positives in node validation -- [ ] Adjust configuration if needed - ---- - -## Rollback Procedure - -If issues occur: - -### Quick Rollback (Docker) -```bash -# Restore old docker-compose -cp docker-compose.yml.backup docker-compose.yml - -# Restart -docker compose down && docker compose up -d -``` - -### Quick Rollback (LXC) -```bash -# Restore old systemd unit -sudo cp /etc/systemd/system/pulse-sensor-proxy.service.backup \ - /etc/systemd/system/pulse-sensor-proxy.service - -# Restore old config (if needed) -sudo cp /etc/pulse-sensor-proxy/config.yaml.backup \ - /etc/pulse-sensor-proxy/config.yaml - -# Reload and restart -sudo systemctl daemon-reload -sudo systemctl restart pulse-sensor-proxy -``` - -### Git Rollback -```bash -# Revert to previous commit -git revert HEAD -git push - -# Rebuild and redeploy -go build ./cmd/pulse-sensor-proxy -sudo systemctl restart pulse-sensor-proxy -``` - ---- - -## Troubleshooting - -### Issue: "Temperature proxy not detected" - -**Symptom:** Pulse logs don't show proxy detection message - -**Check:** -```bash -# 1. Verify proxy is running -sudo systemctl status pulse-sensor-proxy - -# 2. Verify socket exists -ls -la /run/pulse-sensor-proxy/pulse-sensor-proxy.sock - -# 3. Verify container can see socket -docker exec pulse ls -la /run/pulse-sensor-proxy/ # Docker -pct exec -- ls -la /mnt/pulse-proxy/ # LXC - -# 4. Check mount is correct -docker inspect pulse | grep -A 5 Mounts # Docker -pct config | grep mp # LXC -``` - -**Fix:** -- Ensure socket directory is mounted into container -- Verify mount path matches Pulse expectation -- Restart both proxy and Pulse - ---- - -### Issue: "Read-only file system" errors in container - -**Symptom:** Container logs show read-only errors for `/run/pulse-sensor-proxy/` - -**This is EXPECTED and CORRECT!** The mount is intentionally read-only for security. - -**If Pulse itself has issues:** -- Pulse should only READ from the socket, not write -- Check Pulse isn't trying to write to the socket directory -- This should not happen in normal operation - ---- - -### Issue: "Node validation failed" or "potential SSRF attempt" - -**Symptom:** Temperature requests failing, logs show validation errors - -**Check:** -```bash -# Check your allowed_nodes configuration -cat /etc/pulse-sensor-proxy/config.yaml | grep -A 5 allowed_nodes - -# Check what nodes Pulse is trying to connect to -sudo journalctl -u pulse-sensor-proxy | grep "potential SSRF" -``` - -**Fix:** -- Add legitimate nodes to `allowed_nodes` list -- Or set `strict_node_validation: false` temporarily for debugging -- Verify node hostnames/IPs are correct - ---- - -### Issue: High rate limit rejections - -**Symptom:** Metrics show many `pulse_proxy_limiter_rejections_total` - -**Check:** -```bash -# Check which peers are being limited -curl -s http://127.0.0.1:9127/metrics | grep limiter_rejections_total - -# Check logs for details -sudo journalctl -u pulse-sensor-proxy | grep "Rate limit" -``` - -**Fix:** -- This may be normal if you have many nodes -- Rate limits are per-container-range (not per-UID) now -- Adjust `rate_limit` config if needed: - ```yaml - rate_limit: - per_peer_interval_ms: 500 # Increase to 2 req/sec - per_peer_burst: 10 # Allow larger bursts - ``` - ---- - -## Metrics to Monitor - -### Security Metrics -- `pulse_proxy_node_validation_failures_total` - Should be 0 (any value = potential attack) -- `pulse_proxy_read_timeouts_total` - Low is good (high = possible attack) -- `pulse_proxy_limiter_rejections_total` - Some is normal (very high = issue) -- `pulse_proxy_hostkey_changes_total` - Should be 0 (any value = investigate) -- `pulse_proxy_ssh_output_oversized_total` - Should be 0 (any value = investigate) - -### Operational Metrics -- `pulse_proxy_rpc_requests_total{method="get_temperature",result="success"}` - Should increase steadily -- `pulse_proxy_queue_depth` - Should be low (< 5) -- `pulse_proxy_global_concurrency_inflight` - Should be low (< 8) - ---- - -## Success Criteria - -✅ Deployment is successful when: - -- [ ] Proxy service is running and stable -- [ ] Socket is mounted read-only in container -- [ ] Temperature data appears in Pulse UI -- [ ] No errors in proxy or Pulse logs -- [ ] Security metrics show no anomalies -- [ ] All temperature pollers report healthy in scheduler health endpoint - ---- - -## Support - -**If issues persist:** - -1. Check full logs: `journalctl -u pulse-sensor-proxy -n 500 > /tmp/proxy-logs.txt` -2. Check configuration: `cat /etc/pulse-sensor-proxy/config.yaml` -3. Review audit report: `docs/SECURITY_AUDIT_2025-11-07.md` -4. File issue: https://github.com/rcourtman/Pulse/issues - ---- - -**Deployment checklist last updated:** 2025-11-07