Pulse/docs/DEPLOYMENT_CHECKLIST.md

Deployment Checklist - Security Hardening Update

Version: 2025-11-07 Security Hardening Release

---

Pre-Deployment Checklist

1. Review Changes

• Read docs/SECURITY_AUDIT_2025-11-07.md for full context
• Review docs/SECURITY_CHANGELOG.md for breaking changes
• Understand the :ro socket mount requirement

2. Backup Current State

# Backup configuration
cp /etc/pulse-sensor-proxy/config.yaml /etc/pulse-sensor-proxy/config.yaml.backup

# Backup systemd unit
cp /etc/systemd/system/pulse-sensor-proxy.service \
   /etc/systemd/system/pulse-sensor-proxy.service.backup

# Note current service status
systemctl status pulse-sensor-proxy > /tmp/proxy-status-before.txt

# Backup docker-compose (if using Docker)
cp docker-compose.yml docker-compose.yml.backup

3. Verify Test Results

• All tests pass: go test ./cmd/pulse-sensor-proxy ./internal/ssh/knownhosts
• Binary builds: go build ./cmd/pulse-sensor-proxy
• No compilation errors or warnings

---

Deployment Steps

For Docker Deployments

Step 1: Update docker-compose.yml

# Change socket mount from :rw to :ro
sed -i 's|/run/pulse-sensor-proxy:rw|/run/pulse-sensor-proxy:ro|g' docker-compose.yml

# Verify change
grep pulse-sensor-proxy docker-compose.yml
# Should show: - /run/pulse-sensor-proxy:/run/pulse-sensor-proxy:ro

Step 2: Optional Configuration

Create /etc/pulse-sensor-proxy/config.yaml (optional but recommended):

# Node allowlist (prevents SSRF)
allowed_nodes:
  - "10.0.0.0/24"  # Replace with your cluster network
strict_node_validation: true

# Timeouts (prevents DoS)
read_timeout: 5s
write_timeout: 10s

# SSH output limits
max_ssh_output_bytes: 1048576  # 1MB

# Host key management
require_proxmox_hostkeys: false  # Set true for strict mode

# Capability-based authorization (optional)
allowed_peers:
  - uid: 0
    capabilities: [read, write, admin]
  - uid: 1000
    capabilities: [read]  # Docker containers: read-only

Step 3: Restart Services

# Restart proxy on host
sudo systemctl restart pulse-sensor-proxy

# Check proxy status
sudo systemctl status pulse-sensor-proxy

# Restart Pulse container
docker compose down
docker compose up -d

# Check container logs
docker logs pulse -f
# Look for: "Temperature proxy detected - using secure host-side bridge"

---

For LXC Deployments

Step 1: Verify Mount Configuration

# Check mount is already correct (install script uses correct format)
pct config <VMID> | grep mp

# Should see: mp0: /run/pulse-sensor-proxy,mp=/mnt/pulse-proxy
# If shows old format with explicit :rw, update manually

Step 2: Update Systemd Unit

# Copy new hardened unit
sudo cp scripts/pulse-sensor-proxy.service /etc/systemd/system/

# Reload systemd
sudo systemctl daemon-reload

# Verify new directives loaded
systemctl cat pulse-sensor-proxy | grep -E "(MemoryDeny|RestrictRealtime)"

Step 3: Optional Configuration

Create /etc/pulse-sensor-proxy/config.yaml (same as Docker above)

Step 4: Restart Proxy

sudo systemctl restart pulse-sensor-proxy
sudo systemctl status pulse-sensor-proxy

---

Verification Steps

1. Proxy Health Check

# Check proxy is running
sudo systemctl status pulse-sensor-proxy

# Check socket exists and is accessible
ls -la /run/pulse-sensor-proxy/pulse-sensor-proxy.sock

# Test socket from host
echo '{"method":"get_status"}' | \
  sudo socat - UNIX-CONNECT:/run/pulse-sensor-proxy/pulse-sensor-proxy.sock | jq .
# Should return: {"success":true,"data":{"version":"dev","public_key":"...","ssh_dir":"..."}}

2. Container Access Verification

Docker:

# Check socket is visible in container
docker exec pulse ls -la /run/pulse-sensor-proxy/

# Verify container CANNOT write to directory (security fix)
docker exec pulse touch /run/pulse-sensor-proxy/test 2>&1
# Should fail with: "Read-only file system" (GOOD)

# Check Pulse logs for proxy detection
docker logs pulse 2>&1 | grep -i "temperature.*proxy"
# Should see: "Temperature proxy detected - using secure host-side bridge"

LXC:

# Check socket is visible in container
pct exec <VMID> -- ls -la /mnt/pulse-proxy/

# Verify container cannot write
pct exec <VMID> -- touch /mnt/pulse-proxy/test 2>&1
# Should fail (GOOD)

# Check Pulse logs
pct exec <VMID> -- journalctl -u pulse -n 50 | grep -i temperature

3. Temperature Data Collection

# Access Pulse UI: http://your-server:7655
# Navigate to any Proxmox node
# Verify temperature data is shown (if hardware supports it)

# Or check via API
curl http://localhost:7655/api/nodes | jq '.[].temperature'

4. Security Feature Verification

Node Validation (if configured):

# Check logs for validation (if you added allowed_nodes)
sudo journalctl -u pulse-sensor-proxy -n 50 | grep -i validation

Metrics Check:

# Check new security metrics exist
curl -s http://localhost:7655/metrics | grep pulse_proxy_node_validation
curl -s http://localhost:7655/metrics | grep pulse_proxy_read_timeouts
curl -s http://localhost:7655/metrics | grep pulse_proxy_limiter

# Or check proxy metrics directly (if exposed)
curl -s http://127.0.0.1:9127/metrics | grep pulse_proxy

5. Rate Limiting Verification

# Check limiter is using range-based identification for containers
sudo journalctl -u pulse-sensor-proxy -n 100 | grep -i "range:"
# Should see logs like: peer="range:100000-165535" if container is accessing

# Check metrics
curl -s http://127.0.0.1:9127/metrics | grep pulse_proxy_limiter_rejections_total

---

Post-Deployment Monitoring

Immediate (First Hour)

• Watch proxy logs: journalctl -u pulse-sensor-proxy -f
• Watch Pulse logs: docker logs pulse -f or journalctl -u pulse -f
• Check for errors or warnings
• Verify temperature data is updating in UI

First Day

• Monitor for security events in logs
• Check metrics for anomalies
• Verify no legitimate requests being blocked

First Week

• Review security metrics trends
• Check for any false positives in node validation
• Adjust configuration if needed

---

Rollback Procedure

If issues occur:

Quick Rollback (Docker)

# Restore old docker-compose
cp docker-compose.yml.backup docker-compose.yml

# Restart
docker compose down && docker compose up -d

Quick Rollback (LXC)

# Restore old systemd unit
sudo cp /etc/systemd/system/pulse-sensor-proxy.service.backup \
        /etc/systemd/system/pulse-sensor-proxy.service

# Restore old config (if needed)
sudo cp /etc/pulse-sensor-proxy/config.yaml.backup \
        /etc/pulse-sensor-proxy/config.yaml

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart pulse-sensor-proxy

Git Rollback

# Revert to previous commit
git revert HEAD
git push

# Rebuild and redeploy
go build ./cmd/pulse-sensor-proxy
sudo systemctl restart pulse-sensor-proxy

---

Troubleshooting

Issue: "Temperature proxy not detected"

Symptom: Pulse logs don't show proxy detection message

Check:

# 1. Verify proxy is running
sudo systemctl status pulse-sensor-proxy

# 2. Verify socket exists
ls -la /run/pulse-sensor-proxy/pulse-sensor-proxy.sock

# 3. Verify container can see socket
docker exec pulse ls -la /run/pulse-sensor-proxy/  # Docker
pct exec <VMID> -- ls -la /mnt/pulse-proxy/        # LXC

# 4. Check mount is correct
docker inspect pulse | grep -A 5 Mounts  # Docker
pct config <VMID> | grep mp               # LXC

Fix:

• Ensure socket directory is mounted into container
• Verify mount path matches Pulse expectation
• Restart both proxy and Pulse

---

Issue: "Read-only file system" errors in container

Symptom: Container logs show read-only errors for /run/pulse-sensor-proxy/

This is EXPECTED and CORRECT! The mount is intentionally read-only for security.

If Pulse itself has issues:

• Pulse should only READ from the socket, not write
• Check Pulse isn't trying to write to the socket directory
• This should not happen in normal operation

---

Issue: "Node validation failed" or "potential SSRF attempt"

Symptom: Temperature requests failing, logs show validation errors

Check:

# Check your allowed_nodes configuration
cat /etc/pulse-sensor-proxy/config.yaml | grep -A 5 allowed_nodes

# Check what nodes Pulse is trying to connect to
sudo journalctl -u pulse-sensor-proxy | grep "potential SSRF"

Fix:

• Add legitimate nodes to allowed_nodes list
• Or set strict_node_validation: false temporarily for debugging
• Verify node hostnames/IPs are correct

---

Issue: High rate limit rejections

Symptom: Metrics show many pulse_proxy_limiter_rejections_total

Check:

# Check which peers are being limited
curl -s http://127.0.0.1:9127/metrics | grep limiter_rejections_total

# Check logs for details
sudo journalctl -u pulse-sensor-proxy | grep "Rate limit"

Fix:

• This may be normal if you have many nodes
• Rate limits are per-container-range (not per-UID) now
• Adjust rate_limit config if needed:

  rate_limit:
    per_peer_interval_ms: 500  # Increase to 2 req/sec
    per_peer_burst: 10         # Allow larger bursts
  

---

Metrics to Monitor

Security Metrics

• pulse_proxy_node_validation_failures_total - Should be 0 (any value = potential attack)
• pulse_proxy_read_timeouts_total - Low is good (high = possible attack)
• pulse_proxy_limiter_rejections_total - Some is normal (very high = issue)
• pulse_proxy_hostkey_changes_total - Should be 0 (any value = investigate)
• pulse_proxy_ssh_output_oversized_total - Should be 0 (any value = investigate)

Operational Metrics

• pulse_proxy_rpc_requests_total{method="get_temperature",result="success"} - Should increase steadily
• pulse_proxy_queue_depth - Should be low (< 5)
• pulse_proxy_global_concurrency_inflight - Should be low (< 8)

---

Success Criteria

✅ Deployment is successful when:

• Proxy service is running and stable
• Socket is mounted read-only in container
• Temperature data appears in Pulse UI
• No errors in proxy or Pulse logs
• Security metrics show no anomalies
• All temperature pollers report healthy in scheduler health endpoint

---

Support

If issues persist:

1. Check full logs: journalctl -u pulse-sensor-proxy -n 500 > /tmp/proxy-logs.txt
2. Check configuration: cat /etc/pulse-sensor-proxy/config.yaml
3. Review audit report: docs/SECURITY_AUDIT_2025-11-07.md
4. File issue: https://github.com/rcourtman/Pulse/issues

---

Deployment checklist last updated: 2025-11-07