docs: Remove deployment checklist (per user request)
This commit is contained in:
rcourtman
parent
f9dc2f6466
commit
427cb383d8
1 changed files with 0 additions and 404 deletions
|
|
@ -1,404 +0,0 @@
|
|||
# Deployment Checklist - Security Hardening Update
|
||||
## Version: 2025-11-07 Security Hardening Release
|
||||
|
||||
---
|
||||
|
||||
## Pre-Deployment Checklist
|
||||
|
||||
### 1. Review Changes
|
||||
- [ ] Read `docs/SECURITY_AUDIT_2025-11-07.md` for full context
|
||||
- [ ] Review `docs/SECURITY_CHANGELOG.md` for breaking changes
|
||||
- [ ] Understand the `:ro` socket mount requirement
|
||||
|
||||
### 2. Backup Current State
|
||||
```bash
|
||||
# Backup configuration
|
||||
cp /etc/pulse-sensor-proxy/config.yaml /etc/pulse-sensor-proxy/config.yaml.backup
|
||||
|
||||
# Backup systemd unit
|
||||
cp /etc/systemd/system/pulse-sensor-proxy.service \
|
||||
/etc/systemd/system/pulse-sensor-proxy.service.backup
|
||||
|
||||
# Note current service status
|
||||
systemctl status pulse-sensor-proxy > /tmp/proxy-status-before.txt
|
||||
|
||||
# Backup docker-compose (if using Docker)
|
||||
cp docker-compose.yml docker-compose.yml.backup
|
||||
```
|
||||
|
||||
### 3. Verify Test Results
|
||||
- [ ] All tests pass: `go test ./cmd/pulse-sensor-proxy ./internal/ssh/knownhosts`
|
||||
- [ ] Binary builds: `go build ./cmd/pulse-sensor-proxy`
|
||||
- [ ] No compilation errors or warnings
|
||||
|
||||
---
|
||||
|
||||
## Deployment Steps
|
||||
|
||||
### For Docker Deployments
|
||||
|
||||
#### Step 1: Update docker-compose.yml
|
||||
```bash
|
||||
# Change socket mount from :rw to :ro
|
||||
sed -i 's|/run/pulse-sensor-proxy:rw|/run/pulse-sensor-proxy:ro|g' docker-compose.yml
|
||||
|
||||
# Verify change
|
||||
grep pulse-sensor-proxy docker-compose.yml
|
||||
# Should show: - /run/pulse-sensor-proxy:/run/pulse-sensor-proxy:ro
|
||||
```
|
||||
|
||||
#### Step 2: Optional Configuration
|
||||
Create `/etc/pulse-sensor-proxy/config.yaml` (optional but recommended):
|
||||
```yaml
|
||||
# Node allowlist (prevents SSRF)
|
||||
allowed_nodes:
|
||||
- "10.0.0.0/24" # Replace with your cluster network
|
||||
strict_node_validation: true
|
||||
|
||||
# Timeouts (prevents DoS)
|
||||
read_timeout: 5s
|
||||
write_timeout: 10s
|
||||
|
||||
# SSH output limits
|
||||
max_ssh_output_bytes: 1048576 # 1MB
|
||||
|
||||
# Host key management
|
||||
require_proxmox_hostkeys: false # Set true for strict mode
|
||||
|
||||
# Capability-based authorization (optional)
|
||||
allowed_peers:
|
||||
- uid: 0
|
||||
capabilities: [read, write, admin]
|
||||
- uid: 1000
|
||||
capabilities: [read] # Docker containers: read-only
|
||||
```
|
||||
|
||||
#### Step 3: Restart Services
|
||||
```bash
|
||||
# Restart proxy on host
|
||||
sudo systemctl restart pulse-sensor-proxy
|
||||
|
||||
# Check proxy status
|
||||
sudo systemctl status pulse-sensor-proxy
|
||||
|
||||
# Restart Pulse container
|
||||
docker compose down
|
||||
docker compose up -d
|
||||
|
||||
# Check container logs
|
||||
docker logs pulse -f
|
||||
# Look for: "Temperature proxy detected - using secure host-side bridge"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### For LXC Deployments
|
||||
|
||||
#### Step 1: Verify Mount Configuration
|
||||
```bash
|
||||
# Check mount is already correct (install script uses correct format)
|
||||
pct config <VMID> | grep mp
|
||||
|
||||
# Should see: mp0: /run/pulse-sensor-proxy,mp=/mnt/pulse-proxy
|
||||
# If shows old format with explicit :rw, update manually
|
||||
```
|
||||
|
||||
#### Step 2: Update Systemd Unit
|
||||
```bash
|
||||
# Copy new hardened unit
|
||||
sudo cp scripts/pulse-sensor-proxy.service /etc/systemd/system/
|
||||
|
||||
# Reload systemd
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Verify new directives loaded
|
||||
systemctl cat pulse-sensor-proxy | grep -E "(MemoryDeny|RestrictRealtime)"
|
||||
```
|
||||
|
||||
#### Step 3: Optional Configuration
|
||||
Create `/etc/pulse-sensor-proxy/config.yaml` (same as Docker above)
|
||||
|
||||
#### Step 4: Restart Proxy
|
||||
```bash
|
||||
sudo systemctl restart pulse-sensor-proxy
|
||||
sudo systemctl status pulse-sensor-proxy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification Steps
|
||||
|
||||
### 1. Proxy Health Check
|
||||
```bash
|
||||
# Check proxy is running
|
||||
sudo systemctl status pulse-sensor-proxy
|
||||
|
||||
# Check socket exists and is accessible
|
||||
ls -la /run/pulse-sensor-proxy/pulse-sensor-proxy.sock
|
||||
|
||||
# Test socket from host
|
||||
echo '{"method":"get_status"}' | \
|
||||
sudo socat - UNIX-CONNECT:/run/pulse-sensor-proxy/pulse-sensor-proxy.sock | jq .
|
||||
# Should return: {"success":true,"data":{"version":"dev","public_key":"...","ssh_dir":"..."}}
|
||||
```
|
||||
|
||||
### 2. Container Access Verification
|
||||
|
||||
**Docker:**
|
||||
```bash
|
||||
# Check socket is visible in container
|
||||
docker exec pulse ls -la /run/pulse-sensor-proxy/
|
||||
|
||||
# Verify container CANNOT write to directory (security fix)
|
||||
docker exec pulse touch /run/pulse-sensor-proxy/test 2>&1
|
||||
# Should fail with: "Read-only file system" (GOOD)
|
||||
|
||||
# Check Pulse logs for proxy detection
|
||||
docker logs pulse 2>&1 | grep -i "temperature.*proxy"
|
||||
# Should see: "Temperature proxy detected - using secure host-side bridge"
|
||||
```
|
||||
|
||||
**LXC:**
|
||||
```bash
|
||||
# Check socket is visible in container
|
||||
pct exec <VMID> -- ls -la /mnt/pulse-proxy/
|
||||
|
||||
# Verify container cannot write
|
||||
pct exec <VMID> -- touch /mnt/pulse-proxy/test 2>&1
|
||||
# Should fail (GOOD)
|
||||
|
||||
# Check Pulse logs
|
||||
pct exec <VMID> -- journalctl -u pulse -n 50 | grep -i temperature
|
||||
```
|
||||
|
||||
### 3. Temperature Data Collection
|
||||
```bash
|
||||
# Access Pulse UI: http://your-server:7655
|
||||
# Navigate to any Proxmox node
|
||||
# Verify temperature data is shown (if hardware supports it)
|
||||
|
||||
# Or check via API
|
||||
curl http://localhost:7655/api/nodes | jq '.[].temperature'
|
||||
```
|
||||
|
||||
### 4. Security Feature Verification
|
||||
|
||||
**Node Validation (if configured):**
|
||||
```bash
|
||||
# Check logs for validation (if you added allowed_nodes)
|
||||
sudo journalctl -u pulse-sensor-proxy -n 50 | grep -i validation
|
||||
```
|
||||
|
||||
**Metrics Check:**
|
||||
```bash
|
||||
# Check new security metrics exist
|
||||
curl -s http://localhost:7655/metrics | grep pulse_proxy_node_validation
|
||||
curl -s http://localhost:7655/metrics | grep pulse_proxy_read_timeouts
|
||||
curl -s http://localhost:7655/metrics | grep pulse_proxy_limiter
|
||||
|
||||
# Or check proxy metrics directly (if exposed)
|
||||
curl -s http://127.0.0.1:9127/metrics | grep pulse_proxy
|
||||
```
|
||||
|
||||
### 5. Rate Limiting Verification
|
||||
```bash
|
||||
# Check limiter is using range-based identification for containers
|
||||
sudo journalctl -u pulse-sensor-proxy -n 100 | grep -i "range:"
|
||||
# Should see logs like: peer="range:100000-165535" if container is accessing
|
||||
|
||||
# Check metrics
|
||||
curl -s http://127.0.0.1:9127/metrics | grep pulse_proxy_limiter_rejections_total
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Post-Deployment Monitoring
|
||||
|
||||
### Immediate (First Hour)
|
||||
- [ ] Watch proxy logs: `journalctl -u pulse-sensor-proxy -f`
|
||||
- [ ] Watch Pulse logs: `docker logs pulse -f` or `journalctl -u pulse -f`
|
||||
- [ ] Check for errors or warnings
|
||||
- [ ] Verify temperature data is updating in UI
|
||||
|
||||
### First Day
|
||||
- [ ] Monitor for security events in logs
|
||||
- [ ] Check metrics for anomalies
|
||||
- [ ] Verify no legitimate requests being blocked
|
||||
|
||||
### First Week
|
||||
- [ ] Review security metrics trends
|
||||
- [ ] Check for any false positives in node validation
|
||||
- [ ] Adjust configuration if needed
|
||||
|
||||
---
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
If issues occur:
|
||||
|
||||
### Quick Rollback (Docker)
|
||||
```bash
|
||||
# Restore old docker-compose
|
||||
cp docker-compose.yml.backup docker-compose.yml
|
||||
|
||||
# Restart
|
||||
docker compose down && docker compose up -d
|
||||
```
|
||||
|
||||
### Quick Rollback (LXC)
|
||||
```bash
|
||||
# Restore old systemd unit
|
||||
sudo cp /etc/systemd/system/pulse-sensor-proxy.service.backup \
|
||||
/etc/systemd/system/pulse-sensor-proxy.service
|
||||
|
||||
# Restore old config (if needed)
|
||||
sudo cp /etc/pulse-sensor-proxy/config.yaml.backup \
|
||||
/etc/pulse-sensor-proxy/config.yaml
|
||||
|
||||
# Reload and restart
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart pulse-sensor-proxy
|
||||
```
|
||||
|
||||
### Git Rollback
|
||||
```bash
|
||||
# Revert to previous commit
|
||||
git revert HEAD
|
||||
git push
|
||||
|
||||
# Rebuild and redeploy
|
||||
go build ./cmd/pulse-sensor-proxy
|
||||
sudo systemctl restart pulse-sensor-proxy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: "Temperature proxy not detected"
|
||||
|
||||
**Symptom:** Pulse logs don't show proxy detection message
|
||||
|
||||
**Check:**
|
||||
```bash
|
||||
# 1. Verify proxy is running
|
||||
sudo systemctl status pulse-sensor-proxy
|
||||
|
||||
# 2. Verify socket exists
|
||||
ls -la /run/pulse-sensor-proxy/pulse-sensor-proxy.sock
|
||||
|
||||
# 3. Verify container can see socket
|
||||
docker exec pulse ls -la /run/pulse-sensor-proxy/ # Docker
|
||||
pct exec <VMID> -- ls -la /mnt/pulse-proxy/ # LXC
|
||||
|
||||
# 4. Check mount is correct
|
||||
docker inspect pulse | grep -A 5 Mounts # Docker
|
||||
pct config <VMID> | grep mp # LXC
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
- Ensure socket directory is mounted into container
|
||||
- Verify mount path matches Pulse expectation
|
||||
- Restart both proxy and Pulse
|
||||
|
||||
---
|
||||
|
||||
### Issue: "Read-only file system" errors in container
|
||||
|
||||
**Symptom:** Container logs show read-only errors for `/run/pulse-sensor-proxy/`
|
||||
|
||||
**This is EXPECTED and CORRECT!** The mount is intentionally read-only for security.
|
||||
|
||||
**If Pulse itself has issues:**
|
||||
- Pulse should only READ from the socket, not write
|
||||
- Check Pulse isn't trying to write to the socket directory
|
||||
- This should not happen in normal operation
|
||||
|
||||
---
|
||||
|
||||
### Issue: "Node validation failed" or "potential SSRF attempt"
|
||||
|
||||
**Symptom:** Temperature requests failing, logs show validation errors
|
||||
|
||||
**Check:**
|
||||
```bash
|
||||
# Check your allowed_nodes configuration
|
||||
cat /etc/pulse-sensor-proxy/config.yaml | grep -A 5 allowed_nodes
|
||||
|
||||
# Check what nodes Pulse is trying to connect to
|
||||
sudo journalctl -u pulse-sensor-proxy | grep "potential SSRF"
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
- Add legitimate nodes to `allowed_nodes` list
|
||||
- Or set `strict_node_validation: false` temporarily for debugging
|
||||
- Verify node hostnames/IPs are correct
|
||||
|
||||
---
|
||||
|
||||
### Issue: High rate limit rejections
|
||||
|
||||
**Symptom:** Metrics show many `pulse_proxy_limiter_rejections_total`
|
||||
|
||||
**Check:**
|
||||
```bash
|
||||
# Check which peers are being limited
|
||||
curl -s http://127.0.0.1:9127/metrics | grep limiter_rejections_total
|
||||
|
||||
# Check logs for details
|
||||
sudo journalctl -u pulse-sensor-proxy | grep "Rate limit"
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
- This may be normal if you have many nodes
|
||||
- Rate limits are per-container-range (not per-UID) now
|
||||
- Adjust `rate_limit` config if needed:
|
||||
```yaml
|
||||
rate_limit:
|
||||
per_peer_interval_ms: 500 # Increase to 2 req/sec
|
||||
per_peer_burst: 10 # Allow larger bursts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Metrics to Monitor
|
||||
|
||||
### Security Metrics
|
||||
- `pulse_proxy_node_validation_failures_total` - Should be 0 (any value = potential attack)
|
||||
- `pulse_proxy_read_timeouts_total` - Low is good (high = possible attack)
|
||||
- `pulse_proxy_limiter_rejections_total` - Some is normal (very high = issue)
|
||||
- `pulse_proxy_hostkey_changes_total` - Should be 0 (any value = investigate)
|
||||
- `pulse_proxy_ssh_output_oversized_total` - Should be 0 (any value = investigate)
|
||||
|
||||
### Operational Metrics
|
||||
- `pulse_proxy_rpc_requests_total{method="get_temperature",result="success"}` - Should increase steadily
|
||||
- `pulse_proxy_queue_depth` - Should be low (< 5)
|
||||
- `pulse_proxy_global_concurrency_inflight` - Should be low (< 8)
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
✅ Deployment is successful when:
|
||||
|
||||
- [ ] Proxy service is running and stable
|
||||
- [ ] Socket is mounted read-only in container
|
||||
- [ ] Temperature data appears in Pulse UI
|
||||
- [ ] No errors in proxy or Pulse logs
|
||||
- [ ] Security metrics show no anomalies
|
||||
- [ ] All temperature pollers report healthy in scheduler health endpoint
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
**If issues persist:**
|
||||
|
||||
1. Check full logs: `journalctl -u pulse-sensor-proxy -n 500 > /tmp/proxy-logs.txt`
|
||||
2. Check configuration: `cat /etc/pulse-sensor-proxy/config.yaml`
|
||||
3. Review audit report: `docs/SECURITY_AUDIT_2025-11-07.md`
|
||||
4. File issue: https://github.com/rcourtman/Pulse/issues
|
||||
|
||||
---
|
||||
|
||||
**Deployment checklist last updated:** 2025-11-07
|
||||
Loading…
Reference in a new issue