85ffe10aed
Enhance documentation with six Mermaid diagrams to better explain complex system implementations: - Adaptive polling lifecycle flowchart showing enqueue→execute→feedback cycle with scheduler, priority queue, and worker interactions - Circuit breaker state machine diagram illustrating Closed↔Open↔Half-open transitions with triggers and recovery paths - Temperature proxy architecture diagram highlighting trust boundaries, security controls, and data flow between host/container/cluster - Sensor proxy request flow sequence diagram showing auth, rate limiting, validation, and SSH execution pipeline - Alert webhook pipeline flowchart detailing template resolution, URL rendering, HTTP dispatch, and retry logic - Script library workflow diagram illustrating dev→test→bundle→distribute lifecycle emphasizing modular design These visualizations make it easier for operators and contributors to understand Pulse's sophisticated architectural patterns.
108 lines
5.1 KiB
Markdown
108 lines
5.1 KiB
Markdown
# Pulse Sensor Proxy Runbook
|
||
|
||
## Quick Reference
|
||
- Binary: `/opt/pulse/sensor-proxy/bin/pulse-sensor-proxy`
|
||
- Unit: `pulse-sensor-proxy.service`
|
||
- Logs: `/var/log/pulse/sensor-proxy/proxy.log`
|
||
- Audit trail: `/var/log/pulse/sensor-proxy/audit.log` (hash chained, forwarded via rsyslog)
|
||
- Metrics: `http://127.0.0.1:9127/metrics` (set `PULSE_SENSOR_PROXY_METRICS_ADDR` to change/disable)
|
||
- Limiters: ~12 requests/minute per UID (burst 2), per-UID concurrency 2, global concurrency 8, 2 s penalty on validation failures
|
||
|
||
## Monitoring Alerts & Response
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant Backend as Pulse Backend
|
||
participant Proxy as Sensor Proxy RPC Server
|
||
participant Limiter as Limiter (per UID & global)
|
||
participant Validator as Payload Validator
|
||
participant SSH as Cluster Node (forced `sensors -j`)
|
||
participant Metrics as Metrics & Audit Log
|
||
|
||
Backend->>Proxy: RPC request (get_temperature)
|
||
Proxy->>Proxy: Extract SO_PEERCRED (UID/GID/PID)
|
||
Proxy->>Limiter: Check per-UID rate & concurrency
|
||
alt Rate limit exceeded
|
||
Limiter-->>Proxy: reject
|
||
Proxy-->>Backend: 429 Too Many Requests (2 s penalty)
|
||
Proxy->>Metrics: increment limiter_rejections_total
|
||
else Allowed
|
||
Limiter-->>Proxy: permit
|
||
Proxy->>Validator: Validate method & payload
|
||
alt Validation failure
|
||
Validator-->>Proxy: error
|
||
Proxy-->>Backend: 400 validation error
|
||
Proxy->>Metrics: penalty + audit log entry
|
||
else Valid request
|
||
Validator-->>Proxy: ok
|
||
Proxy->>SSH: run `sensors -j` via forced command
|
||
SSH-->>Proxy: temperature JSON
|
||
Proxy-->>Backend: telemetry payload
|
||
Proxy->>Metrics: record success, latency histogram
|
||
Proxy->>Metrics: append audit/audit trail
|
||
end
|
||
end
|
||
```
|
||
|
||
### Rate Limit Hits (`pulse_proxy_limiter_rejections_total`)
|
||
1. Check audit log entries tagged `limiter.rejection` for offending UID.
|
||
2. Confirm workload legitimacy; if expected, consider increasing limits via config override.
|
||
3. If malicious, block source process/user and inspect Pulse audit logs.
|
||
|
||
### Penalty Events (`pulse_proxy_limiter_penalties_total`)
|
||
1. Review corresponding validation failures in audit log (`command.validation_failed`).
|
||
2. If repeated invalid JSON/unknown methods, inspect caller code for regressions or intrusion attempts.
|
||
|
||
### Audit Log Forwarder Down
|
||
1. `journalctl -u rsyslog` to confirm transmission errors.
|
||
2. Ensure `/etc/pulse/log-forwarding` certs valid & remote host reachable.
|
||
3. Forwarding queue stored locally in `/var/log/pulse/sensor-proxy/forwarding.log`; ship manually if outage exceeds 1 hour.
|
||
|
||
### Proxy Health Endpoint Fails
|
||
1. `systemctl status pulse-sensor-proxy`
|
||
2. Check `/var/log/pulse/sensor-proxy/proxy.log` for panic or limiter exhaustion.
|
||
3. Inspect `/var/log/pulse/sensor-proxy/audit.log` for recent privileged method denials.
|
||
|
||
## Standard Procedures
|
||
### Restart Proxy Safely
|
||
```bash
|
||
sudo systemctl stop pulse-sensor-proxy
|
||
sudo apparmor_parser -r /etc/apparmor.d/pulse-sensor-proxy # if updating policy
|
||
sudo systemctl start pulse-sensor-proxy
|
||
```
|
||
Verify:
|
||
```bash
|
||
# Metrics endpoint exposes proxy build/health
|
||
curl -s http://127.0.0.1:9127/metrics | grep pulse_proxy_build_info
|
||
|
||
# Ensure adaptive polling sees the proxy again
|
||
curl -s http://localhost:7655/api/monitoring/scheduler/health \
|
||
| jq '.instances[] | select(.key | contains("temperature")) | {key, pollStatus}'
|
||
```
|
||
Temperature instances should show recent `lastSuccess` timestamps with no DLQ entries.
|
||
|
||
### Rotate SSH Keys
|
||
1. Run `scripts/secure-sensor-files.sh` to regenerate keys (ensure environment locked down).
|
||
2. Use RPC `ensure_cluster_keys` to distribute new public key.
|
||
3. Confirm nodes accept `ssh` from proxy host.
|
||
4. Confirm the scheduler clears any temporary breakers/dlq entries:
|
||
```bash
|
||
curl -s http://localhost:7655/api/monitoring/scheduler/health \
|
||
| jq '.instances[] | select(.key | contains("temperature")) | {key, breaker: .breaker.state, deadLetter: .deadLetter.present}'
|
||
```
|
||
Expect `breaker.state=="closed"` and `deadLetter.present==false` for all proxy-driven pollers.
|
||
|
||
### Adjust Rate Limits
|
||
1. Update `limiter_policy` environment overrides (future config).
|
||
2. Restart proxy; monitor limiter metrics to validate new thresholds.
|
||
3. Document change in security runbook.
|
||
|
||
## Incident Handling
|
||
- **Unauthorized Command Attempt:** audit log shows `command.validation_failed` and limiter penalties; capture correlation ID, check Pulse side for compromised container.
|
||
- **Excessive Temperature Failures:** refer to `pulse_proxy_ssh_requests_total{result="error"}`; validate network ACLs and node health; escalate to Proxmox team if nodes unreachable.
|
||
- **Log Tampering Suspected:** verify audit hash chain by replaying `eventHash` values; compare with remote log store (immutable). Trigger security response if mismatch.
|
||
|
||
## Postmortem Checklist
|
||
- Timeline: command audit entries, limiter stats, rsyslog queue depth.
|
||
- Verify AppArmor/seccomp status (`aa-status`, `systemctl show pulse-sensor-proxy -p AppArmorProfile`).
|
||
- Ensure firewall ACLs match `docs/security/pulse-sensor-proxy-network.md`.
|