Pulse/docs/operations/audit-log-rotation.md

Pulse Sensor Proxy Audit Log Rotation

The sensor proxy writes a tamper-evident audit trail to /var/log/pulse/sensor-proxy/audit.log. Every entry includes the SHA-256 hash of the previous entry, so any modification becomes obvious. Because the process keeps the file open and maintains the running hash in memory, rotation requires special handling.

Rotation Strategy

Use logrotate to rotate the file once it reaches 100 MB. After each rotation, restart the proxy so it opens a new file and starts a fresh hash chain.

Create /etc/logrotate.d/pulse-sensor-proxy with the following contents:

/var/log/pulse/sensor-proxy/audit.log {
    daily
    size 100M
    rotate 90
    compress
    delaycompress
    missingok
    notifempty
    create 0640 pulse pulse
    sharedscripts
    postrotate
        systemctl restart pulse-sensor-proxy.service >/dev/null 2>&1 || true
    endscript
}

Why a Restart Is Mandatory

copytruncate and similar tricks break the chain integrity. Restarting the service ensures:

1. The proxy releases the old file descriptor.
2. A new hash chain starts at sequence 1 with an all-zero prev_hash.

If the proxy is not restarted, it will continue writing to the renamed file and the rotation will have no effect.

Chain Continuity Across Rotations

Each rotated log (audit.log.1.gz, audit.log.2.gz, …) is self-contained. To prove continuity between files:

1. After each rotation, record the final event_hash from the rotated file (for example, store it in the filename or a checksum manifest).
2. When reviewing logs, verify the prev_hash of the first entry in the new file is the zero hash, and reconcile the recorded final hash from the prior file to show no entries were removed.

Maintaining this “final hash ledger” allows auditors to stitch the rotated files together chronologically while preserving the tamper-evident guarantees.

Permissions

Adjust the create directive to match the user and group that run the sensor proxy. The example assumes both user and group are pulse.

---

Post-Rotation Health Checks (v4.24.0+)

After rotating audit logs and restarting pulse-sensor-proxy, verify adaptive polling health:

1. Check Scheduler Health

curl -s http://localhost:7655/api/monitoring/scheduler/health | jq

Verify:

• Temperature proxy pollers appear in instances[] array
• pollStatus.lastSuccess is recent (within last 60 seconds)
• No new entries in deadLetter queue for proxy instances
• breaker.state is closed for proxy nodes

Example check for proxy instances:

curl -s http://localhost:7655/api/monitoring/scheduler/health \
  | jq '.instances[] | select(.type == "proxy" or .connection | contains("proxy")) | {key, lastSuccess: .pollStatus.lastSuccess, breaker: .breaker.state}'

2. Monitor Metrics (10-15 minutes)

Watch these metrics to ensure proxy restart didn't cause issues:

# Queue depth should remain stable
curl -s http://localhost:7655/api/monitoring/scheduler/health | jq '.queue.depth'

# Check staleness for proxy instances
curl -s http://localhost:7655/api/monitoring/scheduler/health \
  | jq '.instances[] | select(.type == "proxy") | {key, staleness: .pollStatus.lastSuccess}'

Expected behavior:

• Queue depth: No significant spike (< 10 temporary increase acceptable)
• Staleness: Proxy instances show fresh polls within 30-60 seconds
• No circuit breaker trips for proxy instances
• No new DLQ entries

3. Cross-Reference Audit Logs

Link rotation events with scheduler health for security review:

# Check Pulse audit log for rotation timing
journalctl -u pulse-sensor-proxy --since "10 minutes ago" | grep -E "restart|rotation"

# Check update history for any concurrent events
curl -s http://localhost:7655/api/updates/history?limit=5 | jq '.entries[] | {action, timestamp, status}'

Why this matters:

• Security auditors can correlate proxy restarts with scheduler behavior
• Update rollbacks may be concurrent with log rotations
• Rollback metadata (new in v4.24.0) provides full operational context
• Ensures restart didn't mask polling failures or breaker trips

4. Troubleshooting Rotation Issues

If proxy instances don't rejoin queue:

1. Check service status

   systemctl status pulse-sensor-proxy
   

2. Verify scheduler sees the proxy

   curl -s http://localhost:7655/api/monitoring/scheduler/health \
     | jq '.instances[] | select(.type == "proxy")'
   

3. Check for circuit breakers

   curl -s http://localhost:7655/api/monitoring/scheduler/health \
     | jq '.instances[] | select(.breaker.state != "closed") | {key, state: .breaker.state, retryAt: .breaker.retryAt}'
   

4. Review logs for errors

   journalctl -u pulse-sensor-proxy -n 50
   journalctl -u pulse | grep -E "proxy|temperature"
   

Recovery actions:

• If breakers are stuck: Restart main Pulse service (systemctl restart pulse)
• If DLQ entries persist: Check proxy credentials and network connectivity
• If polling doesn't resume: Verify proxy configuration in Settings → Sensors

---

Related Documentation

• Scheduler Health API - Complete API reference
• Adaptive Polling Operations - Health monitoring procedures
• Pulse Sensor Proxy Hardening - Security configuration
• Temperature Monitoring Security - Proxy-specific security notes