Pulse/docs/operations/ADAPTIVE_POLLING_ROLLOUT.md

Adaptive Polling Rollout Playbook

This playbook guides operators through enabling the adaptive polling scheduler in production. Follow the steps sequentially and record key checkpoints in the run sheet for audit purposes.

---

1. Prerequisites

1. Test suite status

   • go test ./... and go test -tags=integration ./internal/monitoring -run TestAdaptiveSchedulerIntegration
   • Adaptive polling soak test:

     HARNESS_SOAK_MINUTES=15 go test -tags=integration ./internal/monitoring -run TestAdaptiveSchedulerSoak -soak -timeout 30m
     

   • All tests must pass within the last 24 hours.

2. Monitoring readiness

   • Grafana dashboard updated with:
     • pulse_monitor_poll_queue_depth (gauge)
     • pulse_monitor_poll_staleness_seconds (gauge, per instance)
     • pulse_monitor_poll_total and pulse_monitor_poll_errors_total (rate panels)
     • Alerting panels for circuit breaker state (via scheduler health API).
   • Alerts configured (see §4).

3. Configuration management

   • Ensure staging and production environments are managed via system.json or appropriate env vars.
   • Identify the operator owning flag toggles and service restarts.

4. Rollback plan

   • Confirm ability to set ADAPTIVE_POLLING_ENABLED=false and restart pulse-hot-dev or equivalent service within 5 minutes.
   • Document the systemctl restart pulse-hot-dev command path or container restart procedure.

5. Stakeholder sign-off

   • Adaptive polling feature owner approves rollout window.
   • SRE and on-call engineer acknowledge the playbook.

---

2. Staging Rollout

1. Enable feature flag

   • Update staging configuration:

     export ADAPTIVE_POLLING_ENABLED=true
     

     or edit system.json and set "adaptivePollingEnabled": true.
   • Restart hot-dev service / container to apply:

     systemctl restart pulse-hot-dev
     

     (Adapt to your env if using Docker/K8s.)

2. Verification

   • curl -s http://<staging-host>:7655/api/monitoring/scheduler/health | jq
     • Expect "enabled": true.
   • Check Grafana dashboard for the staging cluster:
     • Queue depth should stabilise near historic baseline (< instances × 1.5).
     • Staleness gauges should stay below 60 s for healthy instances.
     • No persistent circuit breakers (state != "closed") except known failing endpoints.

3. Observation window

   • Monitor for 24–48 hours.
   • Success criteria:
     • No increase in polling failures or alert volume.
     • Queue depth and staleness metrics remain within SLO (queue depth < 1.5× instance count, staleness < 60 s).
     • Scheduler health API shows empty dead-letter queue or expected entries only.
   • Record key metric snapshots at 0 h, 12 h, 24 h.

4. Sign-off

   • If criteria met, proceed to production. Otherwise revert flag to false and investigate (§6).

---

3. Production Rollout

1. Rollout strategy

   • Perform during low-traffic maintenance window.
   • Enable flag gradually by cluster or instance group (e.g., 25 % of nodes every 2 hours):
     1. Update config (ADAPTIVE_POLLING_ENABLED=true) for first subset.
     2. Restart service on those nodes.
     3. Watch metrics for at least 30 minutes before continuing.

2. Monitoring during rollout

   • Grafana dashboard per cluster:
     • poll_queue_depth
     • poll_staleness_seconds
     • poll_total success/error ratio
   • Scheduler health API:

     curl -s http://<prod-host>:7655/api/monitoring/scheduler/health | jq
     

     • Confirm enabled: true, deadLetter.count stable, breakers mostly empty.

3. Success criteria

   • Queue depth rises temporarily but settles within threshold (< 1.5× instance count).
   • Staleness stays below 60 s for healthy instances.
   • No unexplained increase in alert volume or API error rate.
   • Dead-letter queue holds only known failing targets.

4. Completion

   • After all nodes enabled, monitor for an additional 24 h.
   • Record final metric snapshot.

---

4. Grafana & Alert Configuration

1. Dashboard panels

   • Queue Depth: pulse_monitor_poll_queue_depth.
     • Use single-stat with alert if > 1.5× active instances for > 10 min.
   • Instance Staleness: panel per instance type using pulse_monitor_poll_staleness_seconds.
     • Alert threshold: > 60 s for > 5 min (excluding known failing instances).
   • Polling Throughput: rate of pulse_monitor_poll_total{result="success"} vs result="error".
   • Circuit Breakers / DLQ: table from scheduler health API (via scripted datasource) highlighting non-closed breakers or DLQ entries.

2. Alerts

   • Queue depth > threshold for >10 min (Warning), >20 min (Critical).
   • Staleness > 60 s for >5 min (Critical).
   • Dead-letter count increase > N (based on baseline) triggers Warning.
   • Any breaker stuck in open for >10 min triggers Critical.

3. Notification routing

   • Ensure alerts route to on-call + feature owner.

---

5. Rollback Procedure

1. Disable adaptive polling

   • Set ADAPTIVE_POLLING_ENABLED=false (env or system.json).
   • Restart service (systemctl restart pulse-hot-dev or equivalent).

2. Verification

   • Scheduler health API should show "enabled": false.
   • Queue depth returns to pre-feature baseline within 10–15 minutes.
   • Staleness/queue alerts clear.

3. Post-rollback actions

   • Notify stakeholders, capture metric snapshots showing recovery.
   • File incident report if rollback triggered by outage.

---

6. Troubleshooting

Symptom	Possible Cause	Action
Queue depth remains high (> 2× usual)	Insufficient workers, hidden breaker, misconfigured flag	Check scheduler health API for breaker states; consider increasing workers or reverting flag.
Staleness spikes across many instances	Backend API slowdown or connectivity issues	Inspect backend logs, network health; revert flag if duration > 15 min.
Dead-letter count climbs rapidly	Downstream API failures	Investigate specific instances via scheduler health API; fix credential/connectivity issues or rollback.
Circuit breakers stuck half-open/open	Persistent transient failures	Review error logs, ensure backoff/rate limits not starving retries; rollback if unresolved quickly.
Grafana panels flatline	Metrics exporter or job issue	Ensure Prometheus scraping working; verify service restarted with flag.

Accessing Scheduler Health API

curl -s http://<host>:7655/api/monitoring/scheduler/health | jq

Key sections to inspect:

• queue.depth, queue.perType
• instances[].pollStatus (success/failure streaks and last error)
• instances[].breaker (current breaker state, retry windows)
• instances[].deadLetter (reason, retry counts, schedules)
• staleness (normalized freshness score)

Common queries:

Instances with errors

curl -s http://<host>:7655/api/monitoring/scheduler/health \
  | jq '.instances[] | select(.pollStatus.lastError != null) | {key, lastError: .pollStatus.lastError}'

Current dead-letter entries

curl -s http://<host>:7655/api/monitoring/scheduler/health \
  | jq '.instances[] | select(.deadLetter.present) | {key, reason: .deadLetter.reason, retryCount: .deadLetter.retryCount}'

Breakers not closed

curl -s http://<host>:7655/api/monitoring/scheduler/health \
  | jq '.instances[] | select(.breaker.state != "closed") | {key, breaker: .breaker}'

When to Roll Back

Rollback immediately if any of the following occurs:

• Queue depth > 3× baseline for > 15 min.
• Staleness > 120 s on majority of instances.
• Dead-letter count doubles without clear cause.
• Customer-facing alerts or latency regressions attributed to adaptive polling.

Document the incident and notify stakeholders after rollback.