rcourtman
|
cb8be81f1d
|
docs: add adaptive polling production rollout playbook (Phase 2 Task 10)
Add comprehensive operator playbook for production enablement:
**Prerequisites:**
- Test suite validation (unit, integration, soak)
- Monitoring readiness (Grafana dashboards, alerts)
- Configuration management and rollback planning
- Stakeholder sign-off
**Staging Rollout:**
- Feature flag enablement steps
- Verification procedures (scheduler health API)
- 24-48h observation window with success criteria
- Metric checkpoints at 0h, 12h, 24h
**Production Rollout:**
- Gradual strategy (25% nodes every 2 hours)
- Low-traffic maintenance window
- Per-cluster monitoring during rollout
- Success criteria and completion validation
**Grafana/Alert Configuration:**
- Dashboard panels: queue depth, staleness, throughput, breakers/DLQ
- Alert thresholds:
- Queue depth > 1.5× instances for >10min (Warning)
- Staleness > 60s for >5min (Critical)
- DLQ growth (Warning)
- Stuck breakers >10min (Critical)
**Rollback Procedure:**
- Clear disable/restart steps
- Verification of rollback success
- Post-rollback actions and incident reporting
**Troubleshooting:**
- Symptom/cause/action table
- Scheduler health API access guide
- Immediate rollback triggers
Operators can now safely enable adaptive polling following this step-by-step playbook.
Part of Phase 2 Task 10 (Documentation)
|
2025-10-20 15:13:38 +00:00 |
|