danvics/Pulse
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
171 commits 1 branch 0 tags 55 MiB
Commit graph

2 commits

Author SHA1 Message Date
rcourtman
469d11fc7e docs: add comprehensive scheduler health API documentation 
Add detailed API reference and update rollout playbook:

**New: docs/api/SCHEDULER_HEALTH.md**
- Complete endpoint reference for /api/monitoring/scheduler/health
- Request/response structure with field descriptions
- Enhanced "instances" array documentation
- Example responses showing all states (healthy, transient, DLQ)
- Useful jq queries for troubleshooting:
  - Find instances with errors
  - List DLQ entries
  - Show open circuit breakers
  - Sort by failure streaks
- Migration guide (legacy → new fields)
- Troubleshooting examples with real scenarios

**Updated: docs/operations/ADAPTIVE_POLLING_ROLLOUT.md**
- Enhanced "Accessing Scheduler Health API" section (§6)
- Added examples using new instances[] array
- Updated queries to use pollStatus, breaker, deadLetter fields
- Practical jq commands for operators

**Key Documentation Features:**
- Complete JSON schema with examples
- All new fields documented with types and descriptions
- Real-world troubleshooting scenarios
- Copy-paste ready jq queries
- Migration path for existing integrations
- Backward compatibility notes

Operators can now:
- Find error messages without log digging
- Understand circuit breaker states
- Track DLQ entries with full context
- Diagnose issues using single API call

Part of Phase 2 follow-up - enhanced observability
 2025-10-20 15:13:38 +00:00
rcourtman
cb8be81f1d docs: add adaptive polling production rollout playbook (Phase 2 Task 10) 
Add comprehensive operator playbook for production enablement:

**Prerequisites:**
- Test suite validation (unit, integration, soak)
- Monitoring readiness (Grafana dashboards, alerts)
- Configuration management and rollback planning
- Stakeholder sign-off

**Staging Rollout:**
- Feature flag enablement steps
- Verification procedures (scheduler health API)
- 24-48h observation window with success criteria
- Metric checkpoints at 0h, 12h, 24h

**Production Rollout:**
- Gradual strategy (25% nodes every 2 hours)
- Low-traffic maintenance window
- Per-cluster monitoring during rollout
- Success criteria and completion validation

**Grafana/Alert Configuration:**
- Dashboard panels: queue depth, staleness, throughput, breakers/DLQ
- Alert thresholds:
  - Queue depth > 1.5× instances for >10min (Warning)
  - Staleness > 60s for >5min (Critical)
  - DLQ growth (Warning)
  - Stuck breakers >10min (Critical)

**Rollback Procedure:**
- Clear disable/restart steps
- Verification of rollback success
- Post-rollback actions and incident reporting

**Troubleshooting:**
- Symptom/cause/action table
- Scheduler health API access guide
- Immediate rollback triggers

Operators can now safely enable adaptive polling following this step-by-step playbook.

Part of Phase 2 Task 10 (Documentation)
 2025-10-20 15:13:38 +00:00