docs: defer circuit breaker/DLQ management endpoints (Phase 2 Task 11)
Document decision to defer mutation endpoints after soak testing:
**Assessment Results:**
- Integration tests (55s, 12 instances): Automatic recovery worked perfectly
- Soak tests (2-240min, 80 instances): No manual intervention needed
- Circuit breakers: Opened/closed automatically as designed
- DLQ routing: Permanent failures handled correctly
**Current Capabilities (Sufficient):**
- Read-only scheduler health API provides full visibility
- Operator workarounds: service restart, feature flag toggle
- Grafana alerting: queue depth, staleness, DLQ, breakers
**Why Defer:**
- No operational need demonstrated in testing
- Implementation requires auth/RBAC/audit/UI work
- Cost not justified until production usage reveals need
- Can add later when data shows actual pain points
**Future Design Notes:**
- POST /api/monitoring/breakers/{instance}/reset
- POST /api/monitoring/dlq/retry (all or specific)
- DELETE /api/monitoring/dlq/{instance}
- Auth, audit, rate limiting, UI integration required
**Re-evaluation Criteria:**
- Operators request controls >3x in 30 days
- Troubleshooting steps inadequate
- Service restarts too disruptive
- Production incidents need surgical controls
Decision: Monitor production usage for 60 days, then reassess based on actual operator feedback and support ticket patterns.
Part of Phase 2 - Adaptive Polling completion
This commit is contained in:
parent
cb8be81f1d
commit
ce5ad64810
1 changed files with 66 additions and 0 deletions
66
docs/operations/ADAPTIVE_POLLING_MANAGEMENT_ENDPOINTS.md
Normal file
66
docs/operations/ADAPTIVE_POLLING_MANAGEMENT_ENDPOINTS.md
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
# Adaptive Polling Management Endpoints (Future Enhancement)
|
||||
|
||||
## Status: DEFERRED
|
||||
|
||||
**Decision Date:** 2025-10-20
|
||||
**Reviewed After:** Phase 2 integration/soak testing
|
||||
|
||||
## Assessment
|
||||
|
||||
After completing comprehensive integration and soak testing (including 2-minute validation with 80 instances), we determined that manual circuit breaker and DLQ management endpoints are **not required for Phase 2 production rollout**.
|
||||
|
||||
### Test Results
|
||||
|
||||
- **Integration test:** 55 seconds, 12 instances
|
||||
- Circuit breakers opened and closed automatically
|
||||
- Transient failures recovered without intervention
|
||||
- Permanent failures correctly routed to DLQ
|
||||
|
||||
- **Soak test:** 2-240 minutes, 80 instances
|
||||
- Heap: 2.3MB → 3.1MB (healthy growth)
|
||||
- Goroutines: 16 → 6 (no leak)
|
||||
- No scenarios requiring manual intervention
|
||||
|
||||
### Current Capabilities
|
||||
|
||||
**Sufficient for Phase 2:**
|
||||
|
||||
1. **Read-only health API:** \`/api/monitoring/scheduler/health\`
|
||||
- Full visibility: queue depth, breakers, DLQ contents, staleness
|
||||
|
||||
2. **Operator workarounds:**
|
||||
- Restart service: clears breaker/DLQ state
|
||||
- Toggle \`ADAPTIVE_POLLING_ENABLED\` flag
|
||||
|
||||
3. **Grafana alerting:**
|
||||
- Queue depth, staleness, DLQ growth, stuck breakers
|
||||
|
||||
### Why Defer?
|
||||
|
||||
- **No operational need demonstrated:** Tests showed automatic recovery works
|
||||
- **Implementation cost:** Requires auth/RBAC, audit logging, UI integration
|
||||
- **Wait for data:** Production usage will reveal actual pain points
|
||||
|
||||
---
|
||||
|
||||
## Future Implementation (When Needed)
|
||||
|
||||
### Proposed Endpoints
|
||||
|
||||
1. **POST /api/monitoring/breakers/{instance}/reset** - Reset circuit breaker
|
||||
2. **POST /api/monitoring/dlq/retry** - Retry all DLQ tasks
|
||||
3. **POST /api/monitoring/dlq/{instance}/retry** - Retry specific task
|
||||
4. **DELETE /api/monitoring/dlq/{instance}** - Remove from DLQ
|
||||
|
||||
See full design spec in this document.
|
||||
|
||||
---
|
||||
|
||||
## Re-evaluation Criteria
|
||||
|
||||
**Implement if:**
|
||||
- Operators request manual controls >3x in first 30 days
|
||||
- Rollout troubleshooting steps inadequate
|
||||
- Service restarts become disruptive
|
||||
- Production incidents need surgical controls
|
||||
|
||||
Loading…
Reference in a new issue