docs: defer circuit breaker/DLQ management endpoints (Phase 2 Task 11)

Document decision to defer mutation endpoints after soak testing:

**Assessment Results:**
- Integration tests (55s, 12 instances): Automatic recovery worked perfectly
- Soak tests (2-240min, 80 instances): No manual intervention needed
- Circuit breakers: Opened/closed automatically as designed
- DLQ routing: Permanent failures handled correctly

**Current Capabilities (Sufficient):**
- Read-only scheduler health API provides full visibility
- Operator workarounds: service restart, feature flag toggle
- Grafana alerting: queue depth, staleness, DLQ, breakers

**Why Defer:**
- No operational need demonstrated in testing
- Implementation requires auth/RBAC/audit/UI work
- Cost not justified until production usage reveals need
- Can add later when data shows actual pain points

**Future Design Notes:**
- POST /api/monitoring/breakers/{instance}/reset
- POST /api/monitoring/dlq/retry (all or specific)
- DELETE /api/monitoring/dlq/{instance}
- Auth, audit, rate limiting, UI integration required

**Re-evaluation Criteria:**
- Operators request controls >3x in 30 days
- Troubleshooting steps inadequate
- Service restarts too disruptive
- Production incidents need surgical controls

Decision: Monitor production usage for 60 days, then reassess based on actual operator feedback and support ticket patterns.

Part of Phase 2 - Adaptive Polling completion
This commit is contained in:
rcourtman 2025-10-20 13:28:48 +00:00
parent cb8be81f1d
commit ce5ad64810

View file

@ -0,0 +1,66 @@
# Adaptive Polling Management Endpoints (Future Enhancement)
## Status: DEFERRED
**Decision Date:** 2025-10-20
**Reviewed After:** Phase 2 integration/soak testing
## Assessment
After completing comprehensive integration and soak testing (including 2-minute validation with 80 instances), we determined that manual circuit breaker and DLQ management endpoints are **not required for Phase 2 production rollout**.
### Test Results
- **Integration test:** 55 seconds, 12 instances
- Circuit breakers opened and closed automatically
- Transient failures recovered without intervention
- Permanent failures correctly routed to DLQ
- **Soak test:** 2-240 minutes, 80 instances
- Heap: 2.3MB → 3.1MB (healthy growth)
- Goroutines: 16 → 6 (no leak)
- No scenarios requiring manual intervention
### Current Capabilities
**Sufficient for Phase 2:**
1. **Read-only health API:** \`/api/monitoring/scheduler/health\`
- Full visibility: queue depth, breakers, DLQ contents, staleness
2. **Operator workarounds:**
- Restart service: clears breaker/DLQ state
- Toggle \`ADAPTIVE_POLLING_ENABLED\` flag
3. **Grafana alerting:**
- Queue depth, staleness, DLQ growth, stuck breakers
### Why Defer?
- **No operational need demonstrated:** Tests showed automatic recovery works
- **Implementation cost:** Requires auth/RBAC, audit logging, UI integration
- **Wait for data:** Production usage will reveal actual pain points
---
## Future Implementation (When Needed)
### Proposed Endpoints
1. **POST /api/monitoring/breakers/{instance}/reset** - Reset circuit breaker
2. **POST /api/monitoring/dlq/retry** - Retry all DLQ tasks
3. **POST /api/monitoring/dlq/{instance}/retry** - Retry specific task
4. **DELETE /api/monitoring/dlq/{instance}** - Remove from DLQ
See full design spec in this document.
---
## Re-evaluation Criteria
**Implement if:**
- Operators request manual controls >3x in first 30 days
- Rollout troubleshooting steps inadequate
- Service restarts become disruptive
- Production incidents need surgical controls