rcourtman ce5ad64810 docs: defer circuit breaker/DLQ management endpoints (Phase 2 Task 11)

Document decision to defer mutation endpoints after soak testing:

**Assessment Results:**
- Integration tests (55s, 12 instances): Automatic recovery worked perfectly
- Soak tests (2-240min, 80 instances): No manual intervention needed
- Circuit breakers: Opened/closed automatically as designed
- DLQ routing: Permanent failures handled correctly

**Current Capabilities (Sufficient):**
- Read-only scheduler health API provides full visibility
- Operator workarounds: service restart, feature flag toggle
- Grafana alerting: queue depth, staleness, DLQ, breakers

**Why Defer:**
- No operational need demonstrated in testing
- Implementation requires auth/RBAC/audit/UI work
- Cost not justified until production usage reveals need
- Can add later when data shows actual pain points

**Future Design Notes:**
- POST /api/monitoring/breakers/{instance}/reset
- POST /api/monitoring/dlq/retry (all or specific)
- DELETE /api/monitoring/dlq/{instance}
- Auth, audit, rate limiting, UI integration required

**Re-evaluation Criteria:**
- Operators request controls >3x in 30 days
- Troubleshooting steps inadequate
- Service restarts too disruptive
- Production incidents need surgical controls

Decision: Monitor production usage for 60 days, then reassess based on actual operator feedback and support ticket patterns.

Part of Phase 2 - Adaptive Polling completion

2025-10-20 15:13:38 +00:00

2 KiB

Raw Blame History

Adaptive Polling Management Endpoints (Future Enhancement)

Status: DEFERRED

Decision Date: 2025-10-20
Reviewed After: Phase 2 integration/soak testing

Assessment

After completing comprehensive integration and soak testing (including 2-minute validation with 80 instances), we determined that manual circuit breaker and DLQ management endpoints are not required for Phase 2 production rollout.

Test Results

Integration test: 55 seconds, 12 instances
- Circuit breakers opened and closed automatically
- Transient failures recovered without intervention
- Permanent failures correctly routed to DLQ
Soak test: 2-240 minutes, 80 instances
- Heap: 2.3MB → 3.1MB (healthy growth)
- Goroutines: 16 → 6 (no leak)
- No scenarios requiring manual intervention

Current Capabilities

Sufficient for Phase 2:

Read-only health API: `/api/monitoring/scheduler/health`
- Full visibility: queue depth, breakers, DLQ contents, staleness
Operator workarounds:
- Restart service: clears breaker/DLQ state
- Toggle `ADAPTIVE_POLLING_ENABLED` flag
Grafana alerting:
- Queue depth, staleness, DLQ growth, stuck breakers

Why Defer?

No operational need demonstrated: Tests showed automatic recovery works
Implementation cost: Requires auth/RBAC, audit logging, UI integration
Wait for data: Production usage will reveal actual pain points

Future Implementation (When Needed)

Proposed Endpoints

POST /api/monitoring/breakers/{instance}/reset - Reset circuit breaker
POST /api/monitoring/dlq/retry - Retry all DLQ tasks
POST /api/monitoring/dlq/{instance}/retry - Retry specific task
DELETE /api/monitoring/dlq/{instance} - Remove from DLQ

See full design spec in this document.

Re-evaluation Criteria

Implement if:

Operators request manual controls >3x in first 30 days
Rollout troubleshooting steps inadequate
Service restarts become disruptive
Production incidents need surgical controls

2 KiB Raw Blame History