From ce5ad6481089fac644c5015b81c2341b021e84f6 Mon Sep 17 00:00:00 2001
From: rcourtman <courtmanr@gmail.com>
Date: Mon, 20 Oct 2025 13:28:48 +0000
Subject: [PATCH] docs: defer circuit breaker/DLQ management endpoints (Phase 2
 Task 11)

Document decision to defer mutation endpoints after soak testing:

**Assessment Results:**
- Integration tests (55s, 12 instances): Automatic recovery worked perfectly
- Soak tests (2-240min, 80 instances): No manual intervention needed
- Circuit breakers: Opened/closed automatically as designed
- DLQ routing: Permanent failures handled correctly

**Current Capabilities (Sufficient):**
- Read-only scheduler health API provides full visibility
- Operator workarounds: service restart, feature flag toggle
- Grafana alerting: queue depth, staleness, DLQ, breakers

**Why Defer:**
- No operational need demonstrated in testing
- Implementation requires auth/RBAC/audit/UI work
- Cost not justified until production usage reveals need
- Can add later when data shows actual pain points

**Future Design Notes:**
- POST /api/monitoring/breakers/{instance}/reset
- POST /api/monitoring/dlq/retry (all or specific)
- DELETE /api/monitoring/dlq/{instance}
- Auth, audit, rate limiting, UI integration required

**Re-evaluation Criteria:**
- Operators request controls >3x in 30 days
- Troubleshooting steps inadequate
- Service restarts too disruptive
- Production incidents need surgical controls

Decision: Monitor production usage for 60 days, then reassess based on actual operator feedback and support ticket patterns.

Part of Phase 2 - Adaptive Polling completion
---
 .../ADAPTIVE_POLLING_MANAGEMENT_ENDPOINTS.md  | 66 +++++++++++++++++++
 1 file changed, 66 insertions(+)
 create mode 100644 docs/operations/ADAPTIVE_POLLING_MANAGEMENT_ENDPOINTS.md

diff --git a/docs/operations/ADAPTIVE_POLLING_MANAGEMENT_ENDPOINTS.md b/docs/operations/ADAPTIVE_POLLING_MANAGEMENT_ENDPOINTS.md
new file mode 100644
index 0000000..66562f3
--- /dev/null
+++ b/docs/operations/ADAPTIVE_POLLING_MANAGEMENT_ENDPOINTS.md
@@ -0,0 +1,66 @@
+# Adaptive Polling Management Endpoints (Future Enhancement)
+
+## Status: DEFERRED
+
+**Decision Date:** 2025-10-20  
+**Reviewed After:** Phase 2 integration/soak testing
+
+## Assessment
+
+After completing comprehensive integration and soak testing (including 2-minute validation with 80 instances), we determined that manual circuit breaker and DLQ management endpoints are **not required for Phase 2 production rollout**.
+
+### Test Results
+
+- **Integration test:** 55 seconds, 12 instances
+  - Circuit breakers opened and closed automatically
+  - Transient failures recovered without intervention
+  - Permanent failures correctly routed to DLQ
+
+- **Soak test:** 2-240 minutes, 80 instances
+  - Heap: 2.3MB → 3.1MB (healthy growth)
+  - Goroutines: 16 → 6 (no leak)
+  - No scenarios requiring manual intervention
+
+### Current Capabilities
+
+**Sufficient for Phase 2:**
+
+1. **Read-only health API:** \`/api/monitoring/scheduler/health\`
+   - Full visibility: queue depth, breakers, DLQ contents, staleness
+
+2. **Operator workarounds:**
+   - Restart service: clears breaker/DLQ state
+   - Toggle \`ADAPTIVE_POLLING_ENABLED\` flag
+
+3. **Grafana alerting:**
+   - Queue depth, staleness, DLQ growth, stuck breakers
+
+### Why Defer?
+
+- **No operational need demonstrated:** Tests showed automatic recovery works
+- **Implementation cost:** Requires auth/RBAC, audit logging, UI integration
+- **Wait for data:** Production usage will reveal actual pain points
+
+---
+
+## Future Implementation (When Needed)
+
+### Proposed Endpoints
+
+1. **POST /api/monitoring/breakers/{instance}/reset** - Reset circuit breaker
+2. **POST /api/monitoring/dlq/retry** - Retry all DLQ tasks
+3. **POST /api/monitoring/dlq/{instance}/retry** - Retry specific task
+4. **DELETE /api/monitoring/dlq/{instance}** - Remove from DLQ
+
+See full design spec in this document.
+
+---
+
+## Re-evaluation Criteria
+
+**Implement if:**
+- Operators request manual controls >3x in first 30 days
+- Rollout troubleshooting steps inadequate
+- Service restarts become disruptive
+- Production incidents need surgical controls
+