Commit graph

2 commits

Author SHA1 Message Date
rcourtman
160adeb3b8 feat: add scheduler health API endpoint (Phase 2 Task 8)
Task 8 of 10 complete. Exposes read-only scheduler health data including:
- Queue depth and distribution by instance type
- Dead-letter queue inspection (top 25 tasks with error details)
- Circuit breaker states (instance-level)
- Staleness scores per instance

New API endpoint:
  GET /api/monitoring/scheduler/health (requires authentication)

New snapshot methods:
- StalenessTracker.Snapshot() - exports all staleness data
- TaskQueue.Snapshot() - queue depth & per-type distribution
- TaskQueue.PeekAll() - dead-letter task inspection
- circuitBreaker.State() - exports state, failures, retryAt
- Monitor.SchedulerHealth() - aggregates all health data

Documentation updated with API spec, field descriptions, and usage examples.
2025-10-20 15:13:38 +00:00
rcourtman
b1f445b33d feat: implement error handling with circuit breakers and backoff (Phase 2 Task 7)
Adds comprehensive error resilience:
- Circuit breaker with closed/open/half-open states (3 failures = trip)
- Exponential backoff with jitter (2s initial, 2x multiplier, 5min max)
- Dead-letter queue for tasks exceeding 5 retry attempts
- Error classification (transient vs permanent) using internal/errors helpers
- Per-instance failure tracking and breaker state management
- Integration with staleness tracker for outcome recording

Task 7 of 10 complete (70%). Ready for API surfaces and testing.
2025-10-20 15:13:37 +00:00