Pulse/docs/monitoring/ADAPTIVE_POLLING.md

Adaptive Polling Architecture

Overview

Pulse uses an adaptive polling scheduler that adapts poll cadence based on freshness, errors, and workload. The goal is to prioritize stale or changing instances while backing off on healthy, idle targets.

flowchart TD
    PollLoop["PollLoop\n(ticker & config updates)"]
    Scheduler["Scheduler\ncomputes ScheduledTask"]
    Staleness["Staleness Tracker\n(last success, freshness score)"]
    CircuitBreaker["Circuit Breaker\ntracks failure streaks"]
    Backoff["Backoff Policy\nexponential w/ jitter"]
    PriorityQ["Priority Queue\nmin-heap by NextRun"]
    WorkerPool["TaskWorkers\nN concurrent workers"]
    Metrics["Metrics & History\nPrometheus + retention"]
    Success["Poll Success"]
    Failure{"Poll Failure?"}
    Reschedule["Reschedule\n(next interval)"]
    BackoffPath["Backoff / Breaker Open"]
    DeadLetter["Dead-Letter Queue\noperator review"]

    PollLoop --> Scheduler
    Staleness --> Scheduler
    CircuitBreaker --> Scheduler
    Scheduler --> PriorityQ

    PriorityQ -->|due task| WorkerPool
    WorkerPool --> Failure
    WorkerPool -->|result| Metrics
    WorkerPool -->|freshness| Staleness

    Failure -->|No| Success
    Success --> CircuitBreaker
    Success --> Reschedule
    Success --> Metrics
    Reschedule --> Scheduler

    Failure -->|Yes| BackoffPath
    BackoffPath --> CircuitBreaker
    BackoffPath --> Backoff
    Backoff --> Scheduler
    Backoff --> DeadLetter
    DeadLetter -. periodic retry .-> Scheduler
    CircuitBreaker -. state change .-> Scheduler
    Metrics --> Scheduler

• Scheduler computes ScheduledTask entries using adaptive intervals.
• Task queue is a min-heap keyed by NextRun; only due tasks execute.
• Workers execute tasks, capture outcomes, reschedule via scheduler or backoff logic.

Key Components

Component	File	Responsibility
Scheduler	internal/monitoring/scheduler.go	Calculates adaptive intervals per instance.
Staleness tracker	internal/monitoring/staleness_tracker.go	Maintains freshness metadata and scores.
Priority queue	internal/monitoring/task_queue.go	Orders ScheduledTask items by due time + priority.
Circuit breaker	internal/monitoring/circuit_breaker.go	Trips on repeated failures, preventing hot loops.
Backoff	internal/monitoring/backoff.go	Exponential retry delays with jitter.
Workers	internal/monitoring/monitor.go	Pop tasks, execute pollers, reschedule or dead-letter.

Configuration

v4.24.0: Adaptive polling is enabled by default but can be toggled without restart.

Via UI

Navigate to Settings → System → Monitoring to enable/disable adaptive polling. Changes apply immediately without requiring a restart.

Via Environment Variables

Environment variables (default in internal/config/config.go):

Variable	Default	Description
ADAPTIVE_POLLING_ENABLED	true	Changed in v4.24.0: Now enabled by default
ADAPTIVE_POLLING_BASE_INTERVAL	10s	Target cadence when system is healthy
ADAPTIVE_POLLING_MIN_INTERVAL	5s	Lower bound (active instances)
ADAPTIVE_POLLING_MAX_INTERVAL	5m	Upper bound (idle instances)

All settings persist in system.json and respond to environment overrides. Changes apply without restart when modified via UI.

Metrics

v4.24.0: Extended metrics for comprehensive monitoring.

Exposed via Prometheus (:9091/metrics):

Metric	Type	Labels	Description
pulse_monitor_poll_total	counter	instance_type, instance, result	Overall poll attempts (success/error)
pulse_monitor_poll_duration_seconds	histogram	instance_type, instance	Poll latency per instance
pulse_monitor_poll_staleness_seconds	gauge	instance_type, instance	Age since last success (0 on success)
pulse_monitor_poll_queue_depth	gauge	—	Size of priority queue
pulse_monitor_poll_inflight	gauge	instance_type	Concurrent tasks per type
pulse_monitor_poll_errors_total	counter	instance_type, instance, category	New in v4.24.0: Error counts by category (transient/permanent)
pulse_monitor_poll_last_success_timestamp	gauge	instance_type, instance	New in v4.24.0: Unix timestamp of last successful poll

Alerting Recommendations:

• Alert when pulse_monitor_poll_staleness_seconds > 120 for critical instances
• Alert when pulse_monitor_poll_queue_depth > 50 (backlog building)
• Alert when pulse_monitor_poll_errors_total with category=permanent increases (auth/config issues)

Circuit Breaker & Backoff

State	Trigger	Recovery
Closed	Default. Failures counted.	—
Open	≥3 consecutive failures. Poll suppressed.	Exponential delay (max 5 min).
Half-open	Retry window elapsed. Limited re-attempt.	Success ⇒ closed. Failure ⇒ open.

stateDiagram-v2
    [*] --> Closed: Startup / reset
    Closed: Default state\nPolling active\nFailure counter increments
    Closed --> Open: ≥3 consecutive failures
    Open: Polls suppressed\nScheduler schedules backoff (max 5m)
    Open --> HalfOpen: Retry window elapsed
    HalfOpen: Single probe allowed\nBreaker watches probe result
    HalfOpen --> Closed: Probe success\nReset failure streak & delay
    HalfOpen --> Open: Probe failure\nIncrease streak & backoff

Backoff configuration:

• Initial delay: 5 s
• Multiplier: x2 per failure
• Jitter: ±20 %
• Max delay: 5 minutes
• After 5 transient failures or any permanent failure, task moves to dead-letter queue for operator action.

Dead-Letter Queue

Dead-letter entries are kept in memory (same TaskQueue structure) with a 30 min recheck interval. Operators should inspect logs for Routing task to dead-letter queue messages. Future work (Task 8) will add API surfaces for inspection.

API Endpoints

GET /api/monitoring/scheduler/health

Returns comprehensive scheduler health data (authentication required).

Response format:

{
  "updatedAt": "2025-03-21T18:05:00Z",
  "enabled": true,
  "queue": {
    "depth": 7,
    "dueWithinSeconds": 2,
    "perType": {
      "pve": 4,
      "pbs": 2,
      "pmg": 1
    }
  },
  "deadLetter": {
    "count": 2,
    "tasks": [
      {
        "instance": "pbs-nas",
        "type": "pbs",
        "nextRun": "2025-03-21T18:25:00Z",
        "lastError": "connection timeout",
        "failures": 7
      }
    ]
  },
  "breakers": [
    {
      "instance": "pve-core",
      "type": "pve",
      "state": "half_open",
      "failures": 3,
      "retryAt": "2025-03-21T18:05:45Z"
    }
  ],
  "staleness": [
    {
      "instance": "pve-core",
      "type": "pve",
      "score": 0.12,
      "lastSuccess": "2025-03-21T18:04:50Z"
    }
  ]
}

Field descriptions:

• enabled: Feature flag status
• queue.depth: Total queued tasks
• queue.dueWithinSeconds: Tasks due within 12 seconds
• queue.perType: Distribution by instance type
• deadLetter.count: Total dead-letter tasks
• deadLetter.tasks: Up to 25 most recent dead-letter entries
• breakers: Circuit breaker states (only non-default states shown)
• staleness: Freshness scores per instance (0 = fresh, 1 = max stale)

Operational Guidance

1. Enable adaptive polling: set ADAPTIVE_POLLING_ENABLED=true via UI or environment overrides, then restart hot-dev (scripts/hot-dev.sh).
2. Monitor metrics to ensure queue depth and staleness remain within SLA. Configure alerting on poll_staleness_seconds and poll_queue_depth.
3. Inspect scheduler health via API endpoint /api/monitoring/scheduler/health for circuit breaker trips and dead-letter queue status.
4. Review dead-letter logs for persistent failures; resolve underlying connectivity or auth issues before re-enabling.

Rollout Plan

1. Dev/QA: Run hot-dev with feature flag enabled; observe metrics and logs for several cycles.
2. Staged deploy: Enable flag on a subset of clusters; monitor queue depth (<50) and staleness (<45 s).
3. Full rollout: Toggle flag globally once metrics are stable; document any overrides in release notes.
4. Post-launch: Add Grafana panels for queue depth & staleness; alert on circuit breaker trips (future API work).

Known Follow-ups

• Task 8: expose scheduler health & dead-letter statistics via API and UI panels.
• Task 9: add dedicated unit/integration harness for the scheduler & workers.