diff --git a/docs/API.md b/docs/API.md index d2a7c8d..69f4912 100644 --- a/docs/API.md +++ b/docs/API.md @@ -93,6 +93,14 @@ fetch('/api/nodes', { }); ``` +## Common Response Headers + +Most endpoints emit a pair of diagnostic headers to help with troubleshooting: + +- `X-Request-ID` — unique identifier assigned to each HTTP request. The same value appears in Pulse logs, enabling quick correlation when raising support tickets or hunting through log files. +- `X-RateLimit-*` family (`X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Retry-After`) — surfaced when rate limiting is enabled (default in v4.24.0+). +- `X-Diagnostics-Cached-At` — returned only by `/api/diagnostics`; indicates when the current diagnostics payload was generated. + ## Core Endpoints ### Health Check @@ -883,6 +891,8 @@ Returns detailed information about: - Performance metrics - Service health +> **Caching (v4.24.0+):** Diagnostics results are cached for 45 seconds to protect upstream systems. If the cache is fresh it is returned immediately; otherwise a new probe runs, replacing the cache once complete. Inspect the `X-Diagnostics-Cached-At` header to see when the payload was generated. Probe failures surface in the `errors` array and are tracked by Prometheus metrics (`pulse_diagnostics_*`). + ### Network Discovery Discover Proxmox servers on the network. diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index f1f4f55..bf9d4cb 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -384,6 +384,8 @@ These env vars override system.json values. When set, the UI will show a warning - Set explicitly when running in containers or whenever auto-detection picks the wrong address - Example: `PULSE_PUBLIC_URL="http://192.168.1.100:7655"` +> **Log file behaviour:** When `LOG_FILE` is set, Pulse continues to write logs to stderr while also appending to the specified file. Rotation occurs when the file exceeds `LOG_MAX_SIZE` megabytes (default 100 MB). Rotated files older than `LOG_MAX_AGE` days (default 30) are deleted, and compressing is enabled by default (`LOG_COMPRESS=true`), producing `.gz` archives for rotated files. + #### Authentication Variables (from .env file) These should be set in the .env file for security: diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 8d0261b..e0d397d 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -2,6 +2,23 @@ ## Common Issues and Solutions +### Correlate API Calls with Logs + +Every API response includes an `X-Request-ID` header. When escalating issues, capture that value and use it to search the backend logs or log file. The same identifier is emitted as `request_id` in structured logs. + +```bash +# Capture a request ID +curl -i https://pulse.example.com/api/state | grep X-Request-ID + +# Search the rotating log file +grep 'request_id=abc123' /var/log/pulse/pulse.log + +# Docker / kubectl example +docker logs pulse | grep 'request_id=abc123' +``` + +Include the `X-Request-ID` in support tickets or incident notes so responders can jump straight to the relevant log lines. + ### Authentication Problems #### Forgot Password / Lost Access diff --git a/docs/monitoring/PROMETHEUS_METRICS.md b/docs/monitoring/PROMETHEUS_METRICS.md new file mode 100644 index 0000000..aa1c77e --- /dev/null +++ b/docs/monitoring/PROMETHEUS_METRICS.md @@ -0,0 +1,81 @@ +# Pulse Prometheus Metrics (v4.24.0+) + +Pulse exposes multiple metric families that cover HTTP ingress, per-node poll execution, scheduler health, and diagnostics caching. Use the following reference when wiring dashboards or alert rules. + +--- + +## HTTP Request Metrics + +| Metric | Type | Labels | Description | +| --- | --- | --- | --- | +| `pulse_http_request_duration_seconds` | Histogram | `method`, `route`, `status` | Request latency buckets. `route` is a normalised path (dynamic segments collapsed to `:id`, `:uuid`, etc.). | +| `pulse_http_requests_total` | Counter | `method`, `route`, `status` | Total requests handled. | +| `pulse_http_request_errors_total` | Counter | `method`, `route`, `status_class` | Counts 4xx/5xx responses. | + +**Alert suggestion:** +`rate(pulse_http_request_errors_total{status_class="server_error"}[5m]) > 0.05` (more than ~3 server errors/min) should page ops. + +--- + +## Per-Node Poll Metrics + +| Metric | Type | Labels | Description | +| --- | --- | --- | --- | +| `pulse_monitor_node_poll_duration_seconds` | Histogram | `instance_type`, `instance`, `node` | Wall-clock duration for each node poll. | +| `pulse_monitor_node_poll_total` | Counter | `instance_type`, `instance`, `node`, `result` | Success/error counts per node. | +| `pulse_monitor_node_poll_errors_total` | Counter | `instance_type`, `instance`, `node`, `error_type` | Error type breakdown (connection, auth, internal, etc.). | +| `pulse_monitor_node_poll_last_success_timestamp` | Gauge | `instance_type`, `instance`, `node` | Unix timestamp of last successful poll. | +| `pulse_monitor_node_poll_staleness_seconds` | Gauge | `instance_type`, `instance`, `node` | Seconds since last success (−1 means no success yet). | + +**Alert suggestion:** +`max_over_time(pulse_monitor_node_poll_staleness_seconds{node!=""}[10m]) > 300` indicates a node has been stale for 5+ minutes. + +--- + +## Scheduler Health Metrics + +| Metric | Type | Labels | Description | +| --- | --- | --- | --- | +| `pulse_scheduler_queue_due_soon` | Gauge | — | Number of tasks due within 12 seconds. | +| `pulse_scheduler_queue_depth` | Gauge | `instance_type` | Queue depth per instance type (PVE, PBS, PMG). | +| `pulse_scheduler_queue_wait_seconds` | Histogram | `instance_type` | Wait time between when a task should run and when it actually executes. | +| `pulse_scheduler_dead_letter_depth` | Gauge | `instance_type`, `instance` | Dead-letter queue depth per monitored instance. | +| `pulse_scheduler_breaker_state` | Gauge | `instance_type`, `instance` | Circuit breaker state: `0`=closed, `1`=half-open, `2`=open, `-1`=unknown. | +| `pulse_scheduler_breaker_failure_count` | Gauge | `instance_type`, `instance` | Consecutive failures tracked by the breaker. | +| `pulse_scheduler_breaker_retry_seconds` | Gauge | `instance_type`, `instance` | Seconds until the breaker will allow the next attempt. | + +**Alert suggestions:** +- Queue saturation: `max_over_time(pulse_scheduler_queue_depth[10m]) > ` +- DLQ growth: `increase(pulse_scheduler_dead_letter_depth[10m]) > 0` +- Breaker stuck open: `pulse_scheduler_breaker_state == 2` for > 10 minutes. + +--- + +## Diagnostics Cache Metrics + +| Metric | Type | Labels | Description | +| --- | --- | --- | --- | +| `pulse_diagnostics_cache_hits_total` | Counter | — | Diagnostics requests served from cache. | +| `pulse_diagnostics_cache_misses_total` | Counter | — | Requests that triggered a fresh probe. | +| `pulse_diagnostics_refresh_duration_seconds` | Histogram | — | Time taken to refresh diagnostics payload. | + +**Alert suggestion:** +`rate(pulse_diagnostics_cache_misses_total[5m])` spiking alongside `pulse_diagnostics_refresh_duration_seconds` > 20s can signal upstream slowness. + +--- + +## Existing Instance-Level Poll Metrics (for completeness) + +The following metrics pre-date v4.24.0 but remain essential: + +| Metric | Type | Description | +| --- | --- | --- | +| `pulse_monitor_poll_duration_seconds` | Histogram | Poll duration per instance. | +| `pulse_monitor_poll_total` | Counter | Success/error counts per instance. | +| `pulse_monitor_poll_errors_total` | Counter | Error counts per instance. | +| `pulse_monitor_poll_last_success_timestamp` | Gauge | Last successful poll timestamp. | +| `pulse_monitor_poll_staleness_seconds` | Gauge | Seconds since last successful poll (instance-level). | +| `pulse_monitor_poll_queue_depth` | Gauge | Current queue depth. | +| `pulse_monitor_poll_inflight` | Gauge | Polls currently running. | + +Refer to this document whenever you build dashboards or craft alert policies. Scrape all metrics from the Pulse backend `/metrics` endpoint (9091 by default for systemd installs). diff --git a/docs/operations/ADAPTIVE_POLLING_ROLLOUT.md b/docs/operations/ADAPTIVE_POLLING_ROLLOUT.md index 554d128..e78335b 100644 --- a/docs/operations/ADAPTIVE_POLLING_ROLLOUT.md +++ b/docs/operations/ADAPTIVE_POLLING_ROLLOUT.md @@ -13,13 +13,15 @@ Follow these operational procedures for steady-state monitoring, troubleshooting **For new deployments or upgrades to v4.24.0:** 1. **Monitoring readiness** + - Review available scrape series in [Prometheus Metrics](../monitoring/PROMETHEUS_METRICS.md). - Set up Grafana dashboard with: - - `pulse_monitor_poll_queue_depth` (gauge) - - `pulse_monitor_poll_staleness_seconds` (gauge, per instance) - - `pulse_monitor_poll_total` and `pulse_monitor_poll_errors_total` (rate panels) - - `pulse_monitor_poll_last_success_timestamp` (new in v4.24.0) - - Alerting panels for circuit breaker state (via scheduler health API) - - Configure alerts (see §4) + - Instance gauges: `pulse_monitor_poll_queue_depth`, `pulse_monitor_poll_staleness_seconds`, `pulse_monitor_poll_last_success_timestamp` + - **Per-node coverage:** `pulse_monitor_node_poll_staleness_seconds`, `pulse_monitor_node_poll_errors_total`, `pulse_monitor_node_poll_duration_seconds` + - Scheduler health: `pulse_scheduler_queue_depth`, `pulse_scheduler_queue_due_soon`, `pulse_scheduler_dead_letter_depth`, `pulse_scheduler_breaker_state`, and `pulse_scheduler_breaker_failure_count` + - Diagnostics cache sanity: `increase(pulse_diagnostics_cache_misses_total[5m])` vs `pulse_diagnostics_cache_hits_total` + - HTTP SLA: `rate(pulse_http_request_errors_total{status_class="server_error"}[5m])` + - Continue trending `pulse_monitor_poll_total` / `pulse_monitor_poll_errors_total` for throughput + - Configure alerts (see §4) 2. **Baseline metrics** - Record pre-upgrade metrics if upgrading from < v4.24.0: @@ -90,9 +92,12 @@ Follow these operational procedures for steady-state monitoring, troubleshooting - Review scheduler health dashboard or API endpoint - Check for: - Queue depth < 50 (alert if > 50 for 10+ minutes) - - Staleness < 120s for critical instances + - Instance staleness < 60s for healthy instances, < 120s for critical instances + - **Per-node staleness** `pulse_monitor_node_poll_staleness_seconds` < 120s - DLQ count stable (not growing) - Circuit breakers mostly `closed` + - Diagnostics cache misses roughly follow hits (`increase(pulse_diagnostics_cache_misses_total[10m])` in line with hits) + - HTTP error rate (`rate(pulse_http_request_errors_total{status_class="server_error"}[5m])`) near zero 2. **Weekly reviews** - Analyze trends in Grafana: @@ -121,18 +126,23 @@ Follow these operational procedures for steady-state monitoring, troubleshooting 1. **Dashboard panels** - **Queue Depth**: `pulse_monitor_poll_queue_depth` - Use single-stat with alert if > 1.5× active instances for > 10 min - - **Instance Staleness**: panel per instance type using `pulse_monitor_poll_staleness_seconds` + - **Instance & Node Staleness**: combine `pulse_monitor_poll_staleness_seconds` and `pulse_monitor_node_poll_staleness_seconds` - Alert threshold: > 60 s for > 5 min (excluding known failing instances) - **Polling Throughput**: rate of `pulse_monitor_poll_total{result="success"}` vs `result="error"` - - **Circuit Breakers / DLQ**: table from scheduler health API (via scripted datasource) highlighting non-closed breakers or DLQ entries + - **Per-node errors**: table or graph of `pulse_monitor_node_poll_errors_total` to spot noisy nodes + - **Scheduler Health**: panels for `pulse_scheduler_queue_depth`, `pulse_scheduler_queue_due_soon`, `pulse_scheduler_dead_letter_depth`, `pulse_scheduler_breaker_state`, `pulse_scheduler_breaker_failure_count` + - **Diagnostics Cache**: compare `increase(pulse_diagnostics_cache_hits_total[5m])` vs misses so spikes stand out + - **HTTP SLA**: `rate(pulse_http_request_errors_total{status_class="server_error"}[5m])` - **Last Success Timestamp** (v4.24.0+): `pulse_monitor_poll_last_success_timestamp` to detect polling gaps 2. **Alerts** - Queue depth > threshold for >10 min (Warning), >20 min (Critical) - - Staleness > 60 s for >5 min (Critical) + - Instance or node staleness > 60 s for >5 min (Critical) - Dead-letter count increase > N (based on baseline) triggers Warning - - Any breaker stuck in `open` for >10 min triggers Critical + - Any breaker stuck in `open` for >10 min (from `pulse_scheduler_breaker_state`) triggers Critical + - Queue wait > 5 s (95th percentile on `pulse_scheduler_queue_wait_seconds`) triggers Warning - Permanent failures (`pulse_monitor_poll_errors_total{category="permanent"}`) trigger immediate Critical + - Diagnostics refresh duration > 20 s alongside miss spikes should page engineering (`pulse_diagnostics_refresh_duration_seconds`) 3. **Notification routing** - Ensure alerts route to on-call + feature owner