docs: update documentation for diagnostics improvements

Add comprehensive operator documentation for the new observability features introduced in the previous commit. **New Documentation:** - docs/monitoring/PROMETHEUS_METRICS.md - Complete reference for all 18 new Prometheus metrics with alert suggestions **Updated Documentation:** - docs/API.md - Document X-Request-ID and X-Diagnostics-Cached-At headers, explain diagnostics endpoint caching behavior - docs/TROUBLESHOOTING.md - Add section on correlating API calls with logs using request IDs - docs/operations/ADAPTIVE_POLLING_ROLLOUT.md - Update monitoring checklists with new per-node and scheduler metrics - docs/CONFIGURATION.md - Clarify LOG_FILE dual-output behavior and rotation defaults These updates ensure operators understand: - How to set up monitoring/alerting for new metrics - How to configure file logging with rotation - How to troubleshoot using request correlation - What metrics are available for dashboards Related to: 495e6c794 (feat: comprehensive diagnostics improvements)
2025-10-21 12:45:19 +00:00 · 2025-10-21 12:45:19 +00:00 · e0396c1362
commit e0396c1362
parent acedd18c07
5 changed files with 131 additions and 11 deletions
--- a/docs/API.md
+++ b/docs/API.md
@ -93,6 +93,14 @@ fetch('/api/nodes', {
 });
 ```

+## Common Response Headers
+
+Most endpoints emit a pair of diagnostic headers to help with troubleshooting:
+
+- `X-Request-ID` &mdash; unique identifier assigned to each HTTP request. The same value appears in Pulse logs, enabling quick correlation when raising support tickets or hunting through log files.
+- `X-RateLimit-*` family (`X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Retry-After`) &mdash; surfaced when rate limiting is enabled (default in v4.24.0+).
+- `X-Diagnostics-Cached-At` &mdash; returned only by `/api/diagnostics`; indicates when the current diagnostics payload was generated.
+
 ## Core Endpoints

 ### Health Check
@ -883,6 +891,8 @@ Returns detailed information about:
 - Performance metrics
 - Service health

+> **Caching (v4.24.0+):** Diagnostics results are cached for 45 seconds to protect upstream systems. If the cache is fresh it is returned immediately; otherwise a new probe runs, replacing the cache once complete. Inspect the `X-Diagnostics-Cached-At` header to see when the payload was generated. Probe failures surface in the `errors` array and are tracked by Prometheus metrics (`pulse_diagnostics_*`).
+
 ### Network Discovery
 Discover Proxmox servers on the network.

--- a/docs/CONFIGURATION.md
+++ b/docs/CONFIGURATION.md
@ -384,6 +384,8 @@ These env vars override system.json values. When set, the UI will show a warning
  - Set explicitly when running in containers or whenever auto-detection picks the wrong address
  - Example: `PULSE_PUBLIC_URL="http://192.168.1.100:7655"`

+> **Log file behaviour:** When `LOG_FILE` is set, Pulse continues to write logs to stderr while also appending to the specified file. Rotation occurs when the file exceeds `LOG_MAX_SIZE` megabytes (default 100 MB). Rotated files older than `LOG_MAX_AGE` days (default 30) are deleted, and compressing is enabled by default (`LOG_COMPRESS=true`), producing `.gz` archives for rotated files.
+
 #### Authentication Variables (from .env file)
 These should be set in the .env file for security:

--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@ -2,6 +2,23 @@

 ## Common Issues and Solutions

+### Correlate API Calls with Logs
+
+Every API response includes an `X-Request-ID` header. When escalating issues, capture that value and use it to search the backend logs or log file. The same identifier is emitted as `request_id` in structured logs.
+
+```bash
+# Capture a request ID
+curl -i https://pulse.example.com/api/state | grep X-Request-ID
+
+# Search the rotating log file
+grep 'request_id=abc123' /var/log/pulse/pulse.log
+
+# Docker / kubectl example
+docker logs pulse | grep 'request_id=abc123'
+```
+
+Include the `X-Request-ID` in support tickets or incident notes so responders can jump straight to the relevant log lines.
+
 ### Authentication Problems

 #### Forgot Password / Lost Access
--- a/docs/monitoring/PROMETHEUS_METRICS.md
+++ b/docs/monitoring/PROMETHEUS_METRICS.md
@ -0,0 +1,81 @@
+# Pulse Prometheus Metrics (v4.24.0+)
+
+Pulse exposes multiple metric families that cover HTTP ingress, per-node poll execution, scheduler health, and diagnostics caching. Use the following reference when wiring dashboards or alert rules.
+
+---
+
+## HTTP Request Metrics
+
+| Metric | Type | Labels | Description |
+| --- | --- | --- | --- |
+| `pulse_http_request_duration_seconds` | Histogram | `method`, `route`, `status` | Request latency buckets. `route` is a normalised path (dynamic segments collapsed to `:id`, `:uuid`, etc.). |
+| `pulse_http_requests_total` | Counter | `method`, `route`, `status` | Total requests handled. |
+| `pulse_http_request_errors_total` | Counter | `method`, `route`, `status_class` | Counts 4xx/5xx responses. |
+
+**Alert suggestion:**  
+`rate(pulse_http_request_errors_total{status_class="server_error"}[5m]) > 0.05` (more than ~3 server errors/min) should page ops.
+
+---
+
+## Per-Node Poll Metrics
+
+| Metric | Type | Labels | Description |
+| --- | --- | --- | --- |
+| `pulse_monitor_node_poll_duration_seconds` | Histogram | `instance_type`, `instance`, `node` | Wall-clock duration for each node poll. |
+| `pulse_monitor_node_poll_total` | Counter | `instance_type`, `instance`, `node`, `result` | Success/error counts per node. |
+| `pulse_monitor_node_poll_errors_total` | Counter | `instance_type`, `instance`, `node`, `error_type` | Error type breakdown (connection, auth, internal, etc.). |
+| `pulse_monitor_node_poll_last_success_timestamp` | Gauge | `instance_type`, `instance`, `node` | Unix timestamp of last successful poll. |
+| `pulse_monitor_node_poll_staleness_seconds` | Gauge | `instance_type`, `instance`, `node` | Seconds since last success (−1 means no success yet). |
+
+**Alert suggestion:**  
+`max_over_time(pulse_monitor_node_poll_staleness_seconds{node!=""}[10m]) > 300` indicates a node has been stale for 5+ minutes.
+
+---
+
+## Scheduler Health Metrics
+
+| Metric | Type | Labels | Description |
+| --- | --- | --- | --- |
+| `pulse_scheduler_queue_due_soon` | Gauge | — | Number of tasks due within 12 seconds. |
+| `pulse_scheduler_queue_depth` | Gauge | `instance_type` | Queue depth per instance type (PVE, PBS, PMG). |
+| `pulse_scheduler_queue_wait_seconds` | Histogram | `instance_type` | Wait time between when a task should run and when it actually executes. |
+| `pulse_scheduler_dead_letter_depth` | Gauge | `instance_type`, `instance` | Dead-letter queue depth per monitored instance. |
+| `pulse_scheduler_breaker_state` | Gauge | `instance_type`, `instance` | Circuit breaker state: `0`=closed, `1`=half-open, `2`=open, `-1`=unknown. |
+| `pulse_scheduler_breaker_failure_count` | Gauge | `instance_type`, `instance` | Consecutive failures tracked by the breaker. |
+| `pulse_scheduler_breaker_retry_seconds` | Gauge | `instance_type`, `instance` | Seconds until the breaker will allow the next attempt. |
+
+**Alert suggestions:**
+- Queue saturation: `max_over_time(pulse_scheduler_queue_depth[10m]) > <instance count * 1.5>`
+- DLQ growth: `increase(pulse_scheduler_dead_letter_depth[10m]) > 0`
+- Breaker stuck open: `pulse_scheduler_breaker_state == 2` for > 10 minutes.
+
+---
+
+## Diagnostics Cache Metrics
+
+| Metric | Type | Labels | Description |
+| --- | --- | --- | --- |
+| `pulse_diagnostics_cache_hits_total` | Counter | — | Diagnostics requests served from cache. |
+| `pulse_diagnostics_cache_misses_total` | Counter | — | Requests that triggered a fresh probe. |
+| `pulse_diagnostics_refresh_duration_seconds` | Histogram | — | Time taken to refresh diagnostics payload. |
+
+**Alert suggestion:**  
+`rate(pulse_diagnostics_cache_misses_total[5m])` spiking alongside `pulse_diagnostics_refresh_duration_seconds` > 20s can signal upstream slowness.
+
+---
+
+## Existing Instance-Level Poll Metrics (for completeness)
+
+The following metrics pre-date v4.24.0 but remain essential:
+
+| Metric | Type | Description |
+| --- | --- | --- |
+| `pulse_monitor_poll_duration_seconds` | Histogram | Poll duration per instance. |
+| `pulse_monitor_poll_total` | Counter | Success/error counts per instance. |
+| `pulse_monitor_poll_errors_total` | Counter | Error counts per instance. |
+| `pulse_monitor_poll_last_success_timestamp` | Gauge | Last successful poll timestamp. |
+| `pulse_monitor_poll_staleness_seconds` | Gauge | Seconds since last successful poll (instance-level). |
+| `pulse_monitor_poll_queue_depth` | Gauge | Current queue depth. |
+| `pulse_monitor_poll_inflight` | Gauge | Polls currently running. |
+
+Refer to this document whenever you build dashboards or craft alert policies. Scrape all metrics from the Pulse backend `/metrics` endpoint (9091 by default for systemd installs). 
--- a/docs/operations/ADAPTIVE_POLLING_ROLLOUT.md
+++ b/docs/operations/ADAPTIVE_POLLING_ROLLOUT.md
@ -13,13 +13,15 @@ Follow these operational procedures for steady-state monitoring, troubleshooting
 **For new deployments or upgrades to v4.24.0:**

 1. **Monitoring readiness**
+   - Review available scrape series in [Prometheus Metrics](../monitoring/PROMETHEUS_METRICS.md).
   - Set up Grafana dashboard with:
-     - `pulse_monitor_poll_queue_depth` (gauge)
-     - `pulse_monitor_poll_staleness_seconds` (gauge, per instance)
-     - `pulse_monitor_poll_total` and `pulse_monitor_poll_errors_total` (rate panels)
-     - `pulse_monitor_poll_last_success_timestamp` (new in v4.24.0)
-     - Alerting panels for circuit breaker state (via scheduler health API)
-   - Configure alerts (see §4)
+     - Instance gauges: `pulse_monitor_poll_queue_depth`, `pulse_monitor_poll_staleness_seconds`, `pulse_monitor_poll_last_success_timestamp`
+     - **Per-node coverage:** `pulse_monitor_node_poll_staleness_seconds`, `pulse_monitor_node_poll_errors_total`, `pulse_monitor_node_poll_duration_seconds`
+     - Scheduler health: `pulse_scheduler_queue_depth`, `pulse_scheduler_queue_due_soon`, `pulse_scheduler_dead_letter_depth`, `pulse_scheduler_breaker_state`, and `pulse_scheduler_breaker_failure_count`
+     - Diagnostics cache sanity: `increase(pulse_diagnostics_cache_misses_total[5m])` vs `pulse_diagnostics_cache_hits_total`
+     - HTTP SLA: `rate(pulse_http_request_errors_total{status_class="server_error"}[5m])`
+     - Continue trending `pulse_monitor_poll_total` / `pulse_monitor_poll_errors_total` for throughput
+     - Configure alerts (see §4)

 2. **Baseline metrics**
   - Record pre-upgrade metrics if upgrading from < v4.24.0:
@ -90,9 +92,12 @@ Follow these operational procedures for steady-state monitoring, troubleshooting
   - Review scheduler health dashboard or API endpoint
   - Check for:
     - Queue depth < 50 (alert if > 50 for 10+ minutes)
-     - Staleness < 120s for critical instances
+     - Instance staleness < 60s for healthy instances, < 120s for critical instances
+     - **Per-node staleness** `pulse_monitor_node_poll_staleness_seconds` < 120s
     - DLQ count stable (not growing)
     - Circuit breakers mostly `closed`
+     - Diagnostics cache misses roughly follow hits (`increase(pulse_diagnostics_cache_misses_total[10m])` in line with hits)
+     - HTTP error rate (`rate(pulse_http_request_errors_total{status_class="server_error"}[5m])`) near zero

 2. **Weekly reviews**
   - Analyze trends in Grafana:
@ -121,18 +126,23 @@ Follow these operational procedures for steady-state monitoring, troubleshooting
 1. **Dashboard panels**
   - **Queue Depth**: `pulse_monitor_poll_queue_depth`
     - Use single-stat with alert if > 1.5× active instances for > 10 min
-   - **Instance Staleness**: panel per instance type using `pulse_monitor_poll_staleness_seconds`
+   - **Instance & Node Staleness**: combine `pulse_monitor_poll_staleness_seconds` and `pulse_monitor_node_poll_staleness_seconds`
     - Alert threshold: > 60 s for > 5 min (excluding known failing instances)
   - **Polling Throughput**: rate of `pulse_monitor_poll_total{result="success"}` vs `result="error"`
-   - **Circuit Breakers / DLQ**: table from scheduler health API (via scripted datasource) highlighting non-closed breakers or DLQ entries
+   - **Per-node errors**: table or graph of `pulse_monitor_node_poll_errors_total` to spot noisy nodes
+   - **Scheduler Health**: panels for `pulse_scheduler_queue_depth`, `pulse_scheduler_queue_due_soon`, `pulse_scheduler_dead_letter_depth`, `pulse_scheduler_breaker_state`, `pulse_scheduler_breaker_failure_count`
+   - **Diagnostics Cache**: compare `increase(pulse_diagnostics_cache_hits_total[5m])` vs misses so spikes stand out
+   - **HTTP SLA**: `rate(pulse_http_request_errors_total{status_class="server_error"}[5m])`
   - **Last Success Timestamp** (v4.24.0+): `pulse_monitor_poll_last_success_timestamp` to detect polling gaps

 2. **Alerts**
   - Queue depth > threshold for >10 min (Warning), >20 min (Critical)
-   - Staleness > 60 s for >5 min (Critical)
+   - Instance or node staleness > 60 s for >5 min (Critical)
   - Dead-letter count increase > N (based on baseline) triggers Warning
-   - Any breaker stuck in `open` for >10 min triggers Critical
+   - Any breaker stuck in `open` for >10 min (from `pulse_scheduler_breaker_state`) triggers Critical
+   - Queue wait > 5 s (95th percentile on `pulse_scheduler_queue_wait_seconds`) triggers Warning
   - Permanent failures (`pulse_monitor_poll_errors_total{category="permanent"}`) trigger immediate Critical
+   - Diagnostics refresh duration > 20 s alongside miss spikes should page engineering (`pulse_diagnostics_refresh_duration_seconds`)

 3. **Notification routing**
   - Ensure alerts route to on-call + feature owner