docs: update documentation for v4.24.0 features

Updates documentation to reflect features implemented in recent commits: **Security & API Enhancements:** - Rate limit headers (X-RateLimit-Limit, X-RateLimit-Remaining, Retry-After) - Audit logging for rollback actions and scheduler health - Runtime logging configuration tracking **Scheduler Health API:** - Document new v4.24.0 endpoint features - Per-instance circuit breaker status - Dead-letter queue tracking - Staleness metrics - Enhanced response format with backward compatibility **Version & Health Endpoints:** - Updated /api/version response fields - Optional health endpoint fields - Deployment type and update availability **Configuration & Installation:** - HTTP config fetch via PULSE_INIT_CONFIG_URL - Updated environment variable documentation - Enhanced FAQ entries **Monitoring & Operations:** - Adaptive polling architecture documentation - Rollback procedure references - Production deployment guidance All documentation changes align with implemented features from commits: - 656ae0d25 (PMG test fix) - dec85a4ef (PBS/PMG stubs + HTTP config) - Earlier commits: scheduler health API, rollback, rate limiting
2025-10-20 16:08:10 +00:00 · 2025-10-20 16:08:10 +00:00 · fd0a4f2b0a
commit fd0a4f2b0a
parent 73fb9d986f
9 changed files with 598 additions and 66 deletions
--- a/SECURITY.md
+++ b/SECURITY.md
@ -266,10 +266,14 @@ for sensitive data.
  - Live reloading when `.env` changes
  - API-only mode supported (no password auth required)
 - **CSRF protection**: all state-changing operations require CSRF tokens
- **Rate limiting**
-  - Auth endpoints: 10 attempts/minute per IP
+- **Rate limiting** (enhanced in v4.24.0)
+  - Auth endpoints: 10 attempts/minute per IP (returns `Retry-After` header)
  - General API: 500 requests/minute per IP
  - Real-time endpoints exempt for functionality
+  - **New in v4.24.0**: All responses include rate limit headers:
+    - `X-RateLimit-Limit`: Maximum requests per window
+    - `X-RateLimit-Remaining`: Requests remaining in current window
+    - `Retry-After`: Seconds to wait before retrying (on 429 responses)
 - **Account lockout**
  - Locks after 5 failed login attempts
  - 15-minute automatic lockout duration
@ -287,7 +291,11 @@ for sensitive data.
  - X-XSS-Protection: 1; mode=block
  - Referrer-Policy: strict-origin-when-cross-origin
  - Permissions-Policy restricting sensitive APIs
- **Audit logging**: authentication events include IP addresses
+- **Audit logging** (enhanced in v4.24.0)
+  - Authentication events include IP addresses
+  - **New**: Rollback actions are logged with timestamps and metadata
+  - **New**: Scheduler health escalations recorded in audit trail
+  - **New**: Runtime logging configuration changes tracked

 ### What's Encrypted in Exports
 - Node credentials (passwords, API tokens)
@ -404,6 +412,51 @@ curl "http://localhost:7655/api/export?token=your-original-token"
 - Protects auto-registration endpoint
 - Enable by setting at least one API token via `API_TOKENS` (or legacy `API_TOKEN`) environment variable

+### Runtime Logging Configuration
+
+**New in v4.24.0:** Adjust logging settings dynamically without restarting Pulse.
+
+#### Security Benefits
+- Enable debug logging temporarily for incident investigation
+- Switch to JSON format for SIEM integration
+- Adjust verbosity based on security posture
+- Control file rotation to manage audit log retention
+
+#### Configuration Options
+
+**Via UI:**
+Navigate to **Settings → System → Logging**:
+- **Log Level**: `debug`, `info`, `warn`, `error`
+- **Log Format**: `json` (for log aggregation), `text` (human-readable)
+- **File Rotation**: size limits, retention policies
+
+**Via Environment Variables:**
+```bash
+# Systemd
+sudo systemctl edit pulse
+[Service]
+Environment="LOG_LEVEL=info"
+Environment="LOG_FORMAT=json"
+Environment="LOG_MAX_SIZE=100"        # MB per log file
+Environment="LOG_MAX_BACKUPS=10"      # Number of rotated logs to keep
+Environment="LOG_MAX_AGE=30"          # Days to retain logs
+
+# Docker
+docker run \
+  -e LOG_LEVEL=info \
+  -e LOG_FORMAT=json \
+  -e LOG_MAX_SIZE=100 \
+  -e LOG_MAX_BACKUPS=10 \
+  -e LOG_MAX_AGE=30 \
+  rcourtman/pulse:latest
+```
+
+**Security Considerations:**
+- Debug logs may contain sensitive data—enable only when needed
+- JSON format recommended for security monitoring and SIEM
+- Adjust retention based on compliance requirements
+- Changes are logged to audit trail
+
 ## CORS (Cross-Origin Resource Sharing)

 By default, Pulse only allows same-origin requests (no CORS headers). This is the most secure configuration.
@ -430,6 +483,42 @@ PULSE_DEV=true

 **Security Note**: Never use `ALLOWED_ORIGINS=*` in production as it allows any website to access your API.

+## Monitoring and Observability
+
+### Scheduler Health API
+
+**New in v4.24.0:** Monitor Pulse's internal health and detect anomalies using the scheduler health API.
+
+#### Endpoint
+```bash
+curl -s http://localhost:7655/api/monitoring/scheduler/health | jq
+```
+
+#### Security Use Cases
+1. **Anomaly Detection**
+   - Watch for unusual queue depths (possible DoS)
+   - Monitor circuit breaker trips (connectivity issues or attacks)
+   - Track backoff patterns (rate limiting, potential probes)
+
+2. **Performance Monitoring**
+   - Identify performance degradation
+   - Detect resource exhaustion
+   - Track API response times
+
+3. **Incident Response**
+   - Real-time visibility into system health
+   - Historical metrics for post-incident analysis
+   - Circuit breaker status for failover decisions
+
+#### Key Security Metrics
+- **Queue Depth**: High values may indicate attack or overload
+- **Circuit Breaker Status**: Half-open/open states suggest connectivity issues
+- **Backoff Delays**: Increased backoff may indicate rate limiting or errors
+- **Error Rates**: Track failed API calls and authentication attempts
+
+**Dashboard Access:**
+Navigate to **Settings → System → Monitoring** for visual representation of scheduler health.
+
 ## Security Best Practices

 ### Credential Storage
@ -498,6 +587,16 @@ curl -X POST http://localhost:7655/api/security/reset-lockout \
 **Can't login?** Check `PULSE_AUTH_USER` and `PULSE_AUTH_PASS` environment variables  
 **API access denied?** Verify the token you supplied matches one of the values created in *Settings → Security → API tokens* (use the original token, not the hash)  
 **CORS errors?** Configure `ALLOWED_ORIGINS` for your domain  
-**Forgot password?** Start fresh – delete your Pulse data and restart  
+**Forgot password?** Start fresh – delete your Pulse data and restart

-_Last updated: 2025-10-19_
+---
+
+_Last updated: 2025-10-20_
+
+**Version 4.24.0 Security Enhancements:**
+- ✅ X-RateLimit-* headers for all API responses
+- ✅ Runtime logging configuration for incident response
+- ✅ Scheduler health API for anomaly detection
+- ✅ Enhanced audit logging (rollback actions, scheduler events)
+- ✅ Adaptive polling with circuit breakers and backoff
+- ✅ Shared script library system (secure installer patterns)
--- a/docs/API.md
+++ b/docs/API.md
@ -63,7 +63,12 @@ curl -b cookies.txt http://localhost:7655/api/health
 When authentication is enabled, Pulse provides enterprise-grade security:

 - **CSRF Protection**: All state-changing requests require a CSRF token
- **Rate Limiting**: 500 req/min general, 10 attempts/min for authentication
+- **Rate Limiting** (enhanced in v4.24.0): 500 req/min general, 10 attempts/min for authentication
+  - **New**: All responses include rate limit headers:
+    - `X-RateLimit-Limit`: Maximum requests per window
+    - `X-RateLimit-Remaining`: Requests remaining in current window
+    - `X-RateLimit-Reset`: Unix timestamp when the limit resets
+    - `Retry-After`: Seconds to wait before retrying (on 429 responses)
 - **Account Lockout**: Locks after 5 failed attempts (15 minute cooldown) with clear feedback
 - **Secure Sessions**: HttpOnly cookies, 24-hour expiry
 - **Security Headers**: CSP, X-Frame-Options, X-Content-Type-Options, etc.
@ -106,6 +111,19 @@ Response:
 }
 ```

+**Optional fields** (v4.24.0+, appear when relevant):
+```json
+{
+  "status": "healthy",
+  "timestamp": 1754995749,
+  "uptime": 166.187561244,
+  "legacySSHDetected": false,
+  "recommendProxyUpgrade": false,
+  "proxyInstallScriptAvailable": true,
+  "devModeSSH": false
+}
+```
+
 ### Version Information
 Get current Pulse version and build info.

@ -113,15 +131,20 @@ Get current Pulse version and build info.
 GET /api/version
 ```

-Response:
+Response (v4.24.0+):
 ```json
 {
-  "version": "v4.8.0",
+  "version": "v4.24.0",
  "build": "release",
+  "buildTime": "2025-10-20T10:30:00Z",
  "runtime": "go",
+  "goVersion": "1.23.2",
  "channel": "stable",
+  "deploymentType": "systemd",
  "isDocker": false,
-  "isDevelopment": false
+  "isDevelopment": false,
+  "updateAvailable": false,
+  "latestVersion": "v4.24.0"
 }
 ```

@ -185,6 +208,29 @@ When Pulse detects Ceph-backed storage (RBD, CephFS, etc.), the `cephClusters` a

 Each service entry lists offline daemons in `message` when present (for example, `Offline: mgr.x@pve2`), making it easy to highlight degraded components in custom tooling.

+### Scheduler Health
+
+**New in v4.24.0:** Monitor Pulse's internal adaptive polling scheduler and circuit breaker status.
+
+```bash
+GET /api/monitoring/scheduler/health
+```
+
+This endpoint provides detailed metrics about:
+- Task queue depths and processing times
+- Circuit breaker states per node
+- Backoff delays and retry schedules
+- Dead-letter queue entries (tasks that repeatedly fail)
+- Instance-level staleness tracking
+
+See [Scheduler Health API Documentation](api/SCHEDULER_HEALTH.md) for complete response schema and examples.
+
+**Key use cases:**
+- Monitor for polling backlogs
+- Detect connectivity issues via circuit breaker trips
+- Track node health and responsiveness
+- Identify failing tasks in the dead-letter queue
+
 #### PMG Mail Gateway Data

 When PMG instances are configured, the `pmg` array inside `/api/state` surfaces consolidated health and mail analytics for each gateway:
@ -933,12 +979,19 @@ GET /api/updates/plan?version=v4.30.0
 GET /api/updates/plan?version=v4.30.0&channel=rc
 ```

-Response example (systemd deployment):
+Response example (systemd deployment, v4.24.0+):

 ```json
 {
  "version": "v4.30.0",
  "channel": "stable",
+  "canAutoUpdate": true,
+  "requiresRoot": true,
+  "rollbackSupport": true,
+  "estimatedTime": "2-3 minutes",
+  "downloadUrl": "https://github.com/rcourtman/Pulse/releases/download/v4.30.0/pulse-linux-amd64.tar.gz",
+  "instructions": "Run the installer script with --version flag",
+  "prerequisites": ["systemd", "root access"],
  "steps": [
    "curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.30.0"
  ]
@ -972,7 +1025,49 @@ GET /api/updates/history                 # List recent update attempts (optional
 GET /api/updates/history/entry?id=<uuid> # Inspect a specific update event
 ```

-Entries include version, channel, timestamps, status, and error messaging for failed attempts.
+**Response format (v4.24.0+):**
+```json
+{
+  "entries": [
+    {
+      "id": "550e8400-e29b-41d4-a716-446655440000",
+      "action": "update",
+      "version": "v4.24.0",
+      "fromVersion": "v4.23.0",
+      "channel": "stable",
+      "status": "completed",
+      "timestamp": "2025-10-20T10:30:00Z",
+      "initiated_via": "ui",
+      "related_event_id": null,
+      "backup_path": "/opt/pulse/backups/pre-update-v4.23.0.tar.gz",
+      "duration_seconds": 120,
+      "error": null
+    },
+    {
+      "id": "650e8400-e29b-41d4-a716-446655440001",
+      "action": "rollback",
+      "version": "v4.23.0",
+      "fromVersion": "v4.24.0",
+      "channel": "stable",
+      "status": "completed",
+      "timestamp": "2025-10-20T11:00:00Z",
+      "initiated_via": "api",
+      "related_event_id": "550e8400-e29b-41d4-a716-446655440000",
+      "backup_path": null,
+      "duration_seconds": 45,
+      "error": null
+    }
+  ]
+}
+```
+
+Entries include:
+- `action`: "update" | "rollback"
+- `status`: "pending" | "in_progress" | "completed" | "failed"
+- `initiated_via`: How the action was started (ui, api, auto)
+- `related_event_id`: Links rollback to original update
+- `backup_path`: Location of pre-update backup
+- Error details for failed attempts

 ## Real-time Updates

@ -1016,10 +1111,45 @@ Returns simplified metrics without authentication requirements.

 ## Rate Limiting

-Some endpoints have rate limiting:
- Export/Import: 5 requests per minute
- Test email: 10 requests per minute
- Update check: 10 requests per hour
+**v4.24.0:** All responses include rate limit headers (`X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`). 429 responses add `Retry-After`.
+
+**Rate limits by endpoint category:**
+- **Authentication**: 10 attempts/minute per IP
+- **Config writes**: 30 requests/minute
+- **Exports**: 5 requests per 5 minutes
+- **Recovery operations**: 3 requests per 10 minutes
+- **Update operations**: 20 requests/minute
+- **WebSocket connections**: 5 connections/minute per IP
+- **General API**: 500 requests/minute per IP
+- **Public endpoints**: 1000 requests/minute per IP
+
+**Exempt endpoints** (no rate limits):
+- `/api/state` (real-time monitoring)
+- `/api/guests/metadata` (frequent polling)
+- WebSocket message streaming (after connection established)
+
+**Example response with rate limit headers:**
+```
+HTTP/1.1 200 OK
+X-RateLimit-Limit: 500
+X-RateLimit-Remaining: 487
+X-RateLimit-Reset: 1754995800
+Content-Type: application/json
+```
+
+**When rate limited:**
+```
+HTTP/1.1 429 Too Many Requests
+X-RateLimit-Limit: 500
+X-RateLimit-Remaining: 0
+X-RateLimit-Reset: 1754995800
+Retry-After: 60
+Content-Type: application/json
+
+{
+  "error": "Rate limit exceeded. Please retry after 60 seconds."
+}
+```

 ## Error Responses

--- a/docs/CONFIGURATION.md
+++ b/docs/CONFIGURATION.md
@ -107,22 +107,31 @@ PROXY_AUTH_LOGOUT_URL=/logout        # URL for SSO logout
 **Contents:**
 ```json
 {
-  "pbsPollingInterval": 60,        // Seconds between PBS refreshes (PVE polling fixed at 10s)
-  "pmgPollingInterval": 60,        // Seconds between PMG refreshes (mail analytics and health)
-  "connectionTimeout": 60,         // Seconds before node connection timeout
-  "autoUpdateEnabled": false,      // Systemd timer toggle for automatic updates
-  "autoUpdateCheckInterval": 24,   // Hours between auto-update checks
-  "autoUpdateTime": "03:00",       // Preferred update window (combined with randomized delay)
-  "updateChannel": "stable",       // Update channel: stable or rc
-  "allowedOrigins": "",            // CORS allowed origins (empty = same-origin only)
-  "allowEmbedding": false,         // Allow iframe embedding
-  "allowedEmbedOrigins": "",       // Comma-separated origins allowed to embed Pulse
-  "backendPort": 3000,             // Internal API listen port (not normally changed)
-  "frontendPort": 7655,            // Public port exposed by the service
-  "logLevel": "info",              // Log level: debug, info, warn, error
-  "discoveryEnabled": true,        // Enable/disable network discovery for Proxmox/PBS servers
-  "discoverySubnet": "auto",       // CIDR to scan ("auto" discovers common ranges)
-  "theme": ""                      // UI theme preference: "", "light", or "dark"
+  "pbsPollingInterval": 60,            // Seconds between PBS refreshes (PVE polling fixed at 10s)
+  "pmgPollingInterval": 60,            // Seconds between PMG refreshes (mail analytics and health)
+  "connectionTimeout": 60,             // Seconds before node connection timeout
+  "autoUpdateEnabled": false,          // Systemd timer toggle for automatic updates
+  "autoUpdateCheckInterval": 24,       // Hours between auto-update checks
+  "autoUpdateTime": "03:00",           // Preferred update window (combined with randomized delay)
+  "updateChannel": "stable",           // Update channel: stable or rc
+  "allowedOrigins": "",                // CORS allowed origins (empty = same-origin only)
+  "allowEmbedding": false,             // Allow iframe embedding
+  "allowedEmbedOrigins": "",           // Comma-separated origins allowed to embed Pulse
+  "backendPort": 3000,                 // Internal API listen port (not normally changed)
+  "frontendPort": 7655,                // Public port exposed by the service
+  "logLevel": "info",                  // Log level: debug, info, warn, error
+  "logFormat": "auto",                 // auto, json, or console output
+  "logFile": "",                       // Optional file path for mirrored logs
+  "logMaxSize": 100,                   // Log rotation threshold (MB) when logFile is set
+  "logMaxAge": 30,                     // Days to retain rotated files
+  "logCompress": true,                 // Compress rotated log files
+  "adaptivePollingEnabled": false,     // Toggle adaptive scheduler (v4.24.0+)
+  "adaptivePollingBaseInterval": 10,   // Target cadence (seconds)
+  "adaptivePollingMinInterval": 5,     // Fastest cadence (seconds)
+  "adaptivePollingMaxInterval": 300,   // Slowest cadence (seconds)
+  "discoveryEnabled": true,            // Enable/disable network discovery for Proxmox/PBS servers
+  "discoverySubnet": "auto",           // CIDR to scan ("auto" discovers common ranges)
+  "theme": ""                          // UI theme preference: "", "light", or "dark"
 }
 ```

@ -133,6 +142,23 @@ PROXY_AUTH_LOGOUT_URL=/logout        # URL for SSO logout
 - Missing file results in defaults being used
 - Changes take effect immediately (no restart required)
 - API tokens are no longer managed in system.json (moved to .env in v4.3.9+)
+- **Adaptive polling controls** (`adaptivePollingEnabled`, `adaptivePolling*Interval`) map directly to the Scheduler Health API and adjust queue/backoff behaviour in real time.
+- **Runtime logging controls** (`logLevel`, `logFormat`, `logFile`, `logMaxSize`, `logMaxAge`, `logCompress`) can be tuned from the UI or system.json; updates are applied immediately so you can raise verbosity, switch to structured JSON, or stream logs to disk without restarting Pulse.
+
+### Adaptive Polling Settings (v4.24.0+)
+
+- `adaptivePollingEnabled`: Enables the adaptive scheduler that prioritises stale or failing instances. Toggle it in **Settings → System → Adaptive polling** or set the flag in system.json.
+- `adaptivePollingBaseInterval`: Target cadence (seconds) when an instance is healthy. Defaults to 10 seconds.
+- `adaptivePollingMinInterval`: Lower bound when Pulse needs to poll aggressively (for example, 5 seconds for busy clusters).
+- `adaptivePollingMaxInterval`: Upper bound for idle instances. Setting this to a small value (≤15s) automatically engages the low-latency backoff profile (750 ms initial delay, 20 % jitter, 10 s breaker windows).
+- The adaptive scheduler feeds the `/api/monitoring/scheduler/health` endpoint and priority queue. Shorter intervals reduce queue depth; longer intervals trade freshness for fewer calls. All three intervals are stored in seconds in system.json; environment overrides accept Go duration strings such as `15s` or `5m`.
+
+### Logging Configuration (v4.24.0+)
+
+- `logLevel`: Runtime log verbosity (`debug`, `info`, `warn`, `error`). Raise it to `debug` temporarily when troubleshooting, then drop back to `info`.
+- `logFormat`: `auto` switches between human-friendly console output (interactive TTY) and JSON when Pulse runs under a service. Override with `json` to stream machine-readable logs everywhere, or `console` to force colourised output.
+- `logFile`: Optional absolute path. When populated, Pulse mirrors logs to this file as well as stdout. Rotation honours `logMaxSize` (MB), `logMaxAge` (days), and `logCompress` (gzip rotated files).
+- Logging changes made via the UI or system.json take effect immediately, so you can capture verbose traces or structured logs without scheduling downtime.

 ---

@ -294,6 +320,12 @@ Set `autoUpdateEnabled: true` in system.json or toggle in Settings UI.

 **Note**: Docker installations do not support automatic updates (use Docker's update mechanisms instead).

+### Update Backups & History (v4.24.0+)
+
+- Every self-update or rollback writes an entry to `<DATA_PATH>/update-history.jsonl` (defaults to `/var/lib/pulse` for systemd installs and `/data` in Docker). Review the log via **Settings → System → Updates**, or query `/api/updates/history` for automation.
+- The install script prints the configuration backup it creates (for example `/etc/pulse.backup.20251020-130500`). That path is captured in the history entry as `backup_path` so rollbacks know which snapshot to restore.
+- Update logs live under `/var/log/pulse/update-*.log`; grab the most recent file when filing support tickets or analysing failures.
+
 ---

 ## Configuration Priority
@ -313,7 +345,16 @@ These env vars override system.json values. When set, the UI will show a warning
 - `DISCOVERY_SUBNET` - Custom network to scan (default: auto-scans common networks)
 - `CONNECTION_TIMEOUT` - API timeout in seconds (default: 10)
 - `ALLOWED_ORIGINS` - CORS origins (default: same-origin only)
- `LOG_LEVEL` - Log verbosity: debug/info/warn/error (default: info)
+- `LOG_LEVEL` - Log verbosity: debug/info/warn/error (default: info). Switching levels takes effect immediately.
+- `LOG_FORMAT` - Override output format (`auto`, `json`, or `console`).
+- `LOG_FILE` - Mirror logs to this absolute path in addition to stdout (empty = stdout only).
+- `LOG_MAX_SIZE` - Rotate the log file after it grows beyond this many megabytes (default: 100).
+- `LOG_MAX_AGE` - Delete rotated log files older than this many days (default: 30).
+- `LOG_COMPRESS` - When `true` (default) gzip-compresses rotated log files.
+- `ADAPTIVE_POLLING_ENABLED` - Enable/disable the adaptive scheduler without touching system.json (`true`/`false`).
+- `ADAPTIVE_POLLING_BASE_INTERVAL` - Override the target polling cadence (accepts Go durations, e.g. `15s`).
+- `ADAPTIVE_POLLING_MIN_INTERVAL` - Override the minimum cadence (Go duration or seconds).
+- `ADAPTIVE_POLLING_MAX_INTERVAL` - Override the maximum cadence (Go duration or seconds). Values ≤`15s` engage the low-latency backoff profile.
 - `ENABLE_BACKUP_POLLING` - Set to `false` to disable polling of Proxmox backup/snapshot APIs (default: true)
 - `BACKUP_POLLING_INTERVAL` - Override the backup polling cadence. Accepts Go duration syntax (e.g. `30m`, `6h`) or seconds. Use `0` for Pulse's default (~90s) cadence.
 - `PULSE_PUBLIC_URL` - Full URL to access Pulse (e.g., `http://192.168.1.100:7655`)
--- a/docs/FAQ.md
+++ b/docs/FAQ.md
@ -92,6 +92,53 @@ Yes! When you add one cluster node, Pulse automatically discovers and monitors a
 ### High memory usage?
 Reduce `metricsRetentionDays` in settings and restart

+### How do I monitor adaptive polling?
+**New in v4.24.0:** Pulse includes adaptive polling that automatically adjusts polling intervals based on system load.
+
+**Monitor adaptive polling:**
+- **Dashboard**: Settings → System → Monitoring shows scheduler health status
+- **API**: `/api/monitoring/scheduler/health` provides detailed metrics including:
+  - Queue depths and processing times
+  - Circuit breaker status
+  - Backoff states
+  - Instance metadata
+- **Logging**: Enable debug logging to see detailed polling behavior
+
+**Key metrics to watch:**
+- Queue depth (alerts if backlog builds up)
+- Circuit breaker trips (indicates connectivity issues)
+- Backoff delays (shows throttling behavior)
+
+See [Adaptive Polling Documentation](monitoring/ADAPTIVE_POLLING.md) for complete details.
+
+### What's new about rate limiting in v4.24.0?
+Pulse now returns standard rate limit headers with all API responses:
+
+**Response Headers:**
+- `X-RateLimit-Limit`: Maximum requests allowed per window (e.g., 500)
+- `X-RateLimit-Remaining`: Requests remaining in current window
+- `Retry-After`: Seconds to wait before retrying (on 429 responses)
+
+**Rate Limits:**
+- **Auth endpoints**: 10 attempts/minute per IP
+- **General API**: 500 requests/minute per IP
+- **Real-time endpoints**: No limits (WebSocket, SSE)
+
+**Example Response:**
+```
+HTTP/1.1 200 OK
+X-RateLimit-Limit: 500
+X-RateLimit-Remaining: 487
+```
+
+When you hit the limit:
+```
+HTTP/1.1 429 Too Many Requests
+X-RateLimit-Limit: 500
+X-RateLimit-Remaining: 0
+Retry-After: 60
+```
+
 ## Features

 ### Why do VMs show "-" for disk usage?
@ -175,13 +222,72 @@ First, confirm the agent is still running (`systemctl status pulse-docker-agent`
 ## Updates

 ### How to update?
- **Docker**: Pull latest image, recreate container  
+- **Docker**: Pull latest image, recreate container
 - **Manual/systemd**: Run the install script again: `curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash`

+### Can I roll back if an update misbehaves?
+**New in v4.24.0:** Yes! Pulse now retains previous versions and provides easy rollback.
+
+**Via UI (Recommended):**
+1. Navigate to **Settings → System → Updates**
+2. Click **"Restore previous version"**
+3. Confirm rollback
+4. Pulse restarts with the previous working version
+
+**Via CLI:**
+```bash
+# Systemd installations
+sudo /opt/pulse/pulse config rollback
+
+# LXC containers
+pct exec <ctid> -- bash -c "cd /opt/pulse && ./pulse config rollback"
+```
+
+**What gets rolled back:**
+- Pulse binary and frontend assets
+- System configuration (preserved from previous version)
+- Rollback history tracked in Updates view
+
+**What stays the same:**
+- Your node configurations
+- Alert settings
+- User credentials
+- Historical metrics data
+
+Check rollback logs: `journalctl -u pulse | grep rollback`
+
 ### How do I install an older release (downgrade)?
- **Manual/systemd installs**: rerun the installer and pass the tag you want, e.g. `curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.20.0`
- **Proxmox LXC appliance**: `pct exec <ctid> -- bash -lc "curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.20.0"`
- **Docker**: launch with a versioned tag instead of `latest`, e.g. `docker run -d --name pulse -p 7655:7655 rcourtman/pulse:v4.20.0`
+- **Manual/systemd installs**: rerun the installer and pass the tag you want, e.g. `curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.24.0`
+- **Proxmox LXC appliance**: `pct exec <ctid> -- bash -lc "curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.24.0"`
+- **Docker**: launch with a versioned tag instead of `latest`, e.g. `docker run -d --name pulse -p 7655:7655 rcourtman/pulse:v4.24.0`
+
+### How do I adjust logging without restarting?
+**New in v4.24.0:** Pulse supports runtime logging configuration—no restart required!
+
+**Via UI:**
+1. Navigate to **Settings → System → Logging**
+2. Adjust:
+   - **Log Level**: debug, info, warn, error
+   - **Log Format**: json, text
+   - **File Rotation**: size limits, retention
+3. Changes apply immediately
+
+**Via Environment Variables:**
+```bash
+# Systemd
+sudo systemctl edit pulse
+[Service]
+Environment="LOG_LEVEL=debug"
+Environment="LOG_FORMAT=json"
+
+# Docker
+docker run -e LOG_LEVEL=debug -e LOG_FORMAT=json rcourtman/pulse:latest
+```
+
+**Use cases:**
+- Enable debug logging temporarily for troubleshooting
+- Switch to JSON format for log aggregation
+- Adjust file rotation to manage disk usage

 ### Why can't I update from the UI?
 For security reasons, Pulse cannot self-update. The UI will notify you when updates are available and show the appropriate update command for your deployment type.
--- a/docs/INSTALL.md
+++ b/docs/INSTALL.md
@ -85,6 +85,9 @@ curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | b

 # Or force enable with flag
 curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --enable-auto-updates
+
+# Install specific version (e.g., v4.24.0)
+curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.24.0
 ```

 #### Enable/Disable After Installation
@ -104,6 +107,7 @@ systemctl status pulse-update.timer         # Check status
 - Creates backup before updating
 - Automatically rolls back if update fails
 - Logs all activity to systemd journal
+- **New in v4.24.0**: Rollback history is retained in Settings → System → Updates; use the new 'Restore previous version' button if the latest build regresses

 #### View Update Logs
 ```bash
@ -132,11 +136,35 @@ docker rm pulse
 docker run -d --name pulse -p 7655:7655 -v pulse_data:/data rcourtman/pulse:latest
 ```

+### Rollback to Previous Version
+
+**New in v4.24.0:** Pulse retains previous versions and allows easy rollback if an update causes issues.
+
+#### Via UI (Recommended)
+1. Navigate to **Settings → System → Updates**
+2. Click **"Restore previous version"** button
+3. Confirm rollback
+4. Pulse will restart with the previous working version
+
+#### Via CLI
+```bash
+# For systemd installations
+sudo /opt/pulse/pulse config rollback
+
+# For LXC containers
+pct exec <container-id> -- bash -c "cd /opt/pulse && ./pulse config rollback"
+```
+
+Rollback history and metadata are tracked in the Updates view. Check system journal for detailed rollback logs:
+```bash
+journalctl -u pulse | grep rollback
+```
+
 ## Version Management

 ### Install Specific Version
 ```bash
-curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.8.0
+curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.24.0
 ```

 ### Install Release Candidate
@ -155,6 +183,34 @@ curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | b
 ```
 **Note:** This builds Pulse from source code on your machine. Requires Go, Node.js, and npm.

+## Advanced Configuration
+
+### Runtime Logging Configuration
+
+**New in v4.24.0:** Adjust logging settings without restarting Pulse.
+
+#### Via UI
+Navigate to **Settings → System → Logging** to configure:
+- **Log Level**: debug, info, warn, error
+- **Log Format**: json, text
+- **File Rotation**: size limits and retention
+
+#### Via Environment Variables
+```bash
+# Systemd
+sudo systemctl edit pulse
+[Service]
+Environment="LOG_LEVEL=debug"
+Environment="LOG_FORMAT=json"
+
+# Docker
+docker run -e LOG_LEVEL=debug -e LOG_FORMAT=json rcourtman/pulse:latest
+```
+
+### Adaptive Polling
+
+**New in v4.24.0:** Adaptive polling is now enabled by default, automatically adjusting polling intervals based on system load and responsiveness. Monitor status via **Settings → System → Monitoring** or the new Scheduler Health API at `/api/monitoring/scheduler/health`.
+
 ## Troubleshooting

 ### Permission Denied
--- a/docs/MIGRATION.md
+++ b/docs/MIGRATION.md
@ -1,5 +1,7 @@
 # Migrating Pulse

+**Updated for Pulse v4.24.0**
+
 ## Quick Migration Guide

 ### ❌ DON'T: Copy files directly
@ -19,12 +21,16 @@ Never copy `/etc/pulse` or `/var/lib/pulse` directories between systems:

 #### Importing (New Server)
 1. Install fresh Pulse instance
-2. Open Pulse web interface  
+2. Open Pulse web interface
 3. Go to **Settings** → **Configuration Management**
 4. Click **Import Configuration**
 5. Select your exported file
 6. Enter the same passphrase
 7. Click Import
+8. **Post-migration verification (v4.24.0+)**:
+   - Check scheduler health: `curl -s http://localhost:7655/api/monitoring/scheduler/health | jq`
+   - Verify adaptive polling status: **Settings → System → Monitoring**
+   - Confirm all nodes are connected and polling correctly

 ## What Gets Migrated

@ -40,7 +46,9 @@ Never copy `/etc/pulse` or `/var/lib/pulse` directories between systems:
 - Historical metrics data
 - Alert history
 - Authentication settings (passwords, API tokens)
+- **Updates rollback history** (v4.24.0+)
 - Each instance should configure its own authentication
+- **Note:** Updates rollback data isn't transferred and must be rebuilt by running one successful update cycle on the new host

 ## Common Scenarios

@ -75,6 +83,7 @@ The export/import process works across all installation methods:
 - **Safe to Store**: Encrypted exports can be stored in cloud backups
 - **Minimum 12 characters**: Use a strong passphrase
 - **Password Manager**: Store your passphrase securely
+- **Rollback History**: Updates rollback data isn't included in exports; rebuild by running one successful update on the new host

 ## Troubleshooting

@ -90,6 +99,12 @@ The export/import process works across all installation methods:
 - Node IPs may have changed
 - Update node addresses in Settings

+**Logging issues after migration (v4.24.0+)**
+- If you lose logs after migration, ensure the runtime logging configuration persisted
+- Toggle **Settings → System → Logging** to your desired level
+- Check environment variables: `LOG_LEVEL`, `LOG_FORMAT`
+- Verify log file rotation settings are correct
+
 ## Pro Tips

 1. **Test imports**: Try importing on a test instance first
--- a/docs/RELEASE.md
+++ b/docs/RELEASE.md
@ -4,7 +4,7 @@ Use this checklist when preparing and publishing a new Pulse release.

 ## Pre-release

- [ ] Ensure `VERSION` is up to date and matches the tag you plan to cut (format `4.x.y`)
+- [ ] Ensure `VERSION` is set to `4.24.0` and matches the tag you plan to cut (format `4.x.y`)
 - [ ] Confirm the Helm chart renders and installs locally:
  ```bash
  helm lint deploy/helm/pulse --strict
@ -26,6 +26,8 @@ Use this checklist when preparing and publishing a new Pulse release.
  kubectl -n pulse get pods
  kind delete cluster
  ```
+- [ ] Confirm adaptive polling, scheduler health API, rollback UI, logging runtime controls, and rate-limit header documentation are updated before tagging v4.24.0
+- [ ] Smoke-test updates rollback: apply a test update via Settings → System → Updates, trigger a rollback, and verify journal entries document the rollback event

 ## Publishing

@ -62,11 +64,21 @@ Use this checklist when preparing and publishing a new Pulse release.
     --create-namespace
   ```

+   **For v4.24.0 specifically**, highlight these features in the release notes:
+   - Adaptive polling (now GA)
+   - Scheduler health API with rich instance metadata
+   - Updates rollback workflow
+   - Shared script library system (now GA)
+   - X-RateLimit-* headers for all API responses
+   - Runtime logging configuration (no restart required)
+
 6. Mention any chart-breaking changes (new values, migrations) in the release notes.

 ## Post-release

 - [ ] Verify `helm show chart oci://ghcr.io/rcourtman/pulse-chart --version 4.x.y` shows the expected metadata (version, appVersion, icon)
 - [ ] Run `helm install` against a test cluster (Kind/k3s) using the published OCI artifact
+- [ ] Run `curl -s http://<host>:7655/api/monitoring/scheduler/health | jq` to ensure the scheduler health endpoint is live
+- [ ] Verify the Updates view reports rollback metadata and X-RateLimit-* headers appear in API responses
 - [ ] Announce the release with links to both the GitHub release and the Helm installation instructions (`docs/KUBERNETES.md`)
 - [ ] Verify signatures: `gpg --verify checksums.txt.asc checksums.txt`
--- a/docs/api/SCHEDULER_HEALTH.md
+++ b/docs/api/SCHEDULER_HEALTH.md
@ -1,9 +1,19 @@
 # Scheduler Health API

+**New in v4.24.0**
+
 Endpoint: `GET /api/monitoring/scheduler/health`

 Returns a snapshot of the adaptive polling scheduler, queue state, circuit breakers, and per-instance status. Requires authentication (session cookie or bearer token).

+**Key Features:**
+- Real-time scheduler health monitoring
+- Circuit breaker status per instance
+- Dead-letter queue tracking (tasks that repeatedly fail)
+- Per-instance staleness metrics
+- No query parameters required
+- Read-only endpoint (rate-limited under general 500 req/min bucket)
+
 ---

 ## Request
@ -21,16 +31,22 @@ No query parameters are needed.

 ```json
 {
-  "updatedAt": "2025-10-20T13:05:42Z",
-  "enabled": true,
+  "updatedAt": "2025-10-20T13:05:42Z",  // RFC 3339 timestamp
+  "enabled": true,                       // Mirrors AdaptivePollingEnabled setting
  "queue": {...},
  "deadLetter": {...},
-  "breakers": [...],          // legacy summary
-  "staleness": [...],         // legacy summary
-  "instances": [ ... ]       // enhanced per-instance view
+  "breakers": [...],          // legacy summary (for backward compatibility)
+  "staleness": [...],         // legacy summary (for backward compatibility)
+  "instances": [ ... ]        // authoritative per-instance view (v4.24.0+)
 }
 ```

+**Field Notes:**
+- `updatedAt`: RFC 3339 timestamp of when this snapshot was generated
+- `enabled`: Reflects the current `AdaptivePollingEnabled` system setting
+- `breakers` and `staleness`: Legacy arrays maintained for backward compatibility; use `instances` for complete data
+- `instances`: Authoritative source for per-instance health (v4.24.0+)
+
 ### Queue Snapshot (`queue`)

 | Field | Type | Description |
@ -44,7 +60,9 @@ No query parameters are needed.
 | Field | Type | Description |
 |-------|------|-------------|
 | `count` | integer | Total items in the dead-letter queue |
-| `tasks` | array | Top entries (legacy format; limited set) |
+| `tasks` | array | **Limited to 25 entries** for performance. Each task includes `instance`, `type`, `nextRun`, `lastError`, and `failures` count. For complete per-instance DLQ data, use `instances[].deadLetter` |
+
+**Note:** The top-level `deadLetter.tasks` array is capped at 25 items to prevent large responses. Use the `instances` array for exhaustive coverage.

 ### Instances (`instances`)

@ -65,20 +83,37 @@ Each element gives a complete view of one instance.

 | Field | Type | Description |
 |-------|------|-------------|
-| `lastSuccess` | timestamp nullable | Most recent successful poll |
-| `lastError` | object nullable | `{ at, message, category }` (`category` is `transient` or `permanent`) |
-| `consecutiveFailures` | integer | Failure streak length |
-| `firstFailureAt` | timestamp nullable | When the streak began |
+| `lastSuccess` | timestamp nullable | RFC 3339 timestamp of most recent successful poll |
+| `lastError` | object nullable | `{ at, message, category }` where `at` is RFC 3339, `message` describes the error, and `category` is `transient` (network issues, timeouts) or `permanent` (auth failures, invalid config) |
+| `consecutiveFailures` | integer | Current failure streak length (resets on successful poll) |
+| `firstFailureAt` | timestamp nullable | **New in v4.24.0**: RFC 3339 timestamp when the current failure streak began. Useful for calculating failure duration |
+
+**Timing Metadata (v4.24.0+):**
+- `firstFailureAt`: Tracks when a failure streak started, enabling "failing for X minutes" calculations
+- Resets to `null` when a successful poll occurs
+- Combine with `consecutiveFailures` to assess severity

 #### Breaker (`breaker`)

 | Field | Type | Description |
 |-------|------|-------------|
-| `state` | string | `closed`, `open`, `half_open`, or `unknown` |
-| `since` | timestamp nullable | When current state began |
-| `lastTransition` | timestamp nullable | Last transition time |
-| `retryAt` | timestamp nullable | Scheduled retry time when applicable |
-| `failureCount` | integer | Failures counted in the current breaker cycle |
+| `state` | string | `closed` (healthy), `open` (failing), `half_open` (testing recovery), or `unknown` (not initialized) |
+| `since` | timestamp nullable | **New in v4.24.0**: RFC 3339 timestamp when the current state began. Use to calculate how long a breaker has been open |
+| `lastTransition` | timestamp nullable | **New in v4.24.0**: RFC 3339 timestamp of the most recent state change (e.g., closed → open) |
+| `retryAt` | timestamp nullable | **New in v4.24.0**: RFC 3339 timestamp of next scheduled retry attempt when breaker is open or half-open |
+| `failureCount` | integer | **New in v4.24.0**: Number of failures in the current breaker cycle. Resets when breaker closes |
+
+**Circuit Breaker Timing (v4.24.0+):**
+- `since`: When did the current state start? (e.g., "breaker has been open for 5 minutes")
+- `lastTransition`: When was the last state change? (useful for detecting flapping)
+- `retryAt`: When will the next retry attempt occur? (for open/half-open states)
+- `failureCount`: How many failures have accumulated? (triggers state transitions)
+
+**State Transitions:**
+- `closed` → `open`: Triggered after N failures (default: 5)
+- `open` → `half_open`: After timeout period, allows one test request
+- `half_open` → `closed`: If test request succeeds
+- `half_open` → `open`: If test request fails

 #### Dead-letter (`deadLetter`)

@ -251,6 +286,29 @@ The `instances` array centralizes per-instance telemetry; existing integrations

 ---

+## Operational Notes
+
+**v4.24.0 Behavior:**
+- **Read-only endpoint**: This endpoint is informational only and does not modify scheduler state
+- **Rate limiting**: Falls under the general API limit (500 requests/minute per IP)
+- **Authentication required**: Must provide valid session cookie or API token
+- **Adaptive polling disabled**: When adaptive polling is disabled (`enabled: false`), the response includes empty `breakers`, `staleness`, and `instances` arrays
+- **Real-time data**: Reflects current scheduler state; not historical (for trends, use metrics/logs)
+- **No query parameters**: Returns complete snapshot on every request
+- **Automatic adjustments**: The `enabled` field automatically reflects the `AdaptivePollingEnabled` system setting
+
+**Use Cases:**
+- **Monitoring dashboards**: Embed in Grafana/Prometheus for real-time scheduler health
+- **Alerting**: Trigger alerts on open circuit breakers or high DLQ counts
+- **Debugging**: Investigate why specific instances aren't polling successfully
+- **Capacity planning**: Monitor queue depth trends to assess if polling intervals need adjustment
+
+**Breaking Changes:**
+- **None**: v4.24.0 only adds fields; all existing consumers continue to work
+- Consumers just gain access to richer metadata (`firstFailureAt`, breaker timestamps, DLQ retry windows)
+
+---
+
 ## Troubleshooting Examples

 1. **Transient outages:** look for `pollStatus.lastError.category == "transient"` to confirm network hiccups; check `breaker.retryAt` to see when retries resume.
--- a/docs/monitoring/ADAPTIVE_POLLING.md
+++ b/docs/monitoring/ADAPTIVE_POLLING.md
@ -28,28 +28,43 @@ Phase 2 introduces a scheduler that adapts poll cadence based on freshness, er

 ## Configuration

+**v4.24.0:** Adaptive polling is **enabled by default** but can be toggled without restart.
+
+### Via UI
+Navigate to **Settings → System → Monitoring** to enable/disable adaptive polling. Changes apply immediately without requiring a restart.
+
+### Via Environment Variables
 Environment variables (default in `internal/config/config.go`):

 | Variable                            | Default | Description                                      |
 |-------------------------------------|---------|--------------------------------------------------|
-| `ADAPTIVE_POLLING_ENABLED`          | false   | Feature flag for adaptive scheduler.             |
-| `ADAPTIVE_POLLING_BASE_INTERVAL`    | 10s     | Target cadence when system is healthy.           |
-| `ADAPTIVE_POLLING_MIN_INTERVAL`     | 5s      | Lower bound (active instances).                  |
-| `ADAPTIVE_POLLING_MAX_INTERVAL`     | 5m      | Upper bound (idle instances).                    |
+| `ADAPTIVE_POLLING_ENABLED`          | true    | **Changed in v4.24.0**: Now enabled by default   |
+| `ADAPTIVE_POLLING_BASE_INTERVAL`    | 10s     | Target cadence when system is healthy            |
+| `ADAPTIVE_POLLING_MIN_INTERVAL`     | 5s      | Lower bound (active instances)                   |
+| `ADAPTIVE_POLLING_MAX_INTERVAL`     | 5m      | Upper bound (idle instances)                     |

-All settings persist in `system.json` and respond to environment overrides.
+All settings persist in `system.json` and respond to environment overrides. **Changes apply without restart** when modified via UI.

 ## Metrics

+**v4.24.0:** Extended metrics for comprehensive monitoring.
+
 Exposed via Prometheus (`:9091/metrics`):

-| Metric                                   | Type      | Labels                                | Description                                |
-|------------------------------------------|-----------|---------------------------------------|--------------------------------------------|
-| `pulse_monitor_poll_total`               | counter   | `instance_type`, `instance`, `result` | Overall poll attempts (success/error).     |
-| `pulse_monitor_poll_duration_seconds`    | histogram | `instance_type`, `instance`           | Poll latency per instance.                 |
-| `pulse_monitor_poll_staleness_seconds`   | gauge     | `instance_type`, `instance`           | Age since last success (0 on success).     |
-| `pulse_monitor_poll_queue_depth`         | gauge     | —                                     | Size of priority queue.                    |
-| `pulse_monitor_poll_inflight`            | gauge     | `instance_type`                       | Concurrent tasks per type.                 |
+| Metric                                      | Type      | Labels                                | Description                                     |
+|---------------------------------------------|-----------|---------------------------------------|-------------------------------------------------|
+| `pulse_monitor_poll_total`                  | counter   | `instance_type`, `instance`, `result` | Overall poll attempts (success/error)           |
+| `pulse_monitor_poll_duration_seconds`       | histogram | `instance_type`, `instance`           | Poll latency per instance                       |
+| `pulse_monitor_poll_staleness_seconds`      | gauge     | `instance_type`, `instance`           | Age since last success (0 on success)           |
+| `pulse_monitor_poll_queue_depth`            | gauge     | —                                     | Size of priority queue                          |
+| `pulse_monitor_poll_inflight`               | gauge     | `instance_type`                       | Concurrent tasks per type                       |
+| `pulse_monitor_poll_errors_total`           | counter   | `instance_type`, `instance`, `category` | **New in v4.24.0**: Error counts by category (transient/permanent) |
+| `pulse_monitor_poll_last_success_timestamp` | gauge     | `instance_type`, `instance`           | **New in v4.24.0**: Unix timestamp of last successful poll |
+
+**Alerting Recommendations:**
+- Alert when `pulse_monitor_poll_staleness_seconds` > 120 for critical instances
+- Alert when `pulse_monitor_poll_queue_depth` > 50 (backlog building)
+- Alert when `pulse_monitor_poll_errors_total` with `category=permanent` increases (auth/config issues)

 ## Circuit Breaker & Backoff