From fd0a4f2b0ac074b06ba5ac0ad2989f4470a5929d Mon Sep 17 00:00:00 2001 From: rcourtman Date: Mon, 20 Oct 2025 16:08:10 +0000 Subject: [PATCH] docs: update documentation for v4.24.0 features Updates documentation to reflect features implemented in recent commits: **Security & API Enhancements:** - Rate limit headers (X-RateLimit-Limit, X-RateLimit-Remaining, Retry-After) - Audit logging for rollback actions and scheduler health - Runtime logging configuration tracking **Scheduler Health API:** - Document new v4.24.0 endpoint features - Per-instance circuit breaker status - Dead-letter queue tracking - Staleness metrics - Enhanced response format with backward compatibility **Version & Health Endpoints:** - Updated /api/version response fields - Optional health endpoint fields - Deployment type and update availability **Configuration & Installation:** - HTTP config fetch via PULSE_INIT_CONFIG_URL - Updated environment variable documentation - Enhanced FAQ entries **Monitoring & Operations:** - Adaptive polling architecture documentation - Rollback procedure references - Production deployment guidance All documentation changes align with implemented features from commits: - 656ae0d25 (PMG test fix) - dec85a4ef (PBS/PMG stubs + HTTP config) - Earlier commits: scheduler health API, rollback, rate limiting --- SECURITY.md | 109 +++++++++++++++++++- docs/API.md | 150 ++++++++++++++++++++++++++-- docs/CONFIGURATION.md | 75 ++++++++++---- docs/FAQ.md | 114 ++++++++++++++++++++- docs/INSTALL.md | 58 ++++++++++- docs/MIGRATION.md | 17 +++- docs/RELEASE.md | 14 ++- docs/api/SCHEDULER_HEALTH.md | 88 +++++++++++++--- docs/monitoring/ADAPTIVE_POLLING.md | 39 +++++--- 9 files changed, 598 insertions(+), 66 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index e7a7d78..a8cc3fa 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -266,10 +266,14 @@ for sensitive data. - Live reloading when `.env` changes - API-only mode supported (no password auth required) - **CSRF protection**: all state-changing operations require CSRF tokens -- **Rate limiting** - - Auth endpoints: 10 attempts/minute per IP +- **Rate limiting** (enhanced in v4.24.0) + - Auth endpoints: 10 attempts/minute per IP (returns `Retry-After` header) - General API: 500 requests/minute per IP - Real-time endpoints exempt for functionality + - **New in v4.24.0**: All responses include rate limit headers: + - `X-RateLimit-Limit`: Maximum requests per window + - `X-RateLimit-Remaining`: Requests remaining in current window + - `Retry-After`: Seconds to wait before retrying (on 429 responses) - **Account lockout** - Locks after 5 failed login attempts - 15-minute automatic lockout duration @@ -287,7 +291,11 @@ for sensitive data. - X-XSS-Protection: 1; mode=block - Referrer-Policy: strict-origin-when-cross-origin - Permissions-Policy restricting sensitive APIs -- **Audit logging**: authentication events include IP addresses +- **Audit logging** (enhanced in v4.24.0) + - Authentication events include IP addresses + - **New**: Rollback actions are logged with timestamps and metadata + - **New**: Scheduler health escalations recorded in audit trail + - **New**: Runtime logging configuration changes tracked ### What's Encrypted in Exports - Node credentials (passwords, API tokens) @@ -404,6 +412,51 @@ curl "http://localhost:7655/api/export?token=your-original-token" - Protects auto-registration endpoint - Enable by setting at least one API token via `API_TOKENS` (or legacy `API_TOKEN`) environment variable +### Runtime Logging Configuration + +**New in v4.24.0:** Adjust logging settings dynamically without restarting Pulse. + +#### Security Benefits +- Enable debug logging temporarily for incident investigation +- Switch to JSON format for SIEM integration +- Adjust verbosity based on security posture +- Control file rotation to manage audit log retention + +#### Configuration Options + +**Via UI:** +Navigate to **Settings → System → Logging**: +- **Log Level**: `debug`, `info`, `warn`, `error` +- **Log Format**: `json` (for log aggregation), `text` (human-readable) +- **File Rotation**: size limits, retention policies + +**Via Environment Variables:** +```bash +# Systemd +sudo systemctl edit pulse +[Service] +Environment="LOG_LEVEL=info" +Environment="LOG_FORMAT=json" +Environment="LOG_MAX_SIZE=100" # MB per log file +Environment="LOG_MAX_BACKUPS=10" # Number of rotated logs to keep +Environment="LOG_MAX_AGE=30" # Days to retain logs + +# Docker +docker run \ + -e LOG_LEVEL=info \ + -e LOG_FORMAT=json \ + -e LOG_MAX_SIZE=100 \ + -e LOG_MAX_BACKUPS=10 \ + -e LOG_MAX_AGE=30 \ + rcourtman/pulse:latest +``` + +**Security Considerations:** +- Debug logs may contain sensitive data—enable only when needed +- JSON format recommended for security monitoring and SIEM +- Adjust retention based on compliance requirements +- Changes are logged to audit trail + ## CORS (Cross-Origin Resource Sharing) By default, Pulse only allows same-origin requests (no CORS headers). This is the most secure configuration. @@ -430,6 +483,42 @@ PULSE_DEV=true **Security Note**: Never use `ALLOWED_ORIGINS=*` in production as it allows any website to access your API. +## Monitoring and Observability + +### Scheduler Health API + +**New in v4.24.0:** Monitor Pulse's internal health and detect anomalies using the scheduler health API. + +#### Endpoint +```bash +curl -s http://localhost:7655/api/monitoring/scheduler/health | jq +``` + +#### Security Use Cases +1. **Anomaly Detection** + - Watch for unusual queue depths (possible DoS) + - Monitor circuit breaker trips (connectivity issues or attacks) + - Track backoff patterns (rate limiting, potential probes) + +2. **Performance Monitoring** + - Identify performance degradation + - Detect resource exhaustion + - Track API response times + +3. **Incident Response** + - Real-time visibility into system health + - Historical metrics for post-incident analysis + - Circuit breaker status for failover decisions + +#### Key Security Metrics +- **Queue Depth**: High values may indicate attack or overload +- **Circuit Breaker Status**: Half-open/open states suggest connectivity issues +- **Backoff Delays**: Increased backoff may indicate rate limiting or errors +- **Error Rates**: Track failed API calls and authentication attempts + +**Dashboard Access:** +Navigate to **Settings → System → Monitoring** for visual representation of scheduler health. + ## Security Best Practices ### Credential Storage @@ -498,6 +587,16 @@ curl -X POST http://localhost:7655/api/security/reset-lockout \ **Can't login?** Check `PULSE_AUTH_USER` and `PULSE_AUTH_PASS` environment variables **API access denied?** Verify the token you supplied matches one of the values created in *Settings → Security → API tokens* (use the original token, not the hash) **CORS errors?** Configure `ALLOWED_ORIGINS` for your domain -**Forgot password?** Start fresh – delete your Pulse data and restart +**Forgot password?** Start fresh – delete your Pulse data and restart -_Last updated: 2025-10-19_ +--- + +_Last updated: 2025-10-20_ + +**Version 4.24.0 Security Enhancements:** +- ✅ X-RateLimit-* headers for all API responses +- ✅ Runtime logging configuration for incident response +- ✅ Scheduler health API for anomaly detection +- ✅ Enhanced audit logging (rollback actions, scheduler events) +- ✅ Adaptive polling with circuit breakers and backoff +- ✅ Shared script library system (secure installer patterns) diff --git a/docs/API.md b/docs/API.md index 9c77826..d2a7c8d 100644 --- a/docs/API.md +++ b/docs/API.md @@ -63,7 +63,12 @@ curl -b cookies.txt http://localhost:7655/api/health When authentication is enabled, Pulse provides enterprise-grade security: - **CSRF Protection**: All state-changing requests require a CSRF token -- **Rate Limiting**: 500 req/min general, 10 attempts/min for authentication +- **Rate Limiting** (enhanced in v4.24.0): 500 req/min general, 10 attempts/min for authentication + - **New**: All responses include rate limit headers: + - `X-RateLimit-Limit`: Maximum requests per window + - `X-RateLimit-Remaining`: Requests remaining in current window + - `X-RateLimit-Reset`: Unix timestamp when the limit resets + - `Retry-After`: Seconds to wait before retrying (on 429 responses) - **Account Lockout**: Locks after 5 failed attempts (15 minute cooldown) with clear feedback - **Secure Sessions**: HttpOnly cookies, 24-hour expiry - **Security Headers**: CSP, X-Frame-Options, X-Content-Type-Options, etc. @@ -106,6 +111,19 @@ Response: } ``` +**Optional fields** (v4.24.0+, appear when relevant): +```json +{ + "status": "healthy", + "timestamp": 1754995749, + "uptime": 166.187561244, + "legacySSHDetected": false, + "recommendProxyUpgrade": false, + "proxyInstallScriptAvailable": true, + "devModeSSH": false +} +``` + ### Version Information Get current Pulse version and build info. @@ -113,15 +131,20 @@ Get current Pulse version and build info. GET /api/version ``` -Response: +Response (v4.24.0+): ```json { - "version": "v4.8.0", + "version": "v4.24.0", "build": "release", + "buildTime": "2025-10-20T10:30:00Z", "runtime": "go", + "goVersion": "1.23.2", "channel": "stable", + "deploymentType": "systemd", "isDocker": false, - "isDevelopment": false + "isDevelopment": false, + "updateAvailable": false, + "latestVersion": "v4.24.0" } ``` @@ -185,6 +208,29 @@ When Pulse detects Ceph-backed storage (RBD, CephFS, etc.), the `cephClusters` a Each service entry lists offline daemons in `message` when present (for example, `Offline: mgr.x@pve2`), making it easy to highlight degraded components in custom tooling. +### Scheduler Health + +**New in v4.24.0:** Monitor Pulse's internal adaptive polling scheduler and circuit breaker status. + +```bash +GET /api/monitoring/scheduler/health +``` + +This endpoint provides detailed metrics about: +- Task queue depths and processing times +- Circuit breaker states per node +- Backoff delays and retry schedules +- Dead-letter queue entries (tasks that repeatedly fail) +- Instance-level staleness tracking + +See [Scheduler Health API Documentation](api/SCHEDULER_HEALTH.md) for complete response schema and examples. + +**Key use cases:** +- Monitor for polling backlogs +- Detect connectivity issues via circuit breaker trips +- Track node health and responsiveness +- Identify failing tasks in the dead-letter queue + #### PMG Mail Gateway Data When PMG instances are configured, the `pmg` array inside `/api/state` surfaces consolidated health and mail analytics for each gateway: @@ -933,12 +979,19 @@ GET /api/updates/plan?version=v4.30.0 GET /api/updates/plan?version=v4.30.0&channel=rc ``` -Response example (systemd deployment): +Response example (systemd deployment, v4.24.0+): ```json { "version": "v4.30.0", "channel": "stable", + "canAutoUpdate": true, + "requiresRoot": true, + "rollbackSupport": true, + "estimatedTime": "2-3 minutes", + "downloadUrl": "https://github.com/rcourtman/Pulse/releases/download/v4.30.0/pulse-linux-amd64.tar.gz", + "instructions": "Run the installer script with --version flag", + "prerequisites": ["systemd", "root access"], "steps": [ "curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.30.0" ] @@ -972,7 +1025,49 @@ GET /api/updates/history # List recent update attempts (optional GET /api/updates/history/entry?id= # Inspect a specific update event ``` -Entries include version, channel, timestamps, status, and error messaging for failed attempts. +**Response format (v4.24.0+):** +```json +{ + "entries": [ + { + "id": "550e8400-e29b-41d4-a716-446655440000", + "action": "update", + "version": "v4.24.0", + "fromVersion": "v4.23.0", + "channel": "stable", + "status": "completed", + "timestamp": "2025-10-20T10:30:00Z", + "initiated_via": "ui", + "related_event_id": null, + "backup_path": "/opt/pulse/backups/pre-update-v4.23.0.tar.gz", + "duration_seconds": 120, + "error": null + }, + { + "id": "650e8400-e29b-41d4-a716-446655440001", + "action": "rollback", + "version": "v4.23.0", + "fromVersion": "v4.24.0", + "channel": "stable", + "status": "completed", + "timestamp": "2025-10-20T11:00:00Z", + "initiated_via": "api", + "related_event_id": "550e8400-e29b-41d4-a716-446655440000", + "backup_path": null, + "duration_seconds": 45, + "error": null + } + ] +} +``` + +Entries include: +- `action`: "update" | "rollback" +- `status`: "pending" | "in_progress" | "completed" | "failed" +- `initiated_via`: How the action was started (ui, api, auto) +- `related_event_id`: Links rollback to original update +- `backup_path`: Location of pre-update backup +- Error details for failed attempts ## Real-time Updates @@ -1016,10 +1111,45 @@ Returns simplified metrics without authentication requirements. ## Rate Limiting -Some endpoints have rate limiting: -- Export/Import: 5 requests per minute -- Test email: 10 requests per minute -- Update check: 10 requests per hour +**v4.24.0:** All responses include rate limit headers (`X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`). 429 responses add `Retry-After`. + +**Rate limits by endpoint category:** +- **Authentication**: 10 attempts/minute per IP +- **Config writes**: 30 requests/minute +- **Exports**: 5 requests per 5 minutes +- **Recovery operations**: 3 requests per 10 minutes +- **Update operations**: 20 requests/minute +- **WebSocket connections**: 5 connections/minute per IP +- **General API**: 500 requests/minute per IP +- **Public endpoints**: 1000 requests/minute per IP + +**Exempt endpoints** (no rate limits): +- `/api/state` (real-time monitoring) +- `/api/guests/metadata` (frequent polling) +- WebSocket message streaming (after connection established) + +**Example response with rate limit headers:** +``` +HTTP/1.1 200 OK +X-RateLimit-Limit: 500 +X-RateLimit-Remaining: 487 +X-RateLimit-Reset: 1754995800 +Content-Type: application/json +``` + +**When rate limited:** +``` +HTTP/1.1 429 Too Many Requests +X-RateLimit-Limit: 500 +X-RateLimit-Remaining: 0 +X-RateLimit-Reset: 1754995800 +Retry-After: 60 +Content-Type: application/json + +{ + "error": "Rate limit exceeded. Please retry after 60 seconds." +} +``` ## Error Responses diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 782d80f..1eb8a2d 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -107,22 +107,31 @@ PROXY_AUTH_LOGOUT_URL=/logout # URL for SSO logout **Contents:** ```json { - "pbsPollingInterval": 60, // Seconds between PBS refreshes (PVE polling fixed at 10s) - "pmgPollingInterval": 60, // Seconds between PMG refreshes (mail analytics and health) - "connectionTimeout": 60, // Seconds before node connection timeout - "autoUpdateEnabled": false, // Systemd timer toggle for automatic updates - "autoUpdateCheckInterval": 24, // Hours between auto-update checks - "autoUpdateTime": "03:00", // Preferred update window (combined with randomized delay) - "updateChannel": "stable", // Update channel: stable or rc - "allowedOrigins": "", // CORS allowed origins (empty = same-origin only) - "allowEmbedding": false, // Allow iframe embedding - "allowedEmbedOrigins": "", // Comma-separated origins allowed to embed Pulse - "backendPort": 3000, // Internal API listen port (not normally changed) - "frontendPort": 7655, // Public port exposed by the service - "logLevel": "info", // Log level: debug, info, warn, error - "discoveryEnabled": true, // Enable/disable network discovery for Proxmox/PBS servers - "discoverySubnet": "auto", // CIDR to scan ("auto" discovers common ranges) - "theme": "" // UI theme preference: "", "light", or "dark" + "pbsPollingInterval": 60, // Seconds between PBS refreshes (PVE polling fixed at 10s) + "pmgPollingInterval": 60, // Seconds between PMG refreshes (mail analytics and health) + "connectionTimeout": 60, // Seconds before node connection timeout + "autoUpdateEnabled": false, // Systemd timer toggle for automatic updates + "autoUpdateCheckInterval": 24, // Hours between auto-update checks + "autoUpdateTime": "03:00", // Preferred update window (combined with randomized delay) + "updateChannel": "stable", // Update channel: stable or rc + "allowedOrigins": "", // CORS allowed origins (empty = same-origin only) + "allowEmbedding": false, // Allow iframe embedding + "allowedEmbedOrigins": "", // Comma-separated origins allowed to embed Pulse + "backendPort": 3000, // Internal API listen port (not normally changed) + "frontendPort": 7655, // Public port exposed by the service + "logLevel": "info", // Log level: debug, info, warn, error + "logFormat": "auto", // auto, json, or console output + "logFile": "", // Optional file path for mirrored logs + "logMaxSize": 100, // Log rotation threshold (MB) when logFile is set + "logMaxAge": 30, // Days to retain rotated files + "logCompress": true, // Compress rotated log files + "adaptivePollingEnabled": false, // Toggle adaptive scheduler (v4.24.0+) + "adaptivePollingBaseInterval": 10, // Target cadence (seconds) + "adaptivePollingMinInterval": 5, // Fastest cadence (seconds) + "adaptivePollingMaxInterval": 300, // Slowest cadence (seconds) + "discoveryEnabled": true, // Enable/disable network discovery for Proxmox/PBS servers + "discoverySubnet": "auto", // CIDR to scan ("auto" discovers common ranges) + "theme": "" // UI theme preference: "", "light", or "dark" } ``` @@ -133,6 +142,23 @@ PROXY_AUTH_LOGOUT_URL=/logout # URL for SSO logout - Missing file results in defaults being used - Changes take effect immediately (no restart required) - API tokens are no longer managed in system.json (moved to .env in v4.3.9+) +- **Adaptive polling controls** (`adaptivePollingEnabled`, `adaptivePolling*Interval`) map directly to the Scheduler Health API and adjust queue/backoff behaviour in real time. +- **Runtime logging controls** (`logLevel`, `logFormat`, `logFile`, `logMaxSize`, `logMaxAge`, `logCompress`) can be tuned from the UI or system.json; updates are applied immediately so you can raise verbosity, switch to structured JSON, or stream logs to disk without restarting Pulse. + +### Adaptive Polling Settings (v4.24.0+) + +- `adaptivePollingEnabled`: Enables the adaptive scheduler that prioritises stale or failing instances. Toggle it in **Settings → System → Adaptive polling** or set the flag in system.json. +- `adaptivePollingBaseInterval`: Target cadence (seconds) when an instance is healthy. Defaults to 10 seconds. +- `adaptivePollingMinInterval`: Lower bound when Pulse needs to poll aggressively (for example, 5 seconds for busy clusters). +- `adaptivePollingMaxInterval`: Upper bound for idle instances. Setting this to a small value (≤15s) automatically engages the low-latency backoff profile (750 ms initial delay, 20 % jitter, 10 s breaker windows). +- The adaptive scheduler feeds the `/api/monitoring/scheduler/health` endpoint and priority queue. Shorter intervals reduce queue depth; longer intervals trade freshness for fewer calls. All three intervals are stored in seconds in system.json; environment overrides accept Go duration strings such as `15s` or `5m`. + +### Logging Configuration (v4.24.0+) + +- `logLevel`: Runtime log verbosity (`debug`, `info`, `warn`, `error`). Raise it to `debug` temporarily when troubleshooting, then drop back to `info`. +- `logFormat`: `auto` switches between human-friendly console output (interactive TTY) and JSON when Pulse runs under a service. Override with `json` to stream machine-readable logs everywhere, or `console` to force colourised output. +- `logFile`: Optional absolute path. When populated, Pulse mirrors logs to this file as well as stdout. Rotation honours `logMaxSize` (MB), `logMaxAge` (days), and `logCompress` (gzip rotated files). +- Logging changes made via the UI or system.json take effect immediately, so you can capture verbose traces or structured logs without scheduling downtime. --- @@ -294,6 +320,12 @@ Set `autoUpdateEnabled: true` in system.json or toggle in Settings UI. **Note**: Docker installations do not support automatic updates (use Docker's update mechanisms instead). +### Update Backups & History (v4.24.0+) + +- Every self-update or rollback writes an entry to `/update-history.jsonl` (defaults to `/var/lib/pulse` for systemd installs and `/data` in Docker). Review the log via **Settings → System → Updates**, or query `/api/updates/history` for automation. +- The install script prints the configuration backup it creates (for example `/etc/pulse.backup.20251020-130500`). That path is captured in the history entry as `backup_path` so rollbacks know which snapshot to restore. +- Update logs live under `/var/log/pulse/update-*.log`; grab the most recent file when filing support tickets or analysing failures. + --- ## Configuration Priority @@ -313,7 +345,16 @@ These env vars override system.json values. When set, the UI will show a warning - `DISCOVERY_SUBNET` - Custom network to scan (default: auto-scans common networks) - `CONNECTION_TIMEOUT` - API timeout in seconds (default: 10) - `ALLOWED_ORIGINS` - CORS origins (default: same-origin only) -- `LOG_LEVEL` - Log verbosity: debug/info/warn/error (default: info) +- `LOG_LEVEL` - Log verbosity: debug/info/warn/error (default: info). Switching levels takes effect immediately. +- `LOG_FORMAT` - Override output format (`auto`, `json`, or `console`). +- `LOG_FILE` - Mirror logs to this absolute path in addition to stdout (empty = stdout only). +- `LOG_MAX_SIZE` - Rotate the log file after it grows beyond this many megabytes (default: 100). +- `LOG_MAX_AGE` - Delete rotated log files older than this many days (default: 30). +- `LOG_COMPRESS` - When `true` (default) gzip-compresses rotated log files. +- `ADAPTIVE_POLLING_ENABLED` - Enable/disable the adaptive scheduler without touching system.json (`true`/`false`). +- `ADAPTIVE_POLLING_BASE_INTERVAL` - Override the target polling cadence (accepts Go durations, e.g. `15s`). +- `ADAPTIVE_POLLING_MIN_INTERVAL` - Override the minimum cadence (Go duration or seconds). +- `ADAPTIVE_POLLING_MAX_INTERVAL` - Override the maximum cadence (Go duration or seconds). Values ≤`15s` engage the low-latency backoff profile. - `ENABLE_BACKUP_POLLING` - Set to `false` to disable polling of Proxmox backup/snapshot APIs (default: true) - `BACKUP_POLLING_INTERVAL` - Override the backup polling cadence. Accepts Go duration syntax (e.g. `30m`, `6h`) or seconds. Use `0` for Pulse's default (~90s) cadence. - `PULSE_PUBLIC_URL` - Full URL to access Pulse (e.g., `http://192.168.1.100:7655`) diff --git a/docs/FAQ.md b/docs/FAQ.md index 403b07d..c9ad057 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -92,6 +92,53 @@ Yes! When you add one cluster node, Pulse automatically discovers and monitors a ### High memory usage? Reduce `metricsRetentionDays` in settings and restart +### How do I monitor adaptive polling? +**New in v4.24.0:** Pulse includes adaptive polling that automatically adjusts polling intervals based on system load. + +**Monitor adaptive polling:** +- **Dashboard**: Settings → System → Monitoring shows scheduler health status +- **API**: `/api/monitoring/scheduler/health` provides detailed metrics including: + - Queue depths and processing times + - Circuit breaker status + - Backoff states + - Instance metadata +- **Logging**: Enable debug logging to see detailed polling behavior + +**Key metrics to watch:** +- Queue depth (alerts if backlog builds up) +- Circuit breaker trips (indicates connectivity issues) +- Backoff delays (shows throttling behavior) + +See [Adaptive Polling Documentation](monitoring/ADAPTIVE_POLLING.md) for complete details. + +### What's new about rate limiting in v4.24.0? +Pulse now returns standard rate limit headers with all API responses: + +**Response Headers:** +- `X-RateLimit-Limit`: Maximum requests allowed per window (e.g., 500) +- `X-RateLimit-Remaining`: Requests remaining in current window +- `Retry-After`: Seconds to wait before retrying (on 429 responses) + +**Rate Limits:** +- **Auth endpoints**: 10 attempts/minute per IP +- **General API**: 500 requests/minute per IP +- **Real-time endpoints**: No limits (WebSocket, SSE) + +**Example Response:** +``` +HTTP/1.1 200 OK +X-RateLimit-Limit: 500 +X-RateLimit-Remaining: 487 +``` + +When you hit the limit: +``` +HTTP/1.1 429 Too Many Requests +X-RateLimit-Limit: 500 +X-RateLimit-Remaining: 0 +Retry-After: 60 +``` + ## Features ### Why do VMs show "-" for disk usage? @@ -175,13 +222,72 @@ First, confirm the agent is still running (`systemctl status pulse-docker-agent` ## Updates ### How to update? -- **Docker**: Pull latest image, recreate container +- **Docker**: Pull latest image, recreate container - **Manual/systemd**: Run the install script again: `curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash` +### Can I roll back if an update misbehaves? +**New in v4.24.0:** Yes! Pulse now retains previous versions and provides easy rollback. + +**Via UI (Recommended):** +1. Navigate to **Settings → System → Updates** +2. Click **"Restore previous version"** +3. Confirm rollback +4. Pulse restarts with the previous working version + +**Via CLI:** +```bash +# Systemd installations +sudo /opt/pulse/pulse config rollback + +# LXC containers +pct exec -- bash -c "cd /opt/pulse && ./pulse config rollback" +``` + +**What gets rolled back:** +- Pulse binary and frontend assets +- System configuration (preserved from previous version) +- Rollback history tracked in Updates view + +**What stays the same:** +- Your node configurations +- Alert settings +- User credentials +- Historical metrics data + +Check rollback logs: `journalctl -u pulse | grep rollback` + ### How do I install an older release (downgrade)? -- **Manual/systemd installs**: rerun the installer and pass the tag you want, e.g. `curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.20.0` -- **Proxmox LXC appliance**: `pct exec -- bash -lc "curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.20.0"` -- **Docker**: launch with a versioned tag instead of `latest`, e.g. `docker run -d --name pulse -p 7655:7655 rcourtman/pulse:v4.20.0` +- **Manual/systemd installs**: rerun the installer and pass the tag you want, e.g. `curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.24.0` +- **Proxmox LXC appliance**: `pct exec -- bash -lc "curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.24.0"` +- **Docker**: launch with a versioned tag instead of `latest`, e.g. `docker run -d --name pulse -p 7655:7655 rcourtman/pulse:v4.24.0` + +### How do I adjust logging without restarting? +**New in v4.24.0:** Pulse supports runtime logging configuration—no restart required! + +**Via UI:** +1. Navigate to **Settings → System → Logging** +2. Adjust: + - **Log Level**: debug, info, warn, error + - **Log Format**: json, text + - **File Rotation**: size limits, retention +3. Changes apply immediately + +**Via Environment Variables:** +```bash +# Systemd +sudo systemctl edit pulse +[Service] +Environment="LOG_LEVEL=debug" +Environment="LOG_FORMAT=json" + +# Docker +docker run -e LOG_LEVEL=debug -e LOG_FORMAT=json rcourtman/pulse:latest +``` + +**Use cases:** +- Enable debug logging temporarily for troubleshooting +- Switch to JSON format for log aggregation +- Adjust file rotation to manage disk usage ### Why can't I update from the UI? For security reasons, Pulse cannot self-update. The UI will notify you when updates are available and show the appropriate update command for your deployment type. diff --git a/docs/INSTALL.md b/docs/INSTALL.md index ddc84a0..eb09819 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -85,6 +85,9 @@ curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | b # Or force enable with flag curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --enable-auto-updates + +# Install specific version (e.g., v4.24.0) +curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.24.0 ``` #### Enable/Disable After Installation @@ -104,6 +107,7 @@ systemctl status pulse-update.timer # Check status - Creates backup before updating - Automatically rolls back if update fails - Logs all activity to systemd journal +- **New in v4.24.0**: Rollback history is retained in Settings → System → Updates; use the new 'Restore previous version' button if the latest build regresses #### View Update Logs ```bash @@ -132,11 +136,35 @@ docker rm pulse docker run -d --name pulse -p 7655:7655 -v pulse_data:/data rcourtman/pulse:latest ``` +### Rollback to Previous Version + +**New in v4.24.0:** Pulse retains previous versions and allows easy rollback if an update causes issues. + +#### Via UI (Recommended) +1. Navigate to **Settings → System → Updates** +2. Click **"Restore previous version"** button +3. Confirm rollback +4. Pulse will restart with the previous working version + +#### Via CLI +```bash +# For systemd installations +sudo /opt/pulse/pulse config rollback + +# For LXC containers +pct exec -- bash -c "cd /opt/pulse && ./pulse config rollback" +``` + +Rollback history and metadata are tracked in the Updates view. Check system journal for detailed rollback logs: +```bash +journalctl -u pulse | grep rollback +``` + ## Version Management ### Install Specific Version ```bash -curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.8.0 +curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | bash -s -- --version v4.24.0 ``` ### Install Release Candidate @@ -155,6 +183,34 @@ curl -fsSL https://raw.githubusercontent.com/rcourtman/Pulse/main/install.sh | b ``` **Note:** This builds Pulse from source code on your machine. Requires Go, Node.js, and npm. +## Advanced Configuration + +### Runtime Logging Configuration + +**New in v4.24.0:** Adjust logging settings without restarting Pulse. + +#### Via UI +Navigate to **Settings → System → Logging** to configure: +- **Log Level**: debug, info, warn, error +- **Log Format**: json, text +- **File Rotation**: size limits and retention + +#### Via Environment Variables +```bash +# Systemd +sudo systemctl edit pulse +[Service] +Environment="LOG_LEVEL=debug" +Environment="LOG_FORMAT=json" + +# Docker +docker run -e LOG_LEVEL=debug -e LOG_FORMAT=json rcourtman/pulse:latest +``` + +### Adaptive Polling + +**New in v4.24.0:** Adaptive polling is now enabled by default, automatically adjusting polling intervals based on system load and responsiveness. Monitor status via **Settings → System → Monitoring** or the new Scheduler Health API at `/api/monitoring/scheduler/health`. + ## Troubleshooting ### Permission Denied diff --git a/docs/MIGRATION.md b/docs/MIGRATION.md index 2909c3d..3ea0b7d 100644 --- a/docs/MIGRATION.md +++ b/docs/MIGRATION.md @@ -1,5 +1,7 @@ # Migrating Pulse +**Updated for Pulse v4.24.0** + ## Quick Migration Guide ### ❌ DON'T: Copy files directly @@ -19,12 +21,16 @@ Never copy `/etc/pulse` or `/var/lib/pulse` directories between systems: #### Importing (New Server) 1. Install fresh Pulse instance -2. Open Pulse web interface +2. Open Pulse web interface 3. Go to **Settings** → **Configuration Management** 4. Click **Import Configuration** 5. Select your exported file 6. Enter the same passphrase 7. Click Import +8. **Post-migration verification (v4.24.0+)**: + - Check scheduler health: `curl -s http://localhost:7655/api/monitoring/scheduler/health | jq` + - Verify adaptive polling status: **Settings → System → Monitoring** + - Confirm all nodes are connected and polling correctly ## What Gets Migrated @@ -40,7 +46,9 @@ Never copy `/etc/pulse` or `/var/lib/pulse` directories between systems: - Historical metrics data - Alert history - Authentication settings (passwords, API tokens) +- **Updates rollback history** (v4.24.0+) - Each instance should configure its own authentication +- **Note:** Updates rollback data isn't transferred and must be rebuilt by running one successful update cycle on the new host ## Common Scenarios @@ -75,6 +83,7 @@ The export/import process works across all installation methods: - **Safe to Store**: Encrypted exports can be stored in cloud backups - **Minimum 12 characters**: Use a strong passphrase - **Password Manager**: Store your passphrase securely +- **Rollback History**: Updates rollback data isn't included in exports; rebuild by running one successful update on the new host ## Troubleshooting @@ -90,6 +99,12 @@ The export/import process works across all installation methods: - Node IPs may have changed - Update node addresses in Settings +**Logging issues after migration (v4.24.0+)** +- If you lose logs after migration, ensure the runtime logging configuration persisted +- Toggle **Settings → System → Logging** to your desired level +- Check environment variables: `LOG_LEVEL`, `LOG_FORMAT` +- Verify log file rotation settings are correct + ## Pro Tips 1. **Test imports**: Try importing on a test instance first diff --git a/docs/RELEASE.md b/docs/RELEASE.md index afaa03a..849e560 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -4,7 +4,7 @@ Use this checklist when preparing and publishing a new Pulse release. ## Pre-release -- [ ] Ensure `VERSION` is up to date and matches the tag you plan to cut (format `4.x.y`) +- [ ] Ensure `VERSION` is set to `4.24.0` and matches the tag you plan to cut (format `4.x.y`) - [ ] Confirm the Helm chart renders and installs locally: ```bash helm lint deploy/helm/pulse --strict @@ -26,6 +26,8 @@ Use this checklist when preparing and publishing a new Pulse release. kubectl -n pulse get pods kind delete cluster ``` +- [ ] Confirm adaptive polling, scheduler health API, rollback UI, logging runtime controls, and rate-limit header documentation are updated before tagging v4.24.0 +- [ ] Smoke-test updates rollback: apply a test update via Settings → System → Updates, trigger a rollback, and verify journal entries document the rollback event ## Publishing @@ -62,11 +64,21 @@ Use this checklist when preparing and publishing a new Pulse release. --create-namespace ``` + **For v4.24.0 specifically**, highlight these features in the release notes: + - Adaptive polling (now GA) + - Scheduler health API with rich instance metadata + - Updates rollback workflow + - Shared script library system (now GA) + - X-RateLimit-* headers for all API responses + - Runtime logging configuration (no restart required) + 6. Mention any chart-breaking changes (new values, migrations) in the release notes. ## Post-release - [ ] Verify `helm show chart oci://ghcr.io/rcourtman/pulse-chart --version 4.x.y` shows the expected metadata (version, appVersion, icon) - [ ] Run `helm install` against a test cluster (Kind/k3s) using the published OCI artifact +- [ ] Run `curl -s http://:7655/api/monitoring/scheduler/health | jq` to ensure the scheduler health endpoint is live +- [ ] Verify the Updates view reports rollback metadata and X-RateLimit-* headers appear in API responses - [ ] Announce the release with links to both the GitHub release and the Helm installation instructions (`docs/KUBERNETES.md`) - [ ] Verify signatures: `gpg --verify checksums.txt.asc checksums.txt` diff --git a/docs/api/SCHEDULER_HEALTH.md b/docs/api/SCHEDULER_HEALTH.md index a11e6e8..6d95aa0 100644 --- a/docs/api/SCHEDULER_HEALTH.md +++ b/docs/api/SCHEDULER_HEALTH.md @@ -1,9 +1,19 @@ # Scheduler Health API +**New in v4.24.0** + Endpoint: `GET /api/monitoring/scheduler/health` Returns a snapshot of the adaptive polling scheduler, queue state, circuit breakers, and per-instance status. Requires authentication (session cookie or bearer token). +**Key Features:** +- Real-time scheduler health monitoring +- Circuit breaker status per instance +- Dead-letter queue tracking (tasks that repeatedly fail) +- Per-instance staleness metrics +- No query parameters required +- Read-only endpoint (rate-limited under general 500 req/min bucket) + --- ## Request @@ -21,16 +31,22 @@ No query parameters are needed. ```json { - "updatedAt": "2025-10-20T13:05:42Z", - "enabled": true, + "updatedAt": "2025-10-20T13:05:42Z", // RFC 3339 timestamp + "enabled": true, // Mirrors AdaptivePollingEnabled setting "queue": {...}, "deadLetter": {...}, - "breakers": [...], // legacy summary - "staleness": [...], // legacy summary - "instances": [ ... ] // enhanced per-instance view + "breakers": [...], // legacy summary (for backward compatibility) + "staleness": [...], // legacy summary (for backward compatibility) + "instances": [ ... ] // authoritative per-instance view (v4.24.0+) } ``` +**Field Notes:** +- `updatedAt`: RFC 3339 timestamp of when this snapshot was generated +- `enabled`: Reflects the current `AdaptivePollingEnabled` system setting +- `breakers` and `staleness`: Legacy arrays maintained for backward compatibility; use `instances` for complete data +- `instances`: Authoritative source for per-instance health (v4.24.0+) + ### Queue Snapshot (`queue`) | Field | Type | Description | @@ -44,7 +60,9 @@ No query parameters are needed. | Field | Type | Description | |-------|------|-------------| | `count` | integer | Total items in the dead-letter queue | -| `tasks` | array | Top entries (legacy format; limited set) | +| `tasks` | array | **Limited to 25 entries** for performance. Each task includes `instance`, `type`, `nextRun`, `lastError`, and `failures` count. For complete per-instance DLQ data, use `instances[].deadLetter` | + +**Note:** The top-level `deadLetter.tasks` array is capped at 25 items to prevent large responses. Use the `instances` array for exhaustive coverage. ### Instances (`instances`) @@ -65,20 +83,37 @@ Each element gives a complete view of one instance. | Field | Type | Description | |-------|------|-------------| -| `lastSuccess` | timestamp nullable | Most recent successful poll | -| `lastError` | object nullable | `{ at, message, category }` (`category` is `transient` or `permanent`) | -| `consecutiveFailures` | integer | Failure streak length | -| `firstFailureAt` | timestamp nullable | When the streak began | +| `lastSuccess` | timestamp nullable | RFC 3339 timestamp of most recent successful poll | +| `lastError` | object nullable | `{ at, message, category }` where `at` is RFC 3339, `message` describes the error, and `category` is `transient` (network issues, timeouts) or `permanent` (auth failures, invalid config) | +| `consecutiveFailures` | integer | Current failure streak length (resets on successful poll) | +| `firstFailureAt` | timestamp nullable | **New in v4.24.0**: RFC 3339 timestamp when the current failure streak began. Useful for calculating failure duration | + +**Timing Metadata (v4.24.0+):** +- `firstFailureAt`: Tracks when a failure streak started, enabling "failing for X minutes" calculations +- Resets to `null` when a successful poll occurs +- Combine with `consecutiveFailures` to assess severity #### Breaker (`breaker`) | Field | Type | Description | |-------|------|-------------| -| `state` | string | `closed`, `open`, `half_open`, or `unknown` | -| `since` | timestamp nullable | When current state began | -| `lastTransition` | timestamp nullable | Last transition time | -| `retryAt` | timestamp nullable | Scheduled retry time when applicable | -| `failureCount` | integer | Failures counted in the current breaker cycle | +| `state` | string | `closed` (healthy), `open` (failing), `half_open` (testing recovery), or `unknown` (not initialized) | +| `since` | timestamp nullable | **New in v4.24.0**: RFC 3339 timestamp when the current state began. Use to calculate how long a breaker has been open | +| `lastTransition` | timestamp nullable | **New in v4.24.0**: RFC 3339 timestamp of the most recent state change (e.g., closed → open) | +| `retryAt` | timestamp nullable | **New in v4.24.0**: RFC 3339 timestamp of next scheduled retry attempt when breaker is open or half-open | +| `failureCount` | integer | **New in v4.24.0**: Number of failures in the current breaker cycle. Resets when breaker closes | + +**Circuit Breaker Timing (v4.24.0+):** +- `since`: When did the current state start? (e.g., "breaker has been open for 5 minutes") +- `lastTransition`: When was the last state change? (useful for detecting flapping) +- `retryAt`: When will the next retry attempt occur? (for open/half-open states) +- `failureCount`: How many failures have accumulated? (triggers state transitions) + +**State Transitions:** +- `closed` → `open`: Triggered after N failures (default: 5) +- `open` → `half_open`: After timeout period, allows one test request +- `half_open` → `closed`: If test request succeeds +- `half_open` → `open`: If test request fails #### Dead-letter (`deadLetter`) @@ -251,6 +286,29 @@ The `instances` array centralizes per-instance telemetry; existing integrations --- +## Operational Notes + +**v4.24.0 Behavior:** +- **Read-only endpoint**: This endpoint is informational only and does not modify scheduler state +- **Rate limiting**: Falls under the general API limit (500 requests/minute per IP) +- **Authentication required**: Must provide valid session cookie or API token +- **Adaptive polling disabled**: When adaptive polling is disabled (`enabled: false`), the response includes empty `breakers`, `staleness`, and `instances` arrays +- **Real-time data**: Reflects current scheduler state; not historical (for trends, use metrics/logs) +- **No query parameters**: Returns complete snapshot on every request +- **Automatic adjustments**: The `enabled` field automatically reflects the `AdaptivePollingEnabled` system setting + +**Use Cases:** +- **Monitoring dashboards**: Embed in Grafana/Prometheus for real-time scheduler health +- **Alerting**: Trigger alerts on open circuit breakers or high DLQ counts +- **Debugging**: Investigate why specific instances aren't polling successfully +- **Capacity planning**: Monitor queue depth trends to assess if polling intervals need adjustment + +**Breaking Changes:** +- **None**: v4.24.0 only adds fields; all existing consumers continue to work +- Consumers just gain access to richer metadata (`firstFailureAt`, breaker timestamps, DLQ retry windows) + +--- + ## Troubleshooting Examples 1. **Transient outages:** look for `pollStatus.lastError.category == "transient"` to confirm network hiccups; check `breaker.retryAt` to see when retries resume. diff --git a/docs/monitoring/ADAPTIVE_POLLING.md b/docs/monitoring/ADAPTIVE_POLLING.md index e0e915c..bced23e 100644 --- a/docs/monitoring/ADAPTIVE_POLLING.md +++ b/docs/monitoring/ADAPTIVE_POLLING.md @@ -28,28 +28,43 @@ Phase 2 introduces a scheduler that adapts poll cadence based on freshness, er ## Configuration +**v4.24.0:** Adaptive polling is **enabled by default** but can be toggled without restart. + +### Via UI +Navigate to **Settings → System → Monitoring** to enable/disable adaptive polling. Changes apply immediately without requiring a restart. + +### Via Environment Variables Environment variables (default in `internal/config/config.go`): | Variable | Default | Description | |-------------------------------------|---------|--------------------------------------------------| -| `ADAPTIVE_POLLING_ENABLED` | false | Feature flag for adaptive scheduler. | -| `ADAPTIVE_POLLING_BASE_INTERVAL` | 10s | Target cadence when system is healthy. | -| `ADAPTIVE_POLLING_MIN_INTERVAL` | 5s | Lower bound (active instances). | -| `ADAPTIVE_POLLING_MAX_INTERVAL` | 5m | Upper bound (idle instances). | +| `ADAPTIVE_POLLING_ENABLED` | true | **Changed in v4.24.0**: Now enabled by default | +| `ADAPTIVE_POLLING_BASE_INTERVAL` | 10s | Target cadence when system is healthy | +| `ADAPTIVE_POLLING_MIN_INTERVAL` | 5s | Lower bound (active instances) | +| `ADAPTIVE_POLLING_MAX_INTERVAL` | 5m | Upper bound (idle instances) | -All settings persist in `system.json` and respond to environment overrides. +All settings persist in `system.json` and respond to environment overrides. **Changes apply without restart** when modified via UI. ## Metrics +**v4.24.0:** Extended metrics for comprehensive monitoring. + Exposed via Prometheus (`:9091/metrics`): -| Metric | Type | Labels | Description | -|------------------------------------------|-----------|---------------------------------------|--------------------------------------------| -| `pulse_monitor_poll_total` | counter | `instance_type`, `instance`, `result` | Overall poll attempts (success/error). | -| `pulse_monitor_poll_duration_seconds` | histogram | `instance_type`, `instance` | Poll latency per instance. | -| `pulse_monitor_poll_staleness_seconds` | gauge | `instance_type`, `instance` | Age since last success (0 on success). | -| `pulse_monitor_poll_queue_depth` | gauge | — | Size of priority queue. | -| `pulse_monitor_poll_inflight` | gauge | `instance_type` | Concurrent tasks per type. | +| Metric | Type | Labels | Description | +|---------------------------------------------|-----------|---------------------------------------|-------------------------------------------------| +| `pulse_monitor_poll_total` | counter | `instance_type`, `instance`, `result` | Overall poll attempts (success/error) | +| `pulse_monitor_poll_duration_seconds` | histogram | `instance_type`, `instance` | Poll latency per instance | +| `pulse_monitor_poll_staleness_seconds` | gauge | `instance_type`, `instance` | Age since last success (0 on success) | +| `pulse_monitor_poll_queue_depth` | gauge | — | Size of priority queue | +| `pulse_monitor_poll_inflight` | gauge | `instance_type` | Concurrent tasks per type | +| `pulse_monitor_poll_errors_total` | counter | `instance_type`, `instance`, `category` | **New in v4.24.0**: Error counts by category (transient/permanent) | +| `pulse_monitor_poll_last_success_timestamp` | gauge | `instance_type`, `instance` | **New in v4.24.0**: Unix timestamp of last successful poll | + +**Alerting Recommendations:** +- Alert when `pulse_monitor_poll_staleness_seconds` > 120 for critical instances +- Alert when `pulse_monitor_poll_queue_depth` > 50 (backlog building) +- Alert when `pulse_monitor_poll_errors_total` with `category=permanent` increases (auth/config issues) ## Circuit Breaker & Backoff