From 3da07c2c6681e4c70664aa301c1fd930080eeea7 Mon Sep 17 00:00:00 2001 From: "courtmanr@gmail.com" Date: Tue, 25 Nov 2025 00:19:34 +0000 Subject: [PATCH] Refactor integration docs (K8s, Proxy, Webhooks) to be concise --- docs/KUBERNETES.md | 353 +++++--------------------------------- docs/REVERSE_PROXY.md | 368 ++++------------------------------------ docs/WEBHOOKS.md | 384 +++--------------------------------------- 3 files changed, 101 insertions(+), 1004 deletions(-) diff --git a/docs/KUBERNETES.md b/docs/KUBERNETES.md index 01cafa0..c6f03e0 100644 --- a/docs/KUBERNETES.md +++ b/docs/KUBERNETES.md @@ -1,82 +1,45 @@ -# Kubernetes Deployment (Helm) +# ☸️ Kubernetes (Helm) -Deploy Pulse to Kubernetes with the bundled Helm chart under `deploy/helm/pulse`. The chart provisions the Pulse hub (web UI + API) and can optionally run the Docker monitoring agent alongside it. Stable builds are published automatically to the GitHub Container Registry (GHCR) whenever a Pulse release goes out. +Deploy Pulse to Kubernetes using the official Helm chart. -> The Helm chart ships with the release archives and pairs with the upgraded monitoring engine (staleness tracking, circuit breakers, detailed poll metrics) so Kubernetes clusters benefit from the same adaptive scheduling improvements as bare-metal installs. - -## Prerequisites - -- Kubernetes 1.24 or newer with access to a default `StorageClass` -- Helm 3.9+ -- An ingress controller (only if you plan to expose Pulse through an Ingress) -- (Optional) A Docker-compatible runtime on the nodes where you expect to run the Docker agent; the agent talks to `/var/run/docker.sock` - -## Installing from Helm Repository (recommended) - -1. Add the Pulse Helm repository: +## πŸš€ Installation +1. **Add Repo** ```bash helm repo add pulse https://rcourtman.github.io/Pulse/ - helm repo update pulse + helm repo update ``` -2. Install the chart: - +2. **Install** ```bash - # Latest version helm install pulse pulse/pulse \ --namespace pulse \ --create-namespace - - # Or pin to specific version - helm install pulse pulse/pulse \ - --version 4.28.0 \ - --namespace pulse \ - --create-namespace ``` - Check available versions: `helm search repo pulse/pulse --versions` - -3. Port-forward the service to finish the first-time security setup: - +3. **Access** ```bash kubectl -n pulse port-forward svc/pulse 7655:7655 ``` + Open `http://localhost:7655` to complete setup. -4. Browse to `http://localhost:7655`, complete the security bootstrap (admin user + MFA + TLS preferences), and create an API token under **Settings β†’ Security** for any automation or agents you plan to run. +--- -The chart mounts a PersistentVolumeClaim at `/data` so database files, credentials, and configuration survive pod restarts. By default it requests 8β€―GiB with `ReadWriteOnce` accessβ€”adjust via `persistence.*` in `values.yaml`. +## βš™οΈ Configuration -## Working From Source (local packaging) +Configure via `values.yaml` or `--set` flags. -Need to test local modifications or work offline? Install directly from the checked-out repository: +| Parameter | Description | Default | +|-----------|-------------|---------| +| `service.type` | Service type (ClusterIP/LoadBalancer) | `ClusterIP` | +| `ingress.enabled` | Enable Ingress | `false` | +| `persistence.enabled` | Enable PVC for /data | `true` | +| `persistence.size` | PVC Size | `8Gi` | +| `agent.enabled` | Enable Docker Agent sidecar | `false` | -1. Clone the repository and switch into it. - - ```bash - git clone https://github.com/rcourtman/Pulse.git - cd Pulse - ``` - -2. Render or install the chart from `deploy/helm/pulse`: - - ```bash - helm upgrade --install pulse ./deploy/helm/pulse \ - --namespace pulse \ - --create-namespace - ``` - -3. Continue with the port-forward and initial setup steps described above. - -## Common Configuration - -Most day-to-day overrides are done in a custom values file: +### Example `values.yaml` ```yaml -# file: helm-values.yaml -service: - type: ClusterIP - ingress: enabled: true className: nginx @@ -85,284 +48,50 @@ ingress: paths: - path: / pathType: Prefix - tls: - - hosts: [pulse.example.com] - secretName: pulse-tls server: env: - name: TZ - value: Europe/Berlin + value: Europe/London secretEnv: create: true data: - API_TOKENS: docker-agent-token -``` + API_TOKENS: "my-token" -### Runtime Logging Configuration (v4.25.0+) - -Configure logging behavior via environment variables: - -```yaml -server: - env: - - name: LOG_LEVEL - value: info # debug, info, warn, error - - name: LOG_FORMAT - value: json # json, text, auto - - name: LOG_FILE - value: /data/logs/pulse.log # Optional: mirror logs to file - - name: LOG_MAX_SIZE - value: "100" # MB per log file - - name: LOG_MAX_BACKUPS - value: "10" # Number of rotated logs to keep - - name: LOG_MAX_AGE - value: "30" # Days to retain logs -``` - -**Note:** Logging changes via environment variables require pod restart. Use **Settings β†’ System β†’ Logging** in the UI for runtime changes without restart. - -### Adaptive Polling Configuration (v4.25.0+) - -Adaptive polling is **enabled by default** in v4.25.0. Configure via environment variables: - -```yaml -server: - env: - - name: ADAPTIVE_POLLING_ENABLED - value: "true" # Enable/disable adaptive scheduler - - name: ADAPTIVE_POLLING_BASE_INTERVAL - value: "10s" # Target cadence (default: 10s) - - name: ADAPTIVE_POLLING_MIN_INTERVAL - value: "5s" # Fastest cadence (default: 5s) - - name: ADAPTIVE_POLLING_MAX_INTERVAL - value: "5m" # Slowest cadence (default: 5m) -``` - -**Note:** These settings can also be toggled via **Settings β†’ System β†’ Monitoring** in the UI without pod restart. - -Install or upgrade with the overrides: - -```bash -helm upgrade --install pulse ./deploy/helm/pulse \ - --namespace pulse \ - --create-namespace \ - -f helm-values.yaml -``` - -The `server.secretEnv` block above pre-seeds one or more API tokens so the UI is immediately accessible and automation can authenticate. If you prefer to manage credentials separately, set `server.secretEnv.name` to reference an existing secret instead of letting the chart create one. - -### Accessing Pulse - -- **Port forward:** `kubectl -n pulse port-forward svc/pulse 7655:7655` -- **Ingress:** Enable via the snippet above, or supply your own annotations for external DNS, TLS, etc. -- **LoadBalancer:** Set `service.type: LoadBalancer` and, optionally, `service.loadBalancerIP`. - -### Persistence Options - -- `persistence.enabled`: Disable to use an ephemeral `emptyDir` -- `persistence.existingClaim`: Bind to a pre-provisioned PVC -- `persistence.storageClass`: Pin to a specific storage class -- `persistence.size`: Resize the default PVC request - -## Enabling the Docker Agent - -The optional agent reports Docker host metrics back to the Pulse hub. Enable it once you have a valid API token: - -```yaml -# agent-values.yaml agent: enabled: true - env: - - name: PULSE_URL - value: https://pulse.example.com secretEnv: create: true data: - PULSE_TOKEN: docker-agent-token - dockerSocket: - enabled: true - path: /var/run/docker.sock - hostPathType: Socket + PULSE_TOKEN: "my-token" ``` -Apply with (choose one): +Apply with: +```bash +helm upgrade --install pulse pulse/pulse -n pulse -f values.yaml +``` -- **Published chart:** +--- - ```bash - helm upgrade pulse oci://ghcr.io/rcourtman/pulse-chart \ - --install \ - --version \ - --namespace pulse \ - -f agent-values.yaml - ``` - -- **Local checkout:** - - ```bash - helm upgrade --install pulse ./deploy/helm/pulse \ - --namespace pulse \ - -f agent-values.yaml - ``` - -Notes: - -- The agent expects a Docker-compatible runtime and access to the Docker socket. Set `agent.dockerSocket.enabled: false` if you run the agent elsewhere or publish the Docker API securely. -- Use separate API tokens per host; list multiple tokens with `;` or `,` separators in `PULSE_TARGETS` if needed. -- Run the agent as a `DaemonSet` (default) to cover every node, or switch to `agent.kind: Deployment` for a single pod. - -## Upgrades and Removal - -### Upgrading Pulse - -1. **Update the Helm repository:** - - ```bash - helm repo update pulse - ``` - -2. **Check available versions:** - - ```bash - helm search repo pulse/pulse --versions - ``` - -3. **Upgrade to latest:** - - ```bash - helm upgrade pulse pulse/pulse -n pulse - ``` - -4. **Or upgrade to specific version:** - - ```bash - helm upgrade pulse pulse/pulse --version 4.28.0 -n pulse - ``` - -5. **With custom values file:** - - ```bash - helm upgrade pulse pulse/pulse -n pulse -f custom-values.yaml - ``` - -### Rollback - -If an upgrade causes issues: +## πŸ”„ Upgrades ```bash -# List revisions -helm history pulse -n pulse +helm repo update +helm upgrade pulse pulse/pulse -n pulse +``` -# Rollback to previous revision +**Rollback**: +```bash helm rollback pulse -n pulse - -# Or rollback to specific revision -helm rollback pulse 3 -n pulse ``` -### Uninstall +--- -```bash -# Remove Pulse but keep PVCs -helm uninstall pulse -n pulse +## ⚠️ Troubleshooting -# Remove everything including PVCs -helm uninstall pulse -n pulse -kubectl delete pvc -n pulse -l app.kubernetes.io/name=pulse -``` - -### Post-Upgrade Verification (v4.25.0+) - -After upgrading to v4.25.0 or newer, verify the deployment: - -1. **Check update history** - ```bash - # Via UI - # Navigate to Settings β†’ System β†’ Updates - - # Via API - kubectl -n pulse exec deploy/pulse -- curl -s http://localhost:7655/api/updates/history | jq '.entries[0]' - ``` - -2. **Verify scheduler health** (adaptive polling) - ```bash - kubectl -n pulse exec deploy/pulse -- curl -s http://localhost:7655/api/monitoring/scheduler/health | jq - ``` - - **Expected response:** - - `"enabled": true` - - `queue.depth` reasonable (< instances Γ— 1.5) - - `deadLetter.count` = 0 or only known issues - - `instances[]` populated with your nodes - -3. **Check pod logs** - ```bash - kubectl -n pulse logs deploy/pulse --tail=50 - ``` - -4. **Verify rollback capability** - - Pulse v4.24.0+ logs update history - - Rollback available via **Settings β†’ System β†’ Updates β†’ Restore previous version** - - Or via API: `POST /api/updates/rollback` (if supported in Kubernetes deployments) - -## Service Naming - -**Important:** The Helm chart creates a service named `svc/pulse` on port `7655` by default, matching standard Kubernetes naming conventions. - -**Service name variations:** -- **Kubernetes/Helm:** `svc/pulse` (Deployment: `pulse`, Service: `pulse`) -- **Systemd installations:** `pulse.service` or `pulse-backend.service` (legacy) -- **Hot-dev scripts:** `pulse-hot-dev` (development only, not used in production clusters) - -**To check the active service:** -```bash -# Kubernetes -kubectl -n pulse get svc pulse - -# Systemd -systemctl status pulse -``` - -## Troubleshooting - -### Verify Scheduler Health After Rollout - -**v4.24.0+** includes adaptive polling. After any Helm upgrade or rollback, verify: - -```bash -kubectl -n pulse exec deploy/pulse -- curl -s http://localhost:7655/api/monitoring/scheduler/health | jq -``` - -**Look for:** -- `enabled: true` -- Queue depth stable -- No stuck circuit breakers -- Empty or stable dead-letter queue - -### Port Configuration Changes - -Port changes via `service.port` or environment variable `FRONTEND_PORT` take effect immediately but should be documented in change logs. v4.24.0 records restarts and configuration changes in update history. - -**To verify port configuration:** -```bash -# Check service -kubectl -n pulse get svc pulse -o jsonpath='{.spec.ports[0].port}' - -# Check pod environment -kubectl -n pulse exec deploy/pulse -- env | grep PORT -``` - -## Reference - -- Review every available option in `deploy/helm/pulse/values.yaml` -- Inspect published charts without installing: `helm show values oci://ghcr.io/rcourtman/pulse-chart --version ` -- Helm template rendering preview: `helm template pulse ./deploy/helm/pulse -f ` -- `NOTES.txt` emitted by Helm summarizes the service endpoint and agent prerequisites after each install or upgrade - -## Related Documentation - -- [Scheduler Health API](api/SCHEDULER_HEALTH.md) - Monitor adaptive polling status -- [Configuration Guide](CONFIGURATION.md) - System settings and environment variables -- [Adaptive Polling Operations](operations/ADAPTIVE_POLLING_ROLLOUT.md) - Operational procedures -- [Reverse Proxy Setup](REVERSE_PROXY.md) - Configure ingress with rate limit headers +- **Check Pods**: `kubectl -n pulse get pods` +- **Check Logs**: `kubectl -n pulse logs deploy/pulse` +- **Scheduler Health**: + ```bash + kubectl -n pulse exec deploy/pulse -- curl -s http://localhost:7655/api/monitoring/scheduler/health + ``` diff --git a/docs/REVERSE_PROXY.md b/docs/REVERSE_PROXY.md index ee8b28d..651f0a6 100644 --- a/docs/REVERSE_PROXY.md +++ b/docs/REVERSE_PROXY.md @@ -1,356 +1,54 @@ -# Reverse Proxy Configuration +# πŸ”„ Reverse Proxy Setup -Pulse uses WebSockets for real-time updates. Your reverse proxy **MUST** support WebSocket connections or Pulse will not work correctly. +Pulse uses WebSockets for real-time updates. Your proxy **MUST** support WebSockets. -## Important Requirements - -1. **WebSocket Support Required** - Enable WebSocket proxying -2. **Proxy Headers** - Forward original host and IP headers -3. **Timeouts** - Increase timeouts for long-lived connections (7 days recommended) -4. **Buffer Sizes** - Increase for large state updates (64KB recommended) -5. **Rate Limit Headers (v4.24.0+)** - Pass through `X-RateLimit-*` and `Retry-After` headers -6. **Error Pass-through** - Don't intercept 429 responses (rate limits) - -## Authentication with Reverse Proxy - -Pulse always enforces its own authentication. If you want to delegate sign-in to your reverse proxy (Authentik, Authelia, etc.), configure Pulse's **Proxy Authentication** integration under *Settings β†’ Security*. That lets Pulse trust the authenticated user provided by the proxy while keeping per-user roles, API tokens, and audit logging intact. - -> **Note:** The legacy `DISABLE_AUTH` environment variable has been removed. If it still exists in your deployment, Pulse will log a warning at startup and ignore it. Remove the variable and restart Pulse to silence the warning. - -## Nginx +## ⚑ Quick Configs +### Nginx ```nginx -server { - listen 80; - server_name pulse.example.com; +location / { + proxy_pass http://localhost:7655; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; - # Redirect to HTTPS - return 301 https://$server_name$request_uri; -} - -server { - listen 443 ssl http2; - server_name pulse.example.com; - - # SSL configuration - ssl_certificate /path/to/cert.pem; - ssl_certificate_key /path/to/key.pem; - - # Proxy settings - location / { - proxy_pass http://localhost:7655; - proxy_http_version 1.1; - - # Required for WebSocket - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - - # Proxy headers - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - - # Timeouts for WebSocket and adaptive polling (v4.24.0+) - proxy_connect_timeout 7d; - proxy_send_timeout 7d; - proxy_read_timeout 7d; - - # Disable buffering for real-time updates - proxy_buffering off; - - # IMPORTANT (v4.24.0+): Don't intercept error responses - # This ensures 429 rate limit responses reach the client - proxy_intercept_errors off; - - # Increase buffer sizes for large messages - proxy_buffer_size 64k; - proxy_buffers 8 64k; - proxy_busy_buffers_size 128k; - } - - # API endpoints (optional, same config as above) - location /api/ { - proxy_pass http://localhost:7655/api/; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_connect_timeout 7d; - proxy_send_timeout 7d; - proxy_read_timeout 7d; - proxy_buffering off; - } + # Critical for WebSockets + proxy_read_timeout 86400; # 24h } ``` -## Caddy v2 - -Caddy automatically handles WebSocket upgrades when reverse proxying. - +### Caddy ```caddy pulse.example.com { reverse_proxy localhost:7655 } ``` -For more control: - -```caddy -pulse.example.com { - reverse_proxy localhost:7655 { - # Headers automatically handled by Caddy - header_up Host {host} - header_up X-Real-IP {remote} - header_up X-Forwarded-For {remote} - header_up X-Forwarded-Proto {scheme} - - # Increase timeouts for WebSocket - transport http { - dial_timeout 30s - response_header_timeout 30s - read_timeout 0 - } - } -} +### Traefik (Docker Compose) +```yaml +labels: + - "traefik.enable=true" + - "traefik.http.routers.pulse.rule=Host(`pulse.example.com`)" + - "traefik.http.services.pulse.loadbalancer.server.port=7655" ``` -## Apache - +### Apache ```apache - - ServerName pulse.example.com - - SSLEngine on - SSLCertificateFile /path/to/cert.pem - SSLCertificateKeyFile /path/to/key.pem - - # Enable necessary modules: - # a2enmod proxy proxy_http proxy_wstunnel headers - - # WebSocket proxy - RewriteEngine On - RewriteCond %{HTTP:Upgrade} websocket [NC] - RewriteCond %{HTTP:Connection} upgrade [NC] - RewriteRule ^/?(.*) "ws://localhost:7655/$1" [P,L] - - # Regular HTTP proxy - ProxyPass / http://localhost:7655/ - ProxyPassReverse / http://localhost:7655/ - - # Preserve host headers - ProxyPreserveHost On - - # Forward real IP - RequestHeader set X-Real-IP "%{REMOTE_ADDR}s" - RequestHeader set X-Forwarded-For "%{REMOTE_ADDR}s" - RequestHeader set X-Forwarded-Proto "https" - - # Disable buffering - ProxyIOBufferSize 65536 - +RewriteEngine On +RewriteCond %{HTTP:Upgrade} websocket [NC] +RewriteCond %{HTTP:Connection} upgrade [NC] +RewriteRule ^/?(.*) "ws://localhost:7655/$1" [P,L] + +ProxyPass / http://localhost:7655/ +ProxyPassReverse / http://localhost:7655/ ``` -## Traefik +--- -```yaml -# docker-compose.yml -services: - pulse: - image: rcourtman/pulse:latest - labels: - - "traefik.enable=true" - - "traefik.http.routers.pulse.rule=Host(`pulse.example.com`)" - - "traefik.http.routers.pulse.tls=true" - - "traefik.http.services.pulse.loadbalancer.server.port=7655" - # WebSocket support is automatic in Traefik 2.x+ -``` +## ⚠️ Common Issues -Or using Traefik file configuration: - -```yaml -# traefik-dynamic.yml -http: - routers: - pulse: - rule: "Host(`pulse.example.com`)" - service: pulse - tls: {} - - services: - pulse: - loadBalancer: - servers: - - url: "http://localhost:7655" -``` - -## HAProxy - -```haproxy -frontend https - bind *:443 ssl crt /path/to/cert.pem - - # ACL for Pulse - acl host_pulse hdr(host) -i pulse.example.com - - # WebSocket detection - acl is_websocket hdr(Upgrade) -i websocket - - # Use backend - use_backend pulse if host_pulse - -backend pulse - # Health check - option httpchk GET /api/health - - # WebSocket support - option http-server-close - option forwardfor - - # Timeouts for WebSocket - timeout client 3600s - timeout server 3600s - timeout tunnel 3600s - - # Backend server - server pulse1 localhost:7655 check -``` - -## Cloudflare Tunnel - -If using Cloudflare Tunnel (cloudflared): - -```yaml -# config.yml -tunnel: YOUR_TUNNEL_ID -credentials-file: /path/to/credentials.json - -ingress: - - hostname: pulse.example.com - service: http://localhost:7655 - originRequest: - # Enable WebSocket - noTLSVerify: false - connectTimeout: 30s - # No additional config needed - WebSockets work by default - - service: http_status:404 -``` - -## Testing WebSocket Connection - -After configuring your reverse proxy, test that WebSockets work: - -```bash -# Test basic connectivity -curl https://pulse.example.com/api/health - -# Test WebSocket upgrade (should return 101 Switching Protocols) -curl -i -N \ - -H "Connection: Upgrade" \ - -H "Upgrade: websocket" \ - -H "Sec-WebSocket-Version: 13" \ - -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \ - https://pulse.example.com/api/ws -``` - -### v4.24.0+ Verification - -**Test scheduler health API** (adaptive polling): -```bash -curl -s https://pulse.example.com/api/monitoring/scheduler/health | jq -``` - -**Expected response:** -- `enabled: true` -- Queue depth reasonable -- No stuck circuit breakers -- WebSocket connection remains open - -**Verify rate limit headers** (v4.24.0+): -```bash -curl -I https://pulse.example.com/api/version -``` - -**Look for headers:** -``` -HTTP/1.1 200 OK -X-RateLimit-Limit: 500 -X-RateLimit-Remaining: 499 -X-RateLimit-Reset: 1698765432 -``` - -**Test rate limiting:** -```bash -# Trigger rate limit (requires many requests) -for i in {1..600}; do curl -s https://pulse.example.com/api/health > /dev/null; done - -# Should eventually see: -curl -I https://pulse.example.com/api/health -# HTTP/1.1 429 Too Many Requests -# X-RateLimit-Limit: 500 -# X-RateLimit-Remaining: 0 -# Retry-After: 60 -``` - -**Important:** If you don't see rate limit headers or 429 responses are converted to 502/504, verify `proxy_intercept_errors off` (Nginx) or equivalent is set. - -In browser console (F12): -```javascript -// Test WebSocket connection -const ws = new WebSocket('wss://pulse.example.com/api/ws'); -ws.onopen = () => console.log('WebSocket connected!'); -ws.onmessage = (e) => console.log('Received:', e.data); -ws.onerror = (e) => console.error('WebSocket error:', e); -``` - -## Common Issues - -### "Connection Lost" or no real-time updates -- WebSocket upgrade not configured correctly -- Check proxy passes `Upgrade` and `Connection` headers -- Verify timeouts are increased for long connections - -### CORS errors -- Pulse handles CORS internally -- Don't add additional CORS headers in proxy -- If needed, set `ALLOWED_ORIGINS` in Pulse configuration - -### 502 Bad Gateway -- Pulse not running on expected port (default 7655) -- Check with: `curl http://localhost:7655/api/health` -- Verify Pulse service: `systemctl status pulse` (use `pulse-backend` if you're on a legacy unit) - -### WebSocket closes immediately -- Timeout too short in proxy configuration -- Increase `proxy_read_timeout` (Nginx) or equivalent -- Set to at least 3600s (1 hour) or more - -## Security Recommendations - -1. **Always use HTTPS** for production deployments -2. **Set proper headers** to prevent clickjacking: - ```nginx - add_header X-Frame-Options "SAMEORIGIN"; - add_header X-Content-Type-Options "nosniff"; - ``` -3. **Rate limiting** for API endpoints: - ```nginx - limit_req_zone $binary_remote_addr zone=pulse:10m rate=30r/s; - limit_req zone=pulse burst=50 nodelay; - ``` -4. **Hide proxy version**: - ```nginx - proxy_hide_header X-Powered-By; - server_tokens off; - ``` - -## Support - -If WebSockets still don't work after following this guide: -1. Check browser console for errors (F12) -2. Verify Pulse logs: `journalctl -u pulse -f` -3. Test without proxy first: `http://your-server:7655` -4. Report issues: https://github.com/rcourtman/Pulse/issues +- **"Connection Lost"**: WebSocket upgrade failed. Check `Upgrade` and `Connection` headers. +- **502 Bad Gateway**: Pulse is not running on port 7655. +- **CORS Errors**: Do not add CORS headers in the proxy; Pulse handles them. Set `ALLOWED_ORIGINS` env var if needed. diff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.md index e8f5ed3..ef9076e 100644 --- a/docs/WEBHOOKS.md +++ b/docs/WEBHOOKS.md @@ -1,373 +1,43 @@ -# Webhook Configuration Guide +# πŸ”” Webhooks -Pulse supports sending alert notifications to various webhook services including Discord, Slack, Microsoft Teams, Telegram, Gotify, ntfy, PagerDuty, and any custom webhook endpoint. +Pulse supports Discord, Slack, Teams, Telegram, Gotify, ntfy, and generic webhooks. -## Quick Start +## πŸš€ Quick Setup -1. Navigate to **Alerts** β†’ **Notifications** tab -2. Configure email settings or add webhooks -3. Select your service type (Discord, Slack, Teams, Telegram, etc.) -4. Enter the webhook URL and configure settings -5. Test the webhook to ensure it's working -6. Save your configuration +1. Go to **Alerts β†’ Notifications**. +2. Click **Add Webhook**. +3. Select service type and paste the URL. -![Alert Configuration](images/04-alerts.png) -*Alert configuration interface showing notification settings* +## πŸ“ Service URLs -## Supported Services +| Service | URL Format | +|---------|------------| +| **Discord** | `https://discord.com/api/webhooks/{id}/{token}` | +| **Slack** | `https://hooks.slack.com/services/...` | +| **Teams** | `https://{tenant}.webhook.office.com/...` | +| **Telegram** | `https://api.telegram.org/bot{token}/sendMessage?chat_id={id}` | +| **Gotify** | `https://gotify.example.com/message?token={token}` | +| **ntfy** | `https://ntfy.sh/{topic}` | -### Discord -``` -URL Format: https://discord.com/api/webhooks/{webhook_id}/{webhook_token} -``` -1. In Discord, go to Server Settings β†’ Integrations β†’ Webhooks -2. Create a new webhook and copy the URL -3. Paste the URL in Pulse +## 🎨 Custom Templates -### Telegram -``` -URL Format: https://api.telegram.org/bot{bot_token}/sendMessage?chat_id={chat_id} -``` -1. Create a bot with @BotFather on Telegram -2. Get your bot token from BotFather -3. Get your chat ID by messaging the bot and visiting: `https://api.telegram.org/bot/getUpdates` -4. In Pulse, select "Telegram Bot" as the service type -5. Use the URL format: `https://api.telegram.org/bot/sendMessage?chat_id=` -6. **IMPORTANT**: The chat_id MUST be included in the URL as a parameter -7. Pulse automatically sends rich formatted messages with emojis and full alert details +For generic webhooks, use Go templates to format the JSON payload. -### Slack -``` -URL Format: https://hooks.slack.com/services/{webhook_path} -``` -1. In Slack, go to Apps β†’ Incoming Webhooks -2. Add to Slack and choose a channel -3. Copy the webhook URL +**Variables:** +- `{{.Message}}`: Alert text +- `{{.Level}}`: warning/critical +- `{{.Node}}`: Node name +- `{{.Value}}`: Metric value (e.g. 95.5) -### Microsoft Teams -``` -URL Format: https://{tenant}.webhook.office.com/webhookb2/{webhook_path} -``` -1. In Teams channel, click ... β†’ Connectors -2. Configure Incoming Webhook -3. Copy the URL - -### Gotify -``` -URL Format: https://your-gotify-server/message?token={your-app-token} -``` -1. In Gotify, create a new application -2. Copy the application token -3. Use the URL format: `https://your-gotify-server/message?token=YOUR_APP_TOKEN` -4. The token MUST be included as a URL parameter -5. Pulse will send rich markdown-formatted notifications with emojis and full alert details -6. **View in Pulse links**: Automatically detected - links will work out of the box in most cases - -### ntfy -``` -URL Format: https://ntfy.sh/{topic} or https://your-ntfy-server/{topic} -``` -1. Choose a unique topic name (e.g., 'pulse-alerts-x7k9m2') - - **Important**: Anyone who knows your topic name can send you notifications - - Use a unique/random suffix for privacy -2. For ntfy.sh: Use `https://ntfy.sh/YOUR_TOPIC` -3. For self-hosted: Use `https://your-ntfy-server/YOUR_TOPIC` -4. Subscribe to the same topic in your ntfy mobile/desktop app -5. For authentication (optional): - - Click "Custom Headers" section in webhook config - - Add header: `Authorization` - - Value: `Bearer YOUR_TOKEN` or `Basic base64_encoded_credentials` -6. Notifications include dynamic priority levels and emoji tags based on alert severity -7. **View in Pulse links**: Automatically detected - links will work out of the box in most cases - -### PagerDuty -``` -URL: https://events.pagerduty.com/v2/enqueue -``` -1. In PagerDuty, go to Configuration β†’ Services -2. Add an integration β†’ Events API V2 -3. Copy the Integration Key -4. Add the key as a header: `routing_key: YOUR_KEY` - -## Custom Headers - -For webhooks that require authentication or custom headers: - -1. In the webhook configuration, expand the **Custom Headers** section -2. Click **+ Add Header** to add a new header -3. Enter the header name (e.g., `Authorization`, `X-API-Key`, `X-Auth-Token`) -4. Enter the header value (e.g., `Bearer YOUR_TOKEN`, your API key, etc.) -5. Add multiple headers as needed -6. Headers are sent with every webhook request - -### Common Header Examples - -| Service | Header Name | Header Value Format | -|---------|-------------|-------------------| -| Bearer Token | `Authorization` | `Bearer YOUR_TOKEN_HERE` | -| Basic Auth | `Authorization` | `Basic base64_encoded_user:pass` | -| API Key | `X-API-Key` | `your-api-key-here` | -| Custom Token | `X-Auth-Token` | `your-auth-token` | -| ntfy Auth | `Authorization` | `Bearer tk_your_ntfy_token` | -| Custom Service | `X-Service-Key` | `service-specific-key` | - -## Custom Payload Templates - -```mermaid -flowchart LR - Alert[Alert Triggered] - Template[Render Template
Variables & Functions] - Send[HTTP POST] - - Alert --> Template - Template --> Send - Send -->|2xx| Success[Delivered] - Send -->|5xx| Retry[Retry with Backoff] -``` - -For generic webhooks, you can define custom JSON payloads using Go template syntax. - -### Available Variables - -| Variable | Description | Example Value | -|----------|-------------|---------------| -| `{{.ID}}` | Alert ID | "alert-123" | -| `{{.Level}}` | Alert level | "warning", "critical" | -| `{{.Type}}` | Resource type | "cpu", "memory", "disk" | -| `{{.ResourceName}}` | Name of the resource | "Web Server VM" | -| `{{.ResourceID}}` | Resource identifier | "vm-100" | -| `{{.Node}}` | Proxmox node name | "pve-node-01" | -| `{{.Instance}}` | Proxmox instance URL | "https://192.168.1.100:8006" | -| `{{.Message}}` | Alert message | "CPU usage exceeded 90%" | -| `{{.Value}}` | Current metric value | 95.5 | -| `{{.Threshold}}` | Alert threshold | 90.0 | -| `{{.Duration}}` | How long alert has been active | "5m" | -| `{{.Timestamp}}` | Current timestamp | "2024-01-15T10:30:00Z" | -| `{{.StartTime}}` | When alert started | "2024-01-15T10:25:00Z" | - -### Template Functions - -| Function | Description | Example | -|----------|-------------|---------| -| `{{.Level \| title}}` | Capitalize first letter | "Warning" | -| `{{.Level \| upper}}` | Uppercase | "WARNING" | -| `{{.Level \| lower}}` | Lowercase | "warning" | -| `{{printf "%.1f" .Value}}` | Format numbers | "95.5" | -| `{{urlpath .ResourceName}}` | URL-encode a value for path segments | "web%20server" | -| `{{urlquery .Message}}` | URL-encode a value for query parameters | "CPU+usage+%3E+90%25" | - -## Dynamic URL Templates - -Webhook URLs can also reference the same template variables that are available to payloads. Pulse renders the URL before sending the request and safely encodes path segments by default. When placing values inside query parameters, wrap them with the provided helpers to ensure correct encoding: - -```text -https://example.com/hooks/{{urlpath .ResourceName}}?msg={{urlquery .Message}} -``` - -Supported variables are identical to the payload template list (`{{.Message}}`, `{{.Node}}`, custom fields, etc.). This makes it easy to call services that expect alert metadata in the URL itself. - -### Example Templates - -#### Simple JSON +**Example Payload:** ```json { "text": "Alert: {{.Level}} - {{.Message}}", - "resource": "{{.ResourceName}}", - "value": {{.Value}}, - "threshold": {{.Threshold}} + "value": {{.Value}} } ``` -#### Formatted Alert -```json -{ - "alert": { - "level": "{{.Level | upper}}", - "message": "{{.Message}}", - "details": { - "resource": "{{.ResourceName}}", - "node": "{{.Node}}", - "current_value": "{{printf "%.1f" .Value}}%", - "threshold": "{{printf "%.0f" .Threshold}}%", - "duration": "{{.Duration}}" - } - }, - "timestamp": "{{.Timestamp}}" -} -``` +## πŸ›‘οΈ Security -#### Slack-Compatible Custom Format -```json -{ - "text": "Pulse Alert", - "attachments": [{ - "color": "{{if eq .Level "critical"}}danger{{else}}warning{{end}}", - "title": "{{.Level | title}} Alert: {{.ResourceName}}", - "text": "{{.Message}}", - "fields": [ - {"title": "Value", "value": "{{printf "%.1f" .Value}}%", "short": true}, - {"title": "Threshold", "value": "{{printf "%.0f" .Threshold}}%", "short": true}, - {"title": "Node", "value": "{{.Node}}", "short": true}, - {"title": "Duration", "value": "{{.Duration}}", "short": true} - ], - "footer": "Pulse Monitoring", - "ts": {{.Timestamp}} - }] -} -``` - -#### Home Assistant -```json -{ - "title": "Pulse Alert: {{.Level | title}}", - "message": "{{.Message}}", - "data": { - "entity_id": "sensor.{{.Node | lower}}_{{.Type}}", - "state": {{.Value}}, - "attributes": { - "resource": "{{.ResourceName}}", - "threshold": {{.Threshold}}, - "duration": "{{.Duration}}" - } - } -} -``` - -#### n8n / Node-RED -```json -{ - "workflow": "pulse_alert", - "data": { - "alert_id": "{{.ID}}", - "level": "{{.Level}}", - "resource": "{{.ResourceName}}", - "node": "{{.Node}}", - "metric": { - "type": "{{.Type}}", - "value": {{.Value}}, - "threshold": {{.Threshold}} - }, - "message": "{{.Message}}", - "timestamp": "{{.Timestamp}}" - } -} -``` - -## Testing Webhooks - -1. After configuring a webhook, click the **Test** button -2. Pulse will send a test alert to verify the webhook is working -3. Check the receiving service to confirm the message arrived -4. If the test fails, verify: - - The URL is correct and accessible - - Any required authentication tokens are included - - The payload format matches what the service expects - -## Troubleshooting - -### Webhook Returns 400 Bad Request -- Check if the payload format is correct for your service -- For Telegram, ensure chat_id is in the URL (Pulse handles it automatically) -- Verify all required fields are present in custom templates - -### Webhook Returns 401/403 -- Check authentication tokens/keys -- Verify the webhook URL hasn't expired -- Ensure IP restrictions allow Pulse server - -### No Notifications Received -- Verify the webhook is enabled -- Check alert thresholds are configured correctly -- Ensure notification cooldown period has passed -- Test the webhook manually using the Test button -- Temporarily raise log level to `debug` via **Settings β†’ System β†’ Logging** (or set `LOG_LEVEL=debug` and restart) to inspect `webhook.delivery` entries in the logs, then return to `info` once resolved. - -## API Reference - -### Create Webhook -```bash -POST /api/notifications/webhooks -Content-Type: application/json - -{ - "name": "My Webhook", - "url": "https://example.com/webhook", - "method": "POST", - "service": "generic", - "enabled": true, - "template": "{\"alert\": \"{{.Message}}\"}" -} -``` - -### Test Webhook -```bash -POST /api/notifications/webhooks/test -Content-Type: application/json - -{ - "name": "Test", - "url": "https://example.com/webhook", - "service": "generic", - "template": "{\"test\": true}" -} -``` - -### Update Webhook -```bash -PUT /api/notifications/webhooks/{id} -Content-Type: application/json - -{ - "name": "Updated Webhook", - "url": "https://example.com/new-webhook", - "enabled": false -} -``` - -### Delete Webhook -```bash -DELETE /api/notifications/webhooks/{id} -``` - -### List Webhooks -```bash -GET /api/notifications/webhooks -``` - -## Security Considerations - -- **Never expose webhook URLs publicly** - they often contain authentication tokens -- **Use HTTPS URLs** when possible to encrypt data in transit -- **Rotate webhook URLs periodically** if they contain embedded tokens -- **Test webhooks carefully** to avoid sending test data to production channels -- **Limit webhook permissions** in the receiving service where possible - -### Private IP Address Protection - -By default, Pulse blocks webhooks to private IP addresses (192.168.x.x, 10.x.x.x, 172.16-31.x.x) for security. This prevents potential SSRF (Server-Side Request Forgery) attacks. - -**For homelab deployments** where you need to send webhooks to internal services (like Traefik, Home Assistant, or local notification servers): - -1. Navigate to **Settings β†’ System** -2. Find the **Webhook Security** section -3. Enter trusted CIDR ranges in **Allowed Private IP Ranges for Webhooks** - - Example: `192.168.1.0/24,10.0.0.0/8` - - Supports both CIDR notation and individual IPs -4. Save settings - -**Important security notes:** -- Leave empty to block all private IPs (default, most secure) -- Only add networks you fully trust -- Localhost (127.0.0.1), link-local (169.254.x.x), and cloud metadata services remain blocked even with an allowlist -- Changes apply immediately to new webhook configurations -- Existing webhooks are re-validated before each delivery - -**Example configurations:** -``` -Single network: 192.168.1.0/24 -Multiple networks: 192.168.1.0/24,10.0.0.0/8 -Single IP: 192.168.1.100 -Mixed: 192.168.1.0/24,10.5.10.20 -``` - -If you see an error like `"webhook URL resolves to private IP ... - private networks are not allowed"`, you need to configure the allowlist in System Settings. +- **Private IPs**: By default, webhooks to private IPs are blocked. Allow them in **Settings β†’ System β†’ Webhook Security**. +- **Headers**: Add custom headers (e.g., `Authorization: Bearer ...`) in the webhook config.