Refactor integration docs (K8s, Proxy, Webhooks) to be concise

This commit is contained in:
courtmanr@gmail.com 2025-11-25 00:19:34 +00:00
parent 25eaa2ca18
commit 3da07c2c66
3 changed files with 101 additions and 1004 deletions

View file

@ -1,82 +1,45 @@
# Kubernetes Deployment (Helm)
# ☸️ Kubernetes (Helm)
Deploy Pulse to Kubernetes with the bundled Helm chart under `deploy/helm/pulse`. The chart provisions the Pulse hub (web UI + API) and can optionally run the Docker monitoring agent alongside it. Stable builds are published automatically to the GitHub Container Registry (GHCR) whenever a Pulse release goes out.
Deploy Pulse to Kubernetes using the official Helm chart.
> The Helm chart ships with the release archives and pairs with the upgraded monitoring engine (staleness tracking, circuit breakers, detailed poll metrics) so Kubernetes clusters benefit from the same adaptive scheduling improvements as bare-metal installs.
## Prerequisites
- Kubernetes 1.24 or newer with access to a default `StorageClass`
- Helm 3.9+
- An ingress controller (only if you plan to expose Pulse through an Ingress)
- (Optional) A Docker-compatible runtime on the nodes where you expect to run the Docker agent; the agent talks to `/var/run/docker.sock`
## Installing from Helm Repository (recommended)
1. Add the Pulse Helm repository:
## 🚀 Installation
1. **Add Repo**
```bash
helm repo add pulse https://rcourtman.github.io/Pulse/
helm repo update pulse
helm repo update
```
2. Install the chart:
2. **Install**
```bash
# Latest version
helm install pulse pulse/pulse \
--namespace pulse \
--create-namespace
# Or pin to specific version
helm install pulse pulse/pulse \
--version 4.28.0 \
--namespace pulse \
--create-namespace
```
Check available versions: `helm search repo pulse/pulse --versions`
3. Port-forward the service to finish the first-time security setup:
3. **Access**
```bash
kubectl -n pulse port-forward svc/pulse 7655:7655
```
Open `http://localhost:7655` to complete setup.
4. Browse to `http://localhost:7655`, complete the security bootstrap (admin user + MFA + TLS preferences), and create an API token under **Settings → Security** for any automation or agents you plan to run.
---
The chart mounts a PersistentVolumeClaim at `/data` so database files, credentials, and configuration survive pod restarts. By default it requests 8GiB with `ReadWriteOnce` access—adjust via `persistence.*` in `values.yaml`.
## ⚙️ Configuration
## Working From Source (local packaging)
Configure via `values.yaml` or `--set` flags.
Need to test local modifications or work offline? Install directly from the checked-out repository:
| Parameter | Description | Default |
|-----------|-------------|---------|
| `service.type` | Service type (ClusterIP/LoadBalancer) | `ClusterIP` |
| `ingress.enabled` | Enable Ingress | `false` |
| `persistence.enabled` | Enable PVC for /data | `true` |
| `persistence.size` | PVC Size | `8Gi` |
| `agent.enabled` | Enable Docker Agent sidecar | `false` |
1. Clone the repository and switch into it.
```bash
git clone https://github.com/rcourtman/Pulse.git
cd Pulse
```
2. Render or install the chart from `deploy/helm/pulse`:
```bash
helm upgrade --install pulse ./deploy/helm/pulse \
--namespace pulse \
--create-namespace
```
3. Continue with the port-forward and initial setup steps described above.
## Common Configuration
Most day-to-day overrides are done in a custom values file:
### Example `values.yaml`
```yaml
# file: helm-values.yaml
service:
type: ClusterIP
ingress:
enabled: true
className: nginx
@ -85,284 +48,50 @@ ingress:
paths:
- path: /
pathType: Prefix
tls:
- hosts: [pulse.example.com]
secretName: pulse-tls
server:
env:
- name: TZ
value: Europe/Berlin
value: Europe/London
secretEnv:
create: true
data:
API_TOKENS: docker-agent-token
```
API_TOKENS: "my-token"
### Runtime Logging Configuration (v4.25.0+)
Configure logging behavior via environment variables:
```yaml
server:
env:
- name: LOG_LEVEL
value: info # debug, info, warn, error
- name: LOG_FORMAT
value: json # json, text, auto
- name: LOG_FILE
value: /data/logs/pulse.log # Optional: mirror logs to file
- name: LOG_MAX_SIZE
value: "100" # MB per log file
- name: LOG_MAX_BACKUPS
value: "10" # Number of rotated logs to keep
- name: LOG_MAX_AGE
value: "30" # Days to retain logs
```
**Note:** Logging changes via environment variables require pod restart. Use **Settings → System → Logging** in the UI for runtime changes without restart.
### Adaptive Polling Configuration (v4.25.0+)
Adaptive polling is **enabled by default** in v4.25.0. Configure via environment variables:
```yaml
server:
env:
- name: ADAPTIVE_POLLING_ENABLED
value: "true" # Enable/disable adaptive scheduler
- name: ADAPTIVE_POLLING_BASE_INTERVAL
value: "10s" # Target cadence (default: 10s)
- name: ADAPTIVE_POLLING_MIN_INTERVAL
value: "5s" # Fastest cadence (default: 5s)
- name: ADAPTIVE_POLLING_MAX_INTERVAL
value: "5m" # Slowest cadence (default: 5m)
```
**Note:** These settings can also be toggled via **Settings → System → Monitoring** in the UI without pod restart.
Install or upgrade with the overrides:
```bash
helm upgrade --install pulse ./deploy/helm/pulse \
--namespace pulse \
--create-namespace \
-f helm-values.yaml
```
The `server.secretEnv` block above pre-seeds one or more API tokens so the UI is immediately accessible and automation can authenticate. If you prefer to manage credentials separately, set `server.secretEnv.name` to reference an existing secret instead of letting the chart create one.
### Accessing Pulse
- **Port forward:** `kubectl -n pulse port-forward svc/pulse 7655:7655`
- **Ingress:** Enable via the snippet above, or supply your own annotations for external DNS, TLS, etc.
- **LoadBalancer:** Set `service.type: LoadBalancer` and, optionally, `service.loadBalancerIP`.
### Persistence Options
- `persistence.enabled`: Disable to use an ephemeral `emptyDir`
- `persistence.existingClaim`: Bind to a pre-provisioned PVC
- `persistence.storageClass`: Pin to a specific storage class
- `persistence.size`: Resize the default PVC request
## Enabling the Docker Agent
The optional agent reports Docker host metrics back to the Pulse hub. Enable it once you have a valid API token:
```yaml
# agent-values.yaml
agent:
enabled: true
env:
- name: PULSE_URL
value: https://pulse.example.com
secretEnv:
create: true
data:
PULSE_TOKEN: docker-agent-token
dockerSocket:
enabled: true
path: /var/run/docker.sock
hostPathType: Socket
PULSE_TOKEN: "my-token"
```
Apply with (choose one):
Apply with:
```bash
helm upgrade --install pulse pulse/pulse -n pulse -f values.yaml
```
- **Published chart:**
---
```bash
helm upgrade pulse oci://ghcr.io/rcourtman/pulse-chart \
--install \
--version <pulse-version> \
--namespace pulse \
-f agent-values.yaml
```
- **Local checkout:**
```bash
helm upgrade --install pulse ./deploy/helm/pulse \
--namespace pulse \
-f agent-values.yaml
```
Notes:
- The agent expects a Docker-compatible runtime and access to the Docker socket. Set `agent.dockerSocket.enabled: false` if you run the agent elsewhere or publish the Docker API securely.
- Use separate API tokens per host; list multiple tokens with `;` or `,` separators in `PULSE_TARGETS` if needed.
- Run the agent as a `DaemonSet` (default) to cover every node, or switch to `agent.kind: Deployment` for a single pod.
## Upgrades and Removal
### Upgrading Pulse
1. **Update the Helm repository:**
```bash
helm repo update pulse
```
2. **Check available versions:**
```bash
helm search repo pulse/pulse --versions
```
3. **Upgrade to latest:**
```bash
helm upgrade pulse pulse/pulse -n pulse
```
4. **Or upgrade to specific version:**
```bash
helm upgrade pulse pulse/pulse --version 4.28.0 -n pulse
```
5. **With custom values file:**
```bash
helm upgrade pulse pulse/pulse -n pulse -f custom-values.yaml
```
### Rollback
If an upgrade causes issues:
## 🔄 Upgrades
```bash
# List revisions
helm history pulse -n pulse
helm repo update
helm upgrade pulse pulse/pulse -n pulse
```
# Rollback to previous revision
**Rollback**:
```bash
helm rollback pulse -n pulse
# Or rollback to specific revision
helm rollback pulse 3 -n pulse
```
### Uninstall
---
```bash
# Remove Pulse but keep PVCs
helm uninstall pulse -n pulse
## ⚠️ Troubleshooting
# Remove everything including PVCs
helm uninstall pulse -n pulse
kubectl delete pvc -n pulse -l app.kubernetes.io/name=pulse
```
### Post-Upgrade Verification (v4.25.0+)
After upgrading to v4.25.0 or newer, verify the deployment:
1. **Check update history**
```bash
# Via UI
# Navigate to Settings → System → Updates
# Via API
kubectl -n pulse exec deploy/pulse -- curl -s http://localhost:7655/api/updates/history | jq '.entries[0]'
```
2. **Verify scheduler health** (adaptive polling)
```bash
kubectl -n pulse exec deploy/pulse -- curl -s http://localhost:7655/api/monitoring/scheduler/health | jq
```
**Expected response:**
- `"enabled": true`
- `queue.depth` reasonable (< instances × 1.5)
- `deadLetter.count` = 0 or only known issues
- `instances[]` populated with your nodes
3. **Check pod logs**
```bash
kubectl -n pulse logs deploy/pulse --tail=50
```
4. **Verify rollback capability**
- Pulse v4.24.0+ logs update history
- Rollback available via **Settings → System → Updates → Restore previous version**
- Or via API: `POST /api/updates/rollback` (if supported in Kubernetes deployments)
## Service Naming
**Important:** The Helm chart creates a service named `svc/pulse` on port `7655` by default, matching standard Kubernetes naming conventions.
**Service name variations:**
- **Kubernetes/Helm:** `svc/pulse` (Deployment: `pulse`, Service: `pulse`)
- **Systemd installations:** `pulse.service` or `pulse-backend.service` (legacy)
- **Hot-dev scripts:** `pulse-hot-dev` (development only, not used in production clusters)
**To check the active service:**
```bash
# Kubernetes
kubectl -n pulse get svc pulse
# Systemd
systemctl status pulse
```
## Troubleshooting
### Verify Scheduler Health After Rollout
**v4.24.0+** includes adaptive polling. After any Helm upgrade or rollback, verify:
```bash
kubectl -n pulse exec deploy/pulse -- curl -s http://localhost:7655/api/monitoring/scheduler/health | jq
```
**Look for:**
- `enabled: true`
- Queue depth stable
- No stuck circuit breakers
- Empty or stable dead-letter queue
### Port Configuration Changes
Port changes via `service.port` or environment variable `FRONTEND_PORT` take effect immediately but should be documented in change logs. v4.24.0 records restarts and configuration changes in update history.
**To verify port configuration:**
```bash
# Check service
kubectl -n pulse get svc pulse -o jsonpath='{.spec.ports[0].port}'
# Check pod environment
kubectl -n pulse exec deploy/pulse -- env | grep PORT
```
## Reference
- Review every available option in `deploy/helm/pulse/values.yaml`
- Inspect published charts without installing: `helm show values oci://ghcr.io/rcourtman/pulse-chart --version <version>`
- Helm template rendering preview: `helm template pulse ./deploy/helm/pulse -f <values>`
- `NOTES.txt` emitted by Helm summarizes the service endpoint and agent prerequisites after each install or upgrade
## Related Documentation
- [Scheduler Health API](api/SCHEDULER_HEALTH.md) - Monitor adaptive polling status
- [Configuration Guide](CONFIGURATION.md) - System settings and environment variables
- [Adaptive Polling Operations](operations/ADAPTIVE_POLLING_ROLLOUT.md) - Operational procedures
- [Reverse Proxy Setup](REVERSE_PROXY.md) - Configure ingress with rate limit headers
- **Check Pods**: `kubectl -n pulse get pods`
- **Check Logs**: `kubectl -n pulse logs deploy/pulse`
- **Scheduler Health**:
```bash
kubectl -n pulse exec deploy/pulse -- curl -s http://localhost:7655/api/monitoring/scheduler/health
```

View file

@ -1,356 +1,54 @@
# Reverse Proxy Configuration
# 🔄 Reverse Proxy Setup
Pulse uses WebSockets for real-time updates. Your reverse proxy **MUST** support WebSocket connections or Pulse will not work correctly.
Pulse uses WebSockets for real-time updates. Your proxy **MUST** support WebSockets.
## Important Requirements
1. **WebSocket Support Required** - Enable WebSocket proxying
2. **Proxy Headers** - Forward original host and IP headers
3. **Timeouts** - Increase timeouts for long-lived connections (7 days recommended)
4. **Buffer Sizes** - Increase for large state updates (64KB recommended)
5. **Rate Limit Headers (v4.24.0+)** - Pass through `X-RateLimit-*` and `Retry-After` headers
6. **Error Pass-through** - Don't intercept 429 responses (rate limits)
## Authentication with Reverse Proxy
Pulse always enforces its own authentication. If you want to delegate sign-in to your reverse proxy (Authentik, Authelia, etc.), configure Pulse's **Proxy Authentication** integration under *Settings → Security*. That lets Pulse trust the authenticated user provided by the proxy while keeping per-user roles, API tokens, and audit logging intact.
> **Note:** The legacy `DISABLE_AUTH` environment variable has been removed. If it still exists in your deployment, Pulse will log a warning at startup and ignore it. Remove the variable and restart Pulse to silence the warning.
## Nginx
## ⚡ Quick Configs
### Nginx
```nginx
server {
listen 80;
server_name pulse.example.com;
location / {
proxy_pass http://localhost:7655;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
# Redirect to HTTPS
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name pulse.example.com;
# SSL configuration
ssl_certificate /path/to/cert.pem;
ssl_certificate_key /path/to/key.pem;
# Proxy settings
location / {
proxy_pass http://localhost:7655;
proxy_http_version 1.1;
# Required for WebSocket
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# Proxy headers
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeouts for WebSocket and adaptive polling (v4.24.0+)
proxy_connect_timeout 7d;
proxy_send_timeout 7d;
proxy_read_timeout 7d;
# Disable buffering for real-time updates
proxy_buffering off;
# IMPORTANT (v4.24.0+): Don't intercept error responses
# This ensures 429 rate limit responses reach the client
proxy_intercept_errors off;
# Increase buffer sizes for large messages
proxy_buffer_size 64k;
proxy_buffers 8 64k;
proxy_busy_buffers_size 128k;
}
# API endpoints (optional, same config as above)
location /api/ {
proxy_pass http://localhost:7655/api/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 7d;
proxy_send_timeout 7d;
proxy_read_timeout 7d;
proxy_buffering off;
}
# Critical for WebSockets
proxy_read_timeout 86400; # 24h
}
```
## Caddy v2
Caddy automatically handles WebSocket upgrades when reverse proxying.
### Caddy
```caddy
pulse.example.com {
reverse_proxy localhost:7655
}
```
For more control:
```caddy
pulse.example.com {
reverse_proxy localhost:7655 {
# Headers automatically handled by Caddy
header_up Host {host}
header_up X-Real-IP {remote}
header_up X-Forwarded-For {remote}
header_up X-Forwarded-Proto {scheme}
# Increase timeouts for WebSocket
transport http {
dial_timeout 30s
response_header_timeout 30s
read_timeout 0
}
}
}
### Traefik (Docker Compose)
```yaml
labels:
- "traefik.enable=true"
- "traefik.http.routers.pulse.rule=Host(`pulse.example.com`)"
- "traefik.http.services.pulse.loadbalancer.server.port=7655"
```
## Apache
### Apache
```apache
<VirtualHost *:443>
ServerName pulse.example.com
SSLEngine on
SSLCertificateFile /path/to/cert.pem
SSLCertificateKeyFile /path/to/key.pem
# Enable necessary modules:
# a2enmod proxy proxy_http proxy_wstunnel headers
# WebSocket proxy
RewriteEngine On
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteRule ^/?(.*) "ws://localhost:7655/$1" [P,L]
# Regular HTTP proxy
ProxyPass / http://localhost:7655/
ProxyPassReverse / http://localhost:7655/
# Preserve host headers
ProxyPreserveHost On
# Forward real IP
RequestHeader set X-Real-IP "%{REMOTE_ADDR}s"
RequestHeader set X-Forwarded-For "%{REMOTE_ADDR}s"
RequestHeader set X-Forwarded-Proto "https"
# Disable buffering
ProxyIOBufferSize 65536
</VirtualHost>
RewriteEngine On
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteRule ^/?(.*) "ws://localhost:7655/$1" [P,L]
ProxyPass / http://localhost:7655/
ProxyPassReverse / http://localhost:7655/
```
## Traefik
---
```yaml
# docker-compose.yml
services:
pulse:
image: rcourtman/pulse:latest
labels:
- "traefik.enable=true"
- "traefik.http.routers.pulse.rule=Host(`pulse.example.com`)"
- "traefik.http.routers.pulse.tls=true"
- "traefik.http.services.pulse.loadbalancer.server.port=7655"
# WebSocket support is automatic in Traefik 2.x+
```
## ⚠️ Common Issues
Or using Traefik file configuration:
```yaml
# traefik-dynamic.yml
http:
routers:
pulse:
rule: "Host(`pulse.example.com`)"
service: pulse
tls: {}
services:
pulse:
loadBalancer:
servers:
- url: "http://localhost:7655"
```
## HAProxy
```haproxy
frontend https
bind *:443 ssl crt /path/to/cert.pem
# ACL for Pulse
acl host_pulse hdr(host) -i pulse.example.com
# WebSocket detection
acl is_websocket hdr(Upgrade) -i websocket
# Use backend
use_backend pulse if host_pulse
backend pulse
# Health check
option httpchk GET /api/health
# WebSocket support
option http-server-close
option forwardfor
# Timeouts for WebSocket
timeout client 3600s
timeout server 3600s
timeout tunnel 3600s
# Backend server
server pulse1 localhost:7655 check
```
## Cloudflare Tunnel
If using Cloudflare Tunnel (cloudflared):
```yaml
# config.yml
tunnel: YOUR_TUNNEL_ID
credentials-file: /path/to/credentials.json
ingress:
- hostname: pulse.example.com
service: http://localhost:7655
originRequest:
# Enable WebSocket
noTLSVerify: false
connectTimeout: 30s
# No additional config needed - WebSockets work by default
- service: http_status:404
```
## Testing WebSocket Connection
After configuring your reverse proxy, test that WebSockets work:
```bash
# Test basic connectivity
curl https://pulse.example.com/api/health
# Test WebSocket upgrade (should return 101 Switching Protocols)
curl -i -N \
-H "Connection: Upgrade" \
-H "Upgrade: websocket" \
-H "Sec-WebSocket-Version: 13" \
-H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \
https://pulse.example.com/api/ws
```
### v4.24.0+ Verification
**Test scheduler health API** (adaptive polling):
```bash
curl -s https://pulse.example.com/api/monitoring/scheduler/health | jq
```
**Expected response:**
- `enabled: true`
- Queue depth reasonable
- No stuck circuit breakers
- WebSocket connection remains open
**Verify rate limit headers** (v4.24.0+):
```bash
curl -I https://pulse.example.com/api/version
```
**Look for headers:**
```
HTTP/1.1 200 OK
X-RateLimit-Limit: 500
X-RateLimit-Remaining: 499
X-RateLimit-Reset: 1698765432
```
**Test rate limiting:**
```bash
# Trigger rate limit (requires many requests)
for i in {1..600}; do curl -s https://pulse.example.com/api/health > /dev/null; done
# Should eventually see:
curl -I https://pulse.example.com/api/health
# HTTP/1.1 429 Too Many Requests
# X-RateLimit-Limit: 500
# X-RateLimit-Remaining: 0
# Retry-After: 60
```
**Important:** If you don't see rate limit headers or 429 responses are converted to 502/504, verify `proxy_intercept_errors off` (Nginx) or equivalent is set.
In browser console (F12):
```javascript
// Test WebSocket connection
const ws = new WebSocket('wss://pulse.example.com/api/ws');
ws.onopen = () => console.log('WebSocket connected!');
ws.onmessage = (e) => console.log('Received:', e.data);
ws.onerror = (e) => console.error('WebSocket error:', e);
```
## Common Issues
### "Connection Lost" or no real-time updates
- WebSocket upgrade not configured correctly
- Check proxy passes `Upgrade` and `Connection` headers
- Verify timeouts are increased for long connections
### CORS errors
- Pulse handles CORS internally
- Don't add additional CORS headers in proxy
- If needed, set `ALLOWED_ORIGINS` in Pulse configuration
### 502 Bad Gateway
- Pulse not running on expected port (default 7655)
- Check with: `curl http://localhost:7655/api/health`
- Verify Pulse service: `systemctl status pulse` (use `pulse-backend` if you're on a legacy unit)
### WebSocket closes immediately
- Timeout too short in proxy configuration
- Increase `proxy_read_timeout` (Nginx) or equivalent
- Set to at least 3600s (1 hour) or more
## Security Recommendations
1. **Always use HTTPS** for production deployments
2. **Set proper headers** to prevent clickjacking:
```nginx
add_header X-Frame-Options "SAMEORIGIN";
add_header X-Content-Type-Options "nosniff";
```
3. **Rate limiting** for API endpoints:
```nginx
limit_req_zone $binary_remote_addr zone=pulse:10m rate=30r/s;
limit_req zone=pulse burst=50 nodelay;
```
4. **Hide proxy version**:
```nginx
proxy_hide_header X-Powered-By;
server_tokens off;
```
## Support
If WebSockets still don't work after following this guide:
1. Check browser console for errors (F12)
2. Verify Pulse logs: `journalctl -u pulse -f`
3. Test without proxy first: `http://your-server:7655`
4. Report issues: https://github.com/rcourtman/Pulse/issues
- **"Connection Lost"**: WebSocket upgrade failed. Check `Upgrade` and `Connection` headers.
- **502 Bad Gateway**: Pulse is not running on port 7655.
- **CORS Errors**: Do not add CORS headers in the proxy; Pulse handles them. Set `ALLOWED_ORIGINS` env var if needed.

View file

@ -1,373 +1,43 @@
# Webhook Configuration Guide
# 🔔 Webhooks
Pulse supports sending alert notifications to various webhook services including Discord, Slack, Microsoft Teams, Telegram, Gotify, ntfy, PagerDuty, and any custom webhook endpoint.
Pulse supports Discord, Slack, Teams, Telegram, Gotify, ntfy, and generic webhooks.
## Quick Start
## 🚀 Quick Setup
1. Navigate to **Alerts****Notifications** tab
2. Configure email settings or add webhooks
3. Select your service type (Discord, Slack, Teams, Telegram, etc.)
4. Enter the webhook URL and configure settings
5. Test the webhook to ensure it's working
6. Save your configuration
1. Go to **Alerts → Notifications**.
2. Click **Add Webhook**.
3. Select service type and paste the URL.
![Alert Configuration](images/04-alerts.png)
*Alert configuration interface showing notification settings*
## 📝 Service URLs
## Supported Services
| Service | URL Format |
|---------|------------|
| **Discord** | `https://discord.com/api/webhooks/{id}/{token}` |
| **Slack** | `https://hooks.slack.com/services/...` |
| **Teams** | `https://{tenant}.webhook.office.com/...` |
| **Telegram** | `https://api.telegram.org/bot{token}/sendMessage?chat_id={id}` |
| **Gotify** | `https://gotify.example.com/message?token={token}` |
| **ntfy** | `https://ntfy.sh/{topic}` |
### Discord
```
URL Format: https://discord.com/api/webhooks/{webhook_id}/{webhook_token}
```
1. In Discord, go to Server Settings → Integrations → Webhooks
2. Create a new webhook and copy the URL
3. Paste the URL in Pulse
## 🎨 Custom Templates
### Telegram
```
URL Format: https://api.telegram.org/bot{bot_token}/sendMessage?chat_id={chat_id}
```
1. Create a bot with @BotFather on Telegram
2. Get your bot token from BotFather
3. Get your chat ID by messaging the bot and visiting: `https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates`
4. In Pulse, select "Telegram Bot" as the service type
5. Use the URL format: `https://api.telegram.org/bot<BOT_TOKEN>/sendMessage?chat_id=<CHAT_ID>`
6. **IMPORTANT**: The chat_id MUST be included in the URL as a parameter
7. Pulse automatically sends rich formatted messages with emojis and full alert details
For generic webhooks, use Go templates to format the JSON payload.
### Slack
```
URL Format: https://hooks.slack.com/services/{webhook_path}
```
1. In Slack, go to Apps → Incoming Webhooks
2. Add to Slack and choose a channel
3. Copy the webhook URL
**Variables:**
- `{{.Message}}`: Alert text
- `{{.Level}}`: warning/critical
- `{{.Node}}`: Node name
- `{{.Value}}`: Metric value (e.g. 95.5)
### Microsoft Teams
```
URL Format: https://{tenant}.webhook.office.com/webhookb2/{webhook_path}
```
1. In Teams channel, click ... → Connectors
2. Configure Incoming Webhook
3. Copy the URL
### Gotify
```
URL Format: https://your-gotify-server/message?token={your-app-token}
```
1. In Gotify, create a new application
2. Copy the application token
3. Use the URL format: `https://your-gotify-server/message?token=YOUR_APP_TOKEN`
4. The token MUST be included as a URL parameter
5. Pulse will send rich markdown-formatted notifications with emojis and full alert details
6. **View in Pulse links**: Automatically detected - links will work out of the box in most cases
### ntfy
```
URL Format: https://ntfy.sh/{topic} or https://your-ntfy-server/{topic}
```
1. Choose a unique topic name (e.g., 'pulse-alerts-x7k9m2')
- **Important**: Anyone who knows your topic name can send you notifications
- Use a unique/random suffix for privacy
2. For ntfy.sh: Use `https://ntfy.sh/YOUR_TOPIC`
3. For self-hosted: Use `https://your-ntfy-server/YOUR_TOPIC`
4. Subscribe to the same topic in your ntfy mobile/desktop app
5. For authentication (optional):
- Click "Custom Headers" section in webhook config
- Add header: `Authorization`
- Value: `Bearer YOUR_TOKEN` or `Basic base64_encoded_credentials`
6. Notifications include dynamic priority levels and emoji tags based on alert severity
7. **View in Pulse links**: Automatically detected - links will work out of the box in most cases
### PagerDuty
```
URL: https://events.pagerduty.com/v2/enqueue
```
1. In PagerDuty, go to Configuration → Services
2. Add an integration → Events API V2
3. Copy the Integration Key
4. Add the key as a header: `routing_key: YOUR_KEY`
## Custom Headers
For webhooks that require authentication or custom headers:
1. In the webhook configuration, expand the **Custom Headers** section
2. Click **+ Add Header** to add a new header
3. Enter the header name (e.g., `Authorization`, `X-API-Key`, `X-Auth-Token`)
4. Enter the header value (e.g., `Bearer YOUR_TOKEN`, your API key, etc.)
5. Add multiple headers as needed
6. Headers are sent with every webhook request
### Common Header Examples
| Service | Header Name | Header Value Format |
|---------|-------------|-------------------|
| Bearer Token | `Authorization` | `Bearer YOUR_TOKEN_HERE` |
| Basic Auth | `Authorization` | `Basic base64_encoded_user:pass` |
| API Key | `X-API-Key` | `your-api-key-here` |
| Custom Token | `X-Auth-Token` | `your-auth-token` |
| ntfy Auth | `Authorization` | `Bearer tk_your_ntfy_token` |
| Custom Service | `X-Service-Key` | `service-specific-key` |
## Custom Payload Templates
```mermaid
flowchart LR
Alert[Alert Triggered]
Template[Render Template<br/>Variables & Functions]
Send[HTTP POST]
Alert --> Template
Template --> Send
Send -->|2xx| Success[Delivered]
Send -->|5xx| Retry[Retry with Backoff]
```
For generic webhooks, you can define custom JSON payloads using Go template syntax.
### Available Variables
| Variable | Description | Example Value |
|----------|-------------|---------------|
| `{{.ID}}` | Alert ID | "alert-123" |
| `{{.Level}}` | Alert level | "warning", "critical" |
| `{{.Type}}` | Resource type | "cpu", "memory", "disk" |
| `{{.ResourceName}}` | Name of the resource | "Web Server VM" |
| `{{.ResourceID}}` | Resource identifier | "vm-100" |
| `{{.Node}}` | Proxmox node name | "pve-node-01" |
| `{{.Instance}}` | Proxmox instance URL | "https://192.168.1.100:8006" |
| `{{.Message}}` | Alert message | "CPU usage exceeded 90%" |
| `{{.Value}}` | Current metric value | 95.5 |
| `{{.Threshold}}` | Alert threshold | 90.0 |
| `{{.Duration}}` | How long alert has been active | "5m" |
| `{{.Timestamp}}` | Current timestamp | "2024-01-15T10:30:00Z" |
| `{{.StartTime}}` | When alert started | "2024-01-15T10:25:00Z" |
### Template Functions
| Function | Description | Example |
|----------|-------------|---------|
| `{{.Level \| title}}` | Capitalize first letter | "Warning" |
| `{{.Level \| upper}}` | Uppercase | "WARNING" |
| `{{.Level \| lower}}` | Lowercase | "warning" |
| `{{printf "%.1f" .Value}}` | Format numbers | "95.5" |
| `{{urlpath .ResourceName}}` | URL-encode a value for path segments | "web%20server" |
| `{{urlquery .Message}}` | URL-encode a value for query parameters | "CPU+usage+%3E+90%25" |
## Dynamic URL Templates
Webhook URLs can also reference the same template variables that are available to payloads. Pulse renders the URL before sending the request and safely encodes path segments by default. When placing values inside query parameters, wrap them with the provided helpers to ensure correct encoding:
```text
https://example.com/hooks/{{urlpath .ResourceName}}?msg={{urlquery .Message}}
```
Supported variables are identical to the payload template list (`{{.Message}}`, `{{.Node}}`, custom fields, etc.). This makes it easy to call services that expect alert metadata in the URL itself.
### Example Templates
#### Simple JSON
**Example Payload:**
```json
{
"text": "Alert: {{.Level}} - {{.Message}}",
"resource": "{{.ResourceName}}",
"value": {{.Value}},
"threshold": {{.Threshold}}
"value": {{.Value}}
}
```
#### Formatted Alert
```json
{
"alert": {
"level": "{{.Level | upper}}",
"message": "{{.Message}}",
"details": {
"resource": "{{.ResourceName}}",
"node": "{{.Node}}",
"current_value": "{{printf "%.1f" .Value}}%",
"threshold": "{{printf "%.0f" .Threshold}}%",
"duration": "{{.Duration}}"
}
},
"timestamp": "{{.Timestamp}}"
}
```
## 🛡️ Security
#### Slack-Compatible Custom Format
```json
{
"text": "Pulse Alert",
"attachments": [{
"color": "{{if eq .Level "critical"}}danger{{else}}warning{{end}}",
"title": "{{.Level | title}} Alert: {{.ResourceName}}",
"text": "{{.Message}}",
"fields": [
{"title": "Value", "value": "{{printf "%.1f" .Value}}%", "short": true},
{"title": "Threshold", "value": "{{printf "%.0f" .Threshold}}%", "short": true},
{"title": "Node", "value": "{{.Node}}", "short": true},
{"title": "Duration", "value": "{{.Duration}}", "short": true}
],
"footer": "Pulse Monitoring",
"ts": {{.Timestamp}}
}]
}
```
#### Home Assistant
```json
{
"title": "Pulse Alert: {{.Level | title}}",
"message": "{{.Message}}",
"data": {
"entity_id": "sensor.{{.Node | lower}}_{{.Type}}",
"state": {{.Value}},
"attributes": {
"resource": "{{.ResourceName}}",
"threshold": {{.Threshold}},
"duration": "{{.Duration}}"
}
}
}
```
#### n8n / Node-RED
```json
{
"workflow": "pulse_alert",
"data": {
"alert_id": "{{.ID}}",
"level": "{{.Level}}",
"resource": "{{.ResourceName}}",
"node": "{{.Node}}",
"metric": {
"type": "{{.Type}}",
"value": {{.Value}},
"threshold": {{.Threshold}}
},
"message": "{{.Message}}",
"timestamp": "{{.Timestamp}}"
}
}
```
## Testing Webhooks
1. After configuring a webhook, click the **Test** button
2. Pulse will send a test alert to verify the webhook is working
3. Check the receiving service to confirm the message arrived
4. If the test fails, verify:
- The URL is correct and accessible
- Any required authentication tokens are included
- The payload format matches what the service expects
## Troubleshooting
### Webhook Returns 400 Bad Request
- Check if the payload format is correct for your service
- For Telegram, ensure chat_id is in the URL (Pulse handles it automatically)
- Verify all required fields are present in custom templates
### Webhook Returns 401/403
- Check authentication tokens/keys
- Verify the webhook URL hasn't expired
- Ensure IP restrictions allow Pulse server
### No Notifications Received
- Verify the webhook is enabled
- Check alert thresholds are configured correctly
- Ensure notification cooldown period has passed
- Test the webhook manually using the Test button
- Temporarily raise log level to `debug` via **Settings → System → Logging** (or set `LOG_LEVEL=debug` and restart) to inspect `webhook.delivery` entries in the logs, then return to `info` once resolved.
## API Reference
### Create Webhook
```bash
POST /api/notifications/webhooks
Content-Type: application/json
{
"name": "My Webhook",
"url": "https://example.com/webhook",
"method": "POST",
"service": "generic",
"enabled": true,
"template": "{\"alert\": \"{{.Message}}\"}"
}
```
### Test Webhook
```bash
POST /api/notifications/webhooks/test
Content-Type: application/json
{
"name": "Test",
"url": "https://example.com/webhook",
"service": "generic",
"template": "{\"test\": true}"
}
```
### Update Webhook
```bash
PUT /api/notifications/webhooks/{id}
Content-Type: application/json
{
"name": "Updated Webhook",
"url": "https://example.com/new-webhook",
"enabled": false
}
```
### Delete Webhook
```bash
DELETE /api/notifications/webhooks/{id}
```
### List Webhooks
```bash
GET /api/notifications/webhooks
```
## Security Considerations
- **Never expose webhook URLs publicly** - they often contain authentication tokens
- **Use HTTPS URLs** when possible to encrypt data in transit
- **Rotate webhook URLs periodically** if they contain embedded tokens
- **Test webhooks carefully** to avoid sending test data to production channels
- **Limit webhook permissions** in the receiving service where possible
### Private IP Address Protection
By default, Pulse blocks webhooks to private IP addresses (192.168.x.x, 10.x.x.x, 172.16-31.x.x) for security. This prevents potential SSRF (Server-Side Request Forgery) attacks.
**For homelab deployments** where you need to send webhooks to internal services (like Traefik, Home Assistant, or local notification servers):
1. Navigate to **Settings → System**
2. Find the **Webhook Security** section
3. Enter trusted CIDR ranges in **Allowed Private IP Ranges for Webhooks**
- Example: `192.168.1.0/24,10.0.0.0/8`
- Supports both CIDR notation and individual IPs
4. Save settings
**Important security notes:**
- Leave empty to block all private IPs (default, most secure)
- Only add networks you fully trust
- Localhost (127.0.0.1), link-local (169.254.x.x), and cloud metadata services remain blocked even with an allowlist
- Changes apply immediately to new webhook configurations
- Existing webhooks are re-validated before each delivery
**Example configurations:**
```
Single network: 192.168.1.0/24
Multiple networks: 192.168.1.0/24,10.0.0.0/8
Single IP: 192.168.1.100
Mixed: 192.168.1.0/24,10.5.10.20
```
If you see an error like `"webhook URL resolves to private IP ... - private networks are not allowed"`, you need to configure the allowlist in System Settings.
- **Private IPs**: By default, webhooks to private IPs are blocked. Allow them in **Settings → System → Webhook Security**.
- **Headers**: Add custom headers (e.g., `Authorization: Bearer ...`) in the webhook config.