feat: implement priority queue-based task execution (Phase 2 Task 6)
Replaces immediate polling with queue-based scheduling: - TaskQueue with min-heap (container/heap) for NextRun-ordered execution - Worker goroutines that block on WaitNext() until tasks are due - Tasks only execute when NextRun <= now, respecting adaptive intervals - Automatic rescheduling after execution via scheduler.BuildPlan - Queue depth tracking for backpressure-aware interval adjustments - Upsert semantics for updating scheduled tasks without duplicates Task 6 of 10 complete (60%). Ready for error/backoff policies.
This commit is contained in:
rcourtman
parent
c554380cb5
commit
aa5c08ad4a
8 changed files with 726 additions and 645 deletions
526
SECURITY.md
526
SECURITY.md
|
|
@ -1,45 +1,50 @@
|
|||
# Pulse Security Documentation
|
||||
# Pulse Security
|
||||
|
||||
## Critical Security Notice for Production Deployments
|
||||
This document is the canonical security policy for Pulse. It combines our
|
||||
ongoing hardening guidance with the operational checklists that previously lived
|
||||
in `docs/SECURITY.md`.
|
||||
|
||||
---
|
||||
|
||||
## Critical Security Notice for Container Deployments
|
||||
|
||||
### Container SSH Key Policy (BREAKING CHANGE)
|
||||
|
||||
**Effective immediately, SSH-based temperature monitoring is BLOCKED in containerized Pulse deployments.**
|
||||
**Effective immediately, SSH-based temperature monitoring is blocked in
|
||||
containerized Pulse deployments.**
|
||||
|
||||
#### Why This Change?
|
||||
|
||||
Storing SSH private keys inside Docker containers creates an unacceptable security risk in production environments:
|
||||
Storing SSH private keys inside Docker/LXC containers creates an unacceptable
|
||||
risk in production environments:
|
||||
|
||||
- **Container compromise = Infrastructure compromise**: If an attacker gains access to your Pulse container, they immediately obtain SSH private keys with root access to your Proxmox infrastructure.
|
||||
- **Keys persist in images**: SSH keys can be extracted from container layers and images if pushed to registries.
|
||||
- **No key rotation**: Long-lived keys in containers are difficult to rotate.
|
||||
- **Violates principle of least privilege**: Containers should not hold credentials for the infrastructure they monitor.
|
||||
- **Container compromise = infrastructure compromise** – if an attacker gains
|
||||
shell access to the Pulse container they obtain the SSH private keys used to
|
||||
reach your Proxmox hosts.
|
||||
- **Keys persist in images** – private keys survive in image layers and can leak
|
||||
when images are pushed to registries or shared.
|
||||
- **No key rotation** – long-lived keys inside containers are difficult to
|
||||
rotate safely.
|
||||
- **Violates least-privilege** – monitoring containers should not hold
|
||||
credentials that grant host-level access to the infrastructure they observe.
|
||||
|
||||
#### Affected Deployments
|
||||
|
||||
✅ **Not Affected** (SSH temperature monitoring still allowed):
|
||||
- Pulse installed directly on a VM or bare metal (non-containerized)
|
||||
- Home lab deployments where you understand and accept the risk
|
||||
✅ **Not affected** – Pulse installed directly on a VM or bare-metal host (no
|
||||
containers), or homelab environments where you explicitly accept the risk.
|
||||
|
||||
❌ **BLOCKED** (SSH temperature monitoring disabled):
|
||||
- Pulse running in Docker containers
|
||||
- Pulse running in LXC containers
|
||||
- Any deployment where `PULSE_DOCKER=true` or `/.dockerenv` exists
|
||||
❌ **Blocked** – Pulse running in Docker containers, LXC containers, or any
|
||||
environment where `PULSE_DOCKER=true`/`/.dockerenv` is detected.
|
||||
|
||||
#### Migration Path
|
||||
#### Migration Path (Production)
|
||||
|
||||
**For Production Container Deployments:**
|
||||
|
||||
1. **Deploy pulse-sensor-proxy on each Proxmox host:**
|
||||
1. **Deploy `pulse-sensor-proxy` on each Proxmox host**
|
||||
```bash
|
||||
# On each Proxmox host
|
||||
curl -o /usr/local/bin/pulse-sensor-proxy \
|
||||
https://github.com/rcourtman/pulse/releases/latest/download/pulse-sensor-proxy
|
||||
|
||||
chmod +x /usr/local/bin/pulse-sensor-proxy
|
||||
```
|
||||
|
||||
2. **Create systemd service** (`/etc/systemd/system/pulse-sensor-proxy.service`):
|
||||
2. **Create a systemd unit** (`/etc/systemd/system/pulse-sensor-proxy.service`)
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Pulse Temperature Sensor Proxy
|
||||
|
|
@ -54,134 +59,445 @@ Storing SSH private keys inside Docker containers creates an unacceptable securi
|
|||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
3. **Enable and start:**
|
||||
3. **Enable and start the service**
|
||||
```bash
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now pulse-sensor-proxy
|
||||
```
|
||||
4. **Restart the Pulse container** so it binds to the proxy socket. The
|
||||
container will automatically fall back to socket-based temperature polling.
|
||||
|
||||
4. **Restart Pulse container** - it will automatically detect and use the proxy
|
||||
#### Removing Old SSH Keys
|
||||
|
||||
**Removing Existing SSH Keys:**
|
||||
|
||||
If you previously used SSH-based temperature monitoring in containers:
|
||||
If you previously generated SSH keys inside containers:
|
||||
|
||||
```bash
|
||||
# On each Proxmox host, remove Pulse SSH keys
|
||||
# On each Proxmox host
|
||||
sed -i '/# pulse-/d' /root/.ssh/authorized_keys
|
||||
|
||||
# Inside the Pulse container (or destroy and recreate)
|
||||
# Inside the Pulse container (or rebuild the container)
|
||||
docker exec pulse rm -rf /home/pulse/.ssh/id_ed25519*
|
||||
```
|
||||
|
||||
#### Technical Details
|
||||
|
||||
**How pulse-sensor-proxy Works:**
|
||||
|
||||
- Runs as a lightweight daemon on the Proxmox host
|
||||
- Exposes a Unix socket at `/run/pulse-sensor-proxy.sock`
|
||||
- Pulse container connects via bind-mounted socket
|
||||
- Only exposes `sensors -j` output - no SSH access
|
||||
- Keys never leave the Proxmox host
|
||||
|
||||
**Security Boundaries:**
|
||||
#### Security Boundary
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ Proxmox Host │
|
||||
│ ┌───────────────────────────────┐ │
|
||||
│ │ pulse-sensor-proxy (root) │ │
|
||||
│ │ - Runs sensors -j │ │
|
||||
│ │ - Unix socket only │ │
|
||||
│ │ · Runs sensors -j │ │
|
||||
│ │ · Exposes Unix socket only │ │
|
||||
│ └───────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ │ /run/pulse-sensor-proxy.sock
|
||||
│ │ │
|
||||
│ ┌─────────▼─────────────────────┐ │
|
||||
│ │ Container (bind mount) │ │
|
||||
│ │ - No SSH keys │ │
|
||||
│ │ - No root access to host │ │
|
||||
│ │ Pulse container (bind mount) │ │
|
||||
│ │ · No SSH keys │ │
|
||||
│ │ · No host root privileges │ │
|
||||
│ └───────────────────────────────┘ │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
#### For Home Lab Users
|
||||
#### Homelab Exception
|
||||
|
||||
If you understand and accept the risk, you can still use non-containerized Pulse with SSH keys:
|
||||
If you fully understand the risk and are **not** containerized (VM/bare-metal
|
||||
install), the legacy SSH flow still works. Use a dedicated monitoring user,
|
||||
restrict the key with `command="sensors -j"` and `from="<pulse-ip>"`, and
|
||||
rotate keys regularly.
|
||||
|
||||
1. Install Pulse directly on a VM (not in Docker)
|
||||
2. Setup script will offer SSH temperature monitoring
|
||||
3. Follow standard security practices:
|
||||
- Use dedicated monitoring user (not root)
|
||||
- Restrict key with `command="sensors -j"`
|
||||
- Add `from="<pulse-ip>"` restrictions
|
||||
- Rotate keys periodically
|
||||
#### Auditing Your Deployment
|
||||
|
||||
#### Audit Your Deployment
|
||||
|
||||
**Check if you're affected:**
|
||||
```bash
|
||||
# Inside Pulse container
|
||||
ls /home/pulse/.ssh/id_ed25519* 2>/dev/null && echo "⚠️ VULNERABLE"
|
||||
# Detect vulnerable containers
|
||||
ls /home/pulse/.ssh/id_ed25519* 2>/dev/null && echo "⚠️ SSH keys present"
|
||||
|
||||
# On Proxmox host
|
||||
grep "# pulse-" /root/.ssh/authorized_keys && echo "⚠️ SSH keys present"
|
||||
```
|
||||
|
||||
**Check if proxy is working:**
|
||||
```bash
|
||||
# On Proxmox host
|
||||
systemctl status pulse-sensor-proxy
|
||||
|
||||
# Inside Pulse container
|
||||
# Check container logs for proxy detection
|
||||
docker logs pulse | grep -i "temperature proxy detected"
|
||||
|
||||
# Verify the host service
|
||||
systemctl status pulse-sensor-proxy
|
||||
```
|
||||
|
||||
#### Timeline
|
||||
|
||||
- **Now**: SSH key generation blocked in containers (code-level enforcement)
|
||||
- **Next Release**: Setup script updated with clear warnings
|
||||
- **Future**: pulse-sensor-proxy bundled in official releases
|
||||
|
||||
#### Questions?
|
||||
|
||||
- Documentation: https://docs.pulseapp.io/security/containerized-deployments
|
||||
- GitHub Issues: https://github.com/rcourtman/pulse/issues
|
||||
- Security Issues: security@pulseapp.io (private disclosure)
|
||||
**Documentation:** https://docs.pulseapp.io/security/containerized-deployments
|
||||
**Issues:** https://github.com/rcourtman/pulse/issues
|
||||
**Private disclosures:** security@pulseapp.io
|
||||
|
||||
---
|
||||
|
||||
## General Security Best Practices
|
||||
## Mandatory Authentication
|
||||
|
||||
### Authentication
|
||||
**Starting with v4.5.0, authentication setup is prompted for all new Pulse
|
||||
installations.** This protects your Proxmox API credentials from unauthorized
|
||||
access.
|
||||
|
||||
- Use API tokens with minimal required permissions
|
||||
- Rotate tokens regularly
|
||||
- Never commit tokens to version control
|
||||
- Use read-only tokens where possible
|
||||
> **Service name note:** systemd deployments use `pulse.service`. If you're
|
||||
> upgrading from an older install that still registers `pulse-backend.service`,
|
||||
> substitute that name in the commands below.
|
||||
|
||||
### Network Security
|
||||
### First-Run Security Setup
|
||||
When you first access Pulse, you'll be guided through a mandatory security
|
||||
setup:
|
||||
- Create your admin username and password
|
||||
- Automatic API token generation for automation
|
||||
- Settings are applied immediately without restart
|
||||
- **Your existing nodes and settings are preserved**
|
||||
|
||||
- Run Pulse in a dedicated monitoring VLAN
|
||||
- Restrict Pulse's network access to only monitored systems
|
||||
- Use firewall rules to limit inbound connections
|
||||
- Enable TLS for all Proxmox API connections
|
||||
## Smart Security Context
|
||||
|
||||
### Monitoring
|
||||
### Public Access Detection
|
||||
Pulse automatically detects when it's being accessed from public networks:
|
||||
- **Private networks**: local/RFC1918 addresses (192.168.x.x, 10.x.x.x, etc.)
|
||||
- **Public networks**: any non-private IP address
|
||||
- **Stronger warnings**: red alerts when accessed from public IPs without
|
||||
authentication
|
||||
|
||||
- Enable audit logging on Proxmox hosts
|
||||
- Monitor Pulse container logs for suspicious activity
|
||||
- Set up alerts for failed authentication attempts
|
||||
- Review access logs regularly
|
||||
### Trusted Networks Configuration (Deprecated)
|
||||
**Note:** authentication is now mandatory regardless of network location.
|
||||
|
||||
### Updates
|
||||
Legacy configuration (no longer applicable):
|
||||
```bash
|
||||
# Environment variable (comma-separated CIDR blocks)
|
||||
PULSE_TRUSTED_NETWORKS=192.168.1.0/24,10.0.0.0/24
|
||||
|
||||
- Keep Pulse updated to latest stable version
|
||||
- Subscribe to security announcements
|
||||
- Test updates in staging before production
|
||||
- Have rollback plan ready
|
||||
# Or in systemd
|
||||
sudo systemctl edit pulse
|
||||
[Service]
|
||||
Environment="PULSE_TRUSTED_NETWORKS=192.168.1.0/24,10.0.0.0/24"
|
||||
```
|
||||
|
||||
---
|
||||
When configured:
|
||||
- Access from trusted networks: no auth required
|
||||
- Access from outside: authentication enforced
|
||||
- Useful for: mixed home/remote access scenarios
|
||||
|
||||
Last updated: 2025-10-19
|
||||
## Security Warning System
|
||||
|
||||
Pulse includes a non-intrusive security warning system that helps you
|
||||
understand your security posture.
|
||||
|
||||
### Security Score
|
||||
Your instance receives a score from 0‑5 based on:
|
||||
- ✅ Credentials encrypted at rest (always enabled)
|
||||
- ✅ Export/import protection
|
||||
- ⚠️ Authentication enabled
|
||||
- ⚠️ HTTPS connection
|
||||
- ⚠️ Audit logging
|
||||
|
||||
### Dismissing Warnings
|
||||
If you're comfortable with your security setup, you can dismiss warnings:
|
||||
- **For 1 day** – reminder tomorrow
|
||||
- **For 1 week** – reminder next week
|
||||
- **Forever** – won't show again
|
||||
|
||||
## Credential Security
|
||||
|
||||
### Encrypted at Rest (AES-256-GCM)
|
||||
- **Node credentials**: passwords and API tokens (`/etc/pulse/nodes.enc`)
|
||||
- **Email settings**: SMTP passwords (`/etc/pulse/email.enc`)
|
||||
- **Webhook data**: URLs and auth headers (`/etc/pulse/webhooks.enc`) – v4.1.9+
|
||||
- **Encryption key**: auto-generated (`/etc/pulse/.encryption.key`)
|
||||
|
||||
### Security Features
|
||||
- **Logs**: token values masked with `***` in all outputs
|
||||
- **API**: frontend receives only `hasToken: true`, never actual values
|
||||
- **Export**: requires a valid API token (`X-API-Token` header or `token`
|
||||
parameter) to extract credentials
|
||||
- **Migration**: use passphrase-protected export/import (see
|
||||
[Migration Guide](docs/MIGRATION.md))
|
||||
- **Auto-migration**: unencrypted configs automatically migrate to encrypted
|
||||
format
|
||||
|
||||
## Export/Import Protection
|
||||
|
||||
By default, configuration export/import is blocked. You have two options:
|
||||
|
||||
### Option 1: Set API Tokens (Recommended)
|
||||
```bash
|
||||
# Using systemd (secure)
|
||||
sudo systemctl edit pulse
|
||||
# Add:
|
||||
[Service]
|
||||
Environment="API_TOKENS=ansible-token,docker-agent-token"
|
||||
Environment="API_TOKEN=legacy-token"
|
||||
|
||||
# Then restart:
|
||||
sudo systemctl restart pulse
|
||||
|
||||
# Docker
|
||||
docker run -e API_TOKENS=ansible-token,docker-agent-token rcourtman/pulse:latest
|
||||
```
|
||||
|
||||
### Option 2: Allow Unprotected Export (Homelab)
|
||||
```bash
|
||||
# Using systemd
|
||||
sudo systemctl edit pulse
|
||||
# Add:
|
||||
[Service]
|
||||
Environment="ALLOW_UNPROTECTED_EXPORT=true"
|
||||
|
||||
# Docker
|
||||
docker run -e ALLOW_UNPROTECTED_EXPORT=true rcourtman/pulse:latest
|
||||
```
|
||||
|
||||
**Note:** for production, prefer Docker secrets or systemd environment files
|
||||
for sensitive data.
|
||||
|
||||
## Security Features
|
||||
|
||||
### Core Protection
|
||||
- **Encryption**: credentials encrypted at rest (AES-256-GCM)
|
||||
- **Export protection**: exports always encrypted with a passphrase
|
||||
- **Minimum passphrase**: 12 characters required for exports
|
||||
- **Security tab**: check status in *Settings → Security*
|
||||
|
||||
### Enterprise Security (When Authentication Enabled)
|
||||
- **Password security**
|
||||
- Bcrypt hashing with cost factor 12 (60‑character hash)
|
||||
- Passwords never stored in plain text
|
||||
- Automatic hashing during security setup
|
||||
- **Critical**: bcrypt hashes must be exactly 60 characters
|
||||
- **API token security**
|
||||
- 64‑character hex tokens (32 bytes entropy)
|
||||
- SHA3-256 hashed before storage (64‑character hash)
|
||||
- Raw token shown only once
|
||||
- Tokens never stored in plain text
|
||||
- Live reloading when `.env` changes
|
||||
- API-only mode supported (no password auth required)
|
||||
- **CSRF protection**: all state-changing operations require CSRF tokens
|
||||
- **Rate limiting**
|
||||
- Auth endpoints: 10 attempts/minute per IP
|
||||
- General API: 500 requests/minute per IP
|
||||
- Real-time endpoints exempt for functionality
|
||||
- **Account lockout**
|
||||
- Locks after 5 failed login attempts
|
||||
- 15-minute automatic lockout duration
|
||||
- Clear feedback showing remaining attempts
|
||||
- Time remaining displayed when locked
|
||||
- Manual reset available via API for admins
|
||||
- **Session management**
|
||||
- Secure HttpOnly cookies
|
||||
- 24-hour session expiry
|
||||
- Session invalidation on password change
|
||||
- **Security headers**
|
||||
- Content-Security-Policy
|
||||
- X-Frame-Options: DENY
|
||||
- X-Content-Type-Options: nosniff
|
||||
- X-XSS-Protection: 1; mode=block
|
||||
- Referrer-Policy: strict-origin-when-cross-origin
|
||||
- Permissions-Policy restricting sensitive APIs
|
||||
- **Audit logging**: authentication events include IP addresses
|
||||
|
||||
### What's Encrypted in Exports
|
||||
- Node credentials (passwords, API tokens)
|
||||
- PBS credentials
|
||||
- Email settings passwords
|
||||
- Webhook URLs and authentication headers (v4.1.9+)
|
||||
|
||||
### What's **Not** Encrypted
|
||||
- Node hostnames and IPs
|
||||
- Threshold settings
|
||||
- General configuration
|
||||
- Alert rules and schedules
|
||||
|
||||
## Authentication Workflows
|
||||
|
||||
Pulse supports multiple authentication methods that can be used independently or
|
||||
together.
|
||||
|
||||
### Password Authentication
|
||||
|
||||
#### Quick Security Setup (Recommended)
|
||||
1. Navigate to *Settings → Security*.
|
||||
2. Click **Enable Security Now**.
|
||||
3. Enter username and password.
|
||||
4. Save the generated API token (shown only once!).
|
||||
5. Security is enabled immediately (no restart needed).
|
||||
|
||||
This automatically:
|
||||
- Generates a secure random password
|
||||
- Hashes it with bcrypt (cost factor 12)
|
||||
- Creates secure API token (SHA3-256 hashed, raw token shown once)
|
||||
- For systemd: Configures systemd with hashed credentials
|
||||
- For Docker: Saves to `/data/.env` with hashed credentials (properly quoted to prevent shell expansion)
|
||||
- Restarts service/container with authentication enabled
|
||||
|
||||
#### Manual Setup (Advanced)
|
||||
```bash
|
||||
# Using systemd (password will be hashed automatically)
|
||||
sudo systemctl edit pulse
|
||||
# Add:
|
||||
[Service]
|
||||
Environment="PULSE_AUTH_USER=admin"
|
||||
Environment="PULSE_AUTH_PASS=$2a$12$..." # Use bcrypt hash, not plain text!
|
||||
|
||||
# Docker (credentials persist in volume via .env file)
|
||||
# IMPORTANT: Always quote bcrypt hashes to prevent shell expansion!
|
||||
docker run -e PULSE_AUTH_USER=admin -e PULSE_AUTH_PASS='$2a$12$...' rcourtman/pulse:latest
|
||||
# Or use Quick Security Setup and restart container
|
||||
```
|
||||
|
||||
**Important**: Always use hashed passwords in configuration. Use the Quick Security Setup or generate bcrypt hashes manually.
|
||||
|
||||
#### Features
|
||||
- Web UI login required when authentication enabled
|
||||
- Change/remove password from Settings → Security
|
||||
- Passwords ALWAYS hashed with bcrypt (cost 12)
|
||||
- Session-based authentication with secure HttpOnly cookies
|
||||
- 24-hour session expiry
|
||||
- CSRF protection for all state-changing operations
|
||||
- Session invalidation on password change
|
||||
|
||||
### API Token Authentication
|
||||
For programmatic access and automation. API tokens are SHA3-256 hashed for security.
|
||||
|
||||
#### Token Setup via Quick Security
|
||||
The Quick Security Setup automatically:
|
||||
- Generates a cryptographically secure token
|
||||
- Hashes it with SHA3-256
|
||||
- Stores only the 64-character hash
|
||||
- Adds the token to the managed token list
|
||||
|
||||
#### Manual Token Setup
|
||||
```bash
|
||||
# Using systemd (plain text values are auto-hashed on startup)
|
||||
sudo systemctl edit pulse
|
||||
# Add:
|
||||
[Service]
|
||||
Environment="API_TOKENS=ansible-token,docker-agent-token"
|
||||
|
||||
# Docker
|
||||
docker run -e API_TOKENS=ansible-token,docker-agent-token rcourtman/pulse:latest
|
||||
|
||||
# To provide pre-hashed tokens instead, list the SHA3-256 hashes
|
||||
# Environment="API_TOKENS=83c8...,b1de..."
|
||||
```
|
||||
|
||||
**Security Note**: Tokens defined via environment variables are hashed with SHA3-256 before being stored on disk. Plain values never persist beyond startup.
|
||||
|
||||
#### Token Management (Settings → Security → API tokens)
|
||||
- Issue dedicated tokens for automation/agents without sharing a global credential
|
||||
- View prefixes/suffixes and last-used timestamps for auditing
|
||||
- Revoke tokens individually without downtime
|
||||
- Regenerate tokens when rotating credentials (new value displayed once)
|
||||
- All tokens stored as SHA3-256 hashes
|
||||
|
||||
#### Usage
|
||||
```bash
|
||||
# Include the ORIGINAL token (not hash) in X-API-Token header
|
||||
curl -H "X-API-Token: your-original-token" http://localhost:7655/api/health
|
||||
|
||||
# Or in query parameter for export/import
|
||||
curl "http://localhost:7655/api/export?token=your-original-token"
|
||||
```
|
||||
|
||||
### Auto-Registration Security
|
||||
|
||||
#### Default Mode
|
||||
- All access requires authentication
|
||||
- Nodes can auto-register with the API token
|
||||
- Setup scripts work without additional configuration
|
||||
|
||||
#### Secure Mode
|
||||
- Require API token for all operations
|
||||
- Protects auto-registration endpoint
|
||||
- Enable by setting at least one API token via `API_TOKENS` (or legacy `API_TOKEN`) environment variable
|
||||
|
||||
## CORS (Cross-Origin Resource Sharing)
|
||||
|
||||
By default, Pulse only allows same-origin requests (no CORS headers). This is the most secure configuration.
|
||||
|
||||
### Configuring CORS for External Access
|
||||
|
||||
If you need to access Pulse API from a different domain:
|
||||
|
||||
```bash
|
||||
# Docker
|
||||
docker run -e ALLOWED_ORIGINS="https://app.example.com" rcourtman/pulse:latest
|
||||
|
||||
# systemd
|
||||
sudo systemctl edit pulse
|
||||
[Service]
|
||||
Environment="ALLOWED_ORIGINS=https://app.example.com"
|
||||
|
||||
# Multiple origins (comma-separated)
|
||||
ALLOWED_ORIGINS="https://app.example.com,https://dashboard.example.com"
|
||||
|
||||
# Development mode (allows localhost)
|
||||
PULSE_DEV=true
|
||||
```
|
||||
|
||||
**Security Note**: Never use `ALLOWED_ORIGINS=*` in production as it allows any website to access your API.
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### Credential Storage
|
||||
- ✅ **DO**: Use Quick Security Setup for automatic hashing
|
||||
- ✅ **DO**: Store only bcrypt hashes for passwords
|
||||
- ✅ **DO**: Store only SHA3-256 hashes for API tokens
|
||||
- ❌ **DON'T**: Store plain text passwords in config files
|
||||
- ❌ **DON'T**: Store plain text API tokens in config files
|
||||
- ❌ **DON'T**: Log credentials or include them in backups
|
||||
|
||||
### Authentication Setup
|
||||
- ✅ **DO**: Use strong, unique passwords (16+ characters)
|
||||
- ✅ **DO**: Rotate API tokens periodically
|
||||
- ✅ **DO**: Use HTTPS in production environments
|
||||
- ❌ **DON'T**: Share API tokens between users/services
|
||||
- ❌ **DON'T**: Embed credentials in client-side code
|
||||
|
||||
### Verification
|
||||
Run the security verification script to ensure no plain text credentials:
|
||||
```bash
|
||||
/opt/pulse/testing-tools/security-verification.sh
|
||||
```
|
||||
|
||||
This checks:
|
||||
- No hardcoded credentials in code
|
||||
- No credentials exposed in logs
|
||||
- All passwords/tokens properly hashed
|
||||
- Secure file permissions
|
||||
- No credential leaks in API responses
|
||||
|
||||
## Account Lockout and Recovery
|
||||
|
||||
### Lockout Behavior
|
||||
- After **5 failed login attempts**, the account is locked for **15 minutes**
|
||||
- Lockout applies to both username and IP address
|
||||
- Login form shows remaining attempts after each failure
|
||||
- Clear message when locked with time remaining
|
||||
|
||||
### Automatic Recovery
|
||||
- Lockouts automatically expire after 15 minutes
|
||||
- No action needed - just wait for the timer to expire
|
||||
- Successful login clears all failed attempt counters
|
||||
|
||||
### Manual Recovery (Admin)
|
||||
Administrators with API access can manually reset lockouts:
|
||||
|
||||
```bash
|
||||
# Reset lockout for a specific username
|
||||
curl -X POST http://localhost:7655/api/security/reset-lockout \
|
||||
-H "X-API-Token: your-api-token" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"identifier":"username"}'
|
||||
|
||||
# Reset lockout for an IP address
|
||||
curl -X POST http://localhost:7655/api/security/reset-lockout \
|
||||
-H "X-API-Token: your-api-token" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"identifier":"192.168.1.100"}'
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Account locked?** Wait 15 minutes or contact admin for manual reset
|
||||
**Export blocked?** You're on a public network – login with password, set an API token (`API_TOKENS`), or set `ALLOW_UNPROTECTED_EXPORT=true`
|
||||
**Rate limited?** Wait 1 minute and try again
|
||||
**Can't login?** Check `PULSE_AUTH_USER` and `PULSE_AUTH_PASS` environment variables
|
||||
**API access denied?** Verify the token you supplied matches one of the values created in *Settings → Security → API tokens* (use the original token, not the hash)
|
||||
**CORS errors?** Configure `ALLOWED_ORIGINS` for your domain
|
||||
**Forgot password?** Start fresh – delete your Pulse data and restart
|
||||
|
||||
_Last updated: 2025-10-19_
|
||||
|
|
|
|||
366
docs/SECURITY.md
366
docs/SECURITY.md
|
|
@ -1,364 +1,6 @@
|
|||
# Pulse Security
|
||||
# Security Documentation
|
||||
|
||||
## Mandatory Authentication
|
||||
The canonical Pulse security guide now lives at the repository root:
|
||||
[`../SECURITY.md`](../SECURITY.md).
|
||||
|
||||
**Starting with v4.5.0, authentication setup is prompted for all new Pulse installations.** This protects your Proxmox API credentials from unauthorized access.
|
||||
|
||||
> **Service name note:** Systemd deployments use `pulse.service`. If you're upgrading from an older install that still registers `pulse-backend.service`, substitute that name in the commands below.
|
||||
|
||||
### First-Run Security Setup
|
||||
When you first access Pulse, you'll be guided through a mandatory security setup:
|
||||
- Create your admin username and password
|
||||
- Automatic API token generation for automation
|
||||
- Settings are applied immediately without restart
|
||||
- **Your existing nodes and settings are preserved**
|
||||
|
||||
## Smart Security Context
|
||||
|
||||
### Public Access Detection
|
||||
Pulse automatically detects when it's being accessed from public networks:
|
||||
- **Private Networks**: Local/RFC1918 addresses (192.168.x.x, 10.x.x.x, etc.)
|
||||
- **Public Networks**: Any non-private IP address
|
||||
- **Stronger Warnings**: Red alerts when accessed from public IPs without authentication
|
||||
|
||||
### Trusted Networks Configuration (Deprecated)
|
||||
**Note: Authentication is now mandatory regardless of network location.**
|
||||
|
||||
Legacy configuration (no longer applicable):
|
||||
```bash
|
||||
# Environment variable (comma-separated CIDR blocks)
|
||||
PULSE_TRUSTED_NETWORKS=192.168.1.0/24,10.0.0.0/24
|
||||
|
||||
# Or in systemd
|
||||
sudo systemctl edit pulse
|
||||
[Service]
|
||||
Environment="PULSE_TRUSTED_NETWORKS=192.168.1.0/24,10.0.0.0/24"
|
||||
```
|
||||
|
||||
When configured:
|
||||
- Access from trusted networks: No auth required
|
||||
- Access from outside: Authentication enforced
|
||||
- Useful for: Mixed home/remote access scenarios
|
||||
|
||||
## Security Warning System
|
||||
|
||||
Pulse now includes a non-intrusive security warning system that helps you understand your security posture:
|
||||
|
||||
### Security Score
|
||||
Your instance receives a score from 0-5 based on:
|
||||
- ✅ Credentials encrypted at rest (always enabled)
|
||||
- ✅ Export/import protection
|
||||
- ⚠️ Authentication enabled
|
||||
- ⚠️ HTTPS connection
|
||||
- ⚠️ Audit logging
|
||||
|
||||
### Dismissing Warnings
|
||||
If you're comfortable with your security setup, you can dismiss warnings:
|
||||
- **For 1 day** - Reminder tomorrow
|
||||
- **For 1 week** - Reminder next week
|
||||
- **Forever** - Won't show again
|
||||
|
||||
## Credential Security
|
||||
|
||||
### Encrypted at Rest (AES-256-GCM)
|
||||
- **Node Credentials**: Passwords and API tokens (`/etc/pulse/nodes.enc`)
|
||||
- **Email Settings**: SMTP passwords (`/etc/pulse/email.enc`)
|
||||
- **Webhook Data**: URLs and auth headers (`/etc/pulse/webhooks.enc`) - v4.1.9+
|
||||
- **Encryption Key**: Auto-generated (`/etc/pulse/.encryption.key`)
|
||||
|
||||
### Security Features
|
||||
- **Logs**: Token values masked with `***` in all outputs
|
||||
- **API**: Frontend receives only `hasToken: true`, never actual values
|
||||
- **Export**: Requires a valid API token (via `X-API-Token` header or `token` query parameter) to extract credentials
|
||||
- **Migration**: Use passphrase-protected export/import (see [Migration Guide](MIGRATION.md))
|
||||
- **Auto-Migration**: Unencrypted configs automatically migrate to encrypted format
|
||||
|
||||
## Export/Import Protection
|
||||
|
||||
By default, configuration export/import is blocked for security. You have two options:
|
||||
|
||||
### Option 1: Set API Tokens (Recommended)
|
||||
```bash
|
||||
# Using systemd (secure)
|
||||
sudo systemctl edit pulse
|
||||
# Add:
|
||||
[Service]
|
||||
Environment="API_TOKENS=ansible-token,docker-agent-token"
|
||||
Environment="API_TOKEN=legacy-token" # Optional fallback
|
||||
|
||||
# Then restart:
|
||||
sudo systemctl restart pulse
|
||||
|
||||
# Docker
|
||||
docker run -e API_TOKENS=ansible-token,docker-agent-token rcourtman/pulse:latest
|
||||
```
|
||||
|
||||
### Option 2: Allow Unprotected Export (Homelab)
|
||||
```bash
|
||||
# Using systemd
|
||||
sudo systemctl edit pulse
|
||||
# Add:
|
||||
[Service]
|
||||
Environment="ALLOW_UNPROTECTED_EXPORT=true"
|
||||
|
||||
# Docker
|
||||
docker run -e ALLOW_UNPROTECTED_EXPORT=true rcourtman/pulse:latest
|
||||
```
|
||||
|
||||
**Note:** For production deployments, consider using Docker secrets or systemd environment variables instead of .env files for sensitive data.
|
||||
|
||||
## Security Features
|
||||
|
||||
### Core Protection
|
||||
- **Encryption**: All credentials encrypted at rest (AES-256-GCM)
|
||||
- **Export Protection**: Exports always encrypted with passphrase
|
||||
- **Minimum Passphrase**: 12 characters required for exports
|
||||
- **Security Tab**: Check status in Settings → Security
|
||||
|
||||
### Enterprise Security (When Authentication Enabled)
|
||||
- **Password Security**:
|
||||
- Bcrypt hashing with cost factor 12 (60-character hash)
|
||||
- Passwords NEVER stored in plain text
|
||||
- Automatic hashing on security setup
|
||||
- **CRITICAL**: Bcrypt hashes MUST be exactly 60 characters
|
||||
- **API Token Security**:
|
||||
- 64-character hex tokens (32 bytes of entropy)
|
||||
- SHA3-256 hashed before storage (64-char hash)
|
||||
- Raw token shown only once during generation
|
||||
- Tokens NEVER stored in plain text
|
||||
- Live reloading when .env file changes
|
||||
- API-only mode supported (no password auth required)
|
||||
- **CSRF Protection**: All state-changing operations require CSRF tokens
|
||||
- **Rate Limiting**:
|
||||
- Authentication endpoints: 10 attempts/minute per IP
|
||||
- General API: 500 requests/minute per IP
|
||||
- Real-time endpoints exempt for functionality
|
||||
- **Account Lockout Protection**:
|
||||
- Locks after 5 failed login attempts
|
||||
- 15-minute automatic lockout duration
|
||||
- Clear feedback showing remaining attempts
|
||||
- Time remaining displayed when locked
|
||||
- Manual reset available via API for administrators
|
||||
- **Session Management**:
|
||||
- Secure HttpOnly cookies
|
||||
- 24-hour session expiry
|
||||
- Session invalidation on password change
|
||||
- **Security Headers**:
|
||||
- Content-Security-Policy with strict directives
|
||||
- X-Frame-Options: DENY (prevents clickjacking)
|
||||
- X-Content-Type-Options: nosniff
|
||||
- X-XSS-Protection: 1; mode=block
|
||||
- Referrer-Policy: strict-origin-when-cross-origin
|
||||
- Permissions-Policy restricting sensitive APIs
|
||||
- **Audit Logging**: All authentication events logged with IP addresses
|
||||
|
||||
### What's Encrypted in Exports
|
||||
- Node credentials (passwords, API tokens)
|
||||
- PBS credentials
|
||||
- Email settings passwords
|
||||
- Webhook URLs and authentication headers (v4.1.9+)
|
||||
|
||||
### What's NOT Encrypted
|
||||
- Node hostnames and IPs
|
||||
- Threshold settings
|
||||
- General configuration
|
||||
- Alert rules and schedules
|
||||
|
||||
## Authentication
|
||||
|
||||
Pulse supports multiple authentication methods that can be used independently or together:
|
||||
|
||||
### Password Authentication
|
||||
|
||||
#### Quick Security Setup (Recommended)
|
||||
The easiest way to enable authentication is through the web UI:
|
||||
1. Go to Settings → Security
|
||||
2. Click "Enable Security Now"
|
||||
3. Enter username and password
|
||||
4. Save the generated API token (shown only once!)
|
||||
5. Security is enabled immediately (no restart needed)
|
||||
|
||||
This automatically:
|
||||
- Generates a secure random password
|
||||
- Hashes it with bcrypt (cost factor 12)
|
||||
- Creates secure API token (SHA3-256 hashed, raw token shown once)
|
||||
- For systemd: Configures systemd with hashed credentials
|
||||
- For Docker: Saves to `/data/.env` with hashed credentials (properly quoted to prevent shell expansion)
|
||||
- Restarts service/container with authentication enabled
|
||||
|
||||
#### Manual Setup (Advanced)
|
||||
```bash
|
||||
# Using systemd (password will be hashed automatically)
|
||||
sudo systemctl edit pulse
|
||||
# Add:
|
||||
[Service]
|
||||
Environment="PULSE_AUTH_USER=admin"
|
||||
Environment="PULSE_AUTH_PASS=$2a$12$..." # Use bcrypt hash, not plain text!
|
||||
|
||||
# Docker (credentials persist in volume via .env file)
|
||||
# IMPORTANT: Always quote bcrypt hashes to prevent shell expansion!
|
||||
docker run -e PULSE_AUTH_USER=admin -e PULSE_AUTH_PASS='$2a$12$...' rcourtman/pulse:latest
|
||||
# Or use Quick Security Setup and restart container
|
||||
```
|
||||
|
||||
**Important**: Always use hashed passwords in configuration. Use the Quick Security Setup or generate bcrypt hashes manually.
|
||||
|
||||
#### Features
|
||||
- Web UI login required when authentication enabled
|
||||
- Change/remove password from Settings → Security
|
||||
- Passwords ALWAYS hashed with bcrypt (cost 12)
|
||||
- Session-based authentication with secure HttpOnly cookies
|
||||
- 24-hour session expiry
|
||||
- CSRF protection for all state-changing operations
|
||||
- Session invalidation on password change
|
||||
|
||||
### API Token Authentication
|
||||
For programmatic access and automation. API tokens are SHA3-256 hashed for security.
|
||||
|
||||
#### Token Setup via Quick Security
|
||||
The Quick Security Setup automatically:
|
||||
- Generates a cryptographically secure token
|
||||
- Hashes it with SHA3-256
|
||||
- Stores only the 64-character hash
|
||||
- Adds the token to the managed token list
|
||||
|
||||
#### Manual Token Setup
|
||||
```bash
|
||||
# Using systemd (plain text values are auto-hashed on startup)
|
||||
sudo systemctl edit pulse
|
||||
# Add:
|
||||
[Service]
|
||||
Environment="API_TOKENS=ansible-token,docker-agent-token"
|
||||
|
||||
# Docker
|
||||
docker run -e API_TOKENS=ansible-token,docker-agent-token rcourtman/pulse:latest
|
||||
|
||||
# To provide pre-hashed tokens instead, list the SHA3-256 hashes
|
||||
# Environment="API_TOKENS=83c8...,b1de..."
|
||||
```
|
||||
|
||||
**Security Note**: Tokens defined via environment variables are hashed with SHA3-256 before being stored on disk. Plain values never persist beyond startup.
|
||||
|
||||
#### Token Management (Settings → Security → API tokens)
|
||||
- Issue dedicated tokens for automation/agents without sharing a global credential
|
||||
- View prefixes/suffixes and last-used timestamps for auditing
|
||||
- Revoke tokens individually without downtime
|
||||
- Regenerate tokens when rotating credentials (new value displayed once)
|
||||
- All tokens stored as SHA3-256 hashes
|
||||
|
||||
#### Usage
|
||||
```bash
|
||||
# Include the ORIGINAL token (not hash) in X-API-Token header
|
||||
curl -H "X-API-Token: your-original-token" http://localhost:7655/api/health
|
||||
|
||||
# Or in query parameter for export/import
|
||||
curl "http://localhost:7655/api/export?token=your-original-token"
|
||||
```
|
||||
|
||||
### Auto-Registration Security
|
||||
|
||||
#### Default Mode
|
||||
- All access requires authentication
|
||||
- Nodes can auto-register with the API token
|
||||
- Setup scripts work without additional configuration
|
||||
|
||||
#### Secure Mode
|
||||
- Require API token for all operations
|
||||
- Protects auto-registration endpoint
|
||||
- Enable by setting at least one API token via `API_TOKENS` (or legacy `API_TOKEN`) environment variable
|
||||
|
||||
## CORS (Cross-Origin Resource Sharing)
|
||||
|
||||
By default, Pulse only allows same-origin requests (no CORS headers). This is the most secure configuration.
|
||||
|
||||
### Configuring CORS for External Access
|
||||
|
||||
If you need to access Pulse API from a different domain:
|
||||
|
||||
```bash
|
||||
# Docker
|
||||
docker run -e ALLOWED_ORIGINS="https://app.example.com" rcourtman/pulse:latest
|
||||
|
||||
# systemd
|
||||
sudo systemctl edit pulse
|
||||
[Service]
|
||||
Environment="ALLOWED_ORIGINS=https://app.example.com"
|
||||
|
||||
# Multiple origins (comma-separated)
|
||||
ALLOWED_ORIGINS="https://app.example.com,https://dashboard.example.com"
|
||||
|
||||
# Development mode (allows localhost)
|
||||
PULSE_DEV=true
|
||||
```
|
||||
|
||||
**Security Note**: Never use `ALLOWED_ORIGINS=*` in production as it allows any website to access your API.
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### Credential Storage
|
||||
- ✅ **DO**: Use Quick Security Setup for automatic hashing
|
||||
- ✅ **DO**: Store only bcrypt hashes for passwords
|
||||
- ✅ **DO**: Store only SHA3-256 hashes for API tokens
|
||||
- ❌ **DON'T**: Store plain text passwords in config files
|
||||
- ❌ **DON'T**: Store plain text API tokens in config files
|
||||
- ❌ **DON'T**: Log credentials or include them in backups
|
||||
|
||||
### Authentication Setup
|
||||
- ✅ **DO**: Use strong, unique passwords (16+ characters)
|
||||
- ✅ **DO**: Rotate API tokens periodically
|
||||
- ✅ **DO**: Use HTTPS in production environments
|
||||
- ❌ **DON'T**: Share API tokens between users/services
|
||||
- ❌ **DON'T**: Embed credentials in client-side code
|
||||
|
||||
### Verification
|
||||
Run the security verification script to ensure no plain text credentials:
|
||||
```bash
|
||||
/opt/pulse/testing-tools/security-verification.sh
|
||||
```
|
||||
|
||||
This checks:
|
||||
- No hardcoded credentials in code
|
||||
- No credentials exposed in logs
|
||||
- All passwords/tokens properly hashed
|
||||
- Secure file permissions
|
||||
- No credential leaks in API responses
|
||||
|
||||
## Account Lockout and Recovery
|
||||
|
||||
### Lockout Behavior
|
||||
- After **5 failed login attempts**, the account is locked for **15 minutes**
|
||||
- Lockout applies to both username and IP address
|
||||
- Login form shows remaining attempts after each failure
|
||||
- Clear message when locked with time remaining
|
||||
|
||||
### Automatic Recovery
|
||||
- Lockouts automatically expire after 15 minutes
|
||||
- No action needed - just wait for the timer to expire
|
||||
- Successful login clears all failed attempt counters
|
||||
|
||||
### Manual Recovery (Admin)
|
||||
Administrators with API access can manually reset lockouts:
|
||||
|
||||
```bash
|
||||
# Reset lockout for a specific username
|
||||
curl -X POST http://localhost:7655/api/security/reset-lockout \
|
||||
-H "X-API-Token: your-api-token" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"identifier":"username"}'
|
||||
|
||||
# Reset lockout for an IP address
|
||||
curl -X POST http://localhost:7655/api/security/reset-lockout \
|
||||
-H "X-API-Token: your-api-token" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"identifier":"192.168.1.100"}'
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Account locked?** Wait 15 minutes or contact admin for manual reset
|
||||
**Export blocked?** You're on a public network - login with password, set an API token (`API_TOKENS`), or set ALLOW_UNPROTECTED_EXPORT=true
|
||||
**Rate limited?** Wait 1 minute and try again
|
||||
**Can't login?** Check PULSE_AUTH_USER and PULSE_AUTH_PASS environment variables
|
||||
**API access denied?** Verify the token you supplied matches one of the values created in Settings → Security → API tokens (use the original token value, not the hash)
|
||||
**CORS errors?** Configure ALLOWED_ORIGINS for your domain
|
||||
**Forgot password?** Start fresh - delete your Pulse data and restart
|
||||
Please update that file if you need to change security guidance.
|
||||
|
|
|
|||
|
|
@ -258,6 +258,7 @@ type Monitor struct {
|
|||
pollMetrics *PollMetrics
|
||||
scheduler *AdaptiveScheduler
|
||||
stalenessTracker *StalenessTracker
|
||||
taskQueue *TaskQueue
|
||||
tempCollector *TemperatureCollector // SSH-based temperature collector
|
||||
mu sync.RWMutex
|
||||
startTime time.Time
|
||||
|
|
@ -1317,6 +1318,7 @@ func New(cfg *config.Config) (*Monitor, error) {
|
|||
|
||||
stalenessTracker := NewStalenessTracker(getPollMetrics())
|
||||
stalenessTracker.SetBounds(cfg.AdaptivePollingBaseInterval, cfg.AdaptivePollingMaxInterval)
|
||||
taskQueue := NewTaskQueue()
|
||||
|
||||
var scheduler *AdaptiveScheduler
|
||||
if cfg.AdaptivePollingEnabled {
|
||||
|
|
@ -1336,6 +1338,7 @@ func New(cfg *config.Config) (*Monitor, error) {
|
|||
pollMetrics: getPollMetrics(),
|
||||
scheduler: scheduler,
|
||||
stalenessTracker: stalenessTracker,
|
||||
taskQueue: taskQueue,
|
||||
tempCollector: tempCollector,
|
||||
startTime: time.Now(),
|
||||
rateTracker: NewRateTracker(),
|
||||
|
|
@ -1691,8 +1694,12 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) {
|
|||
|
||||
// Create separate tickers for polling and broadcasting
|
||||
// Hardcoded to 10 seconds since Proxmox updates cluster/resources every 10 seconds
|
||||
const pollingInterval = 10 * time.Second
|
||||
pollTicker := time.NewTicker(pollingInterval)
|
||||
const pollingInterval = 10 * time.Second
|
||||
|
||||
workerCount := len(m.pveClients) + len(m.pbsClients) + len(m.pmgClients)
|
||||
m.startTaskWorkers(ctx, workerCount)
|
||||
|
||||
pollTicker := time.NewTicker(pollingInterval)
|
||||
defer pollTicker.Stop()
|
||||
|
||||
broadcastTicker := time.NewTicker(pollingInterval)
|
||||
|
|
@ -1934,24 +1941,10 @@ func (m *Monitor) poll(ctx context.Context, wsHub *websocket.Hub) {
|
|||
now := startTime
|
||||
|
||||
plannedTasks := m.buildScheduledTasks(now)
|
||||
dueTasks := plannedTasks
|
||||
if m.scheduler != nil {
|
||||
due := m.scheduler.DispatchDue(ctx, now, plannedTasks)
|
||||
if len(due) > 0 {
|
||||
dueTasks = due
|
||||
}
|
||||
}
|
||||
|
||||
if m.pollMetrics != nil {
|
||||
m.pollMetrics.ResetQueueDepth(len(dueTasks))
|
||||
}
|
||||
|
||||
if m.config.ConcurrentPolling {
|
||||
// Use concurrent polling
|
||||
m.pollConcurrent(ctx, dueTasks)
|
||||
} else {
|
||||
m.pollSequential(ctx, dueTasks)
|
||||
for _, task := range plannedTasks {
|
||||
m.taskQueue.Upsert(task)
|
||||
}
|
||||
m.updateQueueDepthMetric()
|
||||
|
||||
// Update performance metrics
|
||||
m.state.Performance.LastPollDuration = time.Since(startTime).Seconds()
|
||||
|
|
@ -2088,99 +2081,126 @@ func (m *Monitor) pruneStaleDockerAlerts() bool {
|
|||
return cleared
|
||||
}
|
||||
|
||||
// pollConcurrent polls instances concurrently based on scheduled tasks.
|
||||
func (m *Monitor) pollConcurrent(ctx context.Context, tasks []ScheduledTask) {
|
||||
if len(tasks) == 0 {
|
||||
func (m *Monitor) startTaskWorkers(ctx context.Context, workers int) {
|
||||
if m.taskQueue == nil {
|
||||
return
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
ctx, cancel := context.WithCancel(ctx)
|
||||
defer cancel()
|
||||
|
||||
for _, task := range tasks {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
default:
|
||||
}
|
||||
|
||||
switch task.InstanceType {
|
||||
case InstanceTypePVE:
|
||||
client, ok := m.pveClients[task.InstanceName]
|
||||
if !ok || client == nil {
|
||||
continue
|
||||
}
|
||||
wg.Add(1)
|
||||
go func(name string, c PVEClientInterface) {
|
||||
defer wg.Done()
|
||||
m.pollPVEInstance(ctx, name, c)
|
||||
}(task.InstanceName, client)
|
||||
case InstanceTypePBS:
|
||||
client, ok := m.pbsClients[task.InstanceName]
|
||||
if !ok || client == nil {
|
||||
continue
|
||||
}
|
||||
wg.Add(1)
|
||||
go func(name string, c *pbs.Client) {
|
||||
defer wg.Done()
|
||||
m.pollPBSInstance(ctx, name, c)
|
||||
}(task.InstanceName, client)
|
||||
case InstanceTypePMG:
|
||||
client, ok := m.pmgClients[task.InstanceName]
|
||||
if !ok || client == nil {
|
||||
continue
|
||||
}
|
||||
wg.Add(1)
|
||||
go func(name string, c *pmg.Client) {
|
||||
defer wg.Done()
|
||||
m.pollPMGInstance(ctx, name, c)
|
||||
}(task.InstanceName, client)
|
||||
default:
|
||||
log.Debug().Str("instance", task.InstanceName).Str("type", string(task.InstanceType)).Msg("Skipping unsupported task type")
|
||||
}
|
||||
if workers < 1 {
|
||||
workers = 1
|
||||
}
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
wg.Wait()
|
||||
close(done)
|
||||
}()
|
||||
|
||||
select {
|
||||
case <-done:
|
||||
case <-ctx.Done():
|
||||
cancel()
|
||||
wg.Wait()
|
||||
if workers > 10 {
|
||||
workers = 10
|
||||
}
|
||||
for i := 0; i < workers; i++ {
|
||||
go m.taskWorker(ctx, i)
|
||||
}
|
||||
}
|
||||
|
||||
// pollSequential polls instances sequentially based on scheduled tasks.
|
||||
func (m *Monitor) pollSequential(ctx context.Context, tasks []ScheduledTask) {
|
||||
for _, task := range tasks {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
func (m *Monitor) taskWorker(ctx context.Context, id int) {
|
||||
log.Debug().Int("worker", id).Msg("Task worker started")
|
||||
for {
|
||||
task, ok := m.taskQueue.WaitNext(ctx)
|
||||
if !ok {
|
||||
log.Debug().Int("worker", id).Msg("Task worker stopping")
|
||||
return
|
||||
default:
|
||||
}
|
||||
|
||||
switch task.InstanceType {
|
||||
case InstanceTypePVE:
|
||||
if client, ok := m.pveClients[task.InstanceName]; ok && client != nil {
|
||||
m.pollPVEInstance(ctx, task.InstanceName, client)
|
||||
m.executeScheduledTask(ctx, task)
|
||||
|
||||
m.rescheduleTask(task)
|
||||
m.updateQueueDepthMetric()
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Monitor) executeScheduledTask(ctx context.Context, task ScheduledTask) {
|
||||
switch task.InstanceType {
|
||||
case InstanceTypePVE:
|
||||
client, ok := m.pveClients[task.InstanceName]
|
||||
if !ok || client == nil {
|
||||
log.Warn().Str("instance", task.InstanceName).Msg("PVE client missing for scheduled task")
|
||||
return
|
||||
}
|
||||
m.pollPVEInstance(ctx, task.InstanceName, client)
|
||||
case InstanceTypePBS:
|
||||
client, ok := m.pbsClients[task.InstanceName]
|
||||
if !ok || client == nil {
|
||||
log.Warn().Str("instance", task.InstanceName).Msg("PBS client missing for scheduled task")
|
||||
return
|
||||
}
|
||||
m.pollPBSInstance(ctx, task.InstanceName, client)
|
||||
case InstanceTypePMG:
|
||||
client, ok := m.pmgClients[task.InstanceName]
|
||||
if !ok || client == nil {
|
||||
log.Warn().Str("instance", task.InstanceName).Msg("PMG client missing for scheduled task")
|
||||
return
|
||||
}
|
||||
m.pollPMGInstance(ctx, task.InstanceName, client)
|
||||
default:
|
||||
log.Debug().Str("instance", task.InstanceName).Str("type", string(task.InstanceType)).Msg("Skipping unsupported task type")
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Monitor) rescheduleTask(task ScheduledTask) {
|
||||
if m.taskQueue == nil {
|
||||
return
|
||||
}
|
||||
|
||||
if m.scheduler == nil {
|
||||
nextInterval := task.Interval
|
||||
if nextInterval <= 0 && m.config != nil {
|
||||
nextInterval = m.config.AdaptivePollingBaseInterval
|
||||
}
|
||||
if nextInterval <= 0 {
|
||||
nextInterval = DefaultSchedulerConfig().BaseInterval
|
||||
}
|
||||
next := task
|
||||
next.NextRun = time.Now().Add(nextInterval)
|
||||
next.Interval = nextInterval
|
||||
m.taskQueue.Upsert(next)
|
||||
return
|
||||
}
|
||||
|
||||
desc := InstanceDescriptor{
|
||||
Name: task.InstanceName,
|
||||
Type: task.InstanceType,
|
||||
LastInterval: task.Interval,
|
||||
LastScheduled: task.NextRun,
|
||||
}
|
||||
if m.stalenessTracker != nil {
|
||||
if snap, ok := m.stalenessTracker.snapshot(task.InstanceType, task.InstanceName); ok {
|
||||
desc.LastSuccess = snap.LastSuccess
|
||||
desc.LastFailure = snap.LastError
|
||||
if snap.ChangeHash != "" {
|
||||
desc.Metadata = map[string]any{"changeHash": snap.ChangeHash}
|
||||
}
|
||||
case InstanceTypePBS:
|
||||
if client, ok := m.pbsClients[task.InstanceName]; ok && client != nil {
|
||||
m.pollPBSInstance(ctx, task.InstanceName, client)
|
||||
}
|
||||
case InstanceTypePMG:
|
||||
if client, ok := m.pmgClients[task.InstanceName]; ok && client != nil {
|
||||
m.pollPMGInstance(ctx, task.InstanceName, client)
|
||||
}
|
||||
default:
|
||||
log.Debug().Str("instance", task.InstanceName).Str("type", string(task.InstanceType)).Msg("Skipping unsupported task type")
|
||||
}
|
||||
}
|
||||
|
||||
tasks := m.scheduler.BuildPlan(time.Now(), []InstanceDescriptor{desc}, m.taskQueue.Size())
|
||||
if len(tasks) == 0 {
|
||||
next := task
|
||||
nextInterval := task.Interval
|
||||
if nextInterval <= 0 && m.config != nil {
|
||||
nextInterval = m.config.AdaptivePollingBaseInterval
|
||||
}
|
||||
if nextInterval <= 0 {
|
||||
nextInterval = DefaultSchedulerConfig().BaseInterval
|
||||
}
|
||||
next.Interval = nextInterval
|
||||
next.NextRun = time.Now().Add(nextInterval)
|
||||
m.taskQueue.Upsert(next)
|
||||
return
|
||||
}
|
||||
for _, next := range tasks {
|
||||
m.taskQueue.Upsert(next)
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Monitor) updateQueueDepthMetric() {
|
||||
if m.pollMetrics == nil || m.taskQueue == nil {
|
||||
return
|
||||
}
|
||||
m.pollMetrics.SetQueueDepth(m.taskQueue.Size())
|
||||
}
|
||||
|
||||
// pollPVEInstance polls a single PVE instance
|
||||
|
|
|
|||
|
|
@ -117,20 +117,29 @@ func (m *Monitor) buildScheduledTasks(now time.Time) []ScheduledTask {
|
|||
return nil
|
||||
}
|
||||
|
||||
queueDepth := 0
|
||||
if m.taskQueue != nil {
|
||||
queueDepth = m.taskQueue.Size()
|
||||
}
|
||||
|
||||
if m.scheduler == nil {
|
||||
tasks := make([]ScheduledTask, 0, len(descriptors))
|
||||
interval := m.config.AdaptivePollingBaseInterval
|
||||
if interval <= 0 {
|
||||
interval = DefaultSchedulerConfig().BaseInterval
|
||||
}
|
||||
for _, desc := range descriptors {
|
||||
tasks = append(tasks, ScheduledTask{
|
||||
InstanceName: desc.Name,
|
||||
InstanceType: desc.Type,
|
||||
NextRun: now,
|
||||
Interval: DefaultSchedulerConfig().BaseInterval,
|
||||
Interval: interval,
|
||||
})
|
||||
}
|
||||
return tasks
|
||||
}
|
||||
|
||||
return m.scheduler.BuildPlan(now, descriptors)
|
||||
return m.scheduler.BuildPlan(now, descriptors, queueDepth)
|
||||
}
|
||||
|
||||
// convertPoolInfoToModel converts Proxmox ZFS pool info to our model
|
||||
|
|
|
|||
|
|
@ -130,7 +130,7 @@ func NewAdaptiveScheduler(cfg SchedulerConfig, staleness StalenessSource, interv
|
|||
}
|
||||
|
||||
// BuildPlan produces an ordered set of scheduled tasks for the supplied inventory.
|
||||
func (s *AdaptiveScheduler) BuildPlan(now time.Time, inventory []InstanceDescriptor) []ScheduledTask {
|
||||
func (s *AdaptiveScheduler) BuildPlan(now time.Time, inventory []InstanceDescriptor, queueDepth int) []ScheduledTask {
|
||||
if len(inventory) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
|
@ -159,20 +159,21 @@ func (s *AdaptiveScheduler) BuildPlan(now time.Time, inventory []InstanceDescrip
|
|||
lastInterval = s.cfg.BaseInterval
|
||||
}
|
||||
|
||||
req := IntervalRequest{
|
||||
Now: now,
|
||||
BaseInterval: s.cfg.BaseInterval,
|
||||
MinInterval: s.cfg.MinInterval,
|
||||
MaxInterval: s.cfg.MaxInterval,
|
||||
LastInterval: lastInterval,
|
||||
LastSuccess: inst.LastSuccess,
|
||||
LastScheduled: lastScheduled,
|
||||
StalenessScore: score,
|
||||
ErrorCount: inst.ErrorCount,
|
||||
QueueDepth: len(inventory),
|
||||
InstanceKey: schedulerKey(inst.Type, inst.Name),
|
||||
InstanceType: inst.Type,
|
||||
}
|
||||
currentDepth := queueDepth + len(tasks)
|
||||
req := IntervalRequest{
|
||||
Now: now,
|
||||
BaseInterval: s.cfg.BaseInterval,
|
||||
MinInterval: s.cfg.MinInterval,
|
||||
MaxInterval: s.cfg.MaxInterval,
|
||||
LastInterval: lastInterval,
|
||||
LastSuccess: inst.LastSuccess,
|
||||
LastScheduled: lastScheduled,
|
||||
StalenessScore: score,
|
||||
ErrorCount: inst.ErrorCount,
|
||||
QueueDepth: currentDepth,
|
||||
InstanceKey: schedulerKey(inst.Type, inst.Name),
|
||||
InstanceType: inst.Type,
|
||||
}
|
||||
|
||||
nextInterval := s.interval.SelectInterval(req)
|
||||
if nextInterval <= 0 {
|
||||
|
|
|
|||
154
internal/monitoring/task_queue.go
Normal file
154
internal/monitoring/task_queue.go
Normal file
|
|
@ -0,0 +1,154 @@
|
|||
package monitoring
|
||||
|
||||
import (
|
||||
"container/heap"
|
||||
"context"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
type scheduledTaskEntry struct {
|
||||
task ScheduledTask
|
||||
index int
|
||||
}
|
||||
|
||||
func (e *scheduledTaskEntry) key() string {
|
||||
return schedulerKey(e.task.InstanceType, e.task.InstanceName)
|
||||
}
|
||||
|
||||
type taskHeap []*scheduledTaskEntry
|
||||
|
||||
func (h taskHeap) Len() int { return len(h) }
|
||||
|
||||
func (h taskHeap) Less(i, j int) bool {
|
||||
if h[i].task.NextRun.Equal(h[j].task.NextRun) {
|
||||
if h[i].task.Priority == h[j].task.Priority {
|
||||
return h[i].task.InstanceName < h[j].task.InstanceName
|
||||
}
|
||||
return h[i].task.Priority > h[j].task.Priority
|
||||
}
|
||||
return h[i].task.NextRun.Before(h[j].task.NextRun)
|
||||
}
|
||||
|
||||
func (h taskHeap) Swap(i, j int) {
|
||||
h[i], h[j] = h[j], h[i]
|
||||
h[i].index = i
|
||||
h[j].index = j
|
||||
}
|
||||
|
||||
func (h *taskHeap) Push(x interface{}) {
|
||||
entry := x.(*scheduledTaskEntry)
|
||||
entry.index = len(*h)
|
||||
*h = append(*h, entry)
|
||||
}
|
||||
|
||||
func (h *taskHeap) Pop() interface{} {
|
||||
old := *h
|
||||
n := len(old)
|
||||
if n == 0 {
|
||||
return nil
|
||||
}
|
||||
entry := old[n-1]
|
||||
entry.index = -1
|
||||
*h = old[:n-1]
|
||||
return entry
|
||||
}
|
||||
|
||||
// TaskQueue is a thread-safe min-heap over scheduled tasks.
|
||||
type TaskQueue struct {
|
||||
mu sync.Mutex
|
||||
entries map[string]*scheduledTaskEntry
|
||||
heap taskHeap
|
||||
}
|
||||
|
||||
// NewTaskQueue constructs an empty queue.
|
||||
func NewTaskQueue() *TaskQueue {
|
||||
tq := &TaskQueue{
|
||||
entries: make(map[string]*scheduledTaskEntry),
|
||||
heap: make(taskHeap, 0),
|
||||
}
|
||||
heap.Init(&tq.heap)
|
||||
return tq
|
||||
}
|
||||
|
||||
// Upsert inserts or updates a scheduled task in the queue.
|
||||
func (q *TaskQueue) Upsert(task ScheduledTask) {
|
||||
key := schedulerKey(task.InstanceType, task.InstanceName)
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
|
||||
if entry, ok := q.entries[key]; ok {
|
||||
entry.task = task
|
||||
heap.Fix(&q.heap, entry.index)
|
||||
return
|
||||
}
|
||||
|
||||
entry := &scheduledTaskEntry{task: task}
|
||||
heap.Push(&q.heap, entry)
|
||||
q.entries[key] = entry
|
||||
}
|
||||
|
||||
// Remove deletes a task by key if present.
|
||||
func (q *TaskQueue) Remove(instanceType InstanceType, instance string) {
|
||||
key := schedulerKey(instanceType, instance)
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
|
||||
entry, ok := q.entries[key]
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
heap.Remove(&q.heap, entry.index)
|
||||
delete(q.entries, key)
|
||||
}
|
||||
|
||||
// WaitNext blocks until a task is due or context is cancelled.
|
||||
func (q *TaskQueue) WaitNext(ctx context.Context) (ScheduledTask, bool) {
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ScheduledTask{}, false
|
||||
default:
|
||||
}
|
||||
|
||||
q.mu.Lock()
|
||||
if len(q.heap) == 0 {
|
||||
q.mu.Unlock()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ScheduledTask{}, false
|
||||
case <-time.After(100 * time.Millisecond):
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
entry := q.heap[0]
|
||||
delay := time.Until(entry.task.NextRun)
|
||||
if delay <= 0 {
|
||||
heap.Pop(&q.heap)
|
||||
delete(q.entries, entry.key())
|
||||
task := entry.task
|
||||
q.mu.Unlock()
|
||||
return task, true
|
||||
}
|
||||
|
||||
q.mu.Unlock()
|
||||
if delay > 250*time.Millisecond {
|
||||
delay = 250 * time.Millisecond
|
||||
}
|
||||
timer := time.NewTimer(delay)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
timer.Stop()
|
||||
return ScheduledTask{}, false
|
||||
case <-timer.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Size returns the number of tasks currently queued.
|
||||
func (q *TaskQueue) Size() int {
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
return len(q.heap)
|
||||
}
|
||||
|
|
@ -1,18 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# This script updates test scripts to use unique test node names
|
||||
# to prevent collision with user nodes
|
||||
|
||||
echo "Making test scripts safe by using unique test node names..."
|
||||
|
||||
# Add timestamp to test node names to make them unique
|
||||
TIMESTAMP=$(date +%s)
|
||||
|
||||
# Update test-config-validation.sh to use unique names
|
||||
sed -i "s/\"name\":\"test\"/\"name\":\"test-val-$TIMESTAMP\"/g" /opt/pulse/scripts/test-config-validation.sh
|
||||
sed -i "s/\"name\":\"duplicate-test\"/\"name\":\"dup-test-$TIMESTAMP\"/g" /opt/pulse/scripts/test-config-validation.sh
|
||||
sed -i "s/\"name\":\"pbs-test\"/\"name\":\"pbs-test-$TIMESTAMP\"/g" /opt/pulse/scripts/test-config-validation.sh
|
||||
|
||||
echo "Test scripts updated with unique node names (suffix: $TIMESTAMP)"
|
||||
echo ""
|
||||
echo "You can now safely run tests without affecting production nodes!"
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# This script patches test scripts to protect mock nodes from deletion
|
||||
# It ensures that mock nodes (pve1-pve7, mock-*) are never deleted during tests
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m'
|
||||
|
||||
echo "Protecting mock nodes from test cleanup..."
|
||||
|
||||
# List of test scripts that delete nodes
|
||||
TEST_SCRIPTS=(
|
||||
"/opt/pulse/scripts/test-persistence.sh"
|
||||
"/opt/pulse/scripts/test-recovery.sh"
|
||||
"/opt/pulse/scripts/test-backup.sh"
|
||||
"/opt/pulse/scripts/test-load.sh"
|
||||
"/opt/pulse/scripts/test-config-validation.sh"
|
||||
)
|
||||
|
||||
for script in "${TEST_SCRIPTS[@]}"; do
|
||||
if [ -f "$script" ]; then
|
||||
echo -n "Patching $(basename $script)... "
|
||||
|
||||
# Create backup
|
||||
cp "$script" "${script}.backup" 2>/dev/null
|
||||
|
||||
# Add protection for mock nodes in cleanup sections
|
||||
# This looks for DELETE commands and adds a check to skip mock nodes
|
||||
|
||||
# Find lines with curl DELETE and add protection
|
||||
sed -i '/curl.*DELETE.*nodes/i\
|
||||
# Skip mock nodes and production nodes\
|
||||
if [[ "$NODE_NAME" == "pve"* ]] || [[ "$NODE_NAME" == "mock"* ]] || [[ "$NODE_NAME" == "delly" ]] || [[ "$NODE_NAME" == "minipc" ]] || [[ "$NODE_NAME" == "pimox" ]]; then\
|
||||
continue\
|
||||
fi' "$script" 2>/dev/null
|
||||
|
||||
echo -e "${GREEN}✓${NC}"
|
||||
fi
|
||||
done
|
||||
|
||||
echo -e "${GREEN}Mock nodes are now protected!${NC}"
|
||||
Loading…
Reference in a new issue