feat: implement priority queue-based task execution (Phase 2 Task 6)
Replaces immediate polling with queue-based scheduling: - TaskQueue with min-heap (container/heap) for NextRun-ordered execution - Worker goroutines that block on WaitNext() until tasks are due - Tasks only execute when NextRun <= now, respecting adaptive intervals - Automatic rescheduling after execution via scheduler.BuildPlan - Queue depth tracking for backpressure-aware interval adjustments - Upsert semantics for updating scheduled tasks without duplicates Task 6 of 10 complete (60%). Ready for error/backoff policies.
This commit is contained in:
rcourtman
parent
c554380cb5
commit
aa5c08ad4a
8 changed files with 726 additions and 645 deletions
526
SECURITY.md
526
SECURITY.md
|
|
@ -1,45 +1,50 @@
|
||||||
# Pulse Security Documentation
|
# Pulse Security
|
||||||
|
|
||||||
## Critical Security Notice for Production Deployments
|
This document is the canonical security policy for Pulse. It combines our
|
||||||
|
ongoing hardening guidance with the operational checklists that previously lived
|
||||||
|
in `docs/SECURITY.md`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Critical Security Notice for Container Deployments
|
||||||
|
|
||||||
### Container SSH Key Policy (BREAKING CHANGE)
|
### Container SSH Key Policy (BREAKING CHANGE)
|
||||||
|
|
||||||
**Effective immediately, SSH-based temperature monitoring is BLOCKED in containerized Pulse deployments.**
|
**Effective immediately, SSH-based temperature monitoring is blocked in
|
||||||
|
containerized Pulse deployments.**
|
||||||
|
|
||||||
#### Why This Change?
|
#### Why This Change?
|
||||||
|
|
||||||
Storing SSH private keys inside Docker containers creates an unacceptable security risk in production environments:
|
Storing SSH private keys inside Docker/LXC containers creates an unacceptable
|
||||||
|
risk in production environments:
|
||||||
|
|
||||||
- **Container compromise = Infrastructure compromise**: If an attacker gains access to your Pulse container, they immediately obtain SSH private keys with root access to your Proxmox infrastructure.
|
- **Container compromise = infrastructure compromise** – if an attacker gains
|
||||||
- **Keys persist in images**: SSH keys can be extracted from container layers and images if pushed to registries.
|
shell access to the Pulse container they obtain the SSH private keys used to
|
||||||
- **No key rotation**: Long-lived keys in containers are difficult to rotate.
|
reach your Proxmox hosts.
|
||||||
- **Violates principle of least privilege**: Containers should not hold credentials for the infrastructure they monitor.
|
- **Keys persist in images** – private keys survive in image layers and can leak
|
||||||
|
when images are pushed to registries or shared.
|
||||||
|
- **No key rotation** – long-lived keys inside containers are difficult to
|
||||||
|
rotate safely.
|
||||||
|
- **Violates least-privilege** – monitoring containers should not hold
|
||||||
|
credentials that grant host-level access to the infrastructure they observe.
|
||||||
|
|
||||||
#### Affected Deployments
|
#### Affected Deployments
|
||||||
|
|
||||||
✅ **Not Affected** (SSH temperature monitoring still allowed):
|
✅ **Not affected** – Pulse installed directly on a VM or bare-metal host (no
|
||||||
- Pulse installed directly on a VM or bare metal (non-containerized)
|
containers), or homelab environments where you explicitly accept the risk.
|
||||||
- Home lab deployments where you understand and accept the risk
|
|
||||||
|
|
||||||
❌ **BLOCKED** (SSH temperature monitoring disabled):
|
❌ **Blocked** – Pulse running in Docker containers, LXC containers, or any
|
||||||
- Pulse running in Docker containers
|
environment where `PULSE_DOCKER=true`/`/.dockerenv` is detected.
|
||||||
- Pulse running in LXC containers
|
|
||||||
- Any deployment where `PULSE_DOCKER=true` or `/.dockerenv` exists
|
|
||||||
|
|
||||||
#### Migration Path
|
#### Migration Path (Production)
|
||||||
|
|
||||||
**For Production Container Deployments:**
|
1. **Deploy `pulse-sensor-proxy` on each Proxmox host**
|
||||||
|
|
||||||
1. **Deploy pulse-sensor-proxy on each Proxmox host:**
|
|
||||||
```bash
|
```bash
|
||||||
# On each Proxmox host
|
|
||||||
curl -o /usr/local/bin/pulse-sensor-proxy \
|
curl -o /usr/local/bin/pulse-sensor-proxy \
|
||||||
https://github.com/rcourtman/pulse/releases/latest/download/pulse-sensor-proxy
|
https://github.com/rcourtman/pulse/releases/latest/download/pulse-sensor-proxy
|
||||||
|
|
||||||
chmod +x /usr/local/bin/pulse-sensor-proxy
|
chmod +x /usr/local/bin/pulse-sensor-proxy
|
||||||
```
|
```
|
||||||
|
2. **Create a systemd unit** (`/etc/systemd/system/pulse-sensor-proxy.service`)
|
||||||
2. **Create systemd service** (`/etc/systemd/system/pulse-sensor-proxy.service`):
|
|
||||||
```ini
|
```ini
|
||||||
[Unit]
|
[Unit]
|
||||||
Description=Pulse Temperature Sensor Proxy
|
Description=Pulse Temperature Sensor Proxy
|
||||||
|
|
@ -54,134 +59,445 @@ Storing SSH private keys inside Docker containers creates an unacceptable securi
|
||||||
[Install]
|
[Install]
|
||||||
WantedBy=multi-user.target
|
WantedBy=multi-user.target
|
||||||
```
|
```
|
||||||
|
3. **Enable and start the service**
|
||||||
3. **Enable and start:**
|
|
||||||
```bash
|
```bash
|
||||||
systemctl daemon-reload
|
systemctl daemon-reload
|
||||||
systemctl enable --now pulse-sensor-proxy
|
systemctl enable --now pulse-sensor-proxy
|
||||||
```
|
```
|
||||||
|
4. **Restart the Pulse container** so it binds to the proxy socket. The
|
||||||
|
container will automatically fall back to socket-based temperature polling.
|
||||||
|
|
||||||
4. **Restart Pulse container** - it will automatically detect and use the proxy
|
#### Removing Old SSH Keys
|
||||||
|
|
||||||
**Removing Existing SSH Keys:**
|
If you previously generated SSH keys inside containers:
|
||||||
|
|
||||||
If you previously used SSH-based temperature monitoring in containers:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# On each Proxmox host, remove Pulse SSH keys
|
# On each Proxmox host
|
||||||
sed -i '/# pulse-/d' /root/.ssh/authorized_keys
|
sed -i '/# pulse-/d' /root/.ssh/authorized_keys
|
||||||
|
|
||||||
# Inside the Pulse container (or destroy and recreate)
|
# Inside the Pulse container (or rebuild the container)
|
||||||
docker exec pulse rm -rf /home/pulse/.ssh/id_ed25519*
|
docker exec pulse rm -rf /home/pulse/.ssh/id_ed25519*
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Technical Details
|
#### Security Boundary
|
||||||
|
|
||||||
**How pulse-sensor-proxy Works:**
|
|
||||||
|
|
||||||
- Runs as a lightweight daemon on the Proxmox host
|
|
||||||
- Exposes a Unix socket at `/run/pulse-sensor-proxy.sock`
|
|
||||||
- Pulse container connects via bind-mounted socket
|
|
||||||
- Only exposes `sensors -j` output - no SSH access
|
|
||||||
- Keys never leave the Proxmox host
|
|
||||||
|
|
||||||
**Security Boundaries:**
|
|
||||||
|
|
||||||
```
|
```
|
||||||
┌─────────────────────────────────────┐
|
┌─────────────────────────────────────┐
|
||||||
│ Proxmox Host │
|
│ Proxmox Host │
|
||||||
│ ┌───────────────────────────────┐ │
|
│ ┌───────────────────────────────┐ │
|
||||||
│ │ pulse-sensor-proxy (root) │ │
|
│ │ pulse-sensor-proxy (root) │ │
|
||||||
│ │ - Runs sensors -j │ │
|
│ │ · Runs sensors -j │ │
|
||||||
│ │ - Unix socket only │ │
|
│ │ · Exposes Unix socket only │ │
|
||||||
│ └───────────────────────────────┘ │
|
│ └───────────────────────────────┘ │
|
||||||
│ │ │
|
│ │ │
|
||||||
│ │ /run/pulse-sensor-proxy.sock
|
│ │ /run/pulse-sensor-proxy.sock
|
||||||
│ │ │
|
│ │ │
|
||||||
│ ┌─────────▼─────────────────────┐ │
|
│ ┌─────────▼─────────────────────┐ │
|
||||||
│ │ Container (bind mount) │ │
|
│ │ Pulse container (bind mount) │ │
|
||||||
│ │ - No SSH keys │ │
|
│ │ · No SSH keys │ │
|
||||||
│ │ - No root access to host │ │
|
│ │ · No host root privileges │ │
|
||||||
│ └───────────────────────────────┘ │
|
│ └───────────────────────────────┘ │
|
||||||
└─────────────────────────────────────┘
|
└─────────────────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
#### For Home Lab Users
|
#### Homelab Exception
|
||||||
|
|
||||||
If you understand and accept the risk, you can still use non-containerized Pulse with SSH keys:
|
If you fully understand the risk and are **not** containerized (VM/bare-metal
|
||||||
|
install), the legacy SSH flow still works. Use a dedicated monitoring user,
|
||||||
|
restrict the key with `command="sensors -j"` and `from="<pulse-ip>"`, and
|
||||||
|
rotate keys regularly.
|
||||||
|
|
||||||
1. Install Pulse directly on a VM (not in Docker)
|
#### Auditing Your Deployment
|
||||||
2. Setup script will offer SSH temperature monitoring
|
|
||||||
3. Follow standard security practices:
|
|
||||||
- Use dedicated monitoring user (not root)
|
|
||||||
- Restrict key with `command="sensors -j"`
|
|
||||||
- Add `from="<pulse-ip>"` restrictions
|
|
||||||
- Rotate keys periodically
|
|
||||||
|
|
||||||
#### Audit Your Deployment
|
|
||||||
|
|
||||||
**Check if you're affected:**
|
|
||||||
```bash
|
```bash
|
||||||
# Inside Pulse container
|
# Detect vulnerable containers
|
||||||
ls /home/pulse/.ssh/id_ed25519* 2>/dev/null && echo "⚠️ VULNERABLE"
|
ls /home/pulse/.ssh/id_ed25519* 2>/dev/null && echo "⚠️ SSH keys present"
|
||||||
|
|
||||||
# On Proxmox host
|
# Check container logs for proxy detection
|
||||||
grep "# pulse-" /root/.ssh/authorized_keys && echo "⚠️ SSH keys present"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Check if proxy is working:**
|
|
||||||
```bash
|
|
||||||
# On Proxmox host
|
|
||||||
systemctl status pulse-sensor-proxy
|
|
||||||
|
|
||||||
# Inside Pulse container
|
|
||||||
docker logs pulse | grep -i "temperature proxy detected"
|
docker logs pulse | grep -i "temperature proxy detected"
|
||||||
|
|
||||||
|
# Verify the host service
|
||||||
|
systemctl status pulse-sensor-proxy
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Timeline
|
**Documentation:** https://docs.pulseapp.io/security/containerized-deployments
|
||||||
|
**Issues:** https://github.com/rcourtman/pulse/issues
|
||||||
- **Now**: SSH key generation blocked in containers (code-level enforcement)
|
**Private disclosures:** security@pulseapp.io
|
||||||
- **Next Release**: Setup script updated with clear warnings
|
|
||||||
- **Future**: pulse-sensor-proxy bundled in official releases
|
|
||||||
|
|
||||||
#### Questions?
|
|
||||||
|
|
||||||
- Documentation: https://docs.pulseapp.io/security/containerized-deployments
|
|
||||||
- GitHub Issues: https://github.com/rcourtman/pulse/issues
|
|
||||||
- Security Issues: security@pulseapp.io (private disclosure)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## General Security Best Practices
|
## Mandatory Authentication
|
||||||
|
|
||||||
### Authentication
|
**Starting with v4.5.0, authentication setup is prompted for all new Pulse
|
||||||
|
installations.** This protects your Proxmox API credentials from unauthorized
|
||||||
|
access.
|
||||||
|
|
||||||
- Use API tokens with minimal required permissions
|
> **Service name note:** systemd deployments use `pulse.service`. If you're
|
||||||
- Rotate tokens regularly
|
> upgrading from an older install that still registers `pulse-backend.service`,
|
||||||
- Never commit tokens to version control
|
> substitute that name in the commands below.
|
||||||
- Use read-only tokens where possible
|
|
||||||
|
|
||||||
### Network Security
|
### First-Run Security Setup
|
||||||
|
When you first access Pulse, you'll be guided through a mandatory security
|
||||||
|
setup:
|
||||||
|
- Create your admin username and password
|
||||||
|
- Automatic API token generation for automation
|
||||||
|
- Settings are applied immediately without restart
|
||||||
|
- **Your existing nodes and settings are preserved**
|
||||||
|
|
||||||
- Run Pulse in a dedicated monitoring VLAN
|
## Smart Security Context
|
||||||
- Restrict Pulse's network access to only monitored systems
|
|
||||||
- Use firewall rules to limit inbound connections
|
|
||||||
- Enable TLS for all Proxmox API connections
|
|
||||||
|
|
||||||
### Monitoring
|
### Public Access Detection
|
||||||
|
Pulse automatically detects when it's being accessed from public networks:
|
||||||
|
- **Private networks**: local/RFC1918 addresses (192.168.x.x, 10.x.x.x, etc.)
|
||||||
|
- **Public networks**: any non-private IP address
|
||||||
|
- **Stronger warnings**: red alerts when accessed from public IPs without
|
||||||
|
authentication
|
||||||
|
|
||||||
- Enable audit logging on Proxmox hosts
|
### Trusted Networks Configuration (Deprecated)
|
||||||
- Monitor Pulse container logs for suspicious activity
|
**Note:** authentication is now mandatory regardless of network location.
|
||||||
- Set up alerts for failed authentication attempts
|
|
||||||
- Review access logs regularly
|
|
||||||
|
|
||||||
### Updates
|
Legacy configuration (no longer applicable):
|
||||||
|
```bash
|
||||||
|
# Environment variable (comma-separated CIDR blocks)
|
||||||
|
PULSE_TRUSTED_NETWORKS=192.168.1.0/24,10.0.0.0/24
|
||||||
|
|
||||||
- Keep Pulse updated to latest stable version
|
# Or in systemd
|
||||||
- Subscribe to security announcements
|
sudo systemctl edit pulse
|
||||||
- Test updates in staging before production
|
[Service]
|
||||||
- Have rollback plan ready
|
Environment="PULSE_TRUSTED_NETWORKS=192.168.1.0/24,10.0.0.0/24"
|
||||||
|
```
|
||||||
|
|
||||||
---
|
When configured:
|
||||||
|
- Access from trusted networks: no auth required
|
||||||
|
- Access from outside: authentication enforced
|
||||||
|
- Useful for: mixed home/remote access scenarios
|
||||||
|
|
||||||
Last updated: 2025-10-19
|
## Security Warning System
|
||||||
|
|
||||||
|
Pulse includes a non-intrusive security warning system that helps you
|
||||||
|
understand your security posture.
|
||||||
|
|
||||||
|
### Security Score
|
||||||
|
Your instance receives a score from 0‑5 based on:
|
||||||
|
- ✅ Credentials encrypted at rest (always enabled)
|
||||||
|
- ✅ Export/import protection
|
||||||
|
- ⚠️ Authentication enabled
|
||||||
|
- ⚠️ HTTPS connection
|
||||||
|
- ⚠️ Audit logging
|
||||||
|
|
||||||
|
### Dismissing Warnings
|
||||||
|
If you're comfortable with your security setup, you can dismiss warnings:
|
||||||
|
- **For 1 day** – reminder tomorrow
|
||||||
|
- **For 1 week** – reminder next week
|
||||||
|
- **Forever** – won't show again
|
||||||
|
|
||||||
|
## Credential Security
|
||||||
|
|
||||||
|
### Encrypted at Rest (AES-256-GCM)
|
||||||
|
- **Node credentials**: passwords and API tokens (`/etc/pulse/nodes.enc`)
|
||||||
|
- **Email settings**: SMTP passwords (`/etc/pulse/email.enc`)
|
||||||
|
- **Webhook data**: URLs and auth headers (`/etc/pulse/webhooks.enc`) – v4.1.9+
|
||||||
|
- **Encryption key**: auto-generated (`/etc/pulse/.encryption.key`)
|
||||||
|
|
||||||
|
### Security Features
|
||||||
|
- **Logs**: token values masked with `***` in all outputs
|
||||||
|
- **API**: frontend receives only `hasToken: true`, never actual values
|
||||||
|
- **Export**: requires a valid API token (`X-API-Token` header or `token`
|
||||||
|
parameter) to extract credentials
|
||||||
|
- **Migration**: use passphrase-protected export/import (see
|
||||||
|
[Migration Guide](docs/MIGRATION.md))
|
||||||
|
- **Auto-migration**: unencrypted configs automatically migrate to encrypted
|
||||||
|
format
|
||||||
|
|
||||||
|
## Export/Import Protection
|
||||||
|
|
||||||
|
By default, configuration export/import is blocked. You have two options:
|
||||||
|
|
||||||
|
### Option 1: Set API Tokens (Recommended)
|
||||||
|
```bash
|
||||||
|
# Using systemd (secure)
|
||||||
|
sudo systemctl edit pulse
|
||||||
|
# Add:
|
||||||
|
[Service]
|
||||||
|
Environment="API_TOKENS=ansible-token,docker-agent-token"
|
||||||
|
Environment="API_TOKEN=legacy-token"
|
||||||
|
|
||||||
|
# Then restart:
|
||||||
|
sudo systemctl restart pulse
|
||||||
|
|
||||||
|
# Docker
|
||||||
|
docker run -e API_TOKENS=ansible-token,docker-agent-token rcourtman/pulse:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
### Option 2: Allow Unprotected Export (Homelab)
|
||||||
|
```bash
|
||||||
|
# Using systemd
|
||||||
|
sudo systemctl edit pulse
|
||||||
|
# Add:
|
||||||
|
[Service]
|
||||||
|
Environment="ALLOW_UNPROTECTED_EXPORT=true"
|
||||||
|
|
||||||
|
# Docker
|
||||||
|
docker run -e ALLOW_UNPROTECTED_EXPORT=true rcourtman/pulse:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note:** for production, prefer Docker secrets or systemd environment files
|
||||||
|
for sensitive data.
|
||||||
|
|
||||||
|
## Security Features
|
||||||
|
|
||||||
|
### Core Protection
|
||||||
|
- **Encryption**: credentials encrypted at rest (AES-256-GCM)
|
||||||
|
- **Export protection**: exports always encrypted with a passphrase
|
||||||
|
- **Minimum passphrase**: 12 characters required for exports
|
||||||
|
- **Security tab**: check status in *Settings → Security*
|
||||||
|
|
||||||
|
### Enterprise Security (When Authentication Enabled)
|
||||||
|
- **Password security**
|
||||||
|
- Bcrypt hashing with cost factor 12 (60‑character hash)
|
||||||
|
- Passwords never stored in plain text
|
||||||
|
- Automatic hashing during security setup
|
||||||
|
- **Critical**: bcrypt hashes must be exactly 60 characters
|
||||||
|
- **API token security**
|
||||||
|
- 64‑character hex tokens (32 bytes entropy)
|
||||||
|
- SHA3-256 hashed before storage (64‑character hash)
|
||||||
|
- Raw token shown only once
|
||||||
|
- Tokens never stored in plain text
|
||||||
|
- Live reloading when `.env` changes
|
||||||
|
- API-only mode supported (no password auth required)
|
||||||
|
- **CSRF protection**: all state-changing operations require CSRF tokens
|
||||||
|
- **Rate limiting**
|
||||||
|
- Auth endpoints: 10 attempts/minute per IP
|
||||||
|
- General API: 500 requests/minute per IP
|
||||||
|
- Real-time endpoints exempt for functionality
|
||||||
|
- **Account lockout**
|
||||||
|
- Locks after 5 failed login attempts
|
||||||
|
- 15-minute automatic lockout duration
|
||||||
|
- Clear feedback showing remaining attempts
|
||||||
|
- Time remaining displayed when locked
|
||||||
|
- Manual reset available via API for admins
|
||||||
|
- **Session management**
|
||||||
|
- Secure HttpOnly cookies
|
||||||
|
- 24-hour session expiry
|
||||||
|
- Session invalidation on password change
|
||||||
|
- **Security headers**
|
||||||
|
- Content-Security-Policy
|
||||||
|
- X-Frame-Options: DENY
|
||||||
|
- X-Content-Type-Options: nosniff
|
||||||
|
- X-XSS-Protection: 1; mode=block
|
||||||
|
- Referrer-Policy: strict-origin-when-cross-origin
|
||||||
|
- Permissions-Policy restricting sensitive APIs
|
||||||
|
- **Audit logging**: authentication events include IP addresses
|
||||||
|
|
||||||
|
### What's Encrypted in Exports
|
||||||
|
- Node credentials (passwords, API tokens)
|
||||||
|
- PBS credentials
|
||||||
|
- Email settings passwords
|
||||||
|
- Webhook URLs and authentication headers (v4.1.9+)
|
||||||
|
|
||||||
|
### What's **Not** Encrypted
|
||||||
|
- Node hostnames and IPs
|
||||||
|
- Threshold settings
|
||||||
|
- General configuration
|
||||||
|
- Alert rules and schedules
|
||||||
|
|
||||||
|
## Authentication Workflows
|
||||||
|
|
||||||
|
Pulse supports multiple authentication methods that can be used independently or
|
||||||
|
together.
|
||||||
|
|
||||||
|
### Password Authentication
|
||||||
|
|
||||||
|
#### Quick Security Setup (Recommended)
|
||||||
|
1. Navigate to *Settings → Security*.
|
||||||
|
2. Click **Enable Security Now**.
|
||||||
|
3. Enter username and password.
|
||||||
|
4. Save the generated API token (shown only once!).
|
||||||
|
5. Security is enabled immediately (no restart needed).
|
||||||
|
|
||||||
|
This automatically:
|
||||||
|
- Generates a secure random password
|
||||||
|
- Hashes it with bcrypt (cost factor 12)
|
||||||
|
- Creates secure API token (SHA3-256 hashed, raw token shown once)
|
||||||
|
- For systemd: Configures systemd with hashed credentials
|
||||||
|
- For Docker: Saves to `/data/.env` with hashed credentials (properly quoted to prevent shell expansion)
|
||||||
|
- Restarts service/container with authentication enabled
|
||||||
|
|
||||||
|
#### Manual Setup (Advanced)
|
||||||
|
```bash
|
||||||
|
# Using systemd (password will be hashed automatically)
|
||||||
|
sudo systemctl edit pulse
|
||||||
|
# Add:
|
||||||
|
[Service]
|
||||||
|
Environment="PULSE_AUTH_USER=admin"
|
||||||
|
Environment="PULSE_AUTH_PASS=$2a$12$..." # Use bcrypt hash, not plain text!
|
||||||
|
|
||||||
|
# Docker (credentials persist in volume via .env file)
|
||||||
|
# IMPORTANT: Always quote bcrypt hashes to prevent shell expansion!
|
||||||
|
docker run -e PULSE_AUTH_USER=admin -e PULSE_AUTH_PASS='$2a$12$...' rcourtman/pulse:latest
|
||||||
|
# Or use Quick Security Setup and restart container
|
||||||
|
```
|
||||||
|
|
||||||
|
**Important**: Always use hashed passwords in configuration. Use the Quick Security Setup or generate bcrypt hashes manually.
|
||||||
|
|
||||||
|
#### Features
|
||||||
|
- Web UI login required when authentication enabled
|
||||||
|
- Change/remove password from Settings → Security
|
||||||
|
- Passwords ALWAYS hashed with bcrypt (cost 12)
|
||||||
|
- Session-based authentication with secure HttpOnly cookies
|
||||||
|
- 24-hour session expiry
|
||||||
|
- CSRF protection for all state-changing operations
|
||||||
|
- Session invalidation on password change
|
||||||
|
|
||||||
|
### API Token Authentication
|
||||||
|
For programmatic access and automation. API tokens are SHA3-256 hashed for security.
|
||||||
|
|
||||||
|
#### Token Setup via Quick Security
|
||||||
|
The Quick Security Setup automatically:
|
||||||
|
- Generates a cryptographically secure token
|
||||||
|
- Hashes it with SHA3-256
|
||||||
|
- Stores only the 64-character hash
|
||||||
|
- Adds the token to the managed token list
|
||||||
|
|
||||||
|
#### Manual Token Setup
|
||||||
|
```bash
|
||||||
|
# Using systemd (plain text values are auto-hashed on startup)
|
||||||
|
sudo systemctl edit pulse
|
||||||
|
# Add:
|
||||||
|
[Service]
|
||||||
|
Environment="API_TOKENS=ansible-token,docker-agent-token"
|
||||||
|
|
||||||
|
# Docker
|
||||||
|
docker run -e API_TOKENS=ansible-token,docker-agent-token rcourtman/pulse:latest
|
||||||
|
|
||||||
|
# To provide pre-hashed tokens instead, list the SHA3-256 hashes
|
||||||
|
# Environment="API_TOKENS=83c8...,b1de..."
|
||||||
|
```
|
||||||
|
|
||||||
|
**Security Note**: Tokens defined via environment variables are hashed with SHA3-256 before being stored on disk. Plain values never persist beyond startup.
|
||||||
|
|
||||||
|
#### Token Management (Settings → Security → API tokens)
|
||||||
|
- Issue dedicated tokens for automation/agents without sharing a global credential
|
||||||
|
- View prefixes/suffixes and last-used timestamps for auditing
|
||||||
|
- Revoke tokens individually without downtime
|
||||||
|
- Regenerate tokens when rotating credentials (new value displayed once)
|
||||||
|
- All tokens stored as SHA3-256 hashes
|
||||||
|
|
||||||
|
#### Usage
|
||||||
|
```bash
|
||||||
|
# Include the ORIGINAL token (not hash) in X-API-Token header
|
||||||
|
curl -H "X-API-Token: your-original-token" http://localhost:7655/api/health
|
||||||
|
|
||||||
|
# Or in query parameter for export/import
|
||||||
|
curl "http://localhost:7655/api/export?token=your-original-token"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Auto-Registration Security
|
||||||
|
|
||||||
|
#### Default Mode
|
||||||
|
- All access requires authentication
|
||||||
|
- Nodes can auto-register with the API token
|
||||||
|
- Setup scripts work without additional configuration
|
||||||
|
|
||||||
|
#### Secure Mode
|
||||||
|
- Require API token for all operations
|
||||||
|
- Protects auto-registration endpoint
|
||||||
|
- Enable by setting at least one API token via `API_TOKENS` (or legacy `API_TOKEN`) environment variable
|
||||||
|
|
||||||
|
## CORS (Cross-Origin Resource Sharing)
|
||||||
|
|
||||||
|
By default, Pulse only allows same-origin requests (no CORS headers). This is the most secure configuration.
|
||||||
|
|
||||||
|
### Configuring CORS for External Access
|
||||||
|
|
||||||
|
If you need to access Pulse API from a different domain:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Docker
|
||||||
|
docker run -e ALLOWED_ORIGINS="https://app.example.com" rcourtman/pulse:latest
|
||||||
|
|
||||||
|
# systemd
|
||||||
|
sudo systemctl edit pulse
|
||||||
|
[Service]
|
||||||
|
Environment="ALLOWED_ORIGINS=https://app.example.com"
|
||||||
|
|
||||||
|
# Multiple origins (comma-separated)
|
||||||
|
ALLOWED_ORIGINS="https://app.example.com,https://dashboard.example.com"
|
||||||
|
|
||||||
|
# Development mode (allows localhost)
|
||||||
|
PULSE_DEV=true
|
||||||
|
```
|
||||||
|
|
||||||
|
**Security Note**: Never use `ALLOWED_ORIGINS=*` in production as it allows any website to access your API.
|
||||||
|
|
||||||
|
## Security Best Practices
|
||||||
|
|
||||||
|
### Credential Storage
|
||||||
|
- ✅ **DO**: Use Quick Security Setup for automatic hashing
|
||||||
|
- ✅ **DO**: Store only bcrypt hashes for passwords
|
||||||
|
- ✅ **DO**: Store only SHA3-256 hashes for API tokens
|
||||||
|
- ❌ **DON'T**: Store plain text passwords in config files
|
||||||
|
- ❌ **DON'T**: Store plain text API tokens in config files
|
||||||
|
- ❌ **DON'T**: Log credentials or include them in backups
|
||||||
|
|
||||||
|
### Authentication Setup
|
||||||
|
- ✅ **DO**: Use strong, unique passwords (16+ characters)
|
||||||
|
- ✅ **DO**: Rotate API tokens periodically
|
||||||
|
- ✅ **DO**: Use HTTPS in production environments
|
||||||
|
- ❌ **DON'T**: Share API tokens between users/services
|
||||||
|
- ❌ **DON'T**: Embed credentials in client-side code
|
||||||
|
|
||||||
|
### Verification
|
||||||
|
Run the security verification script to ensure no plain text credentials:
|
||||||
|
```bash
|
||||||
|
/opt/pulse/testing-tools/security-verification.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
This checks:
|
||||||
|
- No hardcoded credentials in code
|
||||||
|
- No credentials exposed in logs
|
||||||
|
- All passwords/tokens properly hashed
|
||||||
|
- Secure file permissions
|
||||||
|
- No credential leaks in API responses
|
||||||
|
|
||||||
|
## Account Lockout and Recovery
|
||||||
|
|
||||||
|
### Lockout Behavior
|
||||||
|
- After **5 failed login attempts**, the account is locked for **15 minutes**
|
||||||
|
- Lockout applies to both username and IP address
|
||||||
|
- Login form shows remaining attempts after each failure
|
||||||
|
- Clear message when locked with time remaining
|
||||||
|
|
||||||
|
### Automatic Recovery
|
||||||
|
- Lockouts automatically expire after 15 minutes
|
||||||
|
- No action needed - just wait for the timer to expire
|
||||||
|
- Successful login clears all failed attempt counters
|
||||||
|
|
||||||
|
### Manual Recovery (Admin)
|
||||||
|
Administrators with API access can manually reset lockouts:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Reset lockout for a specific username
|
||||||
|
curl -X POST http://localhost:7655/api/security/reset-lockout \
|
||||||
|
-H "X-API-Token: your-api-token" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"identifier":"username"}'
|
||||||
|
|
||||||
|
# Reset lockout for an IP address
|
||||||
|
curl -X POST http://localhost:7655/api/security/reset-lockout \
|
||||||
|
-H "X-API-Token: your-api-token" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"identifier":"192.168.1.100"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
**Account locked?** Wait 15 minutes or contact admin for manual reset
|
||||||
|
**Export blocked?** You're on a public network – login with password, set an API token (`API_TOKENS`), or set `ALLOW_UNPROTECTED_EXPORT=true`
|
||||||
|
**Rate limited?** Wait 1 minute and try again
|
||||||
|
**Can't login?** Check `PULSE_AUTH_USER` and `PULSE_AUTH_PASS` environment variables
|
||||||
|
**API access denied?** Verify the token you supplied matches one of the values created in *Settings → Security → API tokens* (use the original token, not the hash)
|
||||||
|
**CORS errors?** Configure `ALLOWED_ORIGINS` for your domain
|
||||||
|
**Forgot password?** Start fresh – delete your Pulse data and restart
|
||||||
|
|
||||||
|
_Last updated: 2025-10-19_
|
||||||
|
|
|
||||||
366
docs/SECURITY.md
366
docs/SECURITY.md
|
|
@ -1,364 +1,6 @@
|
||||||
# Pulse Security
|
# Security Documentation
|
||||||
|
|
||||||
## Mandatory Authentication
|
The canonical Pulse security guide now lives at the repository root:
|
||||||
|
[`../SECURITY.md`](../SECURITY.md).
|
||||||
|
|
||||||
**Starting with v4.5.0, authentication setup is prompted for all new Pulse installations.** This protects your Proxmox API credentials from unauthorized access.
|
Please update that file if you need to change security guidance.
|
||||||
|
|
||||||
> **Service name note:** Systemd deployments use `pulse.service`. If you're upgrading from an older install that still registers `pulse-backend.service`, substitute that name in the commands below.
|
|
||||||
|
|
||||||
### First-Run Security Setup
|
|
||||||
When you first access Pulse, you'll be guided through a mandatory security setup:
|
|
||||||
- Create your admin username and password
|
|
||||||
- Automatic API token generation for automation
|
|
||||||
- Settings are applied immediately without restart
|
|
||||||
- **Your existing nodes and settings are preserved**
|
|
||||||
|
|
||||||
## Smart Security Context
|
|
||||||
|
|
||||||
### Public Access Detection
|
|
||||||
Pulse automatically detects when it's being accessed from public networks:
|
|
||||||
- **Private Networks**: Local/RFC1918 addresses (192.168.x.x, 10.x.x.x, etc.)
|
|
||||||
- **Public Networks**: Any non-private IP address
|
|
||||||
- **Stronger Warnings**: Red alerts when accessed from public IPs without authentication
|
|
||||||
|
|
||||||
### Trusted Networks Configuration (Deprecated)
|
|
||||||
**Note: Authentication is now mandatory regardless of network location.**
|
|
||||||
|
|
||||||
Legacy configuration (no longer applicable):
|
|
||||||
```bash
|
|
||||||
# Environment variable (comma-separated CIDR blocks)
|
|
||||||
PULSE_TRUSTED_NETWORKS=192.168.1.0/24,10.0.0.0/24
|
|
||||||
|
|
||||||
# Or in systemd
|
|
||||||
sudo systemctl edit pulse
|
|
||||||
[Service]
|
|
||||||
Environment="PULSE_TRUSTED_NETWORKS=192.168.1.0/24,10.0.0.0/24"
|
|
||||||
```
|
|
||||||
|
|
||||||
When configured:
|
|
||||||
- Access from trusted networks: No auth required
|
|
||||||
- Access from outside: Authentication enforced
|
|
||||||
- Useful for: Mixed home/remote access scenarios
|
|
||||||
|
|
||||||
## Security Warning System
|
|
||||||
|
|
||||||
Pulse now includes a non-intrusive security warning system that helps you understand your security posture:
|
|
||||||
|
|
||||||
### Security Score
|
|
||||||
Your instance receives a score from 0-5 based on:
|
|
||||||
- ✅ Credentials encrypted at rest (always enabled)
|
|
||||||
- ✅ Export/import protection
|
|
||||||
- ⚠️ Authentication enabled
|
|
||||||
- ⚠️ HTTPS connection
|
|
||||||
- ⚠️ Audit logging
|
|
||||||
|
|
||||||
### Dismissing Warnings
|
|
||||||
If you're comfortable with your security setup, you can dismiss warnings:
|
|
||||||
- **For 1 day** - Reminder tomorrow
|
|
||||||
- **For 1 week** - Reminder next week
|
|
||||||
- **Forever** - Won't show again
|
|
||||||
|
|
||||||
## Credential Security
|
|
||||||
|
|
||||||
### Encrypted at Rest (AES-256-GCM)
|
|
||||||
- **Node Credentials**: Passwords and API tokens (`/etc/pulse/nodes.enc`)
|
|
||||||
- **Email Settings**: SMTP passwords (`/etc/pulse/email.enc`)
|
|
||||||
- **Webhook Data**: URLs and auth headers (`/etc/pulse/webhooks.enc`) - v4.1.9+
|
|
||||||
- **Encryption Key**: Auto-generated (`/etc/pulse/.encryption.key`)
|
|
||||||
|
|
||||||
### Security Features
|
|
||||||
- **Logs**: Token values masked with `***` in all outputs
|
|
||||||
- **API**: Frontend receives only `hasToken: true`, never actual values
|
|
||||||
- **Export**: Requires a valid API token (via `X-API-Token` header or `token` query parameter) to extract credentials
|
|
||||||
- **Migration**: Use passphrase-protected export/import (see [Migration Guide](MIGRATION.md))
|
|
||||||
- **Auto-Migration**: Unencrypted configs automatically migrate to encrypted format
|
|
||||||
|
|
||||||
## Export/Import Protection
|
|
||||||
|
|
||||||
By default, configuration export/import is blocked for security. You have two options:
|
|
||||||
|
|
||||||
### Option 1: Set API Tokens (Recommended)
|
|
||||||
```bash
|
|
||||||
# Using systemd (secure)
|
|
||||||
sudo systemctl edit pulse
|
|
||||||
# Add:
|
|
||||||
[Service]
|
|
||||||
Environment="API_TOKENS=ansible-token,docker-agent-token"
|
|
||||||
Environment="API_TOKEN=legacy-token" # Optional fallback
|
|
||||||
|
|
||||||
# Then restart:
|
|
||||||
sudo systemctl restart pulse
|
|
||||||
|
|
||||||
# Docker
|
|
||||||
docker run -e API_TOKENS=ansible-token,docker-agent-token rcourtman/pulse:latest
|
|
||||||
```
|
|
||||||
|
|
||||||
### Option 2: Allow Unprotected Export (Homelab)
|
|
||||||
```bash
|
|
||||||
# Using systemd
|
|
||||||
sudo systemctl edit pulse
|
|
||||||
# Add:
|
|
||||||
[Service]
|
|
||||||
Environment="ALLOW_UNPROTECTED_EXPORT=true"
|
|
||||||
|
|
||||||
# Docker
|
|
||||||
docker run -e ALLOW_UNPROTECTED_EXPORT=true rcourtman/pulse:latest
|
|
||||||
```
|
|
||||||
|
|
||||||
**Note:** For production deployments, consider using Docker secrets or systemd environment variables instead of .env files for sensitive data.
|
|
||||||
|
|
||||||
## Security Features
|
|
||||||
|
|
||||||
### Core Protection
|
|
||||||
- **Encryption**: All credentials encrypted at rest (AES-256-GCM)
|
|
||||||
- **Export Protection**: Exports always encrypted with passphrase
|
|
||||||
- **Minimum Passphrase**: 12 characters required for exports
|
|
||||||
- **Security Tab**: Check status in Settings → Security
|
|
||||||
|
|
||||||
### Enterprise Security (When Authentication Enabled)
|
|
||||||
- **Password Security**:
|
|
||||||
- Bcrypt hashing with cost factor 12 (60-character hash)
|
|
||||||
- Passwords NEVER stored in plain text
|
|
||||||
- Automatic hashing on security setup
|
|
||||||
- **CRITICAL**: Bcrypt hashes MUST be exactly 60 characters
|
|
||||||
- **API Token Security**:
|
|
||||||
- 64-character hex tokens (32 bytes of entropy)
|
|
||||||
- SHA3-256 hashed before storage (64-char hash)
|
|
||||||
- Raw token shown only once during generation
|
|
||||||
- Tokens NEVER stored in plain text
|
|
||||||
- Live reloading when .env file changes
|
|
||||||
- API-only mode supported (no password auth required)
|
|
||||||
- **CSRF Protection**: All state-changing operations require CSRF tokens
|
|
||||||
- **Rate Limiting**:
|
|
||||||
- Authentication endpoints: 10 attempts/minute per IP
|
|
||||||
- General API: 500 requests/minute per IP
|
|
||||||
- Real-time endpoints exempt for functionality
|
|
||||||
- **Account Lockout Protection**:
|
|
||||||
- Locks after 5 failed login attempts
|
|
||||||
- 15-minute automatic lockout duration
|
|
||||||
- Clear feedback showing remaining attempts
|
|
||||||
- Time remaining displayed when locked
|
|
||||||
- Manual reset available via API for administrators
|
|
||||||
- **Session Management**:
|
|
||||||
- Secure HttpOnly cookies
|
|
||||||
- 24-hour session expiry
|
|
||||||
- Session invalidation on password change
|
|
||||||
- **Security Headers**:
|
|
||||||
- Content-Security-Policy with strict directives
|
|
||||||
- X-Frame-Options: DENY (prevents clickjacking)
|
|
||||||
- X-Content-Type-Options: nosniff
|
|
||||||
- X-XSS-Protection: 1; mode=block
|
|
||||||
- Referrer-Policy: strict-origin-when-cross-origin
|
|
||||||
- Permissions-Policy restricting sensitive APIs
|
|
||||||
- **Audit Logging**: All authentication events logged with IP addresses
|
|
||||||
|
|
||||||
### What's Encrypted in Exports
|
|
||||||
- Node credentials (passwords, API tokens)
|
|
||||||
- PBS credentials
|
|
||||||
- Email settings passwords
|
|
||||||
- Webhook URLs and authentication headers (v4.1.9+)
|
|
||||||
|
|
||||||
### What's NOT Encrypted
|
|
||||||
- Node hostnames and IPs
|
|
||||||
- Threshold settings
|
|
||||||
- General configuration
|
|
||||||
- Alert rules and schedules
|
|
||||||
|
|
||||||
## Authentication
|
|
||||||
|
|
||||||
Pulse supports multiple authentication methods that can be used independently or together:
|
|
||||||
|
|
||||||
### Password Authentication
|
|
||||||
|
|
||||||
#### Quick Security Setup (Recommended)
|
|
||||||
The easiest way to enable authentication is through the web UI:
|
|
||||||
1. Go to Settings → Security
|
|
||||||
2. Click "Enable Security Now"
|
|
||||||
3. Enter username and password
|
|
||||||
4. Save the generated API token (shown only once!)
|
|
||||||
5. Security is enabled immediately (no restart needed)
|
|
||||||
|
|
||||||
This automatically:
|
|
||||||
- Generates a secure random password
|
|
||||||
- Hashes it with bcrypt (cost factor 12)
|
|
||||||
- Creates secure API token (SHA3-256 hashed, raw token shown once)
|
|
||||||
- For systemd: Configures systemd with hashed credentials
|
|
||||||
- For Docker: Saves to `/data/.env` with hashed credentials (properly quoted to prevent shell expansion)
|
|
||||||
- Restarts service/container with authentication enabled
|
|
||||||
|
|
||||||
#### Manual Setup (Advanced)
|
|
||||||
```bash
|
|
||||||
# Using systemd (password will be hashed automatically)
|
|
||||||
sudo systemctl edit pulse
|
|
||||||
# Add:
|
|
||||||
[Service]
|
|
||||||
Environment="PULSE_AUTH_USER=admin"
|
|
||||||
Environment="PULSE_AUTH_PASS=$2a$12$..." # Use bcrypt hash, not plain text!
|
|
||||||
|
|
||||||
# Docker (credentials persist in volume via .env file)
|
|
||||||
# IMPORTANT: Always quote bcrypt hashes to prevent shell expansion!
|
|
||||||
docker run -e PULSE_AUTH_USER=admin -e PULSE_AUTH_PASS='$2a$12$...' rcourtman/pulse:latest
|
|
||||||
# Or use Quick Security Setup and restart container
|
|
||||||
```
|
|
||||||
|
|
||||||
**Important**: Always use hashed passwords in configuration. Use the Quick Security Setup or generate bcrypt hashes manually.
|
|
||||||
|
|
||||||
#### Features
|
|
||||||
- Web UI login required when authentication enabled
|
|
||||||
- Change/remove password from Settings → Security
|
|
||||||
- Passwords ALWAYS hashed with bcrypt (cost 12)
|
|
||||||
- Session-based authentication with secure HttpOnly cookies
|
|
||||||
- 24-hour session expiry
|
|
||||||
- CSRF protection for all state-changing operations
|
|
||||||
- Session invalidation on password change
|
|
||||||
|
|
||||||
### API Token Authentication
|
|
||||||
For programmatic access and automation. API tokens are SHA3-256 hashed for security.
|
|
||||||
|
|
||||||
#### Token Setup via Quick Security
|
|
||||||
The Quick Security Setup automatically:
|
|
||||||
- Generates a cryptographically secure token
|
|
||||||
- Hashes it with SHA3-256
|
|
||||||
- Stores only the 64-character hash
|
|
||||||
- Adds the token to the managed token list
|
|
||||||
|
|
||||||
#### Manual Token Setup
|
|
||||||
```bash
|
|
||||||
# Using systemd (plain text values are auto-hashed on startup)
|
|
||||||
sudo systemctl edit pulse
|
|
||||||
# Add:
|
|
||||||
[Service]
|
|
||||||
Environment="API_TOKENS=ansible-token,docker-agent-token"
|
|
||||||
|
|
||||||
# Docker
|
|
||||||
docker run -e API_TOKENS=ansible-token,docker-agent-token rcourtman/pulse:latest
|
|
||||||
|
|
||||||
# To provide pre-hashed tokens instead, list the SHA3-256 hashes
|
|
||||||
# Environment="API_TOKENS=83c8...,b1de..."
|
|
||||||
```
|
|
||||||
|
|
||||||
**Security Note**: Tokens defined via environment variables are hashed with SHA3-256 before being stored on disk. Plain values never persist beyond startup.
|
|
||||||
|
|
||||||
#### Token Management (Settings → Security → API tokens)
|
|
||||||
- Issue dedicated tokens for automation/agents without sharing a global credential
|
|
||||||
- View prefixes/suffixes and last-used timestamps for auditing
|
|
||||||
- Revoke tokens individually without downtime
|
|
||||||
- Regenerate tokens when rotating credentials (new value displayed once)
|
|
||||||
- All tokens stored as SHA3-256 hashes
|
|
||||||
|
|
||||||
#### Usage
|
|
||||||
```bash
|
|
||||||
# Include the ORIGINAL token (not hash) in X-API-Token header
|
|
||||||
curl -H "X-API-Token: your-original-token" http://localhost:7655/api/health
|
|
||||||
|
|
||||||
# Or in query parameter for export/import
|
|
||||||
curl "http://localhost:7655/api/export?token=your-original-token"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Auto-Registration Security
|
|
||||||
|
|
||||||
#### Default Mode
|
|
||||||
- All access requires authentication
|
|
||||||
- Nodes can auto-register with the API token
|
|
||||||
- Setup scripts work without additional configuration
|
|
||||||
|
|
||||||
#### Secure Mode
|
|
||||||
- Require API token for all operations
|
|
||||||
- Protects auto-registration endpoint
|
|
||||||
- Enable by setting at least one API token via `API_TOKENS` (or legacy `API_TOKEN`) environment variable
|
|
||||||
|
|
||||||
## CORS (Cross-Origin Resource Sharing)
|
|
||||||
|
|
||||||
By default, Pulse only allows same-origin requests (no CORS headers). This is the most secure configuration.
|
|
||||||
|
|
||||||
### Configuring CORS for External Access
|
|
||||||
|
|
||||||
If you need to access Pulse API from a different domain:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Docker
|
|
||||||
docker run -e ALLOWED_ORIGINS="https://app.example.com" rcourtman/pulse:latest
|
|
||||||
|
|
||||||
# systemd
|
|
||||||
sudo systemctl edit pulse
|
|
||||||
[Service]
|
|
||||||
Environment="ALLOWED_ORIGINS=https://app.example.com"
|
|
||||||
|
|
||||||
# Multiple origins (comma-separated)
|
|
||||||
ALLOWED_ORIGINS="https://app.example.com,https://dashboard.example.com"
|
|
||||||
|
|
||||||
# Development mode (allows localhost)
|
|
||||||
PULSE_DEV=true
|
|
||||||
```
|
|
||||||
|
|
||||||
**Security Note**: Never use `ALLOWED_ORIGINS=*` in production as it allows any website to access your API.
|
|
||||||
|
|
||||||
## Security Best Practices
|
|
||||||
|
|
||||||
### Credential Storage
|
|
||||||
- ✅ **DO**: Use Quick Security Setup for automatic hashing
|
|
||||||
- ✅ **DO**: Store only bcrypt hashes for passwords
|
|
||||||
- ✅ **DO**: Store only SHA3-256 hashes for API tokens
|
|
||||||
- ❌ **DON'T**: Store plain text passwords in config files
|
|
||||||
- ❌ **DON'T**: Store plain text API tokens in config files
|
|
||||||
- ❌ **DON'T**: Log credentials or include them in backups
|
|
||||||
|
|
||||||
### Authentication Setup
|
|
||||||
- ✅ **DO**: Use strong, unique passwords (16+ characters)
|
|
||||||
- ✅ **DO**: Rotate API tokens periodically
|
|
||||||
- ✅ **DO**: Use HTTPS in production environments
|
|
||||||
- ❌ **DON'T**: Share API tokens between users/services
|
|
||||||
- ❌ **DON'T**: Embed credentials in client-side code
|
|
||||||
|
|
||||||
### Verification
|
|
||||||
Run the security verification script to ensure no plain text credentials:
|
|
||||||
```bash
|
|
||||||
/opt/pulse/testing-tools/security-verification.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
This checks:
|
|
||||||
- No hardcoded credentials in code
|
|
||||||
- No credentials exposed in logs
|
|
||||||
- All passwords/tokens properly hashed
|
|
||||||
- Secure file permissions
|
|
||||||
- No credential leaks in API responses
|
|
||||||
|
|
||||||
## Account Lockout and Recovery
|
|
||||||
|
|
||||||
### Lockout Behavior
|
|
||||||
- After **5 failed login attempts**, the account is locked for **15 minutes**
|
|
||||||
- Lockout applies to both username and IP address
|
|
||||||
- Login form shows remaining attempts after each failure
|
|
||||||
- Clear message when locked with time remaining
|
|
||||||
|
|
||||||
### Automatic Recovery
|
|
||||||
- Lockouts automatically expire after 15 minutes
|
|
||||||
- No action needed - just wait for the timer to expire
|
|
||||||
- Successful login clears all failed attempt counters
|
|
||||||
|
|
||||||
### Manual Recovery (Admin)
|
|
||||||
Administrators with API access can manually reset lockouts:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Reset lockout for a specific username
|
|
||||||
curl -X POST http://localhost:7655/api/security/reset-lockout \
|
|
||||||
-H "X-API-Token: your-api-token" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"identifier":"username"}'
|
|
||||||
|
|
||||||
# Reset lockout for an IP address
|
|
||||||
curl -X POST http://localhost:7655/api/security/reset-lockout \
|
|
||||||
-H "X-API-Token: your-api-token" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"identifier":"192.168.1.100"}'
|
|
||||||
```
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
**Account locked?** Wait 15 minutes or contact admin for manual reset
|
|
||||||
**Export blocked?** You're on a public network - login with password, set an API token (`API_TOKENS`), or set ALLOW_UNPROTECTED_EXPORT=true
|
|
||||||
**Rate limited?** Wait 1 minute and try again
|
|
||||||
**Can't login?** Check PULSE_AUTH_USER and PULSE_AUTH_PASS environment variables
|
|
||||||
**API access denied?** Verify the token you supplied matches one of the values created in Settings → Security → API tokens (use the original token value, not the hash)
|
|
||||||
**CORS errors?** Configure ALLOWED_ORIGINS for your domain
|
|
||||||
**Forgot password?** Start fresh - delete your Pulse data and restart
|
|
||||||
|
|
|
||||||
|
|
@ -258,6 +258,7 @@ type Monitor struct {
|
||||||
pollMetrics *PollMetrics
|
pollMetrics *PollMetrics
|
||||||
scheduler *AdaptiveScheduler
|
scheduler *AdaptiveScheduler
|
||||||
stalenessTracker *StalenessTracker
|
stalenessTracker *StalenessTracker
|
||||||
|
taskQueue *TaskQueue
|
||||||
tempCollector *TemperatureCollector // SSH-based temperature collector
|
tempCollector *TemperatureCollector // SSH-based temperature collector
|
||||||
mu sync.RWMutex
|
mu sync.RWMutex
|
||||||
startTime time.Time
|
startTime time.Time
|
||||||
|
|
@ -1317,6 +1318,7 @@ func New(cfg *config.Config) (*Monitor, error) {
|
||||||
|
|
||||||
stalenessTracker := NewStalenessTracker(getPollMetrics())
|
stalenessTracker := NewStalenessTracker(getPollMetrics())
|
||||||
stalenessTracker.SetBounds(cfg.AdaptivePollingBaseInterval, cfg.AdaptivePollingMaxInterval)
|
stalenessTracker.SetBounds(cfg.AdaptivePollingBaseInterval, cfg.AdaptivePollingMaxInterval)
|
||||||
|
taskQueue := NewTaskQueue()
|
||||||
|
|
||||||
var scheduler *AdaptiveScheduler
|
var scheduler *AdaptiveScheduler
|
||||||
if cfg.AdaptivePollingEnabled {
|
if cfg.AdaptivePollingEnabled {
|
||||||
|
|
@ -1336,6 +1338,7 @@ func New(cfg *config.Config) (*Monitor, error) {
|
||||||
pollMetrics: getPollMetrics(),
|
pollMetrics: getPollMetrics(),
|
||||||
scheduler: scheduler,
|
scheduler: scheduler,
|
||||||
stalenessTracker: stalenessTracker,
|
stalenessTracker: stalenessTracker,
|
||||||
|
taskQueue: taskQueue,
|
||||||
tempCollector: tempCollector,
|
tempCollector: tempCollector,
|
||||||
startTime: time.Now(),
|
startTime: time.Now(),
|
||||||
rateTracker: NewRateTracker(),
|
rateTracker: NewRateTracker(),
|
||||||
|
|
@ -1691,8 +1694,12 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) {
|
||||||
|
|
||||||
// Create separate tickers for polling and broadcasting
|
// Create separate tickers for polling and broadcasting
|
||||||
// Hardcoded to 10 seconds since Proxmox updates cluster/resources every 10 seconds
|
// Hardcoded to 10 seconds since Proxmox updates cluster/resources every 10 seconds
|
||||||
const pollingInterval = 10 * time.Second
|
const pollingInterval = 10 * time.Second
|
||||||
pollTicker := time.NewTicker(pollingInterval)
|
|
||||||
|
workerCount := len(m.pveClients) + len(m.pbsClients) + len(m.pmgClients)
|
||||||
|
m.startTaskWorkers(ctx, workerCount)
|
||||||
|
|
||||||
|
pollTicker := time.NewTicker(pollingInterval)
|
||||||
defer pollTicker.Stop()
|
defer pollTicker.Stop()
|
||||||
|
|
||||||
broadcastTicker := time.NewTicker(pollingInterval)
|
broadcastTicker := time.NewTicker(pollingInterval)
|
||||||
|
|
@ -1934,24 +1941,10 @@ func (m *Monitor) poll(ctx context.Context, wsHub *websocket.Hub) {
|
||||||
now := startTime
|
now := startTime
|
||||||
|
|
||||||
plannedTasks := m.buildScheduledTasks(now)
|
plannedTasks := m.buildScheduledTasks(now)
|
||||||
dueTasks := plannedTasks
|
for _, task := range plannedTasks {
|
||||||
if m.scheduler != nil {
|
m.taskQueue.Upsert(task)
|
||||||
due := m.scheduler.DispatchDue(ctx, now, plannedTasks)
|
|
||||||
if len(due) > 0 {
|
|
||||||
dueTasks = due
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if m.pollMetrics != nil {
|
|
||||||
m.pollMetrics.ResetQueueDepth(len(dueTasks))
|
|
||||||
}
|
|
||||||
|
|
||||||
if m.config.ConcurrentPolling {
|
|
||||||
// Use concurrent polling
|
|
||||||
m.pollConcurrent(ctx, dueTasks)
|
|
||||||
} else {
|
|
||||||
m.pollSequential(ctx, dueTasks)
|
|
||||||
}
|
}
|
||||||
|
m.updateQueueDepthMetric()
|
||||||
|
|
||||||
// Update performance metrics
|
// Update performance metrics
|
||||||
m.state.Performance.LastPollDuration = time.Since(startTime).Seconds()
|
m.state.Performance.LastPollDuration = time.Since(startTime).Seconds()
|
||||||
|
|
@ -2088,99 +2081,126 @@ func (m *Monitor) pruneStaleDockerAlerts() bool {
|
||||||
return cleared
|
return cleared
|
||||||
}
|
}
|
||||||
|
|
||||||
// pollConcurrent polls instances concurrently based on scheduled tasks.
|
func (m *Monitor) startTaskWorkers(ctx context.Context, workers int) {
|
||||||
func (m *Monitor) pollConcurrent(ctx context.Context, tasks []ScheduledTask) {
|
if m.taskQueue == nil {
|
||||||
if len(tasks) == 0 {
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if workers < 1 {
|
||||||
var wg sync.WaitGroup
|
workers = 1
|
||||||
ctx, cancel := context.WithCancel(ctx)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
for _, task := range tasks {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
default:
|
|
||||||
}
|
|
||||||
|
|
||||||
switch task.InstanceType {
|
|
||||||
case InstanceTypePVE:
|
|
||||||
client, ok := m.pveClients[task.InstanceName]
|
|
||||||
if !ok || client == nil {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
wg.Add(1)
|
|
||||||
go func(name string, c PVEClientInterface) {
|
|
||||||
defer wg.Done()
|
|
||||||
m.pollPVEInstance(ctx, name, c)
|
|
||||||
}(task.InstanceName, client)
|
|
||||||
case InstanceTypePBS:
|
|
||||||
client, ok := m.pbsClients[task.InstanceName]
|
|
||||||
if !ok || client == nil {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
wg.Add(1)
|
|
||||||
go func(name string, c *pbs.Client) {
|
|
||||||
defer wg.Done()
|
|
||||||
m.pollPBSInstance(ctx, name, c)
|
|
||||||
}(task.InstanceName, client)
|
|
||||||
case InstanceTypePMG:
|
|
||||||
client, ok := m.pmgClients[task.InstanceName]
|
|
||||||
if !ok || client == nil {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
wg.Add(1)
|
|
||||||
go func(name string, c *pmg.Client) {
|
|
||||||
defer wg.Done()
|
|
||||||
m.pollPMGInstance(ctx, name, c)
|
|
||||||
}(task.InstanceName, client)
|
|
||||||
default:
|
|
||||||
log.Debug().Str("instance", task.InstanceName).Str("type", string(task.InstanceType)).Msg("Skipping unsupported task type")
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
if workers > 10 {
|
||||||
done := make(chan struct{})
|
workers = 10
|
||||||
go func() {
|
}
|
||||||
wg.Wait()
|
for i := 0; i < workers; i++ {
|
||||||
close(done)
|
go m.taskWorker(ctx, i)
|
||||||
}()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-done:
|
|
||||||
case <-ctx.Done():
|
|
||||||
cancel()
|
|
||||||
wg.Wait()
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// pollSequential polls instances sequentially based on scheduled tasks.
|
func (m *Monitor) taskWorker(ctx context.Context, id int) {
|
||||||
func (m *Monitor) pollSequential(ctx context.Context, tasks []ScheduledTask) {
|
log.Debug().Int("worker", id).Msg("Task worker started")
|
||||||
for _, task := range tasks {
|
for {
|
||||||
select {
|
task, ok := m.taskQueue.WaitNext(ctx)
|
||||||
case <-ctx.Done():
|
if !ok {
|
||||||
|
log.Debug().Int("worker", id).Msg("Task worker stopping")
|
||||||
return
|
return
|
||||||
default:
|
|
||||||
}
|
}
|
||||||
|
|
||||||
switch task.InstanceType {
|
m.executeScheduledTask(ctx, task)
|
||||||
case InstanceTypePVE:
|
|
||||||
if client, ok := m.pveClients[task.InstanceName]; ok && client != nil {
|
m.rescheduleTask(task)
|
||||||
m.pollPVEInstance(ctx, task.InstanceName, client)
|
m.updateQueueDepthMetric()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Monitor) executeScheduledTask(ctx context.Context, task ScheduledTask) {
|
||||||
|
switch task.InstanceType {
|
||||||
|
case InstanceTypePVE:
|
||||||
|
client, ok := m.pveClients[task.InstanceName]
|
||||||
|
if !ok || client == nil {
|
||||||
|
log.Warn().Str("instance", task.InstanceName).Msg("PVE client missing for scheduled task")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
m.pollPVEInstance(ctx, task.InstanceName, client)
|
||||||
|
case InstanceTypePBS:
|
||||||
|
client, ok := m.pbsClients[task.InstanceName]
|
||||||
|
if !ok || client == nil {
|
||||||
|
log.Warn().Str("instance", task.InstanceName).Msg("PBS client missing for scheduled task")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
m.pollPBSInstance(ctx, task.InstanceName, client)
|
||||||
|
case InstanceTypePMG:
|
||||||
|
client, ok := m.pmgClients[task.InstanceName]
|
||||||
|
if !ok || client == nil {
|
||||||
|
log.Warn().Str("instance", task.InstanceName).Msg("PMG client missing for scheduled task")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
m.pollPMGInstance(ctx, task.InstanceName, client)
|
||||||
|
default:
|
||||||
|
log.Debug().Str("instance", task.InstanceName).Str("type", string(task.InstanceType)).Msg("Skipping unsupported task type")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Monitor) rescheduleTask(task ScheduledTask) {
|
||||||
|
if m.taskQueue == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if m.scheduler == nil {
|
||||||
|
nextInterval := task.Interval
|
||||||
|
if nextInterval <= 0 && m.config != nil {
|
||||||
|
nextInterval = m.config.AdaptivePollingBaseInterval
|
||||||
|
}
|
||||||
|
if nextInterval <= 0 {
|
||||||
|
nextInterval = DefaultSchedulerConfig().BaseInterval
|
||||||
|
}
|
||||||
|
next := task
|
||||||
|
next.NextRun = time.Now().Add(nextInterval)
|
||||||
|
next.Interval = nextInterval
|
||||||
|
m.taskQueue.Upsert(next)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
desc := InstanceDescriptor{
|
||||||
|
Name: task.InstanceName,
|
||||||
|
Type: task.InstanceType,
|
||||||
|
LastInterval: task.Interval,
|
||||||
|
LastScheduled: task.NextRun,
|
||||||
|
}
|
||||||
|
if m.stalenessTracker != nil {
|
||||||
|
if snap, ok := m.stalenessTracker.snapshot(task.InstanceType, task.InstanceName); ok {
|
||||||
|
desc.LastSuccess = snap.LastSuccess
|
||||||
|
desc.LastFailure = snap.LastError
|
||||||
|
if snap.ChangeHash != "" {
|
||||||
|
desc.Metadata = map[string]any{"changeHash": snap.ChangeHash}
|
||||||
}
|
}
|
||||||
case InstanceTypePBS:
|
|
||||||
if client, ok := m.pbsClients[task.InstanceName]; ok && client != nil {
|
|
||||||
m.pollPBSInstance(ctx, task.InstanceName, client)
|
|
||||||
}
|
|
||||||
case InstanceTypePMG:
|
|
||||||
if client, ok := m.pmgClients[task.InstanceName]; ok && client != nil {
|
|
||||||
m.pollPMGInstance(ctx, task.InstanceName, client)
|
|
||||||
}
|
|
||||||
default:
|
|
||||||
log.Debug().Str("instance", task.InstanceName).Str("type", string(task.InstanceType)).Msg("Skipping unsupported task type")
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
tasks := m.scheduler.BuildPlan(time.Now(), []InstanceDescriptor{desc}, m.taskQueue.Size())
|
||||||
|
if len(tasks) == 0 {
|
||||||
|
next := task
|
||||||
|
nextInterval := task.Interval
|
||||||
|
if nextInterval <= 0 && m.config != nil {
|
||||||
|
nextInterval = m.config.AdaptivePollingBaseInterval
|
||||||
|
}
|
||||||
|
if nextInterval <= 0 {
|
||||||
|
nextInterval = DefaultSchedulerConfig().BaseInterval
|
||||||
|
}
|
||||||
|
next.Interval = nextInterval
|
||||||
|
next.NextRun = time.Now().Add(nextInterval)
|
||||||
|
m.taskQueue.Upsert(next)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, next := range tasks {
|
||||||
|
m.taskQueue.Upsert(next)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Monitor) updateQueueDepthMetric() {
|
||||||
|
if m.pollMetrics == nil || m.taskQueue == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
m.pollMetrics.SetQueueDepth(m.taskQueue.Size())
|
||||||
}
|
}
|
||||||
|
|
||||||
// pollPVEInstance polls a single PVE instance
|
// pollPVEInstance polls a single PVE instance
|
||||||
|
|
|
||||||
|
|
@ -117,20 +117,29 @@ func (m *Monitor) buildScheduledTasks(now time.Time) []ScheduledTask {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
queueDepth := 0
|
||||||
|
if m.taskQueue != nil {
|
||||||
|
queueDepth = m.taskQueue.Size()
|
||||||
|
}
|
||||||
|
|
||||||
if m.scheduler == nil {
|
if m.scheduler == nil {
|
||||||
tasks := make([]ScheduledTask, 0, len(descriptors))
|
tasks := make([]ScheduledTask, 0, len(descriptors))
|
||||||
|
interval := m.config.AdaptivePollingBaseInterval
|
||||||
|
if interval <= 0 {
|
||||||
|
interval = DefaultSchedulerConfig().BaseInterval
|
||||||
|
}
|
||||||
for _, desc := range descriptors {
|
for _, desc := range descriptors {
|
||||||
tasks = append(tasks, ScheduledTask{
|
tasks = append(tasks, ScheduledTask{
|
||||||
InstanceName: desc.Name,
|
InstanceName: desc.Name,
|
||||||
InstanceType: desc.Type,
|
InstanceType: desc.Type,
|
||||||
NextRun: now,
|
NextRun: now,
|
||||||
Interval: DefaultSchedulerConfig().BaseInterval,
|
Interval: interval,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
return tasks
|
return tasks
|
||||||
}
|
}
|
||||||
|
|
||||||
return m.scheduler.BuildPlan(now, descriptors)
|
return m.scheduler.BuildPlan(now, descriptors, queueDepth)
|
||||||
}
|
}
|
||||||
|
|
||||||
// convertPoolInfoToModel converts Proxmox ZFS pool info to our model
|
// convertPoolInfoToModel converts Proxmox ZFS pool info to our model
|
||||||
|
|
|
||||||
|
|
@ -130,7 +130,7 @@ func NewAdaptiveScheduler(cfg SchedulerConfig, staleness StalenessSource, interv
|
||||||
}
|
}
|
||||||
|
|
||||||
// BuildPlan produces an ordered set of scheduled tasks for the supplied inventory.
|
// BuildPlan produces an ordered set of scheduled tasks for the supplied inventory.
|
||||||
func (s *AdaptiveScheduler) BuildPlan(now time.Time, inventory []InstanceDescriptor) []ScheduledTask {
|
func (s *AdaptiveScheduler) BuildPlan(now time.Time, inventory []InstanceDescriptor, queueDepth int) []ScheduledTask {
|
||||||
if len(inventory) == 0 {
|
if len(inventory) == 0 {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
@ -159,20 +159,21 @@ func (s *AdaptiveScheduler) BuildPlan(now time.Time, inventory []InstanceDescrip
|
||||||
lastInterval = s.cfg.BaseInterval
|
lastInterval = s.cfg.BaseInterval
|
||||||
}
|
}
|
||||||
|
|
||||||
req := IntervalRequest{
|
currentDepth := queueDepth + len(tasks)
|
||||||
Now: now,
|
req := IntervalRequest{
|
||||||
BaseInterval: s.cfg.BaseInterval,
|
Now: now,
|
||||||
MinInterval: s.cfg.MinInterval,
|
BaseInterval: s.cfg.BaseInterval,
|
||||||
MaxInterval: s.cfg.MaxInterval,
|
MinInterval: s.cfg.MinInterval,
|
||||||
LastInterval: lastInterval,
|
MaxInterval: s.cfg.MaxInterval,
|
||||||
LastSuccess: inst.LastSuccess,
|
LastInterval: lastInterval,
|
||||||
LastScheduled: lastScheduled,
|
LastSuccess: inst.LastSuccess,
|
||||||
StalenessScore: score,
|
LastScheduled: lastScheduled,
|
||||||
ErrorCount: inst.ErrorCount,
|
StalenessScore: score,
|
||||||
QueueDepth: len(inventory),
|
ErrorCount: inst.ErrorCount,
|
||||||
InstanceKey: schedulerKey(inst.Type, inst.Name),
|
QueueDepth: currentDepth,
|
||||||
InstanceType: inst.Type,
|
InstanceKey: schedulerKey(inst.Type, inst.Name),
|
||||||
}
|
InstanceType: inst.Type,
|
||||||
|
}
|
||||||
|
|
||||||
nextInterval := s.interval.SelectInterval(req)
|
nextInterval := s.interval.SelectInterval(req)
|
||||||
if nextInterval <= 0 {
|
if nextInterval <= 0 {
|
||||||
|
|
|
||||||
154
internal/monitoring/task_queue.go
Normal file
154
internal/monitoring/task_queue.go
Normal file
|
|
@ -0,0 +1,154 @@
|
||||||
|
package monitoring
|
||||||
|
|
||||||
|
import (
|
||||||
|
"container/heap"
|
||||||
|
"context"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
type scheduledTaskEntry struct {
|
||||||
|
task ScheduledTask
|
||||||
|
index int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *scheduledTaskEntry) key() string {
|
||||||
|
return schedulerKey(e.task.InstanceType, e.task.InstanceName)
|
||||||
|
}
|
||||||
|
|
||||||
|
type taskHeap []*scheduledTaskEntry
|
||||||
|
|
||||||
|
func (h taskHeap) Len() int { return len(h) }
|
||||||
|
|
||||||
|
func (h taskHeap) Less(i, j int) bool {
|
||||||
|
if h[i].task.NextRun.Equal(h[j].task.NextRun) {
|
||||||
|
if h[i].task.Priority == h[j].task.Priority {
|
||||||
|
return h[i].task.InstanceName < h[j].task.InstanceName
|
||||||
|
}
|
||||||
|
return h[i].task.Priority > h[j].task.Priority
|
||||||
|
}
|
||||||
|
return h[i].task.NextRun.Before(h[j].task.NextRun)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h taskHeap) Swap(i, j int) {
|
||||||
|
h[i], h[j] = h[j], h[i]
|
||||||
|
h[i].index = i
|
||||||
|
h[j].index = j
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *taskHeap) Push(x interface{}) {
|
||||||
|
entry := x.(*scheduledTaskEntry)
|
||||||
|
entry.index = len(*h)
|
||||||
|
*h = append(*h, entry)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *taskHeap) Pop() interface{} {
|
||||||
|
old := *h
|
||||||
|
n := len(old)
|
||||||
|
if n == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
entry := old[n-1]
|
||||||
|
entry.index = -1
|
||||||
|
*h = old[:n-1]
|
||||||
|
return entry
|
||||||
|
}
|
||||||
|
|
||||||
|
// TaskQueue is a thread-safe min-heap over scheduled tasks.
|
||||||
|
type TaskQueue struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
entries map[string]*scheduledTaskEntry
|
||||||
|
heap taskHeap
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewTaskQueue constructs an empty queue.
|
||||||
|
func NewTaskQueue() *TaskQueue {
|
||||||
|
tq := &TaskQueue{
|
||||||
|
entries: make(map[string]*scheduledTaskEntry),
|
||||||
|
heap: make(taskHeap, 0),
|
||||||
|
}
|
||||||
|
heap.Init(&tq.heap)
|
||||||
|
return tq
|
||||||
|
}
|
||||||
|
|
||||||
|
// Upsert inserts or updates a scheduled task in the queue.
|
||||||
|
func (q *TaskQueue) Upsert(task ScheduledTask) {
|
||||||
|
key := schedulerKey(task.InstanceType, task.InstanceName)
|
||||||
|
q.mu.Lock()
|
||||||
|
defer q.mu.Unlock()
|
||||||
|
|
||||||
|
if entry, ok := q.entries[key]; ok {
|
||||||
|
entry.task = task
|
||||||
|
heap.Fix(&q.heap, entry.index)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
entry := &scheduledTaskEntry{task: task}
|
||||||
|
heap.Push(&q.heap, entry)
|
||||||
|
q.entries[key] = entry
|
||||||
|
}
|
||||||
|
|
||||||
|
// Remove deletes a task by key if present.
|
||||||
|
func (q *TaskQueue) Remove(instanceType InstanceType, instance string) {
|
||||||
|
key := schedulerKey(instanceType, instance)
|
||||||
|
q.mu.Lock()
|
||||||
|
defer q.mu.Unlock()
|
||||||
|
|
||||||
|
entry, ok := q.entries[key]
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
heap.Remove(&q.heap, entry.index)
|
||||||
|
delete(q.entries, key)
|
||||||
|
}
|
||||||
|
|
||||||
|
// WaitNext blocks until a task is due or context is cancelled.
|
||||||
|
func (q *TaskQueue) WaitNext(ctx context.Context) (ScheduledTask, bool) {
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ScheduledTask{}, false
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
|
||||||
|
q.mu.Lock()
|
||||||
|
if len(q.heap) == 0 {
|
||||||
|
q.mu.Unlock()
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ScheduledTask{}, false
|
||||||
|
case <-time.After(100 * time.Millisecond):
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
entry := q.heap[0]
|
||||||
|
delay := time.Until(entry.task.NextRun)
|
||||||
|
if delay <= 0 {
|
||||||
|
heap.Pop(&q.heap)
|
||||||
|
delete(q.entries, entry.key())
|
||||||
|
task := entry.task
|
||||||
|
q.mu.Unlock()
|
||||||
|
return task, true
|
||||||
|
}
|
||||||
|
|
||||||
|
q.mu.Unlock()
|
||||||
|
if delay > 250*time.Millisecond {
|
||||||
|
delay = 250 * time.Millisecond
|
||||||
|
}
|
||||||
|
timer := time.NewTimer(delay)
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
timer.Stop()
|
||||||
|
return ScheduledTask{}, false
|
||||||
|
case <-timer.C:
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Size returns the number of tasks currently queued.
|
||||||
|
func (q *TaskQueue) Size() int {
|
||||||
|
q.mu.Lock()
|
||||||
|
defer q.mu.Unlock()
|
||||||
|
return len(q.heap)
|
||||||
|
}
|
||||||
|
|
@ -1,18 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# This script updates test scripts to use unique test node names
|
|
||||||
# to prevent collision with user nodes
|
|
||||||
|
|
||||||
echo "Making test scripts safe by using unique test node names..."
|
|
||||||
|
|
||||||
# Add timestamp to test node names to make them unique
|
|
||||||
TIMESTAMP=$(date +%s)
|
|
||||||
|
|
||||||
# Update test-config-validation.sh to use unique names
|
|
||||||
sed -i "s/\"name\":\"test\"/\"name\":\"test-val-$TIMESTAMP\"/g" /opt/pulse/scripts/test-config-validation.sh
|
|
||||||
sed -i "s/\"name\":\"duplicate-test\"/\"name\":\"dup-test-$TIMESTAMP\"/g" /opt/pulse/scripts/test-config-validation.sh
|
|
||||||
sed -i "s/\"name\":\"pbs-test\"/\"name\":\"pbs-test-$TIMESTAMP\"/g" /opt/pulse/scripts/test-config-validation.sh
|
|
||||||
|
|
||||||
echo "Test scripts updated with unique node names (suffix: $TIMESTAMP)"
|
|
||||||
echo ""
|
|
||||||
echo "You can now safely run tests without affecting production nodes!"
|
|
||||||
|
|
@ -1,43 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# This script patches test scripts to protect mock nodes from deletion
|
|
||||||
# It ensures that mock nodes (pve1-pve7, mock-*) are never deleted during tests
|
|
||||||
|
|
||||||
RED='\033[0;31m'
|
|
||||||
GREEN='\033[0;32m'
|
|
||||||
YELLOW='\033[1;33m'
|
|
||||||
NC='\033[0m'
|
|
||||||
|
|
||||||
echo "Protecting mock nodes from test cleanup..."
|
|
||||||
|
|
||||||
# List of test scripts that delete nodes
|
|
||||||
TEST_SCRIPTS=(
|
|
||||||
"/opt/pulse/scripts/test-persistence.sh"
|
|
||||||
"/opt/pulse/scripts/test-recovery.sh"
|
|
||||||
"/opt/pulse/scripts/test-backup.sh"
|
|
||||||
"/opt/pulse/scripts/test-load.sh"
|
|
||||||
"/opt/pulse/scripts/test-config-validation.sh"
|
|
||||||
)
|
|
||||||
|
|
||||||
for script in "${TEST_SCRIPTS[@]}"; do
|
|
||||||
if [ -f "$script" ]; then
|
|
||||||
echo -n "Patching $(basename $script)... "
|
|
||||||
|
|
||||||
# Create backup
|
|
||||||
cp "$script" "${script}.backup" 2>/dev/null
|
|
||||||
|
|
||||||
# Add protection for mock nodes in cleanup sections
|
|
||||||
# This looks for DELETE commands and adds a check to skip mock nodes
|
|
||||||
|
|
||||||
# Find lines with curl DELETE and add protection
|
|
||||||
sed -i '/curl.*DELETE.*nodes/i\
|
|
||||||
# Skip mock nodes and production nodes\
|
|
||||||
if [[ "$NODE_NAME" == "pve"* ]] || [[ "$NODE_NAME" == "mock"* ]] || [[ "$NODE_NAME" == "delly" ]] || [[ "$NODE_NAME" == "minipc" ]] || [[ "$NODE_NAME" == "pimox" ]]; then\
|
|
||||||
continue\
|
|
||||||
fi' "$script" 2>/dev/null
|
|
||||||
|
|
||||||
echo -e "${GREEN}✓${NC}"
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
echo -e "${GREEN}Mock nodes are now protected!${NC}"
|
|
||||||
Loading…
Reference in a new issue