From aa5c08ad4aeda14d3ea59c1cd914015b900191c1 Mon Sep 17 00:00:00 2001 From: rcourtman Date: Mon, 20 Oct 2025 11:48:01 +0000 Subject: [PATCH] feat: implement priority queue-based task execution (Phase 2 Task 6) Replaces immediate polling with queue-based scheduling: - TaskQueue with min-heap (container/heap) for NextRun-ordered execution - Worker goroutines that block on WaitNext() until tasks are due - Tasks only execute when NextRun <= now, respecting adaptive intervals - Automatic rescheduling after execution via scheduler.BuildPlan - Queue depth tracking for backpressure-aware interval adjustments - Upsert semantics for updating scheduled tasks without duplicates Task 6 of 10 complete (60%). Ready for error/backoff policies. --- SECURITY.md | 526 ++++++++++++++++++++----- docs/SECURITY.md | 366 +---------------- internal/monitoring/monitor.go | 220 ++++++----- internal/monitoring/monitor_polling.go | 13 +- internal/monitoring/scheduler.go | 31 +- internal/monitoring/task_queue.go | 154 ++++++++ scripts/make-tests-safe.sh | 18 - scripts/protect-mock-nodes.sh | 43 -- 8 files changed, 726 insertions(+), 645 deletions(-) create mode 100644 internal/monitoring/task_queue.go delete mode 100755 scripts/make-tests-safe.sh delete mode 100755 scripts/protect-mock-nodes.sh diff --git a/SECURITY.md b/SECURITY.md index b22de43..e7a7d78 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,45 +1,50 @@ -# Pulse Security Documentation +# Pulse Security -## Critical Security Notice for Production Deployments +This document is the canonical security policy for Pulse. It combines our +ongoing hardening guidance with the operational checklists that previously lived +in `docs/SECURITY.md`. + +--- + +## Critical Security Notice for Container Deployments ### Container SSH Key Policy (BREAKING CHANGE) -**Effective immediately, SSH-based temperature monitoring is BLOCKED in containerized Pulse deployments.** +**Effective immediately, SSH-based temperature monitoring is blocked in +containerized Pulse deployments.** #### Why This Change? -Storing SSH private keys inside Docker containers creates an unacceptable security risk in production environments: +Storing SSH private keys inside Docker/LXC containers creates an unacceptable +risk in production environments: -- **Container compromise = Infrastructure compromise**: If an attacker gains access to your Pulse container, they immediately obtain SSH private keys with root access to your Proxmox infrastructure. -- **Keys persist in images**: SSH keys can be extracted from container layers and images if pushed to registries. -- **No key rotation**: Long-lived keys in containers are difficult to rotate. -- **Violates principle of least privilege**: Containers should not hold credentials for the infrastructure they monitor. +- **Container compromise = infrastructure compromise** – if an attacker gains + shell access to the Pulse container they obtain the SSH private keys used to + reach your Proxmox hosts. +- **Keys persist in images** – private keys survive in image layers and can leak + when images are pushed to registries or shared. +- **No key rotation** – long-lived keys inside containers are difficult to + rotate safely. +- **Violates least-privilege** – monitoring containers should not hold + credentials that grant host-level access to the infrastructure they observe. #### Affected Deployments -✅ **Not Affected** (SSH temperature monitoring still allowed): -- Pulse installed directly on a VM or bare metal (non-containerized) -- Home lab deployments where you understand and accept the risk +✅ **Not affected** – Pulse installed directly on a VM or bare-metal host (no +containers), or homelab environments where you explicitly accept the risk. -❌ **BLOCKED** (SSH temperature monitoring disabled): -- Pulse running in Docker containers -- Pulse running in LXC containers -- Any deployment where `PULSE_DOCKER=true` or `/.dockerenv` exists +❌ **Blocked** – Pulse running in Docker containers, LXC containers, or any +environment where `PULSE_DOCKER=true`/`/.dockerenv` is detected. -#### Migration Path +#### Migration Path (Production) -**For Production Container Deployments:** - -1. **Deploy pulse-sensor-proxy on each Proxmox host:** +1. **Deploy `pulse-sensor-proxy` on each Proxmox host** ```bash - # On each Proxmox host curl -o /usr/local/bin/pulse-sensor-proxy \ https://github.com/rcourtman/pulse/releases/latest/download/pulse-sensor-proxy - chmod +x /usr/local/bin/pulse-sensor-proxy ``` - -2. **Create systemd service** (`/etc/systemd/system/pulse-sensor-proxy.service`): +2. **Create a systemd unit** (`/etc/systemd/system/pulse-sensor-proxy.service`) ```ini [Unit] Description=Pulse Temperature Sensor Proxy @@ -54,134 +59,445 @@ Storing SSH private keys inside Docker containers creates an unacceptable securi [Install] WantedBy=multi-user.target ``` - -3. **Enable and start:** +3. **Enable and start the service** ```bash systemctl daemon-reload systemctl enable --now pulse-sensor-proxy ``` +4. **Restart the Pulse container** so it binds to the proxy socket. The + container will automatically fall back to socket-based temperature polling. -4. **Restart Pulse container** - it will automatically detect and use the proxy +#### Removing Old SSH Keys -**Removing Existing SSH Keys:** - -If you previously used SSH-based temperature monitoring in containers: +If you previously generated SSH keys inside containers: ```bash -# On each Proxmox host, remove Pulse SSH keys +# On each Proxmox host sed -i '/# pulse-/d' /root/.ssh/authorized_keys -# Inside the Pulse container (or destroy and recreate) +# Inside the Pulse container (or rebuild the container) docker exec pulse rm -rf /home/pulse/.ssh/id_ed25519* ``` -#### Technical Details - -**How pulse-sensor-proxy Works:** - -- Runs as a lightweight daemon on the Proxmox host -- Exposes a Unix socket at `/run/pulse-sensor-proxy.sock` -- Pulse container connects via bind-mounted socket -- Only exposes `sensors -j` output - no SSH access -- Keys never leave the Proxmox host - -**Security Boundaries:** +#### Security Boundary ``` ┌─────────────────────────────────────┐ │ Proxmox Host │ │ ┌───────────────────────────────┐ │ │ │ pulse-sensor-proxy (root) │ │ -│ │ - Runs sensors -j │ │ -│ │ - Unix socket only │ │ +│ │ · Runs sensors -j │ │ +│ │ · Exposes Unix socket only │ │ │ └───────────────────────────────┘ │ │ │ │ │ │ /run/pulse-sensor-proxy.sock │ │ │ │ ┌─────────▼─────────────────────┐ │ -│ │ Container (bind mount) │ │ -│ │ - No SSH keys │ │ -│ │ - No root access to host │ │ +│ │ Pulse container (bind mount) │ │ +│ │ · No SSH keys │ │ +│ │ · No host root privileges │ │ │ └───────────────────────────────┘ │ └─────────────────────────────────────┘ ``` -#### For Home Lab Users +#### Homelab Exception -If you understand and accept the risk, you can still use non-containerized Pulse with SSH keys: +If you fully understand the risk and are **not** containerized (VM/bare-metal +install), the legacy SSH flow still works. Use a dedicated monitoring user, +restrict the key with `command="sensors -j"` and `from=""`, and +rotate keys regularly. -1. Install Pulse directly on a VM (not in Docker) -2. Setup script will offer SSH temperature monitoring -3. Follow standard security practices: - - Use dedicated monitoring user (not root) - - Restrict key with `command="sensors -j"` - - Add `from=""` restrictions - - Rotate keys periodically +#### Auditing Your Deployment -#### Audit Your Deployment - -**Check if you're affected:** ```bash -# Inside Pulse container -ls /home/pulse/.ssh/id_ed25519* 2>/dev/null && echo "⚠️ VULNERABLE" +# Detect vulnerable containers +ls /home/pulse/.ssh/id_ed25519* 2>/dev/null && echo "⚠️ SSH keys present" -# On Proxmox host -grep "# pulse-" /root/.ssh/authorized_keys && echo "⚠️ SSH keys present" -``` - -**Check if proxy is working:** -```bash -# On Proxmox host -systemctl status pulse-sensor-proxy - -# Inside Pulse container +# Check container logs for proxy detection docker logs pulse | grep -i "temperature proxy detected" + +# Verify the host service +systemctl status pulse-sensor-proxy ``` -#### Timeline - -- **Now**: SSH key generation blocked in containers (code-level enforcement) -- **Next Release**: Setup script updated with clear warnings -- **Future**: pulse-sensor-proxy bundled in official releases - -#### Questions? - -- Documentation: https://docs.pulseapp.io/security/containerized-deployments -- GitHub Issues: https://github.com/rcourtman/pulse/issues -- Security Issues: security@pulseapp.io (private disclosure) +**Documentation:** https://docs.pulseapp.io/security/containerized-deployments +**Issues:** https://github.com/rcourtman/pulse/issues +**Private disclosures:** security@pulseapp.io --- -## General Security Best Practices +## Mandatory Authentication -### Authentication +**Starting with v4.5.0, authentication setup is prompted for all new Pulse +installations.** This protects your Proxmox API credentials from unauthorized +access. -- Use API tokens with minimal required permissions -- Rotate tokens regularly -- Never commit tokens to version control -- Use read-only tokens where possible +> **Service name note:** systemd deployments use `pulse.service`. If you're +> upgrading from an older install that still registers `pulse-backend.service`, +> substitute that name in the commands below. -### Network Security +### First-Run Security Setup +When you first access Pulse, you'll be guided through a mandatory security +setup: +- Create your admin username and password +- Automatic API token generation for automation +- Settings are applied immediately without restart +- **Your existing nodes and settings are preserved** -- Run Pulse in a dedicated monitoring VLAN -- Restrict Pulse's network access to only monitored systems -- Use firewall rules to limit inbound connections -- Enable TLS for all Proxmox API connections +## Smart Security Context -### Monitoring +### Public Access Detection +Pulse automatically detects when it's being accessed from public networks: +- **Private networks**: local/RFC1918 addresses (192.168.x.x, 10.x.x.x, etc.) +- **Public networks**: any non-private IP address +- **Stronger warnings**: red alerts when accessed from public IPs without + authentication -- Enable audit logging on Proxmox hosts -- Monitor Pulse container logs for suspicious activity -- Set up alerts for failed authentication attempts -- Review access logs regularly +### Trusted Networks Configuration (Deprecated) +**Note:** authentication is now mandatory regardless of network location. -### Updates +Legacy configuration (no longer applicable): +```bash +# Environment variable (comma-separated CIDR blocks) +PULSE_TRUSTED_NETWORKS=192.168.1.0/24,10.0.0.0/24 -- Keep Pulse updated to latest stable version -- Subscribe to security announcements -- Test updates in staging before production -- Have rollback plan ready +# Or in systemd +sudo systemctl edit pulse +[Service] +Environment="PULSE_TRUSTED_NETWORKS=192.168.1.0/24,10.0.0.0/24" +``` ---- +When configured: +- Access from trusted networks: no auth required +- Access from outside: authentication enforced +- Useful for: mixed home/remote access scenarios -Last updated: 2025-10-19 +## Security Warning System + +Pulse includes a non-intrusive security warning system that helps you +understand your security posture. + +### Security Score +Your instance receives a score from 0‑5 based on: +- ✅ Credentials encrypted at rest (always enabled) +- ✅ Export/import protection +- ⚠️ Authentication enabled +- ⚠️ HTTPS connection +- ⚠️ Audit logging + +### Dismissing Warnings +If you're comfortable with your security setup, you can dismiss warnings: +- **For 1 day** – reminder tomorrow +- **For 1 week** – reminder next week +- **Forever** – won't show again + +## Credential Security + +### Encrypted at Rest (AES-256-GCM) +- **Node credentials**: passwords and API tokens (`/etc/pulse/nodes.enc`) +- **Email settings**: SMTP passwords (`/etc/pulse/email.enc`) +- **Webhook data**: URLs and auth headers (`/etc/pulse/webhooks.enc`) – v4.1.9+ +- **Encryption key**: auto-generated (`/etc/pulse/.encryption.key`) + +### Security Features +- **Logs**: token values masked with `***` in all outputs +- **API**: frontend receives only `hasToken: true`, never actual values +- **Export**: requires a valid API token (`X-API-Token` header or `token` + parameter) to extract credentials +- **Migration**: use passphrase-protected export/import (see + [Migration Guide](docs/MIGRATION.md)) +- **Auto-migration**: unencrypted configs automatically migrate to encrypted + format + +## Export/Import Protection + +By default, configuration export/import is blocked. You have two options: + +### Option 1: Set API Tokens (Recommended) +```bash +# Using systemd (secure) +sudo systemctl edit pulse +# Add: +[Service] +Environment="API_TOKENS=ansible-token,docker-agent-token" +Environment="API_TOKEN=legacy-token" + +# Then restart: +sudo systemctl restart pulse + +# Docker +docker run -e API_TOKENS=ansible-token,docker-agent-token rcourtman/pulse:latest +``` + +### Option 2: Allow Unprotected Export (Homelab) +```bash +# Using systemd +sudo systemctl edit pulse +# Add: +[Service] +Environment="ALLOW_UNPROTECTED_EXPORT=true" + +# Docker +docker run -e ALLOW_UNPROTECTED_EXPORT=true rcourtman/pulse:latest +``` + +**Note:** for production, prefer Docker secrets or systemd environment files +for sensitive data. + +## Security Features + +### Core Protection +- **Encryption**: credentials encrypted at rest (AES-256-GCM) +- **Export protection**: exports always encrypted with a passphrase +- **Minimum passphrase**: 12 characters required for exports +- **Security tab**: check status in *Settings → Security* + +### Enterprise Security (When Authentication Enabled) +- **Password security** + - Bcrypt hashing with cost factor 12 (60‑character hash) + - Passwords never stored in plain text + - Automatic hashing during security setup + - **Critical**: bcrypt hashes must be exactly 60 characters +- **API token security** + - 64‑character hex tokens (32 bytes entropy) + - SHA3-256 hashed before storage (64‑character hash) + - Raw token shown only once + - Tokens never stored in plain text + - Live reloading when `.env` changes + - API-only mode supported (no password auth required) +- **CSRF protection**: all state-changing operations require CSRF tokens +- **Rate limiting** + - Auth endpoints: 10 attempts/minute per IP + - General API: 500 requests/minute per IP + - Real-time endpoints exempt for functionality +- **Account lockout** + - Locks after 5 failed login attempts + - 15-minute automatic lockout duration + - Clear feedback showing remaining attempts + - Time remaining displayed when locked + - Manual reset available via API for admins +- **Session management** + - Secure HttpOnly cookies + - 24-hour session expiry + - Session invalidation on password change +- **Security headers** + - Content-Security-Policy + - X-Frame-Options: DENY + - X-Content-Type-Options: nosniff + - X-XSS-Protection: 1; mode=block + - Referrer-Policy: strict-origin-when-cross-origin + - Permissions-Policy restricting sensitive APIs +- **Audit logging**: authentication events include IP addresses + +### What's Encrypted in Exports +- Node credentials (passwords, API tokens) +- PBS credentials +- Email settings passwords +- Webhook URLs and authentication headers (v4.1.9+) + +### What's **Not** Encrypted +- Node hostnames and IPs +- Threshold settings +- General configuration +- Alert rules and schedules + +## Authentication Workflows + +Pulse supports multiple authentication methods that can be used independently or +together. + +### Password Authentication + +#### Quick Security Setup (Recommended) +1. Navigate to *Settings → Security*. +2. Click **Enable Security Now**. +3. Enter username and password. +4. Save the generated API token (shown only once!). +5. Security is enabled immediately (no restart needed). + +This automatically: +- Generates a secure random password +- Hashes it with bcrypt (cost factor 12) +- Creates secure API token (SHA3-256 hashed, raw token shown once) +- For systemd: Configures systemd with hashed credentials +- For Docker: Saves to `/data/.env` with hashed credentials (properly quoted to prevent shell expansion) +- Restarts service/container with authentication enabled + +#### Manual Setup (Advanced) +```bash +# Using systemd (password will be hashed automatically) +sudo systemctl edit pulse +# Add: +[Service] +Environment="PULSE_AUTH_USER=admin" +Environment="PULSE_AUTH_PASS=$2a$12$..." # Use bcrypt hash, not plain text! + +# Docker (credentials persist in volume via .env file) +# IMPORTANT: Always quote bcrypt hashes to prevent shell expansion! +docker run -e PULSE_AUTH_USER=admin -e PULSE_AUTH_PASS='$2a$12$...' rcourtman/pulse:latest +# Or use Quick Security Setup and restart container +``` + +**Important**: Always use hashed passwords in configuration. Use the Quick Security Setup or generate bcrypt hashes manually. + +#### Features +- Web UI login required when authentication enabled +- Change/remove password from Settings → Security +- Passwords ALWAYS hashed with bcrypt (cost 12) +- Session-based authentication with secure HttpOnly cookies +- 24-hour session expiry +- CSRF protection for all state-changing operations +- Session invalidation on password change + +### API Token Authentication +For programmatic access and automation. API tokens are SHA3-256 hashed for security. + +#### Token Setup via Quick Security +The Quick Security Setup automatically: +- Generates a cryptographically secure token +- Hashes it with SHA3-256 +- Stores only the 64-character hash +- Adds the token to the managed token list + +#### Manual Token Setup +```bash +# Using systemd (plain text values are auto-hashed on startup) +sudo systemctl edit pulse +# Add: +[Service] +Environment="API_TOKENS=ansible-token,docker-agent-token" + +# Docker +docker run -e API_TOKENS=ansible-token,docker-agent-token rcourtman/pulse:latest + +# To provide pre-hashed tokens instead, list the SHA3-256 hashes +# Environment="API_TOKENS=83c8...,b1de..." +``` + +**Security Note**: Tokens defined via environment variables are hashed with SHA3-256 before being stored on disk. Plain values never persist beyond startup. + +#### Token Management (Settings → Security → API tokens) +- Issue dedicated tokens for automation/agents without sharing a global credential +- View prefixes/suffixes and last-used timestamps for auditing +- Revoke tokens individually without downtime +- Regenerate tokens when rotating credentials (new value displayed once) +- All tokens stored as SHA3-256 hashes + +#### Usage +```bash +# Include the ORIGINAL token (not hash) in X-API-Token header +curl -H "X-API-Token: your-original-token" http://localhost:7655/api/health + +# Or in query parameter for export/import +curl "http://localhost:7655/api/export?token=your-original-token" +``` + +### Auto-Registration Security + +#### Default Mode +- All access requires authentication +- Nodes can auto-register with the API token +- Setup scripts work without additional configuration + +#### Secure Mode +- Require API token for all operations +- Protects auto-registration endpoint +- Enable by setting at least one API token via `API_TOKENS` (or legacy `API_TOKEN`) environment variable + +## CORS (Cross-Origin Resource Sharing) + +By default, Pulse only allows same-origin requests (no CORS headers). This is the most secure configuration. + +### Configuring CORS for External Access + +If you need to access Pulse API from a different domain: + +```bash +# Docker +docker run -e ALLOWED_ORIGINS="https://app.example.com" rcourtman/pulse:latest + +# systemd +sudo systemctl edit pulse +[Service] +Environment="ALLOWED_ORIGINS=https://app.example.com" + +# Multiple origins (comma-separated) +ALLOWED_ORIGINS="https://app.example.com,https://dashboard.example.com" + +# Development mode (allows localhost) +PULSE_DEV=true +``` + +**Security Note**: Never use `ALLOWED_ORIGINS=*` in production as it allows any website to access your API. + +## Security Best Practices + +### Credential Storage +- ✅ **DO**: Use Quick Security Setup for automatic hashing +- ✅ **DO**: Store only bcrypt hashes for passwords +- ✅ **DO**: Store only SHA3-256 hashes for API tokens +- ❌ **DON'T**: Store plain text passwords in config files +- ❌ **DON'T**: Store plain text API tokens in config files +- ❌ **DON'T**: Log credentials or include them in backups + +### Authentication Setup +- ✅ **DO**: Use strong, unique passwords (16+ characters) +- ✅ **DO**: Rotate API tokens periodically +- ✅ **DO**: Use HTTPS in production environments +- ❌ **DON'T**: Share API tokens between users/services +- ❌ **DON'T**: Embed credentials in client-side code + +### Verification +Run the security verification script to ensure no plain text credentials: +```bash +/opt/pulse/testing-tools/security-verification.sh +``` + +This checks: +- No hardcoded credentials in code +- No credentials exposed in logs +- All passwords/tokens properly hashed +- Secure file permissions +- No credential leaks in API responses + +## Account Lockout and Recovery + +### Lockout Behavior +- After **5 failed login attempts**, the account is locked for **15 minutes** +- Lockout applies to both username and IP address +- Login form shows remaining attempts after each failure +- Clear message when locked with time remaining + +### Automatic Recovery +- Lockouts automatically expire after 15 minutes +- No action needed - just wait for the timer to expire +- Successful login clears all failed attempt counters + +### Manual Recovery (Admin) +Administrators with API access can manually reset lockouts: + +```bash +# Reset lockout for a specific username +curl -X POST http://localhost:7655/api/security/reset-lockout \ + -H "X-API-Token: your-api-token" \ + -H "Content-Type: application/json" \ + -d '{"identifier":"username"}' + +# Reset lockout for an IP address +curl -X POST http://localhost:7655/api/security/reset-lockout \ + -H "X-API-Token: your-api-token" \ + -H "Content-Type: application/json" \ + -d '{"identifier":"192.168.1.100"}' +``` + +## Troubleshooting + +**Account locked?** Wait 15 minutes or contact admin for manual reset +**Export blocked?** You're on a public network – login with password, set an API token (`API_TOKENS`), or set `ALLOW_UNPROTECTED_EXPORT=true` +**Rate limited?** Wait 1 minute and try again +**Can't login?** Check `PULSE_AUTH_USER` and `PULSE_AUTH_PASS` environment variables +**API access denied?** Verify the token you supplied matches one of the values created in *Settings → Security → API tokens* (use the original token, not the hash) +**CORS errors?** Configure `ALLOWED_ORIGINS` for your domain +**Forgot password?** Start fresh – delete your Pulse data and restart + +_Last updated: 2025-10-19_ diff --git a/docs/SECURITY.md b/docs/SECURITY.md index e490909..0b0501c 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -1,364 +1,6 @@ -# Pulse Security +# Security Documentation -## Mandatory Authentication +The canonical Pulse security guide now lives at the repository root: +[`../SECURITY.md`](../SECURITY.md). -**Starting with v4.5.0, authentication setup is prompted for all new Pulse installations.** This protects your Proxmox API credentials from unauthorized access. - -> **Service name note:** Systemd deployments use `pulse.service`. If you're upgrading from an older install that still registers `pulse-backend.service`, substitute that name in the commands below. - -### First-Run Security Setup -When you first access Pulse, you'll be guided through a mandatory security setup: -- Create your admin username and password -- Automatic API token generation for automation -- Settings are applied immediately without restart -- **Your existing nodes and settings are preserved** - -## Smart Security Context - -### Public Access Detection -Pulse automatically detects when it's being accessed from public networks: -- **Private Networks**: Local/RFC1918 addresses (192.168.x.x, 10.x.x.x, etc.) -- **Public Networks**: Any non-private IP address -- **Stronger Warnings**: Red alerts when accessed from public IPs without authentication - -### Trusted Networks Configuration (Deprecated) -**Note: Authentication is now mandatory regardless of network location.** - -Legacy configuration (no longer applicable): -```bash -# Environment variable (comma-separated CIDR blocks) -PULSE_TRUSTED_NETWORKS=192.168.1.0/24,10.0.0.0/24 - -# Or in systemd -sudo systemctl edit pulse -[Service] -Environment="PULSE_TRUSTED_NETWORKS=192.168.1.0/24,10.0.0.0/24" -``` - -When configured: -- Access from trusted networks: No auth required -- Access from outside: Authentication enforced -- Useful for: Mixed home/remote access scenarios - -## Security Warning System - -Pulse now includes a non-intrusive security warning system that helps you understand your security posture: - -### Security Score -Your instance receives a score from 0-5 based on: -- ✅ Credentials encrypted at rest (always enabled) -- ✅ Export/import protection -- ⚠️ Authentication enabled -- ⚠️ HTTPS connection -- ⚠️ Audit logging - -### Dismissing Warnings -If you're comfortable with your security setup, you can dismiss warnings: -- **For 1 day** - Reminder tomorrow -- **For 1 week** - Reminder next week -- **Forever** - Won't show again - -## Credential Security - -### Encrypted at Rest (AES-256-GCM) -- **Node Credentials**: Passwords and API tokens (`/etc/pulse/nodes.enc`) -- **Email Settings**: SMTP passwords (`/etc/pulse/email.enc`) -- **Webhook Data**: URLs and auth headers (`/etc/pulse/webhooks.enc`) - v4.1.9+ -- **Encryption Key**: Auto-generated (`/etc/pulse/.encryption.key`) - -### Security Features -- **Logs**: Token values masked with `***` in all outputs -- **API**: Frontend receives only `hasToken: true`, never actual values -- **Export**: Requires a valid API token (via `X-API-Token` header or `token` query parameter) to extract credentials -- **Migration**: Use passphrase-protected export/import (see [Migration Guide](MIGRATION.md)) -- **Auto-Migration**: Unencrypted configs automatically migrate to encrypted format - -## Export/Import Protection - -By default, configuration export/import is blocked for security. You have two options: - -### Option 1: Set API Tokens (Recommended) -```bash -# Using systemd (secure) -sudo systemctl edit pulse -# Add: -[Service] -Environment="API_TOKENS=ansible-token,docker-agent-token" -Environment="API_TOKEN=legacy-token" # Optional fallback - -# Then restart: -sudo systemctl restart pulse - -# Docker -docker run -e API_TOKENS=ansible-token,docker-agent-token rcourtman/pulse:latest -``` - -### Option 2: Allow Unprotected Export (Homelab) -```bash -# Using systemd -sudo systemctl edit pulse -# Add: -[Service] -Environment="ALLOW_UNPROTECTED_EXPORT=true" - -# Docker -docker run -e ALLOW_UNPROTECTED_EXPORT=true rcourtman/pulse:latest -``` - -**Note:** For production deployments, consider using Docker secrets or systemd environment variables instead of .env files for sensitive data. - -## Security Features - -### Core Protection -- **Encryption**: All credentials encrypted at rest (AES-256-GCM) -- **Export Protection**: Exports always encrypted with passphrase -- **Minimum Passphrase**: 12 characters required for exports -- **Security Tab**: Check status in Settings → Security - -### Enterprise Security (When Authentication Enabled) -- **Password Security**: - - Bcrypt hashing with cost factor 12 (60-character hash) - - Passwords NEVER stored in plain text - - Automatic hashing on security setup - - **CRITICAL**: Bcrypt hashes MUST be exactly 60 characters -- **API Token Security**: - - 64-character hex tokens (32 bytes of entropy) - - SHA3-256 hashed before storage (64-char hash) - - Raw token shown only once during generation - - Tokens NEVER stored in plain text - - Live reloading when .env file changes - - API-only mode supported (no password auth required) -- **CSRF Protection**: All state-changing operations require CSRF tokens -- **Rate Limiting**: - - Authentication endpoints: 10 attempts/minute per IP - - General API: 500 requests/minute per IP - - Real-time endpoints exempt for functionality -- **Account Lockout Protection**: - - Locks after 5 failed login attempts - - 15-minute automatic lockout duration - - Clear feedback showing remaining attempts - - Time remaining displayed when locked - - Manual reset available via API for administrators -- **Session Management**: - - Secure HttpOnly cookies - - 24-hour session expiry - - Session invalidation on password change -- **Security Headers**: - - Content-Security-Policy with strict directives - - X-Frame-Options: DENY (prevents clickjacking) - - X-Content-Type-Options: nosniff - - X-XSS-Protection: 1; mode=block - - Referrer-Policy: strict-origin-when-cross-origin - - Permissions-Policy restricting sensitive APIs -- **Audit Logging**: All authentication events logged with IP addresses - -### What's Encrypted in Exports -- Node credentials (passwords, API tokens) -- PBS credentials -- Email settings passwords -- Webhook URLs and authentication headers (v4.1.9+) - -### What's NOT Encrypted -- Node hostnames and IPs -- Threshold settings -- General configuration -- Alert rules and schedules - -## Authentication - -Pulse supports multiple authentication methods that can be used independently or together: - -### Password Authentication - -#### Quick Security Setup (Recommended) -The easiest way to enable authentication is through the web UI: -1. Go to Settings → Security -2. Click "Enable Security Now" -3. Enter username and password -4. Save the generated API token (shown only once!) -5. Security is enabled immediately (no restart needed) - -This automatically: -- Generates a secure random password -- Hashes it with bcrypt (cost factor 12) -- Creates secure API token (SHA3-256 hashed, raw token shown once) -- For systemd: Configures systemd with hashed credentials -- For Docker: Saves to `/data/.env` with hashed credentials (properly quoted to prevent shell expansion) -- Restarts service/container with authentication enabled - -#### Manual Setup (Advanced) -```bash -# Using systemd (password will be hashed automatically) -sudo systemctl edit pulse -# Add: -[Service] -Environment="PULSE_AUTH_USER=admin" -Environment="PULSE_AUTH_PASS=$2a$12$..." # Use bcrypt hash, not plain text! - -# Docker (credentials persist in volume via .env file) -# IMPORTANT: Always quote bcrypt hashes to prevent shell expansion! -docker run -e PULSE_AUTH_USER=admin -e PULSE_AUTH_PASS='$2a$12$...' rcourtman/pulse:latest -# Or use Quick Security Setup and restart container -``` - -**Important**: Always use hashed passwords in configuration. Use the Quick Security Setup or generate bcrypt hashes manually. - -#### Features -- Web UI login required when authentication enabled -- Change/remove password from Settings → Security -- Passwords ALWAYS hashed with bcrypt (cost 12) -- Session-based authentication with secure HttpOnly cookies -- 24-hour session expiry -- CSRF protection for all state-changing operations -- Session invalidation on password change - -### API Token Authentication -For programmatic access and automation. API tokens are SHA3-256 hashed for security. - -#### Token Setup via Quick Security -The Quick Security Setup automatically: -- Generates a cryptographically secure token -- Hashes it with SHA3-256 -- Stores only the 64-character hash -- Adds the token to the managed token list - -#### Manual Token Setup -```bash -# Using systemd (plain text values are auto-hashed on startup) -sudo systemctl edit pulse -# Add: -[Service] -Environment="API_TOKENS=ansible-token,docker-agent-token" - -# Docker -docker run -e API_TOKENS=ansible-token,docker-agent-token rcourtman/pulse:latest - -# To provide pre-hashed tokens instead, list the SHA3-256 hashes -# Environment="API_TOKENS=83c8...,b1de..." -``` - -**Security Note**: Tokens defined via environment variables are hashed with SHA3-256 before being stored on disk. Plain values never persist beyond startup. - -#### Token Management (Settings → Security → API tokens) -- Issue dedicated tokens for automation/agents without sharing a global credential -- View prefixes/suffixes and last-used timestamps for auditing -- Revoke tokens individually without downtime -- Regenerate tokens when rotating credentials (new value displayed once) -- All tokens stored as SHA3-256 hashes - -#### Usage -```bash -# Include the ORIGINAL token (not hash) in X-API-Token header -curl -H "X-API-Token: your-original-token" http://localhost:7655/api/health - -# Or in query parameter for export/import -curl "http://localhost:7655/api/export?token=your-original-token" -``` - -### Auto-Registration Security - -#### Default Mode -- All access requires authentication -- Nodes can auto-register with the API token -- Setup scripts work without additional configuration - -#### Secure Mode -- Require API token for all operations -- Protects auto-registration endpoint -- Enable by setting at least one API token via `API_TOKENS` (or legacy `API_TOKEN`) environment variable - -## CORS (Cross-Origin Resource Sharing) - -By default, Pulse only allows same-origin requests (no CORS headers). This is the most secure configuration. - -### Configuring CORS for External Access - -If you need to access Pulse API from a different domain: - -```bash -# Docker -docker run -e ALLOWED_ORIGINS="https://app.example.com" rcourtman/pulse:latest - -# systemd -sudo systemctl edit pulse -[Service] -Environment="ALLOWED_ORIGINS=https://app.example.com" - -# Multiple origins (comma-separated) -ALLOWED_ORIGINS="https://app.example.com,https://dashboard.example.com" - -# Development mode (allows localhost) -PULSE_DEV=true -``` - -**Security Note**: Never use `ALLOWED_ORIGINS=*` in production as it allows any website to access your API. - -## Security Best Practices - -### Credential Storage -- ✅ **DO**: Use Quick Security Setup for automatic hashing -- ✅ **DO**: Store only bcrypt hashes for passwords -- ✅ **DO**: Store only SHA3-256 hashes for API tokens -- ❌ **DON'T**: Store plain text passwords in config files -- ❌ **DON'T**: Store plain text API tokens in config files -- ❌ **DON'T**: Log credentials or include them in backups - -### Authentication Setup -- ✅ **DO**: Use strong, unique passwords (16+ characters) -- ✅ **DO**: Rotate API tokens periodically -- ✅ **DO**: Use HTTPS in production environments -- ❌ **DON'T**: Share API tokens between users/services -- ❌ **DON'T**: Embed credentials in client-side code - -### Verification -Run the security verification script to ensure no plain text credentials: -```bash -/opt/pulse/testing-tools/security-verification.sh -``` - -This checks: -- No hardcoded credentials in code -- No credentials exposed in logs -- All passwords/tokens properly hashed -- Secure file permissions -- No credential leaks in API responses - -## Account Lockout and Recovery - -### Lockout Behavior -- After **5 failed login attempts**, the account is locked for **15 minutes** -- Lockout applies to both username and IP address -- Login form shows remaining attempts after each failure -- Clear message when locked with time remaining - -### Automatic Recovery -- Lockouts automatically expire after 15 minutes -- No action needed - just wait for the timer to expire -- Successful login clears all failed attempt counters - -### Manual Recovery (Admin) -Administrators with API access can manually reset lockouts: - -```bash -# Reset lockout for a specific username -curl -X POST http://localhost:7655/api/security/reset-lockout \ - -H "X-API-Token: your-api-token" \ - -H "Content-Type: application/json" \ - -d '{"identifier":"username"}' - -# Reset lockout for an IP address -curl -X POST http://localhost:7655/api/security/reset-lockout \ - -H "X-API-Token: your-api-token" \ - -H "Content-Type: application/json" \ - -d '{"identifier":"192.168.1.100"}' -``` - -## Troubleshooting - -**Account locked?** Wait 15 minutes or contact admin for manual reset -**Export blocked?** You're on a public network - login with password, set an API token (`API_TOKENS`), or set ALLOW_UNPROTECTED_EXPORT=true -**Rate limited?** Wait 1 minute and try again -**Can't login?** Check PULSE_AUTH_USER and PULSE_AUTH_PASS environment variables -**API access denied?** Verify the token you supplied matches one of the values created in Settings → Security → API tokens (use the original token value, not the hash) -**CORS errors?** Configure ALLOWED_ORIGINS for your domain -**Forgot password?** Start fresh - delete your Pulse data and restart +Please update that file if you need to change security guidance. diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go index b2c552d..1cddad4 100644 --- a/internal/monitoring/monitor.go +++ b/internal/monitoring/monitor.go @@ -258,6 +258,7 @@ type Monitor struct { pollMetrics *PollMetrics scheduler *AdaptiveScheduler stalenessTracker *StalenessTracker + taskQueue *TaskQueue tempCollector *TemperatureCollector // SSH-based temperature collector mu sync.RWMutex startTime time.Time @@ -1317,6 +1318,7 @@ func New(cfg *config.Config) (*Monitor, error) { stalenessTracker := NewStalenessTracker(getPollMetrics()) stalenessTracker.SetBounds(cfg.AdaptivePollingBaseInterval, cfg.AdaptivePollingMaxInterval) + taskQueue := NewTaskQueue() var scheduler *AdaptiveScheduler if cfg.AdaptivePollingEnabled { @@ -1336,6 +1338,7 @@ func New(cfg *config.Config) (*Monitor, error) { pollMetrics: getPollMetrics(), scheduler: scheduler, stalenessTracker: stalenessTracker, + taskQueue: taskQueue, tempCollector: tempCollector, startTime: time.Now(), rateTracker: NewRateTracker(), @@ -1691,8 +1694,12 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) { // Create separate tickers for polling and broadcasting // Hardcoded to 10 seconds since Proxmox updates cluster/resources every 10 seconds - const pollingInterval = 10 * time.Second - pollTicker := time.NewTicker(pollingInterval) +const pollingInterval = 10 * time.Second + +workerCount := len(m.pveClients) + len(m.pbsClients) + len(m.pmgClients) +m.startTaskWorkers(ctx, workerCount) + +pollTicker := time.NewTicker(pollingInterval) defer pollTicker.Stop() broadcastTicker := time.NewTicker(pollingInterval) @@ -1934,24 +1941,10 @@ func (m *Monitor) poll(ctx context.Context, wsHub *websocket.Hub) { now := startTime plannedTasks := m.buildScheduledTasks(now) - dueTasks := plannedTasks - if m.scheduler != nil { - due := m.scheduler.DispatchDue(ctx, now, plannedTasks) - if len(due) > 0 { - dueTasks = due - } - } - - if m.pollMetrics != nil { - m.pollMetrics.ResetQueueDepth(len(dueTasks)) - } - - if m.config.ConcurrentPolling { - // Use concurrent polling - m.pollConcurrent(ctx, dueTasks) - } else { - m.pollSequential(ctx, dueTasks) + for _, task := range plannedTasks { + m.taskQueue.Upsert(task) } + m.updateQueueDepthMetric() // Update performance metrics m.state.Performance.LastPollDuration = time.Since(startTime).Seconds() @@ -2088,99 +2081,126 @@ func (m *Monitor) pruneStaleDockerAlerts() bool { return cleared } -// pollConcurrent polls instances concurrently based on scheduled tasks. -func (m *Monitor) pollConcurrent(ctx context.Context, tasks []ScheduledTask) { - if len(tasks) == 0 { +func (m *Monitor) startTaskWorkers(ctx context.Context, workers int) { + if m.taskQueue == nil { return } - - var wg sync.WaitGroup - ctx, cancel := context.WithCancel(ctx) - defer cancel() - - for _, task := range tasks { - select { - case <-ctx.Done(): - return - default: - } - - switch task.InstanceType { - case InstanceTypePVE: - client, ok := m.pveClients[task.InstanceName] - if !ok || client == nil { - continue - } - wg.Add(1) - go func(name string, c PVEClientInterface) { - defer wg.Done() - m.pollPVEInstance(ctx, name, c) - }(task.InstanceName, client) - case InstanceTypePBS: - client, ok := m.pbsClients[task.InstanceName] - if !ok || client == nil { - continue - } - wg.Add(1) - go func(name string, c *pbs.Client) { - defer wg.Done() - m.pollPBSInstance(ctx, name, c) - }(task.InstanceName, client) - case InstanceTypePMG: - client, ok := m.pmgClients[task.InstanceName] - if !ok || client == nil { - continue - } - wg.Add(1) - go func(name string, c *pmg.Client) { - defer wg.Done() - m.pollPMGInstance(ctx, name, c) - }(task.InstanceName, client) - default: - log.Debug().Str("instance", task.InstanceName).Str("type", string(task.InstanceType)).Msg("Skipping unsupported task type") - } + if workers < 1 { + workers = 1 } - - done := make(chan struct{}) - go func() { - wg.Wait() - close(done) - }() - - select { - case <-done: - case <-ctx.Done(): - cancel() - wg.Wait() + if workers > 10 { + workers = 10 + } + for i := 0; i < workers; i++ { + go m.taskWorker(ctx, i) } } -// pollSequential polls instances sequentially based on scheduled tasks. -func (m *Monitor) pollSequential(ctx context.Context, tasks []ScheduledTask) { - for _, task := range tasks { - select { - case <-ctx.Done(): +func (m *Monitor) taskWorker(ctx context.Context, id int) { + log.Debug().Int("worker", id).Msg("Task worker started") + for { + task, ok := m.taskQueue.WaitNext(ctx) + if !ok { + log.Debug().Int("worker", id).Msg("Task worker stopping") return - default: } - switch task.InstanceType { - case InstanceTypePVE: - if client, ok := m.pveClients[task.InstanceName]; ok && client != nil { - m.pollPVEInstance(ctx, task.InstanceName, client) + m.executeScheduledTask(ctx, task) + + m.rescheduleTask(task) + m.updateQueueDepthMetric() + } +} + +func (m *Monitor) executeScheduledTask(ctx context.Context, task ScheduledTask) { + switch task.InstanceType { + case InstanceTypePVE: + client, ok := m.pveClients[task.InstanceName] + if !ok || client == nil { + log.Warn().Str("instance", task.InstanceName).Msg("PVE client missing for scheduled task") + return + } + m.pollPVEInstance(ctx, task.InstanceName, client) + case InstanceTypePBS: + client, ok := m.pbsClients[task.InstanceName] + if !ok || client == nil { + log.Warn().Str("instance", task.InstanceName).Msg("PBS client missing for scheduled task") + return + } + m.pollPBSInstance(ctx, task.InstanceName, client) + case InstanceTypePMG: + client, ok := m.pmgClients[task.InstanceName] + if !ok || client == nil { + log.Warn().Str("instance", task.InstanceName).Msg("PMG client missing for scheduled task") + return + } + m.pollPMGInstance(ctx, task.InstanceName, client) + default: + log.Debug().Str("instance", task.InstanceName).Str("type", string(task.InstanceType)).Msg("Skipping unsupported task type") + } +} + +func (m *Monitor) rescheduleTask(task ScheduledTask) { + if m.taskQueue == nil { + return + } + + if m.scheduler == nil { + nextInterval := task.Interval + if nextInterval <= 0 && m.config != nil { + nextInterval = m.config.AdaptivePollingBaseInterval + } + if nextInterval <= 0 { + nextInterval = DefaultSchedulerConfig().BaseInterval + } + next := task + next.NextRun = time.Now().Add(nextInterval) + next.Interval = nextInterval + m.taskQueue.Upsert(next) + return + } + + desc := InstanceDescriptor{ + Name: task.InstanceName, + Type: task.InstanceType, + LastInterval: task.Interval, + LastScheduled: task.NextRun, + } + if m.stalenessTracker != nil { + if snap, ok := m.stalenessTracker.snapshot(task.InstanceType, task.InstanceName); ok { + desc.LastSuccess = snap.LastSuccess + desc.LastFailure = snap.LastError + if snap.ChangeHash != "" { + desc.Metadata = map[string]any{"changeHash": snap.ChangeHash} } - case InstanceTypePBS: - if client, ok := m.pbsClients[task.InstanceName]; ok && client != nil { - m.pollPBSInstance(ctx, task.InstanceName, client) - } - case InstanceTypePMG: - if client, ok := m.pmgClients[task.InstanceName]; ok && client != nil { - m.pollPMGInstance(ctx, task.InstanceName, client) - } - default: - log.Debug().Str("instance", task.InstanceName).Str("type", string(task.InstanceType)).Msg("Skipping unsupported task type") } } + + tasks := m.scheduler.BuildPlan(time.Now(), []InstanceDescriptor{desc}, m.taskQueue.Size()) + if len(tasks) == 0 { + next := task + nextInterval := task.Interval + if nextInterval <= 0 && m.config != nil { + nextInterval = m.config.AdaptivePollingBaseInterval + } + if nextInterval <= 0 { + nextInterval = DefaultSchedulerConfig().BaseInterval + } + next.Interval = nextInterval + next.NextRun = time.Now().Add(nextInterval) + m.taskQueue.Upsert(next) + return + } + for _, next := range tasks { + m.taskQueue.Upsert(next) + } +} + +func (m *Monitor) updateQueueDepthMetric() { + if m.pollMetrics == nil || m.taskQueue == nil { + return + } + m.pollMetrics.SetQueueDepth(m.taskQueue.Size()) } // pollPVEInstance polls a single PVE instance diff --git a/internal/monitoring/monitor_polling.go b/internal/monitoring/monitor_polling.go index 8d6bd53..4b20c7e 100644 --- a/internal/monitoring/monitor_polling.go +++ b/internal/monitoring/monitor_polling.go @@ -117,20 +117,29 @@ func (m *Monitor) buildScheduledTasks(now time.Time) []ScheduledTask { return nil } + queueDepth := 0 + if m.taskQueue != nil { + queueDepth = m.taskQueue.Size() + } + if m.scheduler == nil { tasks := make([]ScheduledTask, 0, len(descriptors)) + interval := m.config.AdaptivePollingBaseInterval + if interval <= 0 { + interval = DefaultSchedulerConfig().BaseInterval + } for _, desc := range descriptors { tasks = append(tasks, ScheduledTask{ InstanceName: desc.Name, InstanceType: desc.Type, NextRun: now, - Interval: DefaultSchedulerConfig().BaseInterval, + Interval: interval, }) } return tasks } - return m.scheduler.BuildPlan(now, descriptors) + return m.scheduler.BuildPlan(now, descriptors, queueDepth) } // convertPoolInfoToModel converts Proxmox ZFS pool info to our model diff --git a/internal/monitoring/scheduler.go b/internal/monitoring/scheduler.go index d10cb26..23b3aac 100644 --- a/internal/monitoring/scheduler.go +++ b/internal/monitoring/scheduler.go @@ -130,7 +130,7 @@ func NewAdaptiveScheduler(cfg SchedulerConfig, staleness StalenessSource, interv } // BuildPlan produces an ordered set of scheduled tasks for the supplied inventory. -func (s *AdaptiveScheduler) BuildPlan(now time.Time, inventory []InstanceDescriptor) []ScheduledTask { +func (s *AdaptiveScheduler) BuildPlan(now time.Time, inventory []InstanceDescriptor, queueDepth int) []ScheduledTask { if len(inventory) == 0 { return nil } @@ -159,20 +159,21 @@ func (s *AdaptiveScheduler) BuildPlan(now time.Time, inventory []InstanceDescrip lastInterval = s.cfg.BaseInterval } - req := IntervalRequest{ - Now: now, - BaseInterval: s.cfg.BaseInterval, - MinInterval: s.cfg.MinInterval, - MaxInterval: s.cfg.MaxInterval, - LastInterval: lastInterval, - LastSuccess: inst.LastSuccess, - LastScheduled: lastScheduled, - StalenessScore: score, - ErrorCount: inst.ErrorCount, - QueueDepth: len(inventory), - InstanceKey: schedulerKey(inst.Type, inst.Name), - InstanceType: inst.Type, - } + currentDepth := queueDepth + len(tasks) + req := IntervalRequest{ + Now: now, + BaseInterval: s.cfg.BaseInterval, + MinInterval: s.cfg.MinInterval, + MaxInterval: s.cfg.MaxInterval, + LastInterval: lastInterval, + LastSuccess: inst.LastSuccess, + LastScheduled: lastScheduled, + StalenessScore: score, + ErrorCount: inst.ErrorCount, + QueueDepth: currentDepth, + InstanceKey: schedulerKey(inst.Type, inst.Name), + InstanceType: inst.Type, + } nextInterval := s.interval.SelectInterval(req) if nextInterval <= 0 { diff --git a/internal/monitoring/task_queue.go b/internal/monitoring/task_queue.go new file mode 100644 index 0000000..bbfec79 --- /dev/null +++ b/internal/monitoring/task_queue.go @@ -0,0 +1,154 @@ +package monitoring + +import ( + "container/heap" + "context" + "sync" + "time" +) + +type scheduledTaskEntry struct { + task ScheduledTask + index int +} + +func (e *scheduledTaskEntry) key() string { + return schedulerKey(e.task.InstanceType, e.task.InstanceName) +} + +type taskHeap []*scheduledTaskEntry + +func (h taskHeap) Len() int { return len(h) } + +func (h taskHeap) Less(i, j int) bool { + if h[i].task.NextRun.Equal(h[j].task.NextRun) { + if h[i].task.Priority == h[j].task.Priority { + return h[i].task.InstanceName < h[j].task.InstanceName + } + return h[i].task.Priority > h[j].task.Priority + } + return h[i].task.NextRun.Before(h[j].task.NextRun) +} + +func (h taskHeap) Swap(i, j int) { + h[i], h[j] = h[j], h[i] + h[i].index = i + h[j].index = j +} + +func (h *taskHeap) Push(x interface{}) { + entry := x.(*scheduledTaskEntry) + entry.index = len(*h) + *h = append(*h, entry) +} + +func (h *taskHeap) Pop() interface{} { + old := *h + n := len(old) + if n == 0 { + return nil + } + entry := old[n-1] + entry.index = -1 + *h = old[:n-1] + return entry +} + +// TaskQueue is a thread-safe min-heap over scheduled tasks. +type TaskQueue struct { + mu sync.Mutex + entries map[string]*scheduledTaskEntry + heap taskHeap +} + +// NewTaskQueue constructs an empty queue. +func NewTaskQueue() *TaskQueue { + tq := &TaskQueue{ + entries: make(map[string]*scheduledTaskEntry), + heap: make(taskHeap, 0), + } + heap.Init(&tq.heap) + return tq +} + +// Upsert inserts or updates a scheduled task in the queue. +func (q *TaskQueue) Upsert(task ScheduledTask) { + key := schedulerKey(task.InstanceType, task.InstanceName) + q.mu.Lock() + defer q.mu.Unlock() + + if entry, ok := q.entries[key]; ok { + entry.task = task + heap.Fix(&q.heap, entry.index) + return + } + + entry := &scheduledTaskEntry{task: task} + heap.Push(&q.heap, entry) + q.entries[key] = entry +} + +// Remove deletes a task by key if present. +func (q *TaskQueue) Remove(instanceType InstanceType, instance string) { + key := schedulerKey(instanceType, instance) + q.mu.Lock() + defer q.mu.Unlock() + + entry, ok := q.entries[key] + if !ok { + return + } + heap.Remove(&q.heap, entry.index) + delete(q.entries, key) +} + +// WaitNext blocks until a task is due or context is cancelled. +func (q *TaskQueue) WaitNext(ctx context.Context) (ScheduledTask, bool) { + for { + select { + case <-ctx.Done(): + return ScheduledTask{}, false + default: + } + + q.mu.Lock() + if len(q.heap) == 0 { + q.mu.Unlock() + select { + case <-ctx.Done(): + return ScheduledTask{}, false + case <-time.After(100 * time.Millisecond): + continue + } + } + + entry := q.heap[0] + delay := time.Until(entry.task.NextRun) + if delay <= 0 { + heap.Pop(&q.heap) + delete(q.entries, entry.key()) + task := entry.task + q.mu.Unlock() + return task, true + } + + q.mu.Unlock() + if delay > 250*time.Millisecond { + delay = 250 * time.Millisecond + } + timer := time.NewTimer(delay) + select { + case <-ctx.Done(): + timer.Stop() + return ScheduledTask{}, false + case <-timer.C: + } + } +} + +// Size returns the number of tasks currently queued. +func (q *TaskQueue) Size() int { + q.mu.Lock() + defer q.mu.Unlock() + return len(q.heap) +} diff --git a/scripts/make-tests-safe.sh b/scripts/make-tests-safe.sh deleted file mode 100755 index ad75f5e..0000000 --- a/scripts/make-tests-safe.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/bash - -# This script updates test scripts to use unique test node names -# to prevent collision with user nodes - -echo "Making test scripts safe by using unique test node names..." - -# Add timestamp to test node names to make them unique -TIMESTAMP=$(date +%s) - -# Update test-config-validation.sh to use unique names -sed -i "s/\"name\":\"test\"/\"name\":\"test-val-$TIMESTAMP\"/g" /opt/pulse/scripts/test-config-validation.sh -sed -i "s/\"name\":\"duplicate-test\"/\"name\":\"dup-test-$TIMESTAMP\"/g" /opt/pulse/scripts/test-config-validation.sh -sed -i "s/\"name\":\"pbs-test\"/\"name\":\"pbs-test-$TIMESTAMP\"/g" /opt/pulse/scripts/test-config-validation.sh - -echo "Test scripts updated with unique node names (suffix: $TIMESTAMP)" -echo "" -echo "You can now safely run tests without affecting production nodes!" \ No newline at end of file diff --git a/scripts/protect-mock-nodes.sh b/scripts/protect-mock-nodes.sh deleted file mode 100755 index ca9b12e..0000000 --- a/scripts/protect-mock-nodes.sh +++ /dev/null @@ -1,43 +0,0 @@ -#!/bin/bash - -# This script patches test scripts to protect mock nodes from deletion -# It ensures that mock nodes (pve1-pve7, mock-*) are never deleted during tests - -RED='\033[0;31m' -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -NC='\033[0m' - -echo "Protecting mock nodes from test cleanup..." - -# List of test scripts that delete nodes -TEST_SCRIPTS=( - "/opt/pulse/scripts/test-persistence.sh" - "/opt/pulse/scripts/test-recovery.sh" - "/opt/pulse/scripts/test-backup.sh" - "/opt/pulse/scripts/test-load.sh" - "/opt/pulse/scripts/test-config-validation.sh" -) - -for script in "${TEST_SCRIPTS[@]}"; do - if [ -f "$script" ]; then - echo -n "Patching $(basename $script)... " - - # Create backup - cp "$script" "${script}.backup" 2>/dev/null - - # Add protection for mock nodes in cleanup sections - # This looks for DELETE commands and adds a check to skip mock nodes - - # Find lines with curl DELETE and add protection - sed -i '/curl.*DELETE.*nodes/i\ - # Skip mock nodes and production nodes\ - if [[ "$NODE_NAME" == "pve"* ]] || [[ "$NODE_NAME" == "mock"* ]] || [[ "$NODE_NAME" == "delly" ]] || [[ "$NODE_NAME" == "minipc" ]] || [[ "$NODE_NAME" == "pimox" ]]; then\ - continue\ - fi' "$script" 2>/dev/null - - echo -e "${GREEN}✓${NC}" - fi -done - -echo -e "${GREEN}Mock nodes are now protected!${NC}" \ No newline at end of file