Task 8 of 10 complete. Exposes read-only scheduler health data including:
- Queue depth and distribution by instance type
- Dead-letter queue inspection (top 25 tasks with error details)
- Circuit breaker states (instance-level)
- Staleness scores per instance
New API endpoint:
GET /api/monitoring/scheduler/health (requires authentication)
New snapshot methods:
- StalenessTracker.Snapshot() - exports all staleness data
- TaskQueue.Snapshot() - queue depth & per-type distribution
- TaskQueue.PeekAll() - dead-letter task inspection
- circuitBreaker.State() - exports state, failures, retryAt
- Monitor.SchedulerHealth() - aggregates all health data
Documentation updated with API spec, field descriptions, and usage examples.
Replaces immediate polling with queue-based scheduling:
- TaskQueue with min-heap (container/heap) for NextRun-ordered execution
- Worker goroutines that block on WaitNext() until tasks are due
- Tasks only execute when NextRun <= now, respecting adaptive intervals
- Automatic rescheduling after execution via scheduler.BuildPlan
- Queue depth tracking for backpressure-aware interval adjustments
- Upsert semantics for updating scheduled tasks without duplicates
Task 6 of 10 complete (60%). Ready for error/backoff policies.
Confirms adaptive scheduling logic is fully operational:
- EMA smoothing (alpha=0.6) to prevent interval oscillations
- Staleness-based interpolation between min/max intervals
- Error penalty (0.6x per error) for faster recovery detection
- Queue depth stretch (0.1x per task) for backpressure handling
- ±5% jitter to prevent thundering herd effects
- Per-instance state tracking for smooth transitions
Task 5 of 10 complete. Scheduler foundation ready for queue-based execution.
Adds freshness metadata tracking for all monitored instances:
- StalenessTracker with per-instance last success/error/mutation timestamps
- Change hash detection using SHA1 for detecting data mutations
- Normalized staleness scoring (0-1 scale) based on age vs maxStale
- Integration with PollMetrics for authoritative last-success data
- Wired into all poll functions (PVE/PBS/PMG) via UpdateSuccess/UpdateError
- Connected to scheduler as StalenessSource implementation
Task 4 of 10 complete. Ready for adaptive interval logic.
Implements all remaining Codex recommendations before launch:
1. Privileged Methods Tests:
- TestPrivilegedMethodsCompleteness ensures all host-side RPCs are protected
- Will fail if new privileged RPC is added without authorization
- Verifies read-only methods are NOT in privilegedMethods
2. ID-Mapped Root Detection Tests:
- TestIDMappedRootDetection covers all boundary conditions
- Tests UID/GID range detection (both must be in range)
- Tests multiple ID ranges, edge cases, disabled mode
- 100% coverage of container identification logic
3. Authorization Tests:
- TestPrivilegedMethodsBlocked verifies containers can't call privileged RPCs
- TestIDMappedRootDisabled ensures feature can be disabled
- Tests both container and host credentials
4. Comprehensive Security Documentation (23 KB):
- Architecture overview with diagrams
- Complete authentication & authorization flow
- Rate limiting details (already implemented: 20/min per peer)
- SSH security model and forced commands
- Container isolation mechanisms
- Monitoring & alerting recommendations
- Development mode documentation (PULSE_DEV_ALLOW_CONTAINER_SSH)
- Troubleshooting guide with common issues
- Incident response procedures
Rate Limiting Status:
- Already implemented in throttle.go (20 req/min, burst 10, max 10 concurrent)
- Per-peer rate limiting at line 328 in main.go
- Per-node concurrency control at line 825 in main.go
- Exceeds Codex's requirements
All tests pass. Documentation covers all security aspects.
Addresses final Codex recommendations for production readiness.
RELEASE BLOCKER FIX - Prevents containers from triggering host-level operations.
Added host-only method restrictions:
- RPCEnsureClusterKeys (SSH key distribution)
- RPCRegisterNodes (node registration)
- RPCRequestCleanup (cleanup operations)
Implementation:
- New privilegedMethods map defines host-only methods
- Request handler checks if method is privileged
- If privileged AND caller is from ID-mapped UID range (container), reject
- Host processes (real root, configured UIDs) can still call privileged methods
- Containers can still call get_temperature and get_status
Security impact:
- Prevents compromised containers from:
• Triggering unwanted SSH key distribution to cluster nodes
• Learning about cluster topology via forced registration
• DOS attacks by repeatedly calling key distribution
• Other host-level privileged operations
Without this fix, any container with root could call these methods after
authentication, undermining the security isolation between container and host.
Addresses high-severity finding #2 from security audit.
CRITICAL security fixes for pulse-sensor-proxy:
1. Strengthened hostname validation regex:
- Now requires hostnames to start with alphanumeric character
- Prevents SSH option injection via hostnames starting with '-'
- Pattern: ^[a-zA-Z0-9][a-zA-Z0-9._-]{0,63}$ (1-64 chars total)
- Added IPv4 and IPv6 validation regexes for future use
2. Added validation to vulnerable V1 RPC handlers:
- handleGetTemperature: Now validates node parameter before SSH
- handleRegisterNodes: Now validates discovered cluster nodes
- Previously these handlers passed unsanitized input directly to SSH
3. Defense in depth:
- V2 handlers already had validation (now using improved regex)
- Multiple layers of protection against malicious node identifiers
- Validation prevents container from passing SSH options as hostnames
Without these fixes, a compromised container could potentially inject SSH
options by providing malicious node names, though the 'root@' prefix
provided some mitigation.
Addresses high-severity finding from security audit.
Added clear messaging to explain why the socket bind mount is configured,
focusing on the security benefits rather than technical implementation.
Changes:
- Add explanatory header "Secure Container Communication Setup"
- Explain the three key benefits:
• Container communicates via Unix socket (not SSH)
• No SSH keys exposed inside container (enhanced security)
• Proxy on host manages all temperature collection
- Update technical messages to be more user-friendly:
• "Configuring socket bind mount" instead of "Ensuring..."
• "Restarting container to activate secure communication"
• "Verifying secure communication channel"
• "✓ Secure socket communication ready"
• "Configuring Pulse to use proxy"
This helps users understand WHY the bind mount exists (security) rather
than just seeing technical implementation details.
Adds explicit user consent before installing pulse-sensor-proxy on the
Proxmox host, with support for noninteractive/scripted installations.
Changes:
- Add --proxy flag with yes/no/auto modes
- Add prompt_proxy_installation() function that explains what will be
installed and asks for user confirmation
- Detect Docker in container and preselect 'yes' as default when found
- Support noninteractive mode via --proxy flag for automated installs
- Skip proxy installation if user declines or --proxy=no specified
- Auto-detect mode (--proxy=auto) installs only if Docker is present
Behavior:
- Default (no flag): Prompt user with explanation of what will be installed
- --proxy=yes: Install without prompting (for turnkey workflows)
- --proxy=no: Skip proxy installation entirely
- --proxy=auto: Install only if Docker is detected in container
- Docker detected: Default prompt answer changes to [Y/n] instead of [y/N]
When user declines, clear message explains temperature monitoring will
be unavailable and provides command to enable later.
This provides transparency about host-level changes while preserving
the turnkey workflow for automated/Docker installations.
Instead of warning the user to restart the container manually, the script
now automatically restarts it when the socket mount configuration is
updated. This ensures the mount is immediately active and temperature
monitoring works right away without user intervention.
Uses 'pct restart' if running, 'pct start' if stopped.
When installing with --main flag, the outer install.sh now passes --main
to the inner installation running inside the LXC. This ensures that
pulse-sensor-proxy is built from source inside the container, so the
binary can be copied to the Proxmox host using 'pct pull'.
Previously, the --main flag was not passed through, causing the inner
installation to download the release binary instead of building from
source, which resulted in an empty binary being copied to the host.
When --main flag is specified, install.sh now copies the binary that was
built inside the LXC to the Proxmox host using 'pct pull' and passes it
to install-sensor-proxy.sh with --local-binary flag.
This ensures that when users build from source, no binary downloads are
attempted - everything is built as expected. Release installs continue
to use the download fallback mechanism.
The install-sensor-proxy.sh script now tries GitHub releases first, then falls
back to downloading from the Pulse server if GitHub fails or doesn't have the
binary (common when building from main).
The LXC installer sets PULSE_SENSOR_PROXY_FALLBACK_URL to point to the Pulse
server running inside the newly created LXC, ensuring the proxy binary can be
downloaded from /api/install/pulse-sensor-proxy.
This fixes the issue where installing with --main would fail to install
pulse-sensor-proxy on the host because GitHub releases don't include it yet.
When users install with --main, the install script now:
- Builds pulse-sensor-proxy from source
- Installs it to /opt/pulse/bin/pulse-sensor-proxy
- Copies install-docker.sh and install-sensor-proxy.sh to scripts dir
This ensures the turnkey Docker installer can download pulse-sensor-proxy
from the Pulse server (/api/install/pulse-sensor-proxy) instead of failing.
Previously, building from source would skip pulse-sensor-proxy entirely,
causing the Docker installer to fail when trying to set up temperature
monitoring.
Adds a one-command Docker deployment flow that:
- Detects if running in LXC and installs Docker if needed
- Automatically installs pulse-sensor-proxy on the Proxmox host
- Configures bind mount for proxy socket into LXC
- Generates optimized docker-compose.yml with proxy socket
- Enables temperature monitoring via host-side proxy
The install-docker.sh script handles the complete setup including:
- Docker installation (if needed)
- ACL configuration for container UIDs
- Bind mount setup
- Automatic apparmor=unconfined for socket access
Accessible via: curl -sSL http://pulse:7655/api/install/install-docker.sh | bash
When the setup script detects TEMPERATURE_PROXY_KEY (proxy is available),
it now shows a clear success message instead of attempting SSH verification.
The verification check doesn't work with proxy-based setups since the
container doesn't have SSH keys - all temperature collection happens via
the Unix socket to pulse-sensor-proxy, which handles SSH.
Now shows:
✓ Temperature monitoring configured via pulse-sensor-proxy
Temperature data will appear in the dashboard within 10 seconds
Instead of the misleading:
⚠️ Unable to verify SSH connectivity.
Temperature data will appear once SSH connectivity is configured.
When pulse-sensor-proxy is available, the setup script now automatically
detects and uses the proxy's SSH public key instead of trying to generate
keys inside the container.
This fixes temperature monitoring setup for Docker deployments where:
- Container has proxy socket mounted at /mnt/pulse-proxy
- Proxy handles SSH connections to nodes
- Setup script needs to distribute the proxy's key, not container's key
The fix queries /api/system/proxy-public-key during setup script generation
and overrides SSH_SENSORS_PUBLIC_KEY if the proxy is available.
Tested with Docker on native Proxmox host (delly) - temperatures collected
successfully via proxy socket.
Changed heredoc delimiter from <<'EOF' to <<EOF to allow bash variable
expansion. Previously $SSH_PUBLIC_KEY and $SSH_RESTRICTED_KEY_ENTRY
were being passed as literal strings instead of their actual values,
so cluster nodes never received the correct SSH keys.
This fixes cluster node ProxyJump setup - now both restricted and
unrestricted keys are properly added to cluster nodes.
The setup script now adds both the restricted and unrestricted SSH keys
to ALL cluster nodes, not just the first one. This makes temperature
monitoring truly turnkey - you say 'yes' to configure cluster nodes and
it automatically sets up both keys on each node.
This ensures:
- All nodes can act as ProxyJump hosts if needed
- All nodes can provide temperature data via sensors
- No manual SSH key configuration required
Fixes turnkey cluster temperature monitoring setup.
When using ProxyJump for cluster temperature monitoring, the jump host
(typically the first cluster node) needs an unrestricted SSH key to allow
connection forwarding. Previously only the restricted key with
command="sensors -j" was added, which blocked ProxyJump.
Now the setup script adds TWO keys:
1. Unrestricted key (for ProxyJump/connection forwarding)
2. Restricted key (for running sensors -j directly)
This allows containerized Pulse to:
- Connect through the jump host to other cluster nodes
- Collect temperature data from all cluster members
Fixes cluster temperature monitoring for Docker/LXC deployments.
Added logic to resolve IP addresses for cluster nodes and include them as
HostName entries in the SSH config. Without this, Pulse couldn't connect
to cluster nodes like 'minipc' because the container couldn't resolve
the hostname.
Uses getent to resolve node names to IPs, with fallback to hostname if
resolution fails (for environments where DNS works).
- Changed SSH key generation from RSA 2048 to Ed25519 (more secure, faster, smaller)
- Added openssh-client package to Docker image (required for temperature monitoring)
- Updated SSH config template to use id_ed25519
- Removed unused crypto/rsa and crypto/x509 imports
Ed25519 provides better security with shorter keys and faster operations
compared to RSA. The container now has SSH client tools needed to connect
to Proxmox nodes for temperature data collection.
The setup script was generating SSH config with IdentityFile ~/.ssh/id_ed25519
but Pulse generates id_rsa keys. Updated SSH config template to use id_rsa
to match the actual key type generated by the monitoring system.
Added middleware exception for /api/system/ssh-config when a valid setup
token is provided, matching the pattern used for verify-temperature-ssh.
The middleware was blocking ssh-config requests before they reached the
handler, even though the handler had setup token validation logic.
The ssh-config endpoint was using RequireAuth which only accepts Pulse
API tokens, but the setup script sends a temporary setup token via the
auth_token parameter. Updated to follow the same pattern as
verify-temperature-ssh: check setup token first, then fall back to API auth.
This fixes the 401 error when the setup script tries to configure ProxyJump
for containerized Pulse deployments.
The setup script was passing pulseURL instead of authToken as the last
parameter, causing 'Authentication required' errors when verifying SSH
connectivity. Fixed parameter order in fmt.Sprintf call.
Security improvements to HandleSSHConfig endpoint:
- Add defer r.Body.Close() for proper resource cleanup
- Return 413 status for oversized requests with errors.As check
- Switch from blocklist to allowlist-based directive validation
- Use case-insensitive parsing with comment stripping via bufio.Scanner
- Add Content-Type: application/json header to response
Codex identified that blocklist approach was insufficient and recommended
allowlist validation to prevent unexpected directives. Only permits the
specific SSH directives Pulse needs for ProxyJump configuration.
Make temperature monitoring truly turnkey by automatically configuring
SSH ProxyJump when running in containers without pulse-sensor-proxy.
How it works:
1. Setup script runs on Proxmox host (e.g., delly)
2. Detects Pulse is containerized but proxy unavailable
3. Automatically configures SSH ProxyJump through the current host
4. Writes SSH config to /home/pulse/.ssh/config in container
5. Temperature monitoring "just works" without manual configuration
Changes:
- Track TEMP_MONITORING_AVAILABLE flag during proxy installation
- Auto-configure ProxyJump if proxy installation fails
- Add /api/system/ssh-config endpoint to write SSH config
- Only prompt for temperature monitoring if it can actually work
- Automatic SSH config: ProxyJump through Proxmox host
Before: User had to manually configure ProxyJump or install proxy
After: Temperature monitoring works automatically after setup script
This makes Docker deployments as turnkey as LXC deployments.
Changed the SSH connectivity check failure message from a scary
"FAILED" warning with complex ProxyJump instructions to a simple
informational message.
Before:
- ⚠️ SSH connectivity FAILED for: ...
- Complex multi-line ProxyJump configuration
- Confusing for users who don't need temperature monitoring
After:
- ℹ️ Temperature monitoring will be available once SSH configured
- Simple list of pending nodes
- Brief note about pulse-sensor-proxy for LXC
- Link to docs for details
This makes the setup experience much more turnkey by reducing
noise and focusing on successful completion rather than optional
features that require additional configuration.
Setup Script Improvements:
- Remove confusing "Could not download installer" warning for proxy
- Skip SSH connectivity check in containerized environments without proxy
- Simplify proxy installation prompts (automatic when available)
- Better messaging for containerized setups
These changes make the setup script more turnkey by reducing noise
and warnings that don't apply to test/development environments or
containerized installations.
Discovery Fixes:
- Always update cache even when scan finds no servers (prevents stale data)
- Remove automatic re-add of deleted nodes to discovery (was causing confusion)
- Optimize Docker subnet scanning from 762 IPs to 254 IPs (3x faster)
- Add getHostSubnetFromGateway() to detect host network from container
Frontend Type Fixes:
- Fix ThresholdsTable editScope type errors
- Fix SnapshotAlertConfig index signature
- Remove unused variable in Settings.tsx
These changes make discovery faster, more reliable, and fix the issue where
deleted nodes would persist in the discovery cache or immediately reappear.
Fixes container detection when Docker health checks are enabled.
Previously, the setup script only matched "running" status exactly,
causing it to skip containers showing "running (healthy)" status.
This prevented:
- Proper detection of containerized Pulse installations
- pulse-sensor-proxy installation for temperature monitoring
- Temperature data collection for affected users
The fix captures the full status output and searches for "running"
anywhere in the output, supporting all status variations:
- status: running
- status: running (healthy)
- status: running (unhealthy)
Related to #101
- Add instant-display tooltip on temperature column showing min-max range
- Color-code min/max temperatures individually (green/yellow/red)
- Remove unused NodeCard.tsx component from codebase
- Keep table row height consistent by using tooltip instead of inline display
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Track minimum and maximum CPU temperatures since monitoring started.
This provides better insight into temperature trends and cooling
adequacy over time.
Changes:
- Backend: Add CPUMin, CPUMaxRecord, MinRecorded, MaxRecorded fields
to Temperature model
- Backend: Implement min/max tracking logic in monitoring cycle that
preserves values across polling cycles
- Backend: Initialize min/max on first reading, update on extremes
- Frontend: Update Temperature TypeScript interface with new fields
- Frontend: Display min/max range in NodeCard tooltip (e.g., "52°C
(48-67°C since monitoring started)")
- Frontend: Rebuild dist assets
Temperature display now shows:
- Current temperature with color coding (green/yellow/red)
- Tooltip with full min-max range and context
- Min/max tracked in-memory (resets on Pulse restart)
Example tooltip: "CPU: 52°C (48-67°C since monitoring started)"
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Changed temperature monitoring menu from [K/r/s] to [1/2/3]
- Now all multi-choice menus use numbers consistently
- Main menu: [1/2/3]
- Temperature menu: [1/2/3] (was [K/r/s])
- Yes/no questions still use y/n (standard convention)
