Commit graph

302 commits

Author SHA1 Message Date
rcourtman
059e8bf562 Redirect to login when authentication expires
Related to #626

When authentication expires after some time, users see "Connection lost"
and must refresh the page to see "Authentication required". This commit
implements automatic redirect to login when authentication expires.

Changes:
- Add authentication check to WebSocket endpoint to prevent unauthenticated
  WebSocket connections
- Handle WebSocket close with code 1008 (policy violation) as auth failure
  and redirect to login
- Intercept 401 responses on API calls (except initial auth checks) and
  automatically redirect to login page
- Clear stored credentials and set logout flag before redirect to ensure
  clean login flow

This provides a better user experience by immediately redirecting to the
login page when the session expires, rather than showing a confusing
"Connection lost" message that requires manual page refresh.
2025-11-05 19:36:01 +00:00
rcourtman
b44084af3c Skip false health alerts for Samsung 980/990 SSDs and improve Docker CPU calculation
Related to #547 and #622

## Samsung SSD Fix (#547)
Samsung 980 and 990 series SSDs have known firmware bugs that cause them to
report incorrect health status (typically FAILED or critical warnings) even
when the drives are actually healthy. This is commonly due to incorrect
temperature threshold reporting in the firmware.

This change adds special handling to detect these drives and skip health
status alerts while still monitoring wearout metrics, which remain reliable.
The fix also clears any existing false alerts for these drives.

Users experiencing these false alerts should update their Samsung SSD firmware
to the latest version from Samsung, which typically resolves the issue.

## Docker Agent CPU Fix (#622)
Addresses issue where Docker container CPU usage shows 0%. The Docker
agent uses ContainerStatsOneShot which typically doesn't populate
PreCPUStats, requiring manual delta tracking between collection cycles.

Changes:
- Fix logic bug where prevContainerCPU was updated before checking if
  previous sample existed, causing incorrect delta calculations
- Add comprehensive debug logging showing which calculation method
  succeeded (PreCPUStats, system delta, or time-based fallback)
- Add warning after 10 PreCPUStats failures to inform about manual
  tracking mode (normal for one-shot stats)
- Add detailed failure logging when CPU calculation cannot complete

Expected behavior: First collection cycle returns 0% (no previous
sample), subsequent cycles show accurate CPU metrics.
2025-11-05 19:33:16 +00:00
rcourtman
4c1d7a2797 Fix PMG API parameter issues causing 400 errors
Related to #614

Corrects three issues with PMG monitoring:

1. Remove unsupported timeframe parameter from GetMailStatistics
   - PMG API /statistics/mail does not accept timeframe parameter
   - Previously sent "timeframe=day" causing 400 error
   - API returns current day statistics by default

2. Fix GetMailCount timespan parameter to use seconds
   - Changed from 24 (hours) to 86400 (seconds)
   - PMG API expects timespan in seconds, not hours
   - Previously sent "timespan=24" causing 400 error

3. Update function signature and tests
   - Renamed GetMailCount parameter from timespanHours to timespanSeconds
   - Updated test expectations to match corrected API calls
   - Tests verify parameters are sent correctly

These changes align the PMG client with actual PMG API requirements,
fixing the data population issues reported in v4.25.0.
2025-11-05 19:28:37 +00:00
rcourtman
fcba710183 Guard PBS backups from failed polls
Related to #613

When all PBS datastore queries fail (e.g., due to network issues or PBS
downtime), the system was clearing all backups and showing an empty list.
This adds the same preservation logic that exists for PVE storage backups.

Changes:
- Add shouldPreservePBSBackups() helper function
- Track datastore query success/failure counts in pollPBSBackups()
- Preserve existing backups when all datastore queries fail
- Add comprehensive unit tests for PBS backup preservation logic

This ensures users can still see their backup history even during
temporary connectivity issues with PBS, matching the behavior already
implemented for PVE storage backups.
2025-11-05 19:26:20 +00:00
rcourtman
350828a260 Prefer IP addresses over hostnames for cluster communication
This change modifies the `clusterEndpointEffectiveURL` function to prioritize
IP addresses over hostnames when building cluster endpoint URLs. This eliminates
excessive DNS lookups that can overwhelm DNS servers (e.g., pi-hole), which was
causing hundreds of thousands of unnecessary DNS queries.

When Pulse communicates with Proxmox cluster nodes, it will now:
1. First try to use the IP address from ClusterEndpoint.IP
2. Fall back to ClusterEndpoint.Host only if IP is not available

This is a minimal, backwards-compatible change that maintains existing
functionality while dramatically reducing DNS traffic for clusters where
node IPs are already known and stored.

Related to #620
2025-11-05 19:23:26 +00:00
rcourtman
f0088070be Improve guest agent error classification to prevent false permission errors
Related to #596

**Problem:**
Users were seeing persistent "permission denied" error messages for VMs
that simply didn't have qemu-guest-agent installed or running. The error
detection logic was too broad and classified Proxmox API 500 errors as
permission issues, even when they indicated guest agent unavailability.

**Root Cause:**
When qemu-guest-agent is not installed or not running, Proxmox API returns
various error responses (500, 403) that may contain permission-related text.
The previous error detection logic checked for "permission denied" strings
without considering the HTTP status code context, leading to:
- VMs with guest agent: guest details display correctly
- VMs without guest agent: false "Permission denied" error shown

**Solution:**
Enhanced error classification logic to distinguish between:
1. Actual permission issues (401/403 with permission keywords)
2. Guest agent unavailability (500 errors)
3. Agent timeout issues
4. Other agent errors

The fix ensures that only explicit authentication/authorization errors
(401 Unauthorized, 403 Forbidden with permission keywords) are classified
as permission-denied, while API 500 errors are correctly identified as
agent-not-running issues.

**Changes:**
- Reordered error detection to check most specific patterns first
- Added HTTP status code context to permission error detection
- 500 errors now correctly map to "agent-not-running" status
- Only 401/403 errors with explicit permission keywords trigger "permission-denied"
- Improved log messages to guide users toward correct resolution
- Fixed err.Error() vs errStr variable inconsistency

**Impact:**
Users will now see accurate error messages that guide them to:
- Install qemu-guest-agent when it's missing (most common case)
- Check permissions only when there's an actual auth/authz issue
- Understand the difference between agent problems and permission problems
2025-11-05 19:21:58 +00:00
rcourtman
ddc787418b Round float values in webhook payloads to 1 decimal place
Webhook alert payloads now round Value and Threshold fields to 1 decimal
place before template rendering. This eliminates excessive precision in
webhook messages (e.g., 62.27451680630036 becomes 62.3).

The fix is applied in prepareWebhookData() so all webhook templates
benefit automatically, including Google Space webhooks, generic JSON
webhooks, and custom templates.

Related to #619
2025-11-05 19:19:10 +00:00
rcourtman
b1831d7b3e Add guest URL support for PVE hosts
Related to discussion #615

Add optional GuestURL field to PVE instances and cluster endpoints,
allowing users to specify a separate guest-accessible URL for web UI
navigation that differs from the internal management URL.

Backend changes:
- Add GuestURL field to PVEInstance and ClusterEndpoint structs
- Add GuestURL field to Node model
- Update cluster auto-discovery to preserve existing GuestURL values
- Update node creation logic to populate GuestURL from config
- Update API handlers to accept and persist GuestURL field

Frontend changes:
- Add GuestURL input field to NodeModal for configuration
- Update NodeGroupHeader and NodeSummaryTable to use GuestURL for navigation
- Add GuestURL to Node and PVENodeConfig TypeScript interfaces

When GuestURL is configured, it will be used for navigation links
instead of the Host URL, allowing users to access PVE hosts through
a reverse proxy or different domain while maintaining internal API
connections.
2025-11-05 19:06:08 +00:00
rcourtman
02864f54dd Add test notification functionality for Apprise
- Add support for testing Apprise notifications via /api/notifications/test endpoint
- Users can now test their Apprise configuration (both CLI and HTTP modes) using method="apprise"
- Added comprehensive unit tests for both CLI and HTTP modes
- Tests verify correct behavior when Apprise is enabled/disabled
- Tests validate that notifications are properly sent through Apprise channels

Related to #584
2025-11-05 18:54:18 +00:00
rcourtman
6404b6a5fc Expand temperature sensor compatibility for SuperIO and AMD CPUs
Users with NCT6687 SuperIO chips and AMD processors reporting only chiplet
temperatures were unable to see CPU temperature data. Added support for
Nuvoton/Winbond/Fintek SuperIO chips and AMD Tccd chiplet temperatures,
with debug logging to aid troubleshooting unsupported sensor configurations.

Related to discussion #586
2025-11-05 18:47:21 +00:00
rcourtman
b972b7f05f Fix broken documentation links for containerized deployments
Replace non-functional docs.pulseapp.io URLs with direct GitHub repository
links. The containerized deployment security documentation exists in
SECURITY.md and was previously inaccessible via the external link.

Changes:
- Update SECURITY.md documentation reference
- Fix three documentation links in config_handlers.go (SSH verification,
  setup script, and security block error messages)
- All links now point to GitHub repository where docs actually live

Related to #607
2025-11-05 18:46:41 +00:00
rcourtman
449d77504f Improve PMG connection testing to validate metrics endpoints
Related to #551

Enhanced the PMG connection test to actually validate the metrics
endpoints that Pulse uses for monitoring, rather than only checking
the version endpoint. This provides users with immediate feedback if
their PMG credentials lack the necessary permissions to collect metrics.

Backend changes:
- Test mail statistics, cluster status, and quarantine endpoints during
  connection test (internal/api/config_handlers.go:1695-1714)
- Return warnings array in test response when endpoints are unavailable
- Increased timeout from 10s to 15s to accommodate multiple endpoint checks
- Added warning logs for failed endpoint checks

Frontend changes:
- Added showWarning() toast function for warning messages
- Enhanced NodeModal to display warning status with amber styling
- Added warnings list display in test results UI
- Updated Settings.tsx to show warnings from connection tests

This change helps users identify permission issues immediately rather
than discovering later that metrics aren't being collected despite a
"successful" connection.
2025-11-05 18:40:39 +00:00
rcourtman
c93581e1aa Add DNS caching to reduce excessive DNS queries
Related to #608

Implements DNS caching using rs/dnscache to dramatically reduce DNS query
volume for frequently accessed Proxmox hosts. Users were reporting 260,000+
DNS queries in 37 hours for the same hostnames.

Changes:
- Added rs/dnscache dependency for DNS resolution caching
- Created pkg/tlsutil/dnscache.go with DNS cache wrapper
- Updated HTTP client creation to use cached DNS resolver
- Added DNSCacheTimeout configuration option (default: 5 minutes)
- Made DNS cache timeout configurable via:
  - system.json: dnsCacheTimeout field (seconds)
  - Environment variable: DNS_CACHE_TIMEOUT (duration string)
- DNS cache periodically refreshes to prevent stale entries

Benefits:
- Reduces DNS query load on local DNS servers by ~99%
- Reduces network traffic and DNS query log volume
- Maintains fresh DNS entries through periodic refresh
- Configurable timeout for different network environments

Default behavior: 5-minute cache timeout with automatic refresh
2025-11-05 18:25:38 +00:00
rcourtman
f434a7b9e7 Fix fmt.Sprintf argument count in setup script after Docker/LXC changes
The previous commit added 4 new %s format specifiers for Docker/LXC
instructions but didn't add the corresponding arguments to fmt.Sprintf.

Added 4 pulseURL arguments to match the new format specifiers in the
'unknown environment' section of the setup script.
2025-11-05 18:18:04 +00:00
rcourtman
a1fb79ae6a Fix temperature proxy documentation and setup script for Docker vs LXC clarity
This addresses confusion around temperature monitoring setup for Docker
deployments where users expected a turnkey experience similar to LXC.

The core issue: The setup script and documentation suggested that
temperature monitoring was "automatically configured" for all containerized
deployments, but in reality only LXC containers have a fully automatic
setup. Docker requires manual steps.

Changes:

**Setup Script (config_handlers.go):**
- Fixed "unknown environment" path to show separate instructions for LXC vs Docker
- Docker instructions now correctly show --standalone flag (was incorrectly showing --ctid)
- Added docker-compose.yml bind mount instructions inline
- Added restart command for Docker deployments

**Documentation (TEMPERATURE_MONITORING.md):**
- Added prominent "Deployment-Specific Setup" callout at the top
- Clarified that LXC is fully automatic, Docker requires manual steps
- Reorganized "Setup (Automatic)" section to clearly distinguish:
  - LXC: Fully turnkey (no manual steps)
  - Docker: Manual proxy installation required
  - Node configuration: Works for both
- Updated "Host-side responsibilities" to specify it's Docker-only
- Fixed architecture benefits to reflect LXC vs Docker differences

Why this matters:
- LXC setup script auto-detects the container and runs install-sensor-proxy.sh --ctid
- Docker deployments can't be auto-detected and require --standalone flag
- Users running Docker were getting incorrect instructions (--ctid instead of --standalone)
- Documentation suggested everything was automatic, leading to confusion

Now the documentation and setup script accurately reflect that:
- LXC = Turnkey (automatic)
- Docker = Manual steps required (but well-documented)
- Native = Direct SSH (no proxy)

Related to GitHub Discussion #605
2025-11-05 18:18:04 +00:00
rcourtman
fdf0977be2 Add host agent multi-platform binary distribution and improve host details UI
- Build host agent binaries for all platforms (linux/darwin/windows, amd64/arm64/armv7) in Docker
- Add Makefile target for building agent binaries locally
- Add startup validation to check for missing agent binaries
- Improve download endpoint error messages with troubleshooting guidance
- Enhance host details drawer layout with better organization and visual hierarchy
- Update base images to rolling versions (node:20-alpine, golang:1.24-alpine, alpine:3.20)
2025-11-05 17:38:17 +00:00
rcourtman
7dd7a0b0f9 Fix node/host dropout issue caused by cluster health failures
Implemented comprehensive state preservation to prevent temporary dropouts:

1. Node Grace Period (60s):
   - Track last-online timestamp for each Proxmox node
   - Preserve online status during grace period to prevent flapping
   - Applied to all node status checks throughout codebase

2. Efficient Polling Preservation:
   - Detect when cluster/resources returns empty arrays
   - Preserve previous VMs/containers if had resources before
   - Handles cluster health check failures gracefully

3. Traditional Polling Preservation:
   - Updated preservation logic for per-node VM/container polling
   - Triggers when zero resources returned regardless of node response
   - Fixed issue where nodes responding with empty data bypassed preservation

Root cause: Intermittent Proxmox cluster health failures ("no healthy nodes
available") caused both efficient and traditional polling to return empty
arrays, immediately clearing all VMs/containers from state.

Changes:
- internal/monitoring/monitor.go: Added node grace period, efficient polling preservation
- internal/monitoring/monitor_polling.go: Fixed traditional polling preservation logic

Fixes frequent UI flickering where vmCount/containerCount would briefly drop to zero.
2025-11-05 17:01:20 +00:00
rcourtman
27f2038dab Add per-node temperature monitoring and fix critical config update bug
This commit implements per-node temperature monitoring control and fixes a critical
bug where partial node updates were destroying existing configuration.

Backend changes:
- Add TemperatureMonitoringEnabled field (*bool) to PVEInstance, PBSInstance, and PMGInstance
- Update monitor.go to check per-node temperature setting with global fallback
- Convert all NodeConfigRequest boolean fields to *bool pointers
- Add nil checks in HandleUpdateNode to prevent overwriting unmodified fields
- Fix critical bug where partial updates zeroed out MonitorVMs, MonitorContainers, etc.
- Update NodeResponse, NodeFrontend, and StateSnapshot to include temperature setting
- Fix HandleAddNode and test connection handlers to use pointer-based boolean fields

Frontend changes:
- Add temperatureMonitoringEnabled to Node interface and config types
- Create per-node temperature monitoring toggle handler with optimistic updates
- Update NodeModal to wire up per-node temperature toggle
- Add isTemperatureMonitoringEnabled helper to check effective monitoring state
- Update ConfiguredNodeTables to show/hide temperature badge based on monitoring state
- Update NodeSummaryTable to conditionally show temperature column
- Pass globalTemperatureMonitoringEnabled prop through component tree

The critical bug fix ensures that when updating a single field (like temperature
monitoring), the backend only modifies that specific field instead of zeroing out
all other boolean configuration fields.
2025-11-05 14:11:53 +00:00
rcourtman
e4e915c8a1 Fix temperature data intermittency caused by proxy rate limit retries
Root Cause:
The classifyError() function in tempproxy/client.go was returning nil
when err was nil, even if respError contained "rate limit exceeded".
This caused the retry logic to treat rate limit errors as retryable,
triggering 3 retries with exponential backoff (100ms, 200ms, 400ms)
for each rate-limited request.

With multiple nodes polling simultaneously and hitting the proxy's
1 req/sec default rate limit, this created a retry storm:
- 3 nodes polling every 10 seconds
- 1-2 requests rate limited per cycle
- Each rate limit triggered 3 retries
- Result: 6+ extra requests per cycle, causing temperature data to
  flicker in and out as requests were dropped

Solution:
1. Reordered classifyError() to check respError first before checking
   if err is nil, ensuring rate limit errors are properly classified
2. Added explicit rate limit detection that marks these errors as
   non-retryable
3. Added stub EnableTemperatureMonitoring/DisableTemperatureMonitoring
   methods to Monitor for interface compatibility

Impact:
- Rate limit retry attempts reduced from 151 in 10 minutes to 0
- Temperature data now stable for all nodes
- No more flickering temperature displays in dashboard
2025-11-05 10:20:15 +00:00
rcourtman
7a185c4ab3 Improve guest agent timeout handling for high-load environments (refs #592)
This change addresses intermittent "Guest details unavailable" and "Disk stats
unavailable" errors affecting users with large VM deployments (50+ VMs) or
high-load Proxmox environments.

Changes:
- Increased default guest agent timeouts (3-5s → 10-15s) to better handle
  environments under load
- Added automatic retry logic (1 retry by default) for transient timeout failures
- Made all timeouts and retry count configurable via environment variables:
  * GUEST_AGENT_FSINFO_TIMEOUT (default: 15s)
  * GUEST_AGENT_NETWORK_TIMEOUT (default: 10s)
  * GUEST_AGENT_OSINFO_TIMEOUT (default: 10s)
  * GUEST_AGENT_VERSION_TIMEOUT (default: 10s)
  * GUEST_AGENT_RETRIES (default: 1)
- Added comprehensive documentation in VM_DISK_MONITORING.md with configuration
  examples for different deployment scenarios

These improvements allow Pulse to gracefully handle intermittent API timeouts
without immediately displaying errors, while remaining configurable for
different network conditions and environment sizes.

Fixes: https://github.com/rcourtman/Pulse/discussions/592
2025-11-05 09:40:58 +00:00
rcourtman
d52ac6d8b5 Fix CSRF token validation and improve token management
- Add Access-Control-Expose-Headers to allow frontend to read X-CSRF-Token response header
- Implement proactive CSRF token issuance on GET requests when session exists but CSRF cookie is missing
- Ensures frontend always has valid CSRF token before making POST requests
- Fixes 403 Forbidden errors when toggling system settings

This resolves CSRF validation failures that occurred when CSRF tokens expired or were missing while valid sessions existed.
2025-11-05 09:23:44 +00:00
rcourtman
10862db4e4 Enhance container detection for temperature SSH safeguards (refs #601) 2025-11-04 22:30:35 +00:00
rcourtman
adda6eea38 Update docker CPU metrics and add OpenRC installer support (Refs #255) 2025-11-04 22:16:50 +00:00
rcourtman
6eb1a10d9b Refactor: Code cleanup and localStorage consolidation
This commit includes comprehensive codebase cleanup and refactoring:

## Code Cleanup
- Remove dead TypeScript code (types/monitoring.ts - 194 lines duplicate)
- Remove unused Go functions (GetClusterNodes, MigratePassword, GetClusterHealthInfo)
- Clean up commented-out code blocks across multiple files
- Remove unused TypeScript exports (helpTextClass, private tag color helpers)
- Delete obsolete test files and components

## localStorage Consolidation
- Centralize all storage keys into STORAGE_KEYS constant
- Update 5 files to use centralized keys:
  * utils/apiClient.ts (AUTH, LEGACY_TOKEN)
  * components/Dashboard/Dashboard.tsx (GUEST_METADATA)
  * components/Docker/DockerHosts.tsx (DOCKER_METADATA)
  * App.tsx (PLATFORMS_SEEN)
  * stores/updates.ts (UPDATES)
- Benefits: Single source of truth, prevents typos, better maintainability

## Previous Work Committed
- Docker monitoring improvements and disk metrics
- Security enhancements and setup fixes
- API refactoring and cleanup
- Documentation updates
- Build system improvements

## Testing
- All frontend tests pass (29 tests)
- All Go tests pass (15 packages)
- Production build successful
- Zero breaking changes

Total: 186 files changed, 5825 insertions(+), 11602 deletions(-)
2025-11-04 21:50:46 +00:00
rcourtman
5c4be1921c chore: snapshot current changes 2025-11-02 22:47:55 +00:00
rcourtman
fb22469eb0 Add disk usage threshold support for Docker containers
Extends the Docker monitoring and alerting system to track writable layer
usage as a percentage of the container's root filesystem. This helps
identify containers with bloated copy-on-write layers before they
consume excessive disk space.

- Add disk threshold to DockerThresholdConfig (default: 85% trigger, 80% clear)
- Evaluate disk alerts for running containers when RootFilesystemBytes > 0
- Include disk metadata (writable layer, total filesystem, block I/O stats)
- Update frontend to display and configure disk thresholds
- Add test coverage for disk usage alert hysteresis
- Document disk monitoring in DOCKER_MONITORING.md

Per-container and per-host overrides apply to disk thresholds the same
way they do for CPU and memory.
2025-10-29 14:52:25 +00:00
rcourtman
6b670a7af3 Auto-clear removal block after successful Docker host stop
When a Docker host successfully completes a stop command and confirms
it has disabled itself, automatically clear the removal block to allow
immediate re-enrollment.

This fixes the UX issue where users who remove a Docker host cannot
immediately reinstall it with a new token, as the host ID remains
blocked for 24 hours. The block is still needed to prevent zombie
reports from stale agents, but once the agent confirms it stopped
successfully, there's no need to keep the block.

Changes:
- Clear removal block in HandleCommandAck after successful host removal
- Allows remove → reinstall workflow without manual intervention
- Block remains for forced removals or offline hosts (as intended)
2025-10-29 12:40:22 +00:00
rcourtman
730c6bf864 Fix Docker agent removal and improve security
This commit addresses multiple issues in the Docker/host agent removal flow:

Agent Stop Fix:
- Add systemctl stop command after agent acknowledgement to prevent systemd restart
- Previous behavior: agent disabled but systemd immediately restarted it (Restart=always)
- New behavior: agent disables itself, sends ack, then stops systemd service completely

UX Improvements:
- Add real-time elapsed time counter during removal wait
- Show progress indicators prominently (no longer hidden in dropdown)
- Display expected time range (30-60 seconds) and last heartbeat
- Auto-show timeout warning after 2 minutes with actionable "Force remove" button
- Add contextual help explaining what's happening at each stage

Security Enhancement:
- Automatically revoke API tokens when removing Docker/host agents
- Previous behavior: tokens remained valid after agent removal
- New behavior: tokens are revoked and persisted immediately on removal
- Prevents removed agents from re-authenticating with old credentials
2025-10-29 12:27:36 +00:00
rcourtman
32392d1212 Add disk metrics, block I/O, and mount details to Docker monitoring
Extends Docker container monitoring with comprehensive disk and storage information:
- Writable layer size and root filesystem usage displayed in new Disk column
- Block I/O statistics (read/write bytes totals) shown in container drawer
- Mount metadata including type, source, destination, mode, and driver details
- Configurable via --collect-disk flag (enabled by default, can be disabled for large fleets)

Also fixes config watcher to consistently use production auth config path instead of following PULSE_DATA_DIR when in mock mode.
2025-10-29 12:05:36 +00:00
rcourtman
b3285c05c8 Consolidate pending changes
- Add Docker metadata test comment
- Update alerts configuration and thresholds
- Enhance config file watcher
- Update documentation
- Refine settings UI
2025-10-28 23:20:44 +00:00
rcourtman
99b11760ac Implement Docker metadata API endpoints
Add backend support for storing and managing Docker resource metadata:

- Create DockerMetadataStore for managing Docker container/service metadata
- Implement DockerMetadataHandler with GET/PUT/DELETE operations
- Register /api/docker/metadata routes with proper authentication
- Store metadata in docker_metadata.json file
- Validate custom URLs (http/https scheme, valid host)
- Supports resource IDs in format: {hostId}:container:{containerId}

Enables the frontend Docker URL editing feature to persist data.
2025-10-28 22:56:53 +00:00
rcourtman
f2acdd59af Normalize docker agent version handling 2025-10-28 08:42:58 +00:00
rcourtman
e07336dd9f refactor: remove legacy DISABLE_AUTH flag and enhance authentication UX
Major authentication system improvements:

- Remove deprecated DISABLE_AUTH environment variable support
- Update all documentation to remove DISABLE_AUTH references
- Add auth recovery instructions to docs (create .auth_recovery file)
- Improve first-run setup and Quick Security wizard flows
- Enhance login page with better error messaging and validation
- Refactor Docker hosts view with new unified table and tree components
- Add useDebouncedValue hook for better search performance
- Improve Settings page with better security configuration UX
- Update mock mode and development scripts for consistency
- Add ScrollableTable persistence and improved responsive design

Backend changes:
- Remove DISABLE_AUTH flag detection and handling
- Improve auth configuration validation and error messages
- Enhance security status endpoint responses
- Update router integration tests

Frontend changes:
- New Docker components: DockerUnifiedTable, DockerTree, DockerSummaryStats
- Better connection status indicator positioning
- Improved authentication state management
- Enhanced CSRF and session handling
- Better loading states and error recovery

This completes the migration away from the insecure DISABLE_AUTH pattern
toward proper authentication with recovery mechanisms.
2025-10-27 19:46:51 +00:00
rcourtman
68ce8e7520 feat: finalize swarm service monitoring (#598) 2025-10-26 09:35:49 +00:00
rcourtman
8e83eaf823 Add container state filtering to Docker agent 2025-10-25 21:40:59 +00:00
rcourtman
334ed3aedc Improve setup script auth usability 2025-10-25 19:08:48 +00:00
rcourtman
5a2d808aa1 Harden setup token flow and enforce encrypted persistence 2025-10-25 16:00:37 +00:00
rcourtman
a279e6720e Add auth enforcement integration tests 2025-10-25 15:02:48 +00:00
rcourtman
7075cef326 Harden API auth and token handling 2025-10-25 14:54:03 +00:00
rcourtman
77282bd3a6 Implement Pulse tag overrides and alert clear persistence 2025-10-25 14:28:32 +00:00
rcourtman
d643dcf0bc perf: reduce polling allocations and guest metadata load 2025-10-25 13:12:47 +00:00
rcourtman
138d8facd2 Improve host agent onboarding flow 2025-10-25 09:37:29 +00:00
rcourtman
b4247fc095 feat: add server-side support for agent installation improvements
API Enhancements:
- Add SHA256 checksum endpoint for binary downloads
  - Computes checksum on-the-fly when .sha256 suffix is requested
  - Example: /download/pulse-host-agent?platform=linux&arch=amd64.sha256
  - Enables installer scripts to verify binary integrity
- Add /uninstall-host-agent.sh endpoint for Linux/macOS uninstall script
- Add endpoint to public paths (no auth required)

Checksum Implementation:
- New serveChecksum() function computes SHA256 hash using crypto/sha256
- Returns plain text checksum in hex format
- Supports all binary download endpoints
- Zero performance impact (only computed when requested)

Install Script Updates:
- Add --force/-f flag to skip all interactive prompts
  - URL/token prompts skipped with --force
  - Reinstall confirmation skipped with --force
  - Checksum mismatch still aborts (security first)
- Force mode auto-accepts updates and reinstalls
- Usage: ./install-host-agent.sh --url $URL --token $TOKEN --force

Security Notes:
- Checksum verification protects against:
  - Corrupted downloads due to network issues
  - Man-in-the-middle binary tampering
  - Storage corruption on server
- Force mode maintains security by aborting on checksum mismatch
- No bypass for security-critical validations

These improvements enable:
- Automated deployments (--force flag)
- Binary integrity verification (checksums)
- Better security posture (tamper detection)
- Standardized uninstall process (endpoint)

The /api/version endpoint already exists and returns version info
for update checks (no changes needed).
2025-10-23 22:27:02 +00:00
rcourtman
6333a445e9 feat: add native Windows service support and expandable host details
Windows Host Agent Enhancements:
- Implement native Windows service support using golang.org/x/sys/windows/svc
- Add Windows Event Log integration for troubleshooting
- Create professional PowerShell installation/uninstallation scripts
- Add process termination and retry logic to handle Windows file locking
- Register uninstall endpoint at /uninstall-host-agent.ps1

Host Agent UI Improvements:
- Add expandable drawer to Hosts page (click row to view details)
- Display system info, network interfaces, disks, and temperatures in cards
- Replace status badges with subtle colored indicators
- Remove redundant master-detail sidebar layout
- Add search filtering for hosts

Technical Details:
- service_windows.go: Windows service lifecycle management with graceful shutdown
- service_stub.go: Cross-platform compatibility for non-Windows builds
- install-host-agent.ps1: Full Windows installation with validation
- uninstall-host-agent.ps1: Clean removal with process termination and retries
- HostsOverview.tsx: Expandable row pattern matching Docker/Proxmox pages

Files Added:
- cmd/pulse-host-agent/service_windows.go
- cmd/pulse-host-agent/service_stub.go
- scripts/install-host-agent.ps1
- scripts/uninstall-host-agent.ps1
- frontend-modern/src/components/Hosts/HostsOverview.tsx
- frontend-modern/src/components/Hosts/HostsFilter.tsx

The Windows service now starts reliably with automatic restart on failure,
and the uninstall script handles file locking gracefully without requiring reboots.
2025-10-23 22:11:56 +00:00
rcourtman
5c54685f04 Add API token scopes and standalone host agent
Introduces granular permission scopes for API tokens (docker:report, docker:manage, host-agent:report, monitoring:read/write, settings:read/write) allowing tokens to be restricted to minimum required access. Legacy tokens default to full access until scopes are explicitly configured.

Adds standalone host agent for monitoring Linux, macOS, and Windows servers outside Proxmox/Docker estates. New Servers workspace in UI displays uptime, OS metadata, and capacity metrics from enrolled agents.

Includes comprehensive token management UI overhaul with scope presets, inline editing, and visual scope indicators.
2025-10-23 11:40:31 +00:00
rcourtman
e76ab5eec0 Strip IPv6 scopes from container metadata (#596) 2025-10-23 08:55:18 +00:00
rcourtman
a885fb5472 Surface LXC interface IPs via PVE interfaces API (#596) 2025-10-23 08:07:32 +00:00
rcourtman
b95c01066e Capture dynamic LXC IP metrics (#596) 2025-10-23 07:50:45 +00:00
rcourtman
be85459db2 Add LXC config metadata for guest drawers (#596) 2025-10-23 07:30:32 +00:00
rcourtman
f4ead79c82 Ensure LXC drawers populate without metrics (#596) 2025-10-22 22:27:19 +00:00
rcourtman
aac3dacd63 Improve LXC guest metrics visibility (#596) 2025-10-22 22:24:33 +00:00
rcourtman
cdba742884 Stabilize diagnostics test VM selection 2025-10-22 19:48:56 +00:00
rcourtman
dd2beffc8c Stop legacy temperature SSH retries when auth fails (#595) 2025-10-22 19:35:51 +00:00
rcourtman
fe1533ea13 Improve PMG metric ingestion refs #551 2025-10-22 18:15:27 +00:00
rcourtman
d813f2396f Respect custom ports when discovering Proxmox clusters 2025-10-22 17:42:52 +00:00
rcourtman
20ff56aceb Add coverage for PVE memused fallback #553 2025-10-22 17:14:12 +00:00
rcourtman
3a3e0e080c Add replication monitoring plumbing and UI
Refs #395
2025-10-22 16:10:15 +00:00
rcourtman
7ae393c8ec Refine Proxmox node memory fallback (#582) 2025-10-22 15:36:26 +00:00
rcourtman
c9543e8a7e Add qemu guest agent version metadata 2025-10-22 15:24:07 +00:00
rcourtman
cdf80dbab5 Align backup node column with current host (#577) 2025-10-22 13:47:51 +00:00
rcourtman
77108abc65 Propagate config updates to settings nodes (#588) 2025-10-22 13:45:13 +00:00
rcourtman
be26f957c0 Add snapshot size alert thresholds (#585) 2025-10-22 13:30:40 +00:00
rcourtman
30879c3b7b Handle AMD Tctl temperature readings (refs #586) 2025-10-22 12:58:34 +00:00
rcourtman
f83caf8933 Add collision-safe Docker host identifiers (#590) 2025-10-22 12:30:25 +00:00
rcourtman
bc479643e4 release: prepare v4.25.0 2025-10-22 10:46:18 +00:00
rcourtman
4eb8bed9b5 Fix initial setup caching and container discovery defaults 2025-10-22 07:34:32 +00:00
rcourtman
ff4dc49ae4 Update Pulse install flow and related components 2025-10-21 19:58:53 +00:00
rcourtman
999da6d900 feat: production-ready import/export with API tokens and transactional rollback
Export/import payload bumped to v4.1 to include API tokens alongside existing
config bundle, eliminating blind spots in disaster recovery scenarios.

## Key Features

**API Tokens in Exports (v4.1)**
- Exports now include API token metadata (ID, name, hash, prefix, suffix, timestamps)
- Export format version bumped from 4.0 to 4.1
- Fixes gap where API tokens were lost during config migrations

**Transactional Atomic Imports**
- New importTransaction helper stages all writes before committing
- On failure, automatic rollback restores original configs
- Prevents partial/corrupted imports that could break running systems
- All config writes (nodes, alerts, email, webhooks, apprise, system, OIDC, API tokens, guest metadata) now transaction-aware

**Backward Compatibility**
- Version 4.0 exports (without API tokens) still import successfully
- System logs notice but proceeds, leaving existing API tokens untouched
- No breaking changes to existing export/import workflows

## Implementation

**Files Added:**
- internal/config/import_transaction.go - Transaction helper with staging/rollback

**Files Modified:**
- internal/config/export.go - v4.1 export, transactional ImportConfig wrapper
- internal/config/persistence.go - Transaction-aware Save* methods, beginTransaction/endTransaction helpers
- internal/config/persistence_test.go - 4 comprehensive unit tests

**Testing:**
- TestExportConfigIncludesAPITokens - Verifies API tokens in v4.1 exports
- TestImportConfigTransactionalSuccess - Validates atomic import success path
- TestImportConfigRollbackOnFailure - Confirms rollback on mid-import failure
- TestImportAcceptsVersion40Bundle - Ensures backward compatibility with v4.0

All tests passing 

## Migration Notes

- No manual migration required
- Users can re-export to generate v4.1 bundles with API tokens
- Existing 4.0 bundles remain valid for import
- Recommended: Re-run export after upgrade to ensure API tokens are captured

Co-authored-by: Codex (implementation)
Co-authored-by: Claude (coordination and testing)
2025-10-21 14:37:44 +00:00
rcourtman
2786afdff0 feat: comprehensive diagnostics and observability improvements
Upgrade diagnostics infrastructure from 5/10 to 8/10 production readiness
with enhanced metrics, logging, and request correlation capabilities.

**Request Correlation**
- Wire request IDs through context in middleware
- Return X-Request-ID header in all API responses
- Enable downstream log correlation across request lifecycle

**HTTP/API Metrics** (18 new Prometheus metrics)
- pulse_http_request_duration_seconds - API latency histogram
- pulse_http_requests_total - request counter by method/route/status
- pulse_http_request_errors_total - error counter by type
- Path normalization to control label cardinality

**Per-Node Poll Metrics**
- pulse_monitor_node_poll_duration_seconds - per-node timing
- pulse_monitor_node_poll_total - success/error counts per node
- pulse_monitor_node_poll_errors_total - error breakdown per node
- pulse_monitor_node_poll_last_success_timestamp - freshness tracking
- pulse_monitor_node_poll_staleness_seconds - age since last success
- Enables multi-node hotspot identification

**Scheduler Health Metrics**
- pulse_scheduler_queue_due_soon - ready queue depth
- pulse_scheduler_queue_depth - by instance type
- pulse_scheduler_queue_wait_seconds - time in queue histogram
- pulse_scheduler_dead_letter_depth - failed task tracking
- pulse_scheduler_breaker_state - circuit breaker state
- pulse_scheduler_breaker_failure_count - consecutive failures
- pulse_scheduler_breaker_retry_seconds - time until retry
- Enable alerting on DLQ spikes, breaker opens, queue backlogs

**Diagnostics Endpoint Caching**
- pulse_diagnostics_cache_hits_total - cache performance
- pulse_diagnostics_cache_misses_total - cache misses
- pulse_diagnostics_refresh_duration_seconds - probe timing
- 45-second TTL prevents thundering herd on /api/diagnostics
- Thread-safe with RWMutex
- X-Diagnostics-Cached-At header shows cache freshness

**Debug Log Performance**
- Gate high-frequency debug logs behind IsLevelEnabled() checks
- Reduces CPU waste in production when debug disabled
- Covers scheduler loops, poll cycles, API handlers

**Persistent Logging**
- File logging with automatic rotation
- LOG_FILE, LOG_MAX_SIZE, LOG_MAX_AGE, LOG_COMPRESS env vars
- MultiWriter sends logs to both stderr and file
- Gzip compression support for rotated logs

Files modified:
- internal/api/diagnostics.go (caching layer)
- internal/api/middleware.go (request IDs, HTTP metrics)
- internal/api/http_metrics.go (NEW - HTTP metric definitions)
- internal/logging/logging.go (file logging with rotation)
- internal/monitoring/metrics.go (node + scheduler metrics)
- internal/monitoring/monitor.go (instrumentation, debug gating)

Impact: Dramatically improved production troubleshooting with per-node
visibility, scheduler health metrics, persistent logs, and cached
diagnostics. Fast incident response now possible for multi-node deployments.
2025-10-21 12:37:39 +00:00
rcourtman
bd13b966d0 feat: complete API token export/import with version handling
Complete the API token export/import feature with proper version
handling and backward compatibility:

- Bump export format to version 4.1 to indicate API token support
- Import API tokens when loading v4.1 exports
- Handle version compatibility gracefully:
  - v4.1: Full support including API tokens
  - v4.0: Notice that tokens weren't included (backward compatible)
  - Other: Warning but best-effort import
- Initialize empty array instead of nil for cleaner JSON

This ensures API tokens are properly preserved when migrating or
restoring Pulse instances while maintaining backward compatibility
with older exports.
2025-10-21 11:38:23 +00:00
rcourtman
59cd456428 feat: improve request ID handling in middleware
Enhance request ID middleware to support distributed tracing:

- Honor incoming X-Request-ID headers from upstream proxies/load balancers
- Use logging.WithRequestID() for consistent ID generation across codebase
- Return X-Request-ID in response headers for client correlation
- Include request_id in panic recovery logs for debugging

This enables better request tracing across multiple Pulse instances
and integrates with standard distributed tracing practices.
2025-10-21 11:37:57 +00:00
rcourtman
cdbc6057b0 feat: export API tokens in config export
Add API tokens to the export data so they are included when
exporting/backing up configuration. This ensures API tokens are
preserved when migrating or restoring Pulse instances.

Changes:
- Add APITokens field to ExportData struct
- Load API tokens during export process
- Include tokens in exported JSON (omitempty if none exist)
2025-10-21 11:37:25 +00:00
rcourtman
ad371bf412 feat: improve alert system performance, UX, and edge case handling
Implement 5 medium/low priority improvements identified in systematic review:

UX IMPROVEMENTS:
- Notify existing critical alerts when activating from pending_review state
  Previously: critical alerts during observation window would never notify
  Now: users receive notifications for active critical alerts after activation
  Implementation: Added NotifyExistingAlert() method and logic in ActivateAlerts()

PERFORMANCE OPTIMIZATIONS:
- Replace per-alert cleanup goroutines with periodic batch cleanup
  Prevents spawning 1000s of goroutines during alert flapping
  recentlyResolved entries now cleaned up once per minute instead of 1 goroutine per alert
- Simplify GetActiveAlerts() implementation
  Removed intermediate map copy, holds lock slightly longer but operation is fast
  Cleaner code with reduced memory allocation

CONFIGURATION VALIDATION:
- Validate timezone in quiet hours configuration
  Invalid timezones now disable quiet hours with error log instead of silent fallback
  Prevents unexpected behavior when timezone is typo'd or invalid

GRACEFUL SHUTDOWN:
- Add 100ms delay in Stop() for background goroutine cleanup
  Reduces risk of state corruption during shutdown
  Allows escalation checker and periodic save to exit cleanly

Technical details:
- internal/alerts/alerts.go: Added NotifyExistingAlert(), optimized cleanup patterns
- internal/api/alerts.go: Enhanced ActivateAlerts() to notify existing critical alerts
- Removed ~20 lines of goroutine spawning code
- Added periodic cleanup for recentlyResolved map
- All changes preserve backward compatibility

Testing: Verified compilation with 'go build -o /dev/null ./...'
2025-10-21 11:05:45 +00:00
rcourtman
06b5d5153b fix: resolve critical alert system bugs preventing crashes and memory leaks
Fix 5 critical bugs identified through systematic code review:

CRITICAL FIXES (prevent service crashes):
- Add panic recovery to all alert callbacks (onAlert, onResolved, onEscalate)
- Clone alerts before passing to escalation callback to prevent data races
- Make clearAlertNoLock callback async to prevent deadlock

HIGH PRIORITY FIXES (prevent memory leaks):
- Add cleanup for stale pendingAlerts entries (deleted resources)
- Add cleanup for dockerRestartTracking (ephemeral containers in CI/CD)

MEDIUM PRIORITY FIXES (prevent stuck alerts):
- Validate hysteresis thresholds (ensure clear < trigger)
- Auto-fix invalid configurations with warning logs

Impact:
- Service stability: Malformed webhook URLs or email configs can no longer crash Pulse
- Memory management: Prevents unbounded growth in dynamic environments
- Alert reliability: Prevents alerts that never clear due to invalid thresholds
- Concurrency safety: Eliminates data races in escalation path

Technical details:
- Created safeCallResolvedCallback() and safeCallEscalateCallback() wrappers
- Added ensureValidHysteresis() validation helper
- Extended Cleanup() with pendingAlerts and dockerRestartTracking pruning
- All callbacks now have defer/recover panic handlers with detailed logging

Testing: Verified compilation with 'go build -o /dev/null ./...'
2025-10-21 10:55:57 +00:00
rcourtman
85ffe10aed docs: add Mermaid diagrams to improve visual documentation
Enhance documentation with six Mermaid diagrams to better explain
complex system implementations:

- Adaptive polling lifecycle flowchart showing enqueue→execute→feedback
  cycle with scheduler, priority queue, and worker interactions
- Circuit breaker state machine diagram illustrating Closed↔Open↔Half-open
  transitions with triggers and recovery paths
- Temperature proxy architecture diagram highlighting trust boundaries,
  security controls, and data flow between host/container/cluster
- Sensor proxy request flow sequence diagram showing auth, rate limiting,
  validation, and SSH execution pipeline
- Alert webhook pipeline flowchart detailing template resolution, URL
  rendering, HTTP dispatch, and retry logic
- Script library workflow diagram illustrating dev→test→bundle→distribute
  lifecycle emphasizing modular design

These visualizations make it easier for operators and contributors to
understand Pulse's sophisticated architectural patterns.
2025-10-21 10:40:33 +00:00
rcourtman
66b97333f7 fix: skip update check for source builds and show appropriate UI message
Source builds use commit hashes (main-c147fa1) not semantic versions
(v4.23.0), so update checks would always fail or show misleading
"Update Available" banners.

Changes:
- Add IsSourceBuild flag to VersionInfo struct
- Detect source builds via BUILD_FROM_SOURCE marker file
- Skip update check for source builds (like Docker)
- Update frontend to show "Built from source" message
- Disable manual update check button for source builds
- Return "source" deployment type for source builds

Backend:
- internal/updates/version.go: Add isSourceBuildEnvironment() detection
- internal/updates/manager.go: Skip check with appropriate message
- internal/api/types.go: Add isSourceBuild to API response
- internal/api/router.go: Include isSourceBuild in version endpoint

Frontend:
- src/api/updates.ts: Add isSourceBuild to VersionInfo type
- src/stores/updates.ts: Don't poll for updates on source builds
- src/components/Settings/Settings.tsx: Show "Built from source" message

Fixes the confusing "Update Available" banner for users who explicitly
chose --source to get latest main branch code.

Co-authored-by: Codex AI
2025-10-21 10:08:00 +00:00
rcourtman
56c6c0cc0c feat: improve discovery with progress tracking, validation, and structured errors
Significantly enhanced network discovery feature to eliminate false positives,
provide real-time progress updates, and better error reporting.

Key improvements:
- Require positive Proxmox identification (version data, auth headers, or certificates)
  instead of reporting any service on ports 8006/8007
- Add real-time progress tracking with phase/target counts and completion percentage
- Implement structured error reporting with IP, phase, type, and timestamp details
- Fix TLS timeout handling to prevent hangs on unresponsive hosts
- Expose progress and structured errors via WebSocket for UI consumption
- Reduce log verbosity by moving discovery logs to debug level
- Fix duplicate IP counting to ensure progress reaches 100%

Breaking changes: None (backward compatible with legacy API methods)
2025-10-20 22:29:30 +00:00
rcourtman
8194ce9e7a feat: add containerization detection to version endpoint
Added containerized and containerId fields to /api/version endpoint
to enable automatic temperature proxy installation for LXC containers.

Changes:
- Added Containerized bool field to VersionResponse
- Added ContainerId string field to VersionResponse
- Detect containerization by checking /run/systemd/container file
- Extract container ID from hostname for LXC containers
- Set deployment type from container type (lxc/docker)

This allows the PVE setup script to:
1. Detect that Pulse is running in a container
2. Find the container ID by matching IPs
3. Automatically install pulse-sensor-proxy on the host
4. Configure bind mount for secure socket communication

Fixes the issue where setup script showed 'Proxy not available'
even when Pulse was containerized.
2025-10-20 22:14:03 +00:00
rcourtman
d430efcecb fix: correct fmt.Sprintf argument alignment in PVE setup script
Critical bug fix: The setup script's format string had 33 placeholders
but was only receiving 27 arguments, causing:
- INSTALLER_URL to receive authToken instead of pulseURL
- This made curl try to resolve the token value as a hostname
- Error: 'curl: (6) Could not resolve host: N7AE3P'
- Token ID showed '%!s(MISSING)' in manual setup instructions

Fixed by:
- Added missing tokenName at position 7
- Added literal '%s' strings for version_ge printf placeholders
- Added authToken arguments for Authorization headers (positions 29, 31)
- Ensured all 33 format placeholders have corresponding arguments

Now generates correct URLs:
- INSTALLER_URL: http://192.168.0.160:7655/api/install/install-sensor-proxy.sh
- --pulse-server: http://192.168.0.160:7655
- Token ID: pulse-monitor@pam!pulse-192-168-0-160-[timestamp]
2025-10-20 21:58:37 +00:00
rcourtman
d421f101ba feat: harden temperature proxy installation with better validation and error handling
Setup script improvements (config_handlers.go):
- Remove redundant mount configuration and container restart logic
- Let installer handle all mount/restart operations (single source of truth)
- Eliminate hard-coded mp0 assumption

Installer improvements (install-sensor-proxy.sh):
- Add mount configuration persistence validation via pct config check
- Surface pct set errors instead of silencing with 2>/dev/null
- Capture and display curl download errors with temp files
- Check systemd daemon-reload/enable/restart exit codes
- Show journalctl output when service fails to start
- Make socket verification fatal (was warning)
- Provide clear manual steps when hot-plug fails on running container

This makes the installation fail fast with actionable error messages
instead of silently proceeding with broken configuration.
2025-10-20 21:14:00 +00:00
rcourtman
07f198da63 fix: pass Pulse server URL as argument instead of env var for proxy installer
Changes:
- Replace PULSE_SENSOR_PROXY_FALLBACK_URL env export with --pulse-server argument
- Remove --quiet flag from installer invocation to show download progress
- More reliable than environment variable inheritance in subshells

This ensures the proxy installer can reliably download the binary from the
Pulse server fallback when GitHub is unavailable.
2025-10-20 20:58:25 +00:00
rcourtman
db54233769 fix: show full installer output instead of filtering
The setup script was filtering installer output to only show lines with
✓|⚠️|ERROR, which hid successful download messages like:
'Downloading pulse-sensor-proxy-linux-amd64 from Pulse server...'

This made it appear the installer failed even when the Pulse server
fallback download succeeded. Changed to show all installer output for
better visibility and debugging.

Users will now see the complete installation flow including:
- GitHub download attempt (expected to fail for dev builds)
- Pulse server fallback download (should succeed)
- All setup steps and validations

Improves transparency and reduces confusion during setup
2025-10-20 20:47:41 +00:00
rcourtman
dcad3a3a27 fix: allow dev/main builds to bypass version check
Version check was blocking dev/main builds (e.g., '0.0.0-main-da9da6f')
from using temperature proxy, even though they have the latest code.

Added regex to skip version check for builds matching:
- ^0\.0\.0-main (main branch builds)
- ^dev (dev builds)
- ^main (main version strings)

These builds are assumed to have proxy support since they're built from
the latest codebase.

Fixes testing workflow when installing Pulse with --main flag
2025-10-20 18:19:31 +00:00
rcourtman
93a601d7c7 fix: only check Pulse version for containerized deployments
The version check was blocking ALL v4.23.0 users from temperature monitoring,
even non-containerized ones who don't need the proxy.

Changed to only check version when PULSE_IS_CONTAINERIZED=true, since:
- Non-containerized Pulse can use direct SSH on any version
- Containerized Pulse requires v4.24.0+ for proxy support

This ensures non-containerized v4.23.0 users can still use temperature monitoring
via direct SSH while properly blocking proxy setup for containerized v4.23.0.

Fixes regression introduced in commit fbe4ab83a
2025-10-20 18:03:09 +00:00
rcourtman
001d7f5f1c fix: comprehensive temperature proxy setup improvements
Addresses multiple issues that prevented successful temperature monitoring setup:

1. **Missing log directory (install-sensor-proxy.sh)**
   - Added LogsDirectory=pulse/sensor-proxy to both systemd service templates
   - Fixes crash: "open /var/log/pulse/sensor-proxy/audit.log: read-only file system"
   - Uses systemd's LogsDirectory directive for proper permissions

2. **Invalid pct restart command (install-sensor-proxy.sh:822)**
   - Changed from `pct restart` (doesn't exist) to `pct stop && sleep 2 && pct start`
   - Fixes container restart failures during proxy setup

3. **Version compatibility check (config_handlers.go)**
   - Added const minProxyReadyVersion = "4.24.0"
   - Setup script now queries /api/version endpoint
   - Blocks proxy setup on Pulse < v4.24.0 with clear upgrade message
   - Prevents users from attempting proxy setup on incompatible versions

4. **Proxy service health validation (config_handlers.go)**
   - Verifies pulse-sensor-proxy service is actually running
   - Checks socket exists at /run/pulse-sensor-proxy/pulse-sensor-proxy.sock
   - Shows journalctl command for troubleshooting on failure
   - Sets TEMP_MONITORING_AVAILABLE=false to skip remaining steps

5. **Interactive LXC restart prompt (config_handlers.go)**
   - Replaced passive "please restart" message with interactive prompt
   - Default action is "yes" for easy acceptance
   - Actually executes pct stop/start on confirmation
   - Handles non-interactive environments gracefully

6. **Post-restart socket verification (config_handlers.go)**
   - Validates socket is accessible inside container after restart
   - Provides clear error if mount didn't work
   - Prevents claiming success when setup is incomplete

All changes tested with fresh LXC installation. Temperature monitoring now
works end-to-end with proper error handling and user guidance.

Fixes temperature proxy setup flow for v4.24.0+
2025-10-20 18:00:21 +00:00
rcourtman
5ebb32ce10 feat: enhance runtime configuration and system settings management
Improves configuration handling and system settings APIs to support
v4.24.0 features including runtime logging controls, adaptive polling
configuration, and enhanced config export/persistence.

Changes:
- Add config override system for discovery service
- Enhance system settings API with runtime logging controls
- Improve config persistence and export functionality
- Update security setup handling
- Refine monitoring and discovery service integration

These changes provide the backend support for the configuration
features documented in the v4.24.0 release.
2025-10-20 17:41:19 +00:00
rcourtman
c91b7874ac docs: comprehensive v4.24.0 documentation audit and updates
Complete documentation overhaul for Pulse v4.24.0 release covering all new
features and operational procedures.

Documentation Updates (19 files):

P0 Release-Critical:
- Operations: Rewrote ADAPTIVE_POLLING_ROLLOUT.md as GA operations runbook
- Operations: Updated ADAPTIVE_POLLING_MANAGEMENT_ENDPOINTS.md with DEFERRED status
- Operations: Enhanced audit-log-rotation.md with scheduler health checks
- Security: Updated proxy hardening docs with rate limit defaults
- Docker: Added runtime logging and rollback procedures

P1 Deployment & Integration:
- KUBERNETES.md: Runtime logging config, adaptive polling, post-upgrade verification
- PORT_CONFIGURATION.md: Service naming, change tracking via update history
- REVERSE_PROXY.md: Rate limit headers, error pass-through, v4.24.0 verification
- PROXY_AUTH.md, OIDC.md, WEBHOOKS.md: Runtime logging integration
- TROUBLESHOOTING.md, VM_DISK_MONITORING.md, zfs-monitoring.md: Updated workflows

Features Documented:
- X-RateLimit-* headers for all API responses
- Updates rollback workflow (UI & CLI)
- Scheduler health API with rich metadata
- Runtime logging configuration (no restart required)
- Adaptive polling (GA, enabled by default)
- Enhanced audit logging
- Circuit breakers and dead-letter queue

Supporting Changes:
- Discovery service enhancements
- Config handlers updates
- Sensor proxy installer improvements

Total Changes: 1,626 insertions(+), 622 deletions(-)
Files Modified: 24 (19 docs, 5 code)

All documentation is production-ready for v4.24.0 release.
2025-10-20 17:20:13 +00:00
rcourtman
73fb9d986f feat: add PBS/PMG stubs to test harness and implement HTTP config fetch
Resolves two remaining TODOs from codebase audit.

## 1. PBS/PMG Test Harness Stubs

**Location:** internal/monitoring/harness_integration.go:149-151

**Changes:**
- Added PBS client stub registration: `monitor.pbsClients[inst.Name] = &pbs.Client{}`
- Added PMG client stub registration: `monitor.pmgClients[inst.Name] = &pmg.Client{}`
- Added imports for pkg/pbs and pkg/pmg

**Purpose:**
Enables integration test scenarios to include PBS and PMG instance types
alongside existing PVE support. Stubs allow scheduler to register and
execute tasks for these instance types during integration testing.

**Testing:**
 TestAdaptiveSchedulerIntegration passes (55.5s)
 Integration test harness now supports all three instance types

## 2. HTTP Config URL Fetch

**Location:** cmd/pulse/config.go:226-261

**Problem:**
`PULSE_INIT_CONFIG_URL` was recognized but not implemented, returning
"URL import not yet implemented" error.

**Implementation:**
- URL validation (http/https schemes only)
- HTTP client with 15 second timeout
- Status code validation (2xx required)
- Empty response detection
- Base64 decoding with fallback to raw data
- Matches existing env-var behavior for `PULSE_INIT_CONFIG_DATA`

**Security:**
- Both HTTP and HTTPS supported (HTTPS recommended for production)
- URL scheme validation prevents file:// or other protocols
- Timeout prevents hanging on unresponsive servers

**Usage:**
```bash
export PULSE_INIT_CONFIG_URL="https://config-server/encrypted-config"
export PULSE_INIT_CONFIG_PASSPHRASE="secret"
pulse config auto-import
```

**Testing:**
 Code compiles cleanly
 Follows same pattern as existing PULSE_INIT_CONFIG_DATA handling

## Impact

- Completes integration test infrastructure for all instance types
- Enables automated config distribution via HTTP(S) for container deployments
- Removes last TODOs from codebase (no TODO/FIXME remaining in Go files)
2025-10-20 16:05:45 +00:00
rcourtman
c1bf03fe39 fix: use proper Monitor constructor in PMG tests to initialize all maps
Fixes panic: assignment to entry in nil map in PMG polling tests.

**Problem:**
Tests were manually creating Monitor structs without initializing internal
maps like pollStatusMap, causing nil map panics when recordTaskResult()
tried to update task status.

**Root Cause:**
- TestPollPMGInstancePopulatesState (line 90)
- TestPollPMGInstanceRecordsAuthFailures (line 189)

Both created Monitor with only partial field initialization, missing:
- pollStatusMap
- dlqInsightMap
- instanceInfoCache
- Other internal state maps

**Solution:**
Changed both tests to use New() constructor which properly initializes all
maps and internal state (monitor.go:1541). This ensures tests match production
initialization and will automatically pick up any future map additions.

**Tests:**
 TestPollPMGInstancePopulatesState - now passes
 TestPollPMGInstanceRecordsAuthFailures - now passes
 All monitoring tests pass (0.125s)

Follows best practice: use constructors instead of manual struct creation
to maintain initialization invariants.
2025-10-20 15:22:23 +00:00
rcourtman
039a07b8b0 test: add X-RateLimit-Limit header regression test (#578)
test: add X-RateLimit-Limit header regression test
2025-10-20 16:14:40 +01:00
rcourtman
97871bec82 feat: implement updates rollback logic (Phase 1 follow-up)
Implement complete rollback functionality for systemd/LXC deployments:

**Rollback Strategy:**
- Downloads old binary from GitHub releases
- Restores config from timestamped backups
- Service detection (pulse/pulse-backend/pulse-hot-dev)
- Comprehensive health verification

**Implementation:**

Main rollback flow:
1. Create rollback history entry
2. Detect active service name
3. Download old binary version from GitHub
4. Stop Pulse service
5. Create safety backup of current config
6. Restore config from backup directory
7. Install old binary
8. Start service
9. Wait for health check (30s timeout)
10. Update rollback history (success/failure)

**Helper Functions:**

- detectServiceName(): Auto-detect active service from candidates
- downloadBinary(): Download specific version from GitHub releases
  - Auto-detects architecture (amd64/arm64)
  - Validates download success
  - Sets executable permissions
- stopService/startService(): Systemctl service management
- restoreConfig(): Atomic config restoration
- installBinary(): Safe binary installation with backup
- waitForHealth(): Retry health endpoint with timeout

**Safety Features:**
- Safety backup before restore (rollback-safety timestamp)
- Pre-rollback binary backup (.pre-rollback)
- Health check verification post-rollback
- Comprehensive error logging
- History tracking for audit

**Limitations:**
- Binary backup deleted by install.sh (downloads from GitHub)
- Network dependency for binary retrieval
- Config-only backups from current install.sh

**Testing:**
- Compiles cleanly
- Ready for unit/integration tests

Closes Phase 1 technical debt - rollback capability now functional.

Part of Phase 1 Security Hardening follow-up work
2025-10-20 15:13:38 +00:00
rcourtman
9b1709a05b feat: enhance scheduler health API with rich instance metadata
Add comprehensive instance-level diagnostics to /api/monitoring/scheduler/health

**New Response Structure:**

Enhanced "instances" array with per-instance details:
- Instance metadata: displayName, type, connection URL
- Poll status: last success/error timestamps, error messages, error category
- Circuit breaker: state, timestamps, failure counts, retry windows
- Dead letter: present flag, reason, attempt history, retry schedule

**Implementation:**

Data structures:
- instanceInfo: cache of display names, URLs, types
- pollStatus: tracks successes/errors with timestamps and categories
- dlqInsight: DLQ entry metadata (reason, attempts, schedule)
- circuitBreaker: enhanced with stateSince, lastTransition

Tracking logic:
- buildInstanceInfoCache: populate metadata from config on startup
- recordTaskResult: track poll outcomes, error details, categories
- sendToDeadLetter: capture DLQ insights (reason, timestamps)
- circuitBreaker: record state transitions with timestamps

**Backward Compatible:**
- Existing fields (deadLetter, breakers, staleness) unchanged
- New "instances" array is additive
- Old clients can ignore new fields

**Testing:**
- Unit test: TestSchedulerHealth_EnhancedResponse validates all fields
- Integration tests: still passing (55s)
- All error tracking and breaker history verified

**Operator Benefits:**
- Diagnose issues without log digging
- See error messages directly in API
- Understand breaker states and retry schedules
- Track DLQ entries with full context
- Single API call for complete instance health view

Example: Quickly identify "401 unauthorized" on specific PBS instance,
see it's in DLQ after 5 retries, and know when next retry scheduled.

Part of Phase 2 follow-up work to improve observability.
2025-10-20 15:13:38 +00:00
rcourtman
14d06a1654 test: add soak test with runtime instrumentation (Phase 2 Task 9d)
Add comprehensive soak testing capabilities:

**Runtime Instrumentation:**
- Periodic sampling of heap, stack, goroutines, GC count
- Sample every 10s during harness runs
- HarnessReport includes full RuntimeSamples history
- Detect memory leaks (>10% sustained growth)
- Detect goroutine leaks (>20 leaked goroutines)

**Soak Test:**
- TestAdaptiveSchedulerSoak with 15min+ duration
- Skip unless -soak flag or HARNESS_SOAK_MINUTES set
- 80 synthetic instances (60 healthy, 15 transient, 5 permanent)
- Configurable duration via env var
- Validates: heap growth <10%, goroutines stable, queue depth bounded
- Staleness threshold: 45s for long-running tests

**Wrapper Script:**
- testing-tools/run_adaptive_soak.sh for easy execution
- Accepts duration in minutes: ./run_adaptive_soak.sh 30
- Logs to tmp/adaptive_soak_<timestamp>.log
- Sets proper timeout (duration + 5min buffer)

**Test Results (2-minute validation):**
- 80 instances, 17 samples
- Heap: 2.3MB → 3.1MB (healthy)
- Goroutines: 16 → 6 (no leak, actually decreased)
- Circuit breakers: correctly blocking transient failures

Run with: go test -tags=integration ./internal/monitoring -run TestAdaptiveSchedulerSoak -soak -timeout 20m

Part of Phase 2 Task 9 (Integration/Soak Testing)
2025-10-20 15:13:38 +00:00
rcourtman
2636ba9137 test: add comprehensive integration test harness for adaptive polling (Phase 2 Task 9c)
Add PollExecutor seam and integration test infrastructure:

**PollExecutor Interface:**
- Add pluggable executor interface for testability
- Implement realExecutor wrapping existing poll functions
- Add SetExecutor() for test injection
- Zero impact on production behavior

**Integration Test Harness:**
- Build-tagged integration tests (go:build integration)
- Synthetic workload generator with configurable scenarios
- Fake executor simulating latencies, failures, recovery
- Runtime metrics collection (queue depth, staleness, goroutines)

**Comprehensive Assertions:**
- Queue depth bounds: stays within 1.5× instance count
- Staleness: healthy instances <20s, multiple poll cycles
- Circuit breakers: transient failures recover, permanent stay blocked
- Dead-letter queue: only permanent failures routed
- Scheduler health: snapshot consistency validation

**Test Scenarios:**
- 10 healthy PVE instances (rapid polling)
- 1 transient failure instance (fail → recover)
- 1 permanent failure instance (DLQ routing)
- 55s test duration with 3s base intervals
- Validates full adaptive scheduler lifecycle

Runs with: go test -tags=integration ./internal/monitoring -run TestAdaptiveSchedulerIntegration

Part of Phase 2 Task 9 (Integration/Soak Testing)
2025-10-20 15:13:38 +00:00
rcourtman
7d422d2909 feat: add professional logging with runtime configuration and performance optimization
Implements structured logging package with LOG_LEVEL/LOG_FORMAT env support, debug level guards for hot paths, enriched error messages with actionable context, and stack trace capture for production debugging. Improves observability and reduces log overhead in high-frequency polling loops.
2025-10-20 15:13:38 +00:00
rcourtman
25b797f18d test: add comprehensive staleness tracker unit tests (Phase 2 Task 9b)
Added 17 test cases covering:
- UpdateSuccess/UpdateError state management
- Staleness scoring (fresh, stale, max-stale, never-succeeded)
- Score normalization and capping (0.0 to 1.0 range)
- SetBounds behavior and defaults
- Snapshot merging logic
- Snapshot() API for full state export
- Nil safety and concurrent access

All tests verify correct freshness calculation based on lastSuccess
timestamps and configurable maxStale bounds.

Phase 2 testing status:
-  Backoff exponential growth and jitter (13 tests)
-  Circuit breaker state machine (10 tests)
-  Staleness tracker scoring (17 tests)
- Total: 40+ unit tests covering core scheduling logic
2025-10-20 15:13:38 +00:00
rcourtman
24ae6d8d78 test: add comprehensive unit tests for backoff and circuit breaker (Phase 2 Task 9a)
Added 30+ test cases covering:

Backoff tests (backoff_test.go):
- Exponential growth with multiplier
- Jitter distribution and bounds
- Max delay capping
- Edge cases (negative attempts, zero config values)
- Realistic production scenarios

Circuit breaker tests (circuit_breaker_test.go):
- State transitions: closed → open → half-open → closed
- Retry interval backoff with bit-shifting (5s << failureCount)
- Half-open window behavior
- Concurrent access safety
- Default parameter validation

All tests pass with proper handling of time-based state transitions
and exponential backoff mechanics (bit-shift based retry intervals).
2025-10-20 15:13:38 +00:00
rcourtman
160adeb3b8 feat: add scheduler health API endpoint (Phase 2 Task 8)
Task 8 of 10 complete. Exposes read-only scheduler health data including:
- Queue depth and distribution by instance type
- Dead-letter queue inspection (top 25 tasks with error details)
- Circuit breaker states (instance-level)
- Staleness scores per instance

New API endpoint:
  GET /api/monitoring/scheduler/health (requires authentication)

New snapshot methods:
- StalenessTracker.Snapshot() - exports all staleness data
- TaskQueue.Snapshot() - queue depth & per-type distribution
- TaskQueue.PeekAll() - dead-letter task inspection
- circuitBreaker.State() - exports state, failures, retryAt
- Monitor.SchedulerHealth() - aggregates all health data

Documentation updated with API spec, field descriptions, and usage examples.
2025-10-20 15:13:38 +00:00
rcourtman
b1f445b33d feat: implement error handling with circuit breakers and backoff (Phase 2 Task 7)
Adds comprehensive error resilience:
- Circuit breaker with closed/open/half-open states (3 failures = trip)
- Exponential backoff with jitter (2s initial, 2x multiplier, 5min max)
- Dead-letter queue for tasks exceeding 5 retry attempts
- Error classification (transient vs permanent) using internal/errors helpers
- Per-instance failure tracking and breaker state management
- Integration with staleness tracker for outcome recording

Task 7 of 10 complete (70%). Ready for API surfaces and testing.
2025-10-20 15:13:37 +00:00
rcourtman
aa5c08ad4a feat: implement priority queue-based task execution (Phase 2 Task 6)
Replaces immediate polling with queue-based scheduling:
- TaskQueue with min-heap (container/heap) for NextRun-ordered execution
- Worker goroutines that block on WaitNext() until tasks are due
- Tasks only execute when NextRun <= now, respecting adaptive intervals
- Automatic rescheduling after execution via scheduler.BuildPlan
- Queue depth tracking for backpressure-aware interval adjustments
- Upsert semantics for updating scheduled tasks without duplicates

Task 6 of 10 complete (60%). Ready for error/backoff policies.
2025-10-20 15:13:37 +00:00
rcourtman
c554380cb5 feat: verify adaptive interval logic implementation (Phase 2 Task 5)
Confirms adaptive scheduling logic is fully operational:
- EMA smoothing (alpha=0.6) to prevent interval oscillations
- Staleness-based interpolation between min/max intervals
- Error penalty (0.6x per error) for faster recovery detection
- Queue depth stretch (0.1x per task) for backpressure handling
- ±5% jitter to prevent thundering herd effects
- Per-instance state tracking for smooth transitions

Task 5 of 10 complete. Scheduler foundation ready for queue-based execution.
2025-10-20 15:13:37 +00:00
rcourtman
c7d1abf874 feat: implement staleness tracker for adaptive polling (Phase 2 Task 4)
Adds freshness metadata tracking for all monitored instances:
- StalenessTracker with per-instance last success/error/mutation timestamps
- Change hash detection using SHA1 for detecting data mutations
- Normalized staleness scoring (0-1 scale) based on age vs maxStale
- Integration with PollMetrics for authoritative last-success data
- Wired into all poll functions (PVE/PBS/PMG) via UpdateSuccess/UpdateError
- Connected to scheduler as StalenessSource implementation

Task 4 of 10 complete. Ready for adaptive interval logic.
2025-10-20 15:13:37 +00:00
rcourtman
57429900a6 feat: add adaptive polling scheduler infrastructure (Phase 2 Tasks 1-3)
Implements adaptive scheduling foundation for Phase 2:
- Poll cycle metrics: duration, staleness, queue depth, in-flight counters
- Adaptive scheduler with pluggable staleness/interval/enqueue interfaces
- Config support: ADAPTIVE_POLLING_ENABLED flag + min/max/base intervals
- Feature flag defaults to disabled for safe rollout
- Scheduler wiring into Monitor with conditional instantiation

Tasks 1-3 of 10 complete. Ready for staleness tracker implementation.
2025-10-20 15:13:37 +00:00
rcourtman
524f42cc28 security: complete Phase 1 sensor proxy hardening
Implements comprehensive security hardening for pulse-sensor-proxy:
- Privilege drop from root to unprivileged user (UID 995)
- Hash-chained tamper-evident audit logging with remote forwarding
- Per-UID rate limiting (0.2 QPS, burst 2) with concurrency caps
- Enhanced command validation with 10+ attack pattern tests
- Fuzz testing (7M+ executions, 0 crashes)
- SSH hardening, AppArmor/seccomp profiles, operational runbooks

All 27 Phase 1 tasks complete. Ready for production deployment.
2025-10-20 15:13:37 +00:00
rcourtman
6619dc803e refactor: use strconv.Itoa instead of string(rune()) in test
Replace string(rune(i)) with strconv.Itoa(i) in hub_concurrency_test.go
for generating client IDs. While this is test code and not a production bug,
it uses the same incorrect pattern that caused the PR #575 bug.

This ensures consistent best practices across the codebase and avoids
confusion for developers who might copy this pattern.

Related: #575
2025-10-20 15:12:14 +00:00
rcourtman
8d6346a008 test: add X-RateLimit-Limit header regression test
Add regression test for PR #575 to ensure rate limit headers are formatted
as decimal strings (e.g., "10") instead of Unicode control characters.

Also fixes pre-existing fmt.Sprintf argument count mismatch in PVE setup
script (internal/api/config_handlers.go:3077). The template had 28 format
specifiers (excluding %%s escape sequence) but was only receiving 24
arguments. Added missing pulseURL and tokenName arguments to match template.

Related: #575
2025-10-20 15:10:59 +00:00
rcourtman
20d94f4c90 Fix X-RateLimit-Limit header value (#575)
Fix X-RateLimit-Limit header value
2025-10-20 15:57:28 +01:00
rcourtman
1e25fa572a security: add resilience and error handling to tempproxy client
Implements comprehensive client-side improvements for production reliability:

1. Context Support with Deadlines:
   - Added callWithContext() for context-aware RPC calls
   - Respects context deadlines and cancellation
   - Prevents goroutine pileup under network issues

2. Exponential Backoff with Jitter:
   - Automatic retry with exponential backoff (100ms → 10s)
   - ±10% jitter to prevent thundering herd
   - Max 3 retries for transient failures
   - Smart retry decision based on error classification

3. Error Classification:
   - ProxyError type with classification (Transport, Auth, SSH, Sensor, Timeout)
   - Retryable vs non-retryable error identification
   - Better error messages for debugging
   - Structured error handling throughout

4. Improved Connection Handling:
   - DialContext for cancellable connections
   - Proper deadline propagation
   - Clean separation of single-attempt vs retry logic
   - Legacy call() method preserved for backwards compatibility

Security Notes:
- SSH fallback already blocked in containers (temperature.go:69-77)
- Per-client token auth not needed after method-level authz (commit d55112ac4)
- ID-mapped root blocked from privileged methods

Performance:
- No retry on non-retryable errors (auth, sensor failures)
- Context cancellation short-circuits retry loops
- Jitter prevents synchronized retry storms

Addresses Codex findings #4 and #5 from security audit.
2025-10-19 16:37:11 +00:00
rcourtman
049f79987f feat: add turnkey Docker installer with automatic proxy setup
Adds a one-command Docker deployment flow that:
- Detects if running in LXC and installs Docker if needed
- Automatically installs pulse-sensor-proxy on the Proxmox host
- Configures bind mount for proxy socket into LXC
- Generates optimized docker-compose.yml with proxy socket
- Enables temperature monitoring via host-side proxy

The install-docker.sh script handles the complete setup including:
- Docker installation (if needed)
- ACL configuration for container UIDs
- Bind mount setup
- Automatic apparmor=unconfined for socket access

Accessible via: curl -sSL http://pulse:7655/api/install/install-docker.sh | bash
2025-10-19 15:03:24 +00:00
rcourtman
a841a1a6fe fix: show success message instead of warning when using pulse-sensor-proxy
When the setup script detects TEMPERATURE_PROXY_KEY (proxy is available),
it now shows a clear success message instead of attempting SSH verification.

The verification check doesn't work with proxy-based setups since the
container doesn't have SSH keys - all temperature collection happens via
the Unix socket to pulse-sensor-proxy, which handles SSH.

Now shows:
✓ Temperature monitoring configured via pulse-sensor-proxy
  Temperature data will appear in the dashboard within 10 seconds

Instead of the misleading:
⚠️  Unable to verify SSH connectivity.
   Temperature data will appear once SSH connectivity is configured.
2025-10-19 14:06:18 +00:00
rcourtman
557eedb247 fix: detect and use proxy SSH key in setup script for Docker deployments
When pulse-sensor-proxy is available, the setup script now automatically
detects and uses the proxy's SSH public key instead of trying to generate
keys inside the container.

This fixes temperature monitoring setup for Docker deployments where:
- Container has proxy socket mounted at /mnt/pulse-proxy
- Proxy handles SSH connections to nodes
- Setup script needs to distribute the proxy's key, not container's key

The fix queries /api/system/proxy-public-key during setup script generation
and overrides SSH_SENSORS_PUBLIC_KEY if the proxy is available.

Tested with Docker on native Proxmox host (delly) - temperatures collected
successfully via proxy socket.
2025-10-19 13:50:08 +00:00
Sangar
ce21a6b94f Fix X-RateLimit-Limit header value 2025-10-19 11:43:03 +02:00
rcourtman
21712111e7 fix: enable variable expansion in cluster node SSH key heredoc
Changed heredoc delimiter from <<'EOF' to <<EOF to allow bash variable
expansion. Previously $SSH_PUBLIC_KEY and $SSH_RESTRICTED_KEY_ENTRY
were being passed as literal strings instead of their actual values,
so cluster nodes never received the correct SSH keys.

This fixes cluster node ProxyJump setup - now both restricted and
unrestricted keys are properly added to cluster nodes.
2025-10-19 09:08:00 +00:00
rcourtman
c17059ca8e fix: add ProxyJump key to all cluster nodes automatically
The setup script now adds both the restricted and unrestricted SSH keys
to ALL cluster nodes, not just the first one. This makes temperature
monitoring truly turnkey - you say 'yes' to configure cluster nodes and
it automatically sets up both keys on each node.

This ensures:
- All nodes can act as ProxyJump hosts if needed
- All nodes can provide temperature data via sensors
- No manual SSH key configuration required

Fixes turnkey cluster temperature monitoring setup.
2025-10-19 09:02:28 +00:00
rcourtman
bfde490ad4 fix: add unrestricted SSH key for ProxyJump on jump host
When using ProxyJump for cluster temperature monitoring, the jump host
(typically the first cluster node) needs an unrestricted SSH key to allow
connection forwarding. Previously only the restricted key with
command="sensors -j" was added, which blocked ProxyJump.

Now the setup script adds TWO keys:
1. Unrestricted key (for ProxyJump/connection forwarding)
2. Restricted key (for running sensors -j directly)

This allows containerized Pulse to:
- Connect through the jump host to other cluster nodes
- Collect temperature data from all cluster members

Fixes cluster temperature monitoring for Docker/LXC deployments.
2025-10-19 08:56:52 +00:00
rcourtman
78c2228b89 fix: add HostName entries for cluster nodes in SSH config
Added logic to resolve IP addresses for cluster nodes and include them as
HostName entries in the SSH config. Without this, Pulse couldn't connect
to cluster nodes like 'minipc' because the container couldn't resolve
the hostname.

Uses getent to resolve node names to IPs, with fallback to hostname if
resolution fails (for environments where DNS works).
2025-10-19 08:48:25 +00:00
rcourtman
dd70bdee08 feat: switch to Ed25519 SSH keys and add openssh-client to container
- Changed SSH key generation from RSA 2048 to Ed25519 (more secure, faster, smaller)
- Added openssh-client package to Docker image (required for temperature monitoring)
- Updated SSH config template to use id_ed25519
- Removed unused crypto/rsa and crypto/x509 imports

Ed25519 provides better security with shorter keys and faster operations
compared to RSA. The container now has SSH client tools needed to connect
to Proxmox nodes for temperature data collection.
2025-10-19 08:43:20 +00:00
rcourtman
6acfc3f121 fix: use id_rsa in SSH config instead of id_ed25519
The setup script was generating SSH config with IdentityFile ~/.ssh/id_ed25519
but Pulse generates id_rsa keys. Updated SSH config template to use id_rsa
to match the actual key type generated by the monitoring system.
2025-10-19 08:39:55 +00:00
rcourtman
759a3b7d2f fix: bypass middleware auth for ssh-config with setup token
Added middleware exception for /api/system/ssh-config when a valid setup
token is provided, matching the pattern used for verify-temperature-ssh.

The middleware was blocking ssh-config requests before they reached the
handler, even though the handler had setup token validation logic.
2025-10-19 08:35:39 +00:00
rcourtman
4b1d0013c0 fix: allow setup token auth for SSH config endpoint
The ssh-config endpoint was using RequireAuth which only accepts Pulse
API tokens, but the setup script sends a temporary setup token via the
auth_token parameter. Updated to follow the same pattern as
verify-temperature-ssh: check setup token first, then fall back to API auth.

This fixes the 401 error when the setup script tries to configure ProxyJump
for containerized Pulse deployments.
2025-10-19 08:31:05 +00:00
rcourtman
8c51ba727d fix: pass authToken to verify-temperature-ssh endpoint
The setup script was passing pulseURL instead of authToken as the last
parameter, causing 'Authentication required' errors when verifying SSH
connectivity. Fixed parameter order in fmt.Sprintf call.
2025-10-19 08:23:31 +00:00
rcourtman
74c426b87a feat: implement allowlist-based SSH config validation per Codex review
Security improvements to HandleSSHConfig endpoint:
- Add defer r.Body.Close() for proper resource cleanup
- Return 413 status for oversized requests with errors.As check
- Switch from blocklist to allowlist-based directive validation
- Use case-insensitive parsing with comment stripping via bufio.Scanner
- Add Content-Type: application/json header to response

Codex identified that blocklist approach was insufficient and recommended
allowlist validation to prevent unexpected directives. Only permits the
specific SSH directives Pulse needs for ProxyJump configuration.
2025-10-18 23:27:14 +00:00
rcourtman
71abcb2a37 fix: harden SSH config endpoint per Codex security review
Addressed security concerns identified by Codex code review:

1. **Memory exhaustion protection**
   - Added http.MaxBytesReader with 32KB limit
   - Prevents malicious large POST from killing server

2. **Dangerous directive blocking**
   - Reject ProxyCommand, LocalCommand, RemoteCommand
   - Prevents command injection via SSH config

3. **Improved error handling**
   - Check all error returns properly
   - Return 5xx on failures
   - Log file size and path for debugging

4. **Scoped SSH config (critical fix)**
   - Changed from `Host *` to specific cluster nodes
   - Prevents overriding ALL SSH connections
   - Only affects Proxmox nodes for temperature monitoring
   - Preserves other SSH functionality (git, etc.)

Before: Host * broke all SSH connections from Pulse
After: Only Proxmox cluster nodes use ProxyJump

Credit: Codex code review identified these issues
2025-10-18 23:21:59 +00:00
rcourtman
8595b4c001 feat: automatic ProxyJump for turnkey temperature monitoring
Make temperature monitoring truly turnkey by automatically configuring
SSH ProxyJump when running in containers without pulse-sensor-proxy.

How it works:
1. Setup script runs on Proxmox host (e.g., delly)
2. Detects Pulse is containerized but proxy unavailable
3. Automatically configures SSH ProxyJump through the current host
4. Writes SSH config to /home/pulse/.ssh/config in container
5. Temperature monitoring "just works" without manual configuration

Changes:
- Track TEMP_MONITORING_AVAILABLE flag during proxy installation
- Auto-configure ProxyJump if proxy installation fails
- Add /api/system/ssh-config endpoint to write SSH config
- Only prompt for temperature monitoring if it can actually work
- Automatic SSH config: ProxyJump through Proxmox host

Before: User had to manually configure ProxyJump or install proxy
After: Temperature monitoring works automatically after setup script

This makes Docker deployments as turnkey as LXC deployments.
2025-10-18 23:17:38 +00:00
rcourtman
77b4ccf592 feat: simplify SSH verification failure messaging
Changed the SSH connectivity check failure message from a scary
"FAILED" warning with complex ProxyJump instructions to a simple
informational message.

Before:
- ⚠️ SSH connectivity FAILED for: ...
- Complex multi-line ProxyJump configuration
- Confusing for users who don't need temperature monitoring

After:
- ℹ️  Temperature monitoring will be available once SSH configured
- Simple list of pending nodes
- Brief note about pulse-sensor-proxy for LXC
- Link to docs for details

This makes the setup experience much more turnkey by reducing
noise and focusing on successful completion rather than optional
features that require additional configuration.
2025-10-18 23:08:11 +00:00
rcourtman
92215c012e feat: improve setup script turnkey experience
Setup Script Improvements:
- Remove confusing "Could not download installer" warning for proxy
- Skip SSH connectivity check in containerized environments without proxy
- Simplify proxy installation prompts (automatic when available)
- Better messaging for containerized setups

These changes make the setup script more turnkey by reducing noise
and warnings that don't apply to test/development environments or
containerized installations.
2025-10-18 23:01:02 +00:00
rcourtman
b640347a78 fix: improve discovery performance and reliability
Discovery Fixes:
- Always update cache even when scan finds no servers (prevents stale data)
- Remove automatic re-add of deleted nodes to discovery (was causing confusion)
- Optimize Docker subnet scanning from 762 IPs to 254 IPs (3x faster)
- Add getHostSubnetFromGateway() to detect host network from container

Frontend Type Fixes:
- Fix ThresholdsTable editScope type errors
- Fix SnapshotAlertConfig index signature
- Remove unused variable in Settings.tsx

These changes make discovery faster, more reliable, and fix the issue where
deleted nodes would persist in the discovery cache or immediately reappear.
2025-10-18 22:59:40 +00:00
rcourtman
2045bcfdd6 fix: detect containerized Pulse with healthy status
Fixes container detection when Docker health checks are enabled.
Previously, the setup script only matched "running" status exactly,
causing it to skip containers showing "running (healthy)" status.

This prevented:
- Proper detection of containerized Pulse installations
- pulse-sensor-proxy installation for temperature monitoring
- Temperature data collection for affected users

The fix captures the full status output and searches for "running"
anywhere in the output, supporting all status variations:
- status: running
- status: running (healthy)
- status: running (unhealthy)

Related to #101
2025-10-18 20:23:05 +00:00
Pulse Automation Bot
cfdfe896be Adjust backup and snapshot alert handling 2025-10-18 20:11:01 +00:00
Pulse Automation Bot
80b9d0602a Add Apprise notification integration (#570) 2025-10-18 16:39:39 +00:00
Pulse Automation Bot
0b4e4f9c59 Add configurable backup polling interval 2025-10-18 13:06:41 +00:00
Richard Courtman
97b9c6739c feat: add min/max temperature tracking for nodes
Track minimum and maximum CPU temperatures since monitoring started.
This provides better insight into temperature trends and cooling
adequacy over time.

Changes:
- Backend: Add CPUMin, CPUMaxRecord, MinRecorded, MaxRecorded fields
  to Temperature model
- Backend: Implement min/max tracking logic in monitoring cycle that
  preserves values across polling cycles
- Backend: Initialize min/max on first reading, update on extremes
- Frontend: Update Temperature TypeScript interface with new fields
- Frontend: Display min/max range in NodeCard tooltip (e.g., "52°C
  (48-67°C since monitoring started)")
- Frontend: Rebuild dist assets

Temperature display now shows:
- Current temperature with color coding (green/yellow/red)
- Tooltip with full min-max range and context
- Min/max tracked in-memory (resets on Pulse restart)

Example tooltip: "CPU: 52°C (48-67°C since monitoring started)"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-18 08:15:10 +00:00
Richard Courtman
27db397f62 fix: use consistent number-based prompts in setup script
- Changed temperature monitoring menu from [K/r/s] to [1/2/3]
- Now all multi-choice menus use numbers consistently
- Main menu: [1/2/3]
- Temperature menu: [1/2/3] (was [K/r/s])
- Yes/no questions still use y/n (standard convention)
2025-10-18 07:39:19 +00:00
Richard Courtman
2ba50c24c0 fix: remove extra sprintf arguments causing setup script syntax error 2025-10-18 07:11:19 +00:00
Richard Courtman
de3bb47930 fix: improve turnkey temperature monitoring for standalone nodes
- Fix script input handling to work with standard curl | bash pattern by prioritizing /dev/tty
- Add Raspberry Pi temperature sensor support (cpu_thermal chip and generic temp sensors)
- Add comprehensive documentation for turnkey standalone node setup
- Fix printf formatting error in setup script
2025-10-18 06:51:56 +00:00
Richard Courtman
669d7dc05c feat: add turnkey temperature monitoring for standalone nodes
Implements automatic temperature monitoring setup for standalone
Proxmox/Pimox nodes without manual SSH key configuration.

Changes:
- Add /api/system/proxy-public-key endpoint to expose proxy's SSH public key
- Setup script now detects standalone nodes (non-cluster)
- Auto-fetches and installs proxy SSH key with forced commands
- Add Raspberry Pi temperature support via cpu_thermal and /sys/class/thermal
- Enhance setup script with better error handling for lm-sensors installation
- Add RPi detection to skip lm-sensors and use native thermal interface

Security:
- Public key endpoint is safe (public keys are meant to be public)
- All installed keys use forced command="sensors -j" with full restrictions
- No shell access, port forwarding, or other SSH features enabled
2025-10-17 22:15:50 +00:00
rcourtman
5886b920ba fix: improve sensor proxy install script reliability
Fixes two issues with the sensor proxy installation:
1. Local node IP detection now uses exact matching instead of substring matching to avoid false negatives
2. Removes duplicate output filtering in the setup script wrapper

These changes ensure that the proxy SSH key is correctly configured on the local node during cluster installations.
2025-10-17 19:09:54 +00:00
rcourtman
123e0f04ca feat: add comprehensive node cleanup system
Implements automated cleanup workflow when nodes are deleted from Pulse, removing all monitoring footprint from the host. Changes include a new RPC handler in the sensor proxy for cleanup requests, enhanced node deletion modal with detailed cleanup explanations, and improved SSH key management with proper tagging for atomic updates.
2025-10-17 18:53:45 +00:00
rcourtman
d0f7fd6404 fix: setup script now configures proxy socket bind mount
The setup script was restarting the container but never running the
pct set command to configure the bind mount. This meant the socket
was never accessible inside the container.

Now runs: pct set <ctid> -mp0 /run/pulse-sensor-proxy,mp=/mnt/pulse-proxy
before restarting the container to ensure the mount is configured.
2025-10-17 14:25:30 +00:00
rcourtman
65b696f2d6 fix: remove unused log import from tempproxy client
Leftover from removing EnsureClusterKeys() method. Caused compile failure
preventing hot-dev from starting.
2025-10-17 14:15:37 +00:00
rcourtman
f141f7db33 feat: enhance sensor proxy with improved cluster discovery and SSH management
Improvements to pulse-sensor-proxy:
- Fix cluster discovery to use pvecm status for IP addresses instead of node names
- Add standalone node support for non-clustered Proxmox hosts
- Enhanced SSH key push with detailed logging, success/failure tracking, and error reporting
- Add --pulse-server flag to installer for custom Pulse URLs
- Configure www-data group membership for Proxmox IPC access

UI and API cleanup:
- Remove unused "Ensure cluster keys" button from Settings
- Remove /api/diagnostics/temperature-proxy/ensure-cluster-keys endpoint
- Remove EnsureClusterKeys method from tempproxy client

The setup script already handles SSH key distribution during initial configuration,
making the manual refresh button redundant.
2025-10-17 11:43:26 +00:00
rcourtman
e367cd384e fix: normalize version strings with build metadata in tests 2025-10-16 09:07:40 +00:00
rcourtman
f7c4e9cdde fix: resolve race condition in mock mode update loop 2025-10-16 09:04:50 +00:00
rcourtman
219fcc6de5 Stop disabled metrics from sending webhooks
Refs #561
2025-10-16 08:57:12 +00:00
rcourtman
79dc620b34 Docker agent: add arch-aware self-update download
Refs #526
2025-10-16 08:43:59 +00:00
rcourtman
61b492e93b Fix race in mock mode and address frontend type checks 2025-10-16 08:28:17 +00:00
rcourtman
6fdef61710 Expand monitoring and discovery test coverage 2025-10-16 08:17:08 +00:00
rcourtman
3a4fc044ea Add guest agent caching and update doc hints (refs #560) 2025-10-16 08:15:49 +00:00
rcourtman
1d580c658d chore: bump version to v4.24.0 2025-10-15 22:26:24 +00:00
rcourtman
4838793677 feat: enhance alerts system with tests and improved thresholds
- Add comprehensive test coverage for alerts package with 285+ new tests
- Implement ThresholdsTable component with metric thresholds display
- Enhance Alerts page UI with improved layout and metric filtering
- Add frontend component tests for Alerts page and ThresholdsTable
- Set up Vitest testing infrastructure for SolidJS components
- Improve config persistence with better validation
- Expand discovery tests with 333+ test cases
- Update API, configuration, and Docker monitoring documentation
2025-10-15 22:25:04 +00:00