Commit graph

251 commits

Author SHA1 Message Date
rcourtman
f8647b53ff Fix mp mount detection pattern for pulse-sensor-proxy
The grep pattern was looking for 'pulse-sensor-proxy' as a standalone
string, but the actual mount line contains paths like:
  mp0: /run/pulse-sensor-proxy,mp=/mnt/pulse-proxy,replicate=0

This caused the removal logic to never execute, leaving the old mp
mount in place and preventing the migration to lxc.mount.entry format.

Changed pattern to match either path component:
- /pulse-sensor-proxy (source path)
- /mnt/pulse-proxy (mount point)

Also removed space after colon in pattern to match actual format.

This completes the fix for temperature proxy setup on LXC containers.
2025-11-22 22:34:26 +00:00
rcourtman
080453b68a Fix LXC config modification for Proxmox pmxcfs filesystem
The /etc/pve/ directory is a clustered FUSE filesystem (pmxcfs) managed
by Proxmox. Direct modifications using sed -i or echo >> don't work
reliably on this filesystem, and LXC config files contain snapshot
sections that must be preserved.

Changes:
- Use temp file approach: copy config, modify temp, copy back to trigger sync
- Only modify main config section (before first [snapshot] marker)
- Properly handle both mp mount removal and lxc.mount.entry addition
- Apply fix to both install.sh and install-sensor-proxy.sh

This fixes temperature proxy setup failures where the socket mount
entry wasn't being persisted to the container configuration.

Related to #628
2025-11-22 22:19:00 +00:00
rcourtman
ee3f1e0cdb Handle standby SMART temps and capture disk identity 2025-11-22 07:35:13 +00:00
rcourtman
bbbb417b01 Prevent token manager auth swap and fix docker agent perms (Related to #740) 2025-11-22 07:18:42 +00:00
rcourtman
91c51689c9 Guard host-agent installs on noexec filesystems (Related to #718) 2025-11-21 23:00:47 +00:00
rcourtman
19408fdb2b Related to #738: make pulse proxy mount migration-safe 2025-11-21 21:29:14 +00:00
rcourtman
548c65c1a3 Harden release validation for host agent downloads (related to #735) 2025-11-21 10:47:53 +00:00
rcourtman
30a069dfa6 Add TrueNAS SCALE persistence for host agent (Related to #718) 2025-11-21 10:07:14 +00:00
rcourtman
0adc4e7838 Initialize ObservedValues in Windows installer 2025-11-20 21:01:44 +00:00
rcourtman
34b4b0a4d6 fix(installer): handle legacy sensor-proxy config commands 2025-11-20 20:33:51 +00:00
courtmanr@gmail.com
17d55b8cf4 feat: implement atomic config management in sensor proxy 2025-11-20 19:01:24 +00:00
rcourtman
18b0323340 Add agent-id support to host agent installers (Related to #721) 2025-11-20 18:14:18 +00:00
courtmanr@gmail.com
838993cf40 Implement sensor proxy installation and configuration updates 2025-11-20 13:23:21 +00:00
courtmanr@gmail.com
3f974c06ca Fix macOS build for sensor-proxy and improve hot-dev script 2025-11-20 12:28:01 +00:00
courtmanr@gmail.com
1e229d5f38 Improve login page UI and fix hot-dev script for macOS 2025-11-20 12:21:49 +00:00
courtmanr@gmail.com
8bfdf3769b Update config persistence, crypto, and dev script 2025-11-20 11:46:20 +00:00
rcourtman
f4e2cbca48 Improve token collision handling and installer subnet support 2025-11-20 09:45:36 +00:00
rcourtman
4067b0717a Improve Windows installer arch detection (related to #723) 2025-11-20 09:37:45 +00:00
rcourtman
844c895521 WIP: Fix temperature proxy registration persistence (incomplete)
This commit contains multiple fixes for temperature proxy registration,
but the core issue remains unresolved.

## What's Fixed:
1. Added config pointer and reloadFunc to TemperatureProxyHandlers
2. Added SetConfig method to keep handler in sync with router config changes
3. Added config reload after registration to prevent monitor from overwriting
4. Fixed installer port conflict detection and duplicate YAML key issues
5. Added comprehensive debug logging throughout registration flow

## What's Still Broken:
The TemperatureProxyURL, TemperatureProxyToken, and TemperatureProxyControlToken
fields are NOT persisting to nodes.enc after SaveNodesConfig is called.

Debug logs confirm:
- HandleRegister correctly updates nodesConfig.PVEInstances[matchedIndex]
- The correct data is passed to SaveNodesConfig (verified in logs)
- SaveNodesConfig completes without errors
- Config reload executes successfully
- BUT after Pulse restart, the fields are empty when loaded from disk

The bug is in SaveNodesConfig serialization or file writing logic itself.

Related files:
- internal/api/temperature_proxy.go: Registration handler
- internal/config/persistence.go: SaveNodesConfig implementation
- internal/config/config.go: PVEInstance struct definition
2025-11-19 20:12:19 +00:00
rcourtman
320ef6da71 fix(sensor-proxy): ensure correct config.yaml permissions after modifications
Fixed bug where config.yaml would end up with root:root 600 permissions
after the installer modified it, causing service startup failures with
"permission denied" errors.

Root cause: Two code paths modified config.yaml without resetting ownership:
1. ensure_control_plane_config() - used mktemp (creates root-owned file),
   then mv'd it over config.yaml without chown/chmod
2. HTTP mode configuration - appended to config.yaml without resetting perms

Fix: Added chown/chmod after both modifications:
- Line 1601-1602: After control-plane config update
- Line 1860-1861: After HTTP mode config append

Now config.yaml maintains pulse-sensor-proxy:pulse-sensor-proxy 644
permissions after all modifications, allowing the service to start correctly.

This bug was discovered during repair logic testing - the service failed
to start after the installer ran, even though the fmt.Sprintf argument
alignment fix was working correctly.
2025-11-19 14:53:44 +00:00
courtmanr@gmail.com
a39fdc9e99 fix: local dev setup, encryption key generation, and pnpm lockfile 2025-11-19 14:48:09 +00:00
rcourtman
8e07b831a3 fix(docker): always reinstall sensor-proxy to refresh tokens/config
Same turnkey UX issue as PVE quick setup: when local sensor-proxy socket
exists and validates, install-docker.sh would skip reinstallation (line 498).
This left stale control-plane tokens and URLs.

Fix: Always reinstall when LOCAL_PROXY_EXISTED_AT_START=true
- Track socket existence at script start
- Change message: 'will refresh to update tokens/config'
- Set SKIP_INSTALLATION=false to force reinstall
- Installer is idempotent (Phase 2), safe to rerun

This completes the turnkey repair fix across all entry points (PVE quick
setup and Docker installer).
2025-11-19 13:37:06 +00:00
rcourtman
c5c1d64a7b feat(sensor-proxy): improve turnkey setup experience with Pulse restart handling
- Update installer to use v4.32.0 Phase 2 binaries with file-based config
- Add automatic detection of Pulse service (systemd/hot-dev/docker)
- Add --restart-pulse flag for automatic Pulse restart in dev/test environments
- Default behavior shows clear instructions to restart Pulse manually (safe for production)
- Add prominent restart notice with command suggestions based on detected deployment
- Improve UX by making restart step impossible to miss

Related to Phase 2 sensor-proxy architecture improvements
2025-11-19 12:44:07 +00:00
rcourtman
879de50131 fix(sensor-proxy): eliminate race in migration script
Stop pulse-sensor-proxy service before modifying config.yaml to prevent
races with the running daemon or concurrent CLI commands.

The migration script now:
1. Stops service if running
2. Updates config atomically (temp + rename in same directory)
3. Restarts service if it was running

This achieves complete architectural isolation - ALL config file writers
are now coordinated (either through Phase 2 CLI locking or by ensuring
the service is stopped during modification).

Addresses final Codex review feedback.
2025-11-19 11:04:58 +00:00
rcourtman
a24e9053d1 fix(sensor-proxy): ensure migrate script atomic write is same-filesystem
Create temp file in same directory as config.yaml to ensure mv is truly
atomic (won't degrade to copy+unlink on different filesystems).

Added comments noting this is a legacy migration script with minor race
risk (no file locking) that should be deprecated once all users upgrade
to v4.32+.
2025-11-19 11:02:14 +00:00
rcourtman
de7fd72b47 fix(sensor-proxy): fix remaining unsafe config writers
1. Self-heal script: Add BINARY_PATH variable so CLI migration actually runs
   - Previously logged "Binary not available" and skipped migration

2. migrate-sensor-proxy-control-plane.sh: Use atomic write (temp + rename)
   - Prevents partial writes if script is interrupted
   - Reduces race window with running service

These were the remaining gaps identified by Codex review.

NOTE: migrate-sensor-proxy-control-plane.sh still uses Python manipulation
instead of the Phase 2 CLI, but as a one-time migration script for upgrades
from v4.31, the atomic write provides sufficient protection. Future versions
can deprecate this script entirely.
2025-11-19 10:59:54 +00:00
rcourtman
9d20acd35e fix(sensor-proxy): eliminate all uncoordinated config writers
Remove all code paths that manipulate config files without Phase 2 locking:

1. Installer: Remove ensure_allowed_nodes_file_reference() call (line 1674)
   - Migration now handled exclusively by config migrate-to-file

2. Installer: Make migration failures fatal in update_allowed_nodes()
   - Prevents fallback to unsafe Python manipulation

3. Daemon sanitizer: Remove os.WriteFile() call
   - Now only sanitizes in-memory copy, doesn't write back to disk
   - Logs warning instructing admin to run `config migrate-to-file`

4. Self-heal script: Replace 132 lines of Python with CLI call
   - sanitize_allowed_nodes() now calls `config migrate-to-file`
   - Eliminates uncoordinated Python-based config rewriting

All config mutations now flow exclusively through Phase 2 CLI with
atomic operations and file locking. No code paths remain that can
create duplicate allowed_nodes blocks.

Addresses Codex review feedback on Phase 2 gaps.
2025-11-19 10:55:01 +00:00
rcourtman
c1185ebc9d feat(sensor-proxy): complete Phase 2 with CLI-based config migration
Add `config migrate-to-file` command and update installer to eliminate
all shell/Python config manipulation, ensuring atomic operations throughout.

Changes:
- Add `config migrate-to-file` command to atomically migrate inline
  allowed_nodes blocks to file-based configuration
- Update installer's update_allowed_nodes() to call CLI exclusively
- Simplify migrate_inline_allowed_nodes_to_file() to use CLI
- Remove dependency on Python/sed for config manipulation
- Implement dual-file locking (config.yaml + allowed_nodes.yaml) to
  prevent race conditions during migration

All config mutations now flow through the Phase 2 CLI with:
- File locking (flock)
- Atomic writes (temp + rename + fsync)
- Proper YAML parsing/generation

This completes Phase 2 architecture and eliminates the root cause of
config corruption issues.

Related to prior commits: 53dec6010, 3dc073a28, 804a638ea, 131666bc1
2025-11-19 10:35:49 +00:00
rcourtman
804a638ea3 fix(sensor-proxy): critical Phase 2 locking and validation fixes
Fixes critical issues found by Codex code review:

**1. Fixed file locking race condition (CRITICAL)**
- Lock file was being replaced by atomic rename, invalidating the lock
- New approach: lock a separate `.lock` file that persists across renames
- Ensures concurrent writers (installer + self-heal timer) are properly serialized
- Without this fix, corruption was still possible despite Phase 2

**2. Fixed validation to honor configured allowed_nodes_file path**
- validate command now uses loadConfig() to read actual config
- Respects allowed_nodes_file setting instead of assuming default path
- Prevents false positives/negatives when path is customized

**3. Allow empty allowed_nodes lists**
- Empty lists are valid (admin may clear for security, or rely on IPC validation)
- validate no longer fails on empty lists
- set-allowed-nodes --replace with zero nodes now supported
- Critical for operational flexibility

**4. Installer error propagation**
- update_allowed_nodes failures now exit installer with error
- Prevents silent failures that leave stale allowlists
- Self-heal will abort instead of masking CLI errors

**Technical Details:**
- withLockedFile() now locks `<path>.lock` instead of target file
- Lock held for entire duration of read-modify-write-rename
- atomicWriteFile() completes while lock is still held
- Empty lists represented as `allowed_nodes: []` in YAML

**Testing:**
 Lock file created and persists across operations
 Empty list can be written with --replace
 Validation passes with empty lists
 Config path from allowed_nodes_file honored
 Concurrent operations properly serialized

These fixes ensure Phase 2 actually eliminates corruption by design.

Identified by Codex code review
Related to Phase 2 commit 3dc073a28
2025-11-19 09:47:43 +00:00
rcourtman
3dc073a285 feat(sensor-proxy): Phase 2 - atomic config management with CLI
Implements bullet-proof configuration management to completely eliminate
allowed_nodes corruption by design. This builds on Phase 1 (file-only mode)
by replacing all shell/Python config manipulation with proper Go tooling.

**New Features:**
- `pulse-sensor-proxy config validate` - parse and validate config files
- `pulse-sensor-proxy config set-allowed-nodes` - atomic node list updates
- File locking via flock prevents concurrent write races
- Atomic writes (temp file + rename) ensure consistency
- systemd ExecStartPre validation prevents startup with bad config

**Architectural Changes:**
1. Installer now calls config CLI instead of embedded Python/shell scripts
2. All config mutations go through single authoritative writer
3. Deduplication and normalization handled in Go (reuses existing logic)
4. Sanitizer kept as noisy failsafe (warns if corruption still occurs)

**Implementation Details:**
- New cmd/pulse-sensor-proxy/config_cmd.go with cobra commands
- withLockedFile() wrapper ensures exclusive access
- atomicWriteFile() uses temp + rename pattern
- Installer update_allowed_nodes() simplified to CLI calls
- Both systemd service modes include ExecStartPre validation

**Why This Works:**
- Single code path for all writes (no shell/Python divergence)
- File locking serializes self-heal timer + manual installer runs
- Validation gate prevents proxy from starting with corrupt config
- CLI uses same YAML parser as the daemon (guaranteed compatibility)

**Phase 2 Benefits:**
- Corruption impossible by design (not just detected and fixed)
- No more Python dependency for config management
- Atomic operations prevent partial writes
- Clear error messages on validation failures

The defensive sanitizer remains active but now logs loudly if triggered,
allowing us to confirm Phase 2 eliminates corruption in production before
removing the safety net entirely.

This completes the fix for the recurring temperature monitoring outages.

Related to Phase 1 commit 53dec6010
2025-11-19 09:37:49 +00:00
rcourtman
53dec6010c fix(sensor-proxy): eliminate allowed_nodes config corruption
Phase 1 hotfix to address recurring config file corruption that causes
99% of temperature monitoring failures. The root cause was the installer
oscillating between inline and file-based allowlist modes, creating
duplicate `allowed_nodes:` keys in config.yaml.

Changes:
- Force file-based allowlist mode exclusively (refuse versions < v4.31.1)
- Add automatic migration from inline to file-based config
- Remove inline mode code path from update_allowed_nodes()
- Migration runs on every install/self-heal to clean up existing corruption

The self-heal timer runs every 5 minutes and was the primary source of
corruption when version detection failed or encountered edge cases.

This eliminates the dual code paths and ensures config.yaml is never
edited for allowlist changes - only /etc/pulse-sensor-proxy/allowed_nodes.yaml
is modified.

Phase 2 (next release) will implement proper Go-based config management
with atomic writes, locking, and systemd validation to prevent corruption
by design.

Related to recurring temperature monitoring outages
2025-11-19 09:21:54 +00:00
rcourtman
6b5ae18dfe fix: sanitize sensor proxy config during self-heal
Related to #714.
2025-11-18 22:51:40 +00:00
rcourtman
d93a2c1053 Improve host agent binary handling and docker installer purge (Related to #693) 2025-11-18 22:11:44 +00:00
rcourtman
4a8f743d02 Deduplicate inline proxy allow list 2025-11-18 14:58:50 +00:00
rcourtman
54c47fe2f5 Improve temperature proxy detection 2025-11-18 14:25:09 +00:00
rcourtman
91dbe31ea5 Include control-plane allow list in proxy config 2025-11-18 10:42:13 +00:00
rcourtman
7fa39467c0 Fix setup-script tokens and proxy registration timing 2025-11-18 10:22:54 +00:00
rcourtman
f7261f3080 Harden turnkey install and proxy auto-registration 2025-11-18 00:24:50 +00:00
rcourtman
ce595b69fa Avoid blocking self-heal start during install 2025-11-17 23:14:51 +00:00
rcourtman
3ca2bf308e Break self-heal recursion when proxy unregistered 2025-11-17 23:01:57 +00:00
rcourtman
94a0c82439 Fix inline allowed_nodes cleanup 2025-11-17 22:50:25 +00:00
rcourtman
09ee968422 Improve temp proxy install UX 2025-11-17 22:30:32 +00:00
rcourtman
011c7094b2 Restore pending control-plane helpers 2025-11-17 22:04:30 +00:00
rcourtman
16bd34513e Fix pending control-plane helpers 2025-11-17 22:01:11 +00:00
rcourtman
d08322e5f8 Speed up proxy self-heal reconciliation 2025-11-17 21:56:21 +00:00
rcourtman
75d231bf53 Add automatic control-plane reconciliation 2025-11-17 21:55:47 +00:00
rcourtman
2a3ee72b7f Improve sensor proxy installer compatibility 2025-11-17 21:38:28 +00:00
rcourtman
f0f44231a1 Fix proxy install summary and allowed_nodes cleanup 2025-11-17 14:38:01 +00:00
rcourtman
430c98c5ef Move allowed_nodes to managed file 2025-11-16 10:06:58 +00:00
rcourtman
5951a364f7 Auto-deploy proxy for standalone temp monitoring 2025-11-16 09:47:07 +00:00
rcourtman
e375a21d4b Fix backup UX and proxy config dedupe 2025-11-15 23:26:44 +00:00
rcourtman
3223a3a432 Fix allowed_nodes sanitizer indentation handling 2025-11-15 22:42:08 +00:00
rcourtman
2452674c3c Rewrite proxy allowed_nodes sanitizer 2025-11-15 22:34:29 +00:00
rcourtman
daa70280f5 Gracefully handle missing PVE instance during proxy registration 2025-11-15 22:25:50 +00:00
rcourtman
871c2988ff Fix proxy installer dedupe 2025-11-15 22:04:36 +00:00
rcourtman
de5b314842 Improve temperature proxy control-plane flow 2025-11-15 21:49:51 +00:00
rcourtman
1b37de22f6 Ensure sensor proxy installer configures Pulse env 2025-11-15 18:28:42 +00:00
rcourtman
fa4d5de7bd Ensure sensor proxy installer configures Pulse env 2025-11-15 18:23:40 +00:00
rcourtman
671703a0f4 Deduplicate allowed_nodes when installing sensor proxy 2025-11-15 18:14:38 +00:00
rcourtman
dd9089b26c Fix installer adding invalid hostname entries to allowed_nodes
The installer was adding node hostnames (and accidentally the header "Name")
to allowed_nodes in addition to IPs. This caused:
1. Invalid entries like "Name", "minipc", "delly" in config
2. These are not valid for SSH temperature collection

Only IPs should be in allowed_nodes since that's what the proxy uses for SSH.
Removed the loop that added CLUSTER_NODE_NAMES to the array.

Also fixed: Removed extraction of CLUSTER_NODE_NAMES since it's no longer used.
2025-11-15 10:07:22 +00:00
rcourtman
ec2bd8daf6 Fix cleanup systemd-run deadlock
Problem:
Cleanup script uses systemd-run with both --wait and
--property="After=pulse-sensor-cleanup.service", creating a circular
dependency:
- cleanup.service runs and waits for uninstaller to complete
- uninstaller has After=cleanup.service, so it waits for cleanup to finish
- Result: Both services stuck waiting for each other

Fix:
Remove the --property="After=pulse-sensor-cleanup.service" line. The
Conflicts=pulse-sensor-proxy.service is sufficient to ensure the proxy
stops before uninstallation. The cleanup script doesn't need to finish
before the uninstaller starts.

Testing:
Cleanup now completes successfully, removing all artifacts:
- Systemd units removed
- Binaries deleted from /opt/pulse/sensor-proxy/
- Data directory /var/lib/pulse-sensor-proxy/ removed
- SSH keys cleaned from authorized_keys
- pulse-monitor user and API tokens deleted
- LXC bind mounts removed from container configs

Related to #605 (temperature monitoring cleanup)
2025-11-15 09:03:17 +00:00
rcourtman
fbfd086724 Fix missed /usr/local path migration and add backward compatibility
**Missed Migration**:
- Line 2204 still used /usr/local/bin/pulse-sensor-wrapper.sh in fallback path
- Updated to use /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh

**Backward Compatibility**:
- When pushing SSH keys to cluster nodes, installer now checks if remote node
  has old installation (/usr/local/bin wrapper exists but /opt path doesn't)
- Automatically creates symlink on remote nodes to maintain compatibility
- Prevents temperature collection failures when cluster has mixed old/new installs

**Root Cause**:
When installer runs on upgraded node (delly), it pushes SSH keys with new forced
command path to all cluster nodes. If remote node (minipc) has old installation,
the forced command fails because wrapper doesn't exist at new path.

This fix ensures "it works straight out the box" by bridging old and new paths
automatically during SSH key deployment.
2025-11-15 08:37:44 +00:00
rcourtman
5b7d1726d7 Fix update_allowed_nodes to be properly idempotent
Rewrote AWK state machine to correctly handle:
- Multiple allowed_nodes sections (removes all of them)
- Comment lines immediately preceding allowed_nodes (discards them)
- Empty lines within allowed_nodes section
- Indented list items and comments

The function now:
1. Buffers comment lines that might precede allowed_nodes
2. When allowed_nodes: is detected, discards buffered comments
3. Skips all content until hitting a non-indented, non-comment line
4. Flushes buffered comments when hitting non-comment content

This ensures running the installer multiple times won't create duplicate
allowed_nodes sections in config.yaml.

Tested with script that verifies duplicate sections are removed correctly.
2025-11-15 08:03:28 +00:00
rcourtman
fe53d64730 Address remaining Codex review findings
**LXC Bind Mount Removal**:
- Changed from sed to `pct set -delete` for safer mount removal
- Validates syntax and prevents breaking container configs
- Finds mount points by grepping for pulse-sensor-proxy, extracts mp number

**API Token Parsing** (three-tier fallback):
1. Try `pveum --output-format json` with python3 JSON parsing
2. Fall back to `pvesh get /access/users/pulse-monitor@pam/token` JSON API
3. Last resort: parse table output with improved filtering (handles more Unicode chars)

**Retry Logic**:
- Rename cleanup-request.json to .processing instead of deleting immediately
- Allows retry on failure (processing file persists if script crashes)
- Remove .processing file only on successful completion
- Prevents loops while enabling failure recovery

These complete all 8 issues identified by Codex review.
2025-11-15 00:35:22 +00:00
rcourtman
bcd8d4e0fa Fix critical cleanup implementation issues found by Codex review
**Host Detection**:
- Now detects localhost by hostname and FQDN, not just IP
- Fixes issue where nodes configured as https://hostname:8006 would skip
  localhost cleanup (API tokens, bind mounts, service removal)

**Systemd Sandbox**:
- Added /etc/pve and /etc/systemd/system to ReadWritePaths
- Allows cleanup script to modify Proxmox configs and systemd units

**Uninstaller Improvements**:
- Use UUID for transient unit names (prevents same-second collisions)
- Added --purge flag for complete removal
- Added --wait and --collect flags to capture exit code
- Now fails cleanup if uninstaller exits non-zero

**Path Migration**:
- Fixed all /usr/local references to use /opt/pulse/sensor-proxy
- Updated forced command in SSH authorized_keys
- Updated self-heal script installer path
- Updated Go backend removal helpers (supports both new and legacy paths)

These fixes address Codex findings: hostname detection, sandbox permissions,
transient unit collisions, incomplete purging, and incomplete path migration.

Related to cleanup implementation testing.
2025-11-15 00:33:41 +00:00
rcourtman
17d2e6876e Fix unbound SHARE_DIR variable in installer
After relocating binaries to INSTALL_ROOT, the SHARE_DIR variable was removed
but one reference remained in cache_installer_for_self_heal() causing
'unbound variable' error.

Changed to use INSTALL_ROOT directly since that's where the cached installer
is stored (STORED_INSTALLER=${INSTALL_ROOT}/install-sensor-proxy.sh).
2025-11-15 00:09:34 +00:00
rcourtman
ed48d7555a Fix directory creation order in sensor-proxy installer
The installer was trying to write binaries to /opt/pulse/sensor-proxy/bin/
before creating the directory structure, causing 'No such file or directory'
errors on fresh installs.

Moved directory creation for INSTALL_ROOT and bin/ to before binary installation
section (before line 657), ensuring directories exist before use.

Related to cleanup implementation testing.
2025-11-15 00:08:25 +00:00
rcourtman
6692228e0f Implement full cleanup when nodes are removed from Pulse
Extends cleanup script to completely remove Pulse footprint from hosts
when nodes are removed, not just SSH keys. Now removes: SSH keys, proxy
service, binaries, API tokens, pulse-monitor user, and LXC bind mounts.

Key improvements:

1. **flock Serialization**: Prevents concurrent cleanup runs
   - Acquires exclusive lock on cleanup.lock file
   - Prevents race conditions and cleanup loops

2. **Immediate Request File Deletion**: Delete cleanup-request.json
   before any long-running operations to prevent re-triggering

3. **API Token Cleanup**: Removes all pulse-monitor@pam API tokens
   - Tries JSON output first (Proxmox 7.0+)
   - Falls back to table parsing with proper filtering (no decoration chars)
   - Deletes pulse-monitor@pam user after removing all tokens

4. **LXC Bind Mount Removal**: Scans all container configs and removes
   pulse-sensor-proxy bind mount entries

5. **Process Isolation for Uninstaller**: Uses systemd-run to spawn
   isolated transient unit that won't be killed when proxy service stops
   - Unit name: pulse-uninstall-{timestamp}
   - Properties: Type=oneshot, Conflicts=pulse-sensor-proxy.service
   - Runs non-blocking so cleanup service can exit cleanly
   - Falls back to direct call if systemd-run unavailable

6. **Complete Service/Binary Removal**: Calls installer's --uninstall
   - Stops and disables pulse-sensor-proxy.service
   - Removes all systemd units
   - Deletes all binaries from /opt/pulse/sensor-proxy/
   - Removes configuration files
   - Cleans up directories

Changes to cleanup script logic:
- Added LOCKFILE and INSTALLER_PATH configuration
- Acquire flock before processing (prevents concurrent runs)
- Delete request file immediately after reading
- Full localhost cleanup: SSH keys → API tokens → bind mounts → uninstall
- Remote cleanup still SSH-key-only (can't orchestrate uninstall remotely)
- Better error handling with appropriate log levels

Updated cleanup service unit:
- ExecStart now uses ${CLEANUP_SCRIPT_PATH} variable (new /opt location)
- Changed heredoc from 'SERVICE_EOF' to SERVICE_EOF for variable expansion

Addresses all issues documented in CLEANUP_TODO.md:
-  Read-only filesystem (binaries now in /opt, removable)
-  Process isolation (systemd-run transient unit)
-  Cleanup loops (flock + immediate file deletion)
-  API token parsing (JSON first, filtered table fallback)

The UI message is now accurate: "Removing this proxmox ve node also
scrubs the Pulse footprint on the host — the proxy service, SSH key,
API token, and bind mount are all cleaned up automatically."

Part of: CLEANUP_TODO.md Phase 2-4
Supersedes: ed65fda74 (original cleanup attempt with process issues)
Depends on: b192c60e9 (binary relocation to /opt)
2025-11-15 00:03:09 +00:00
rcourtman
b192c60e90 Relocate sensor-proxy binaries to /opt for guaranteed cleanup
Moves all sensor-proxy binaries and scripts from /usr/local/bin to
/opt/pulse/sensor-proxy/bin to ensure they can be removed during cleanup
even on systems with read-only /usr (hardened Proxmox setups).

Changes:
- INSTALL_ROOT=/opt/pulse/sensor-proxy (new writable location)
- Binary path: /opt/pulse/sensor-proxy/bin/pulse-sensor-proxy
- Wrapper script: /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh
- Cleanup script: /opt/pulse/sensor-proxy/bin/pulse-sensor-cleanup.sh
- Selfheal script: /opt/pulse/sensor-proxy/bin/pulse-sensor-proxy-selfheal.sh
- Installer storage: /opt/pulse/sensor-proxy/install-sensor-proxy.sh

Updated:
- Directory creation to include ${INSTALL_ROOT}/bin
- Systemd service ExecStart paths to use ${BINARY_PATH}
- Self-heal service ExecStart to use ${SELFHEAL_SCRIPT}
- Changed heredoc delimiters from 'EOF' to EOF for variable expansion

Rationale:
Proxmox VE can mount /usr as read-only in hardened configurations.
The previous /usr/local/bin location prevented complete uninstallation
on these systems, violating Pulse's correctness principle. The /opt
location is guaranteed writable and appropriate for third-party software.

This is Phase 1 of implementing full cleanup functionality per
CLEANUP_TODO.md. Subsequent commits will add process isolation,
API token deletion, and bind mount removal.

Part of: #CLEANUP_TODO.md Phase 1
Related: ed65fda74 (original cleanup attempt)
2025-11-14 23:59:25 +00:00
rcourtman
8204b54051 Fix permission denied error when updating allowed_nodes
The update_allowed_nodes function was changing ownership of the temp file
before all writes were complete, causing 'Permission denied' errors when
appending the allowed_nodes section.

Root cause:
- mktemp creates file owned by script runner (root)
- chown changed ownership to pulse-sensor-proxy:pulse-sensor-proxy
- Subsequent append (>>) failed because root can't write to the file

Fix: Defer chown until after all writes complete and file is moved to
final location. Ownership is still correctly set on the final config file.
2025-11-14 23:21:46 +00:00
rcourtman
ed65fda74d Extend node cleanup to fully remove Pulse footprint
When a Proxmox node is removed from Pulse, the cleanup now performs full uninstallation:

- SSH keys removal (existing functionality)
- Uninstalls pulse-sensor-proxy service
- Removes LXC bind mounts from container configs
- Deletes Proxmox API tokens
- Removes pulse-monitor@pam user

This aligns with security best practices and user expectations - "remove node"
should completely sever trust with that machine, not leave credentials and
privileged services behind.

The cleanup script now calls the uninstaller (--uninstall) and uses pveum
to remove API tokens. This prevents leftover artifacts if the host is
repurposed or compromised.

Related: config_handlers.go triggerPVEHostCleanup() at node deletion
2025-11-14 22:58:50 +00:00
rcourtman
c5a5eb2534 Use unique temp file prefix to avoid permission conflicts
- Change mktemp to use /tmp/pulse-config.XXXXXXXXXX template
- Prevents conflicts with stale temp files from previous runs
- Fixes 'Permission denied' errors when script re-runs
2025-11-14 22:44:33 +00:00
rcourtman
3a381e764a Add cleanup trap and better error handling to update_allowed_nodes
- Add trap to remove temp file on function return (success or failure)
- Add error check for mv command with descriptive message
- Ensure config file has proper permissions after update

This prevents orphaned temp files when errors occur and provides
better diagnostics when file operations fail.
2025-11-14 22:13:35 +00:00
rcourtman
d0080e4adf Remove invalid 'local' keyword from main script scope
The all_nodes arrays were declared with 'local' keyword outside of
functions, causing bash syntax error:
  'local: can only be used in a function'

Fixed by removing 'local' keyword - arrays in main script scope don't
need it and it's actually invalid syntax.
2025-11-14 22:06:44 +00:00
rcourtman
ddf0733353 Fix update_allowed_nodes to remove comment headers
The awk logic was removing allowed_nodes sections but leaving their
comment headers behind. When multiple sections existed, comments would
accumulate.

New approach:
- Buffer all comment lines encountered outside sections
- When a non-comment line is found, flush buffered comments
- When allowed_nodes is found, discard buffered comments (they belonged
  to the section we're removing)
- This cleanly removes section headers like:
  '# Cluster nodes (auto-discovered during installation)'
  '# These nodes are allowed to request...'

Tested with config containing duplicate allowed_nodes sections - now
correctly produces clean output with all duplicates and headers removed.
2025-11-14 22:02:10 +00:00
rcourtman
245d6f5b47 Ensure base config exists before updating allowed_nodes
The installer was only creating base config.yaml in standalone mode,
but update_allowed_nodes() is also called in LXC mode. When the config
didn't exist, update_allowed_nodes() would create an empty file and only
add the allowed_nodes section, missing required fields like
allowed_peer_uids, metrics_address, rate_limit, etc.

This caused the proxy to fail when it tried to parse the incomplete config.

Now creates a proper base config with all required fields if the file
doesn't exist, before any mode-specific configuration is added.
2025-11-14 21:57:16 +00:00
rcourtman
f7936cb540 Fix duplicate allowed_nodes in sensor proxy installer
The install-sensor-proxy.sh script was blindly appending allowed_nodes
sections to the config file without checking if they already existed.
When the script was re-run or if the initial config already had an
allowed_nodes section, this created duplicate YAML keys that caused
the proxy service to fail with parse errors.

Changes:
- Add update_allowed_nodes() helper function that safely updates the
  allowed_nodes section by removing any existing ones first
- Replace all three cat >> config.yaml heredocs with calls to the
  helper function (cluster nodes, standalone mode, pvecm fallback)
- Uses awk to properly parse and remove multi-line YAML sections

This makes the installer idempotent and prevents config corruption on
re-runs.

Fixes issue where proxy service crashed with:
  'mapping key "allowed_nodes" already defined at line X'
2025-11-14 21:43:31 +00:00
rcourtman
e708727725 Improve sensor proxy socket verification 2025-11-14 20:14:25 +00:00
rcourtman
1d0d23d237 Related to #710: harden Windows installer arch detection 2025-11-14 10:50:56 +00:00
rcourtman
1ed69d35b6 docs: highlight runbooks in index and script verification checklist 2025-11-14 10:39:10 +00:00
rcourtman
4d021f8a50 Related to #701: improve sensor proxy installer caching 2025-11-14 00:51:54 +00:00
rcourtman
4f8b807f0c Refine Windows host installer logging (related to #709) 2025-11-13 23:09:22 +00:00
rcourtman
70673c1fdc Improve temperature proxy diagnostics and tests 2025-11-13 22:31:53 +00:00
rcourtman
9cb151d836 Add localhost to allowed_source_subnets for self-monitoring
The HTTP mode installer now includes 127.0.0.1/32 in allowed_source_subnets
to permit self-monitoring queries from localhost. This fixes 403 Forbidden
errors when nodes query their own sensor-proxy instance.

Related to HTTP mode implementation for external PVE hosts.
2025-11-13 19:34:11 +00:00
rcourtman
e2bd514899 Fix HTTP mode for pulse-sensor-proxy and improve installer safety
## HTTP Server Fixes
- Add source IP middleware to enforce allowed_source_subnets
- Fix missing source subnet validation for external HTTP requests
- HTTP health endpoint now respects subnet restrictions

## Installer Improvements
- Auto-configure allowed_source_subnets with Pulse server IP
- Add cluster node hostnames to allowed_nodes (not just IPs)
- Fix node validation to accept both hostnames and IPs
- Add Pulse server reachability check before installation
- Add port availability check for HTTP mode
- Add automatic rollback on service startup failure
- Add HTTP endpoint health check after installation
- Fix config backup and deduplication (prevent duplicate keys)
- Fix IPv4 validation with loopback rejection
- Improve registration retry logic with detailed errors
- Add automatic LXC bind mount cleanup on uninstall

## Temperature Collection Fixes
- Add local temperature collection for self-monitoring nodes
- Fix node identifier matching (use hostname not SSH host)
- Fix JSON double-encoding in HTTP client response

Related to #XXX (temperature monitoring fixes)
2025-11-13 18:22:36 +00:00
rcourtman
71217dfae4 Add HTTP mode support to sensor-proxy installer
Implements complete HTTP mode installation workflow for external PVE hosts.

New installer features:
- `--http-mode` flag: Enable HTTP server mode for remote temperature monitoring
- `--http-addr <addr>` flag: Configure listen address (default :8443)
- Auto-generates self-signed TLS certificates (4096-bit RSA, 10-year validity)
- Registers with Pulse API and receives authentication token
- Configures systemd service with proper security hardening

Installation workflow (HTTP mode):
1. Validate --pulse-server parameter is provided
2. Generate TLS certificate with SAN (hostname + IPs)
3. Call Pulse API POST /api/temperature-proxy/register
4. Receive and store auth token securely (mode 600)
5. Append HTTP config to config.yaml
6. Update systemd service with TLS paths
7. Start service

TLS certificate generation:
- Uses openssl req with RSA 4096-bit keys
- 10-year validity period
- SubjectAltName includes hostname + all IPs
- Files stored in /etc/pulse-sensor-proxy/tls/
- Permissions: 640 root:pulse-sensor-proxy
- Logs SHA256 fingerprint for audit

API registration:
- Calls POST /api/temperature-proxy/register
- Payload: {"hostname": "...", "proxy_url": "https://..."}
- Response: {"token": "...", "pve_instance": "..."}
- Aborts installation on registration failure (fail-fast)
- Token stored in config.yaml

Systemd service updates:
- Adds ReadOnlyPaths=/etc/pulse-sensor-proxy/tls for HTTP mode
- RestrictAddressFamilies already includes AF_INET/AF_INET6
- Maintains all existing security hardening

Error handling:
- Validates required parameters before starting
- Aborts on TLS generation failure
- Aborts on API registration failure
- Provides actionable troubleshooting guidance
- Logs clear error messages

Security:
- Tokens stored with mode 600, owned by service user
- TLS keys protected with mode 640
- Service runs as unprivileged pulse-sensor-proxy user
- Full systemd hardening maintained

Usage example:
  curl -fsSL https://pulse-server/download/install-sensor-proxy.sh | \
    bash -s -- --http-mode --pulse-server https://pulse.example.com:7655

Related to #571
2025-11-13 16:33:12 +00:00
rcourtman
cd9d63d464 Fix #571: Installer now auto-configures allowed_nodes for temperature monitoring
When cluster IPC validation fails (due to systemd hardening), the proxy
falls back to allowlist-based validation. The installer now automatically
populates allowed_nodes with:

- Cluster mode: All discovered cluster member IPs
- Standalone mode: localhost IP addresses (including 127.0.0.1/localhost)
- Fallback mode: localhost IPs when pvecm unavailable

This ensures out-of-the-box temperature monitoring works on fresh installs
without manual configuration.
2025-11-13 15:37:30 +00:00
rcourtman
5ef6ca16fe Fix temperature monitoring for clustered and LXC Proxmox environments (addresses #571)
Root cause: pulse-sensor-proxy runs with strict systemd hardening that prevents
access to Proxmox corosync IPC (abstract UNIX sockets). When pvecm fails with
IPC errors, the code incorrectly treated it as "standalone mode" and only
discovered localhost addresses, rejecting legitimate cluster members and external
nodes.

Changes:

1. **Distinguish IPC failures from true standalone mode**
   - Detect ipcc_send_rec and access control list errors specifically
   - These indicate a cluster exists but isn't accessible (LXC, systemd restrictions)
   - Return error to disable cluster validation instead of misusing standalone logic

2. **Graceful degradation when cluster validation fails**
   - When cluster IPC is unavailable, fall through to permissive mode
   - Log debug message suggesting allowed_nodes configuration
   - Allows requests to proceed rather than blocking all temperature monitoring

3. **Improve local address discovery for true standalone nodes**
   - Use Go's native net.Interfaces() instead of shelling out to 'ip addr'
   - More reliable and works with AF_NETLINK restrictions
   - Add helpful logging when only hostnames are discovered

4. **Systemd hardening adjustments**
   - Add AF_NETLINK to RestrictAddressFamilies (for net.Interfaces())
   - Remove RemoveIPC=true (attempted fix for corosync, insufficient)
   - Add ReadWritePaths=-/run/corosync (optional path, corosync uses abstract sockets anyway)

Result: Temperature monitoring now works in:
- Clustered Proxmox hosts (falls back to permissive when IPC blocked)
- LXC containers (correctly detects IPC failure, allows requests)
- Standalone nodes (proper local address discovery with IPs)

Workaround for maximum security: Configure allowed_nodes in /etc/pulse-sensor-proxy/config.yaml
when cluster validation cannot be used.
2025-11-13 13:25:27 +00:00
rcourtman
ca8cc0844a Fix temperature monitoring on standalone Proxmox nodes (addresses #571)
Root cause: The systemd service hardening blocked AF_NETLINK sockets,
preventing IP address discovery on standalone nodes. The proxy could
only discover hostnames, causing node_not_cluster_member rejections
when users configured Pulse with IP addresses.

Changes:
1. Add AF_NETLINK to RestrictAddressFamilies in all systemd services
   - pulse-sensor-proxy.service
   - install-sensor-proxy.sh (both modes)
   - pulse-sensor-cleanup.service

2. Replace shell-based 'ip addr' with Go native net.Interfaces() API
   - More reliable and doesn't require external commands
   - Works even with strict systemd restrictions
   - Properly filters loopback, link-local, and down interfaces

3. Improve error logging and user guidance
   - Warn when no IP addresses can be discovered
   - Provide clear instructions about allowed_nodes workaround
   - Include address counts in logs for debugging

This fix ensures standalone Proxmox nodes can properly validate
temperature requests by IP address without requiring manual
allowed_nodes configuration.
2025-11-13 13:02:15 +00:00
rcourtman
e822ab7ae1 Fix remote sync check in release trigger script
- Replace unreliable git fetch --dry-run check
- Use git rev-parse to compare local and remote commits
- Prevents false warnings about diverged branches
2025-11-13 11:43:36 +00:00
rcourtman
d400befd35 Add pre-flight validation script for releases
- Check VERSION file matches before triggering workflow
- Validate working directory is clean
- Confirm on main branch and up to date
- Load release notes from /tmp/release_notes_X.Y.Z.md
- Prevents wasting CI time on misconfigured releases
2025-11-13 11:36:53 +00:00
rcourtman
1dd0e74133 Dramatically improve temperature proxy installation robustness
Users were abandoning Pulse due to catastrophic temperature monitoring setup failures. This commit addresses the root causes:

**Problem 1: Silent Failures**
- Installations reported "SUCCESS" even when proxy never started
- UI showed green checkmarks with no temperature data
- Zero feedback when things went wrong

**Problem 2: Missing Diagnostics**
- Service failures logged only in journald
- Users saw "Something going on with the proxy" with no actionable guidance
- No way to troubleshoot from error messages

**Problem 3: Standalone Node Issues**
- Proxy daemon logged continuous pvecm errors as warnings
- "ipcc_send_rec" and "Unknown error -1" messages confused users
- These are expected for non-clustered/LXC setups

**Solutions Implemented:**

1. **Health Gate in install.sh (lines 1588-1629)**
   - Verify service is running after installation
   - Check socket exists on host
   - Confirm socket visible inside container via bind mount
   - Fail loudly with specific diagnostics if any check fails

2. **Actionable Error Messages in install-sensor-proxy.sh (lines 822-877)**
   - When service fails to start: dump full systemctl status + 40 lines of logs
   - When socket missing: show permissions, service status, and remediation command
   - Include common issues checklist (missing user, permission errors, lm-sensors, etc.)
   - Direct link to troubleshooting docs

3. **Better Standalone Node Detection in ssh.go (lines 585-595)**
   - Recognize "Unknown error -1" and "Unable to load access control list" as LXC indicators
   - Log at INFO level (not WARN) since this is expected behavior
   - Clarify message: "using localhost for temperature collection"

**Impact:**
- Eliminates "green checkmark but no temps" scenario
- Users get immediate actionable feedback on failures
- Standalone/LXC installations work silently without error spam
- Reduces support burden from #571 (15+ comments of user frustration)

Related to #571
2025-11-13 10:14:19 +00:00
rcourtman
221c41b2b0 Polish release notes fallback 2025-11-13 09:10:43 +00:00
rcourtman
8330664ed2 Add deterministic release notes fallback 2025-11-13 00:00:25 +00:00
rcourtman
a6a8f0a8ef Improve release notes fallback 2025-11-12 23:40:26 +00:00
rcourtman
92e155ed3f Handle Snap Docker home restrictions (Related to #693) 2025-11-12 19:20:04 +00:00
rcourtman
b829d86d2c Handle docker validation under errexit
Related to #693
2025-11-12 18:34:53 +00:00
rcourtman
b3e8e43736 Related to #698: harden installer release detection 2025-11-12 17:56:16 +00:00
rcourtman
9dd6357328 Improve sensor-proxy release detection (related to #701) 2025-11-12 17:49:20 +00:00
rcourtman
92a5d74ba9 Add Snap Docker support to install-docker-agent.sh
Snap-installed Docker does not automatically create a docker group,
causing permission denied errors when the pulse-docker service user
tries to access /var/run/docker.sock.

Changes:
- Auto-detect Snap Docker installations
- Create docker group if missing when Snap Docker is detected
- Restart Snap Docker after group creation to refresh socket ACLs
- Add socket access validation before starting the service
- Handle symlinked Docker sockets in systemd unit ReadWritePaths
- Document troubleshooting steps in DOCKER_MONITORING.md
2025-11-11 23:07:29 +00:00