**LXC Bind Mount Removal**:
- Changed from sed to `pct set -delete` for safer mount removal
- Validates syntax and prevents breaking container configs
- Finds mount points by grepping for pulse-sensor-proxy, extracts mp number
**API Token Parsing** (three-tier fallback):
1. Try `pveum --output-format json` with python3 JSON parsing
2. Fall back to `pvesh get /access/users/pulse-monitor@pam/token` JSON API
3. Last resort: parse table output with improved filtering (handles more Unicode chars)
**Retry Logic**:
- Rename cleanup-request.json to .processing instead of deleting immediately
- Allows retry on failure (processing file persists if script crashes)
- Remove .processing file only on successful completion
- Prevents loops while enabling failure recovery
These complete all 8 issues identified by Codex review.
**Host Detection**:
- Now detects localhost by hostname and FQDN, not just IP
- Fixes issue where nodes configured as https://hostname:8006 would skip
localhost cleanup (API tokens, bind mounts, service removal)
**Systemd Sandbox**:
- Added /etc/pve and /etc/systemd/system to ReadWritePaths
- Allows cleanup script to modify Proxmox configs and systemd units
**Uninstaller Improvements**:
- Use UUID for transient unit names (prevents same-second collisions)
- Added --purge flag for complete removal
- Added --wait and --collect flags to capture exit code
- Now fails cleanup if uninstaller exits non-zero
**Path Migration**:
- Fixed all /usr/local references to use /opt/pulse/sensor-proxy
- Updated forced command in SSH authorized_keys
- Updated self-heal script installer path
- Updated Go backend removal helpers (supports both new and legacy paths)
These fixes address Codex findings: hostname detection, sandbox permissions,
transient unit collisions, incomplete purging, and incomplete path migration.
Related to cleanup implementation testing.
After relocating binaries to INSTALL_ROOT, the SHARE_DIR variable was removed
but one reference remained in cache_installer_for_self_heal() causing
'unbound variable' error.
Changed to use INSTALL_ROOT directly since that's where the cached installer
is stored (STORED_INSTALLER=${INSTALL_ROOT}/install-sensor-proxy.sh).
The installer was trying to write binaries to /opt/pulse/sensor-proxy/bin/
before creating the directory structure, causing 'No such file or directory'
errors on fresh installs.
Moved directory creation for INSTALL_ROOT and bin/ to before binary installation
section (before line 657), ensuring directories exist before use.
Related to cleanup implementation testing.
All phases of full cleanup implementation are now complete:
- Phase 1: Binary relocation to /opt ✅
- Phase 2: Process isolation via systemd-run ✅
- Phase 3: flock serialization and immediate file deletion ✅
- Phase 4: Improved API token parsing ✅
- Phase 5: Testing (pending)
Updated status from 'requires architectural changes' to 'complete, ready for testing'.
Extends cleanup script to completely remove Pulse footprint from hosts
when nodes are removed, not just SSH keys. Now removes: SSH keys, proxy
service, binaries, API tokens, pulse-monitor user, and LXC bind mounts.
Key improvements:
1. **flock Serialization**: Prevents concurrent cleanup runs
- Acquires exclusive lock on cleanup.lock file
- Prevents race conditions and cleanup loops
2. **Immediate Request File Deletion**: Delete cleanup-request.json
before any long-running operations to prevent re-triggering
3. **API Token Cleanup**: Removes all pulse-monitor@pam API tokens
- Tries JSON output first (Proxmox 7.0+)
- Falls back to table parsing with proper filtering (no decoration chars)
- Deletes pulse-monitor@pam user after removing all tokens
4. **LXC Bind Mount Removal**: Scans all container configs and removes
pulse-sensor-proxy bind mount entries
5. **Process Isolation for Uninstaller**: Uses systemd-run to spawn
isolated transient unit that won't be killed when proxy service stops
- Unit name: pulse-uninstall-{timestamp}
- Properties: Type=oneshot, Conflicts=pulse-sensor-proxy.service
- Runs non-blocking so cleanup service can exit cleanly
- Falls back to direct call if systemd-run unavailable
6. **Complete Service/Binary Removal**: Calls installer's --uninstall
- Stops and disables pulse-sensor-proxy.service
- Removes all systemd units
- Deletes all binaries from /opt/pulse/sensor-proxy/
- Removes configuration files
- Cleans up directories
Changes to cleanup script logic:
- Added LOCKFILE and INSTALLER_PATH configuration
- Acquire flock before processing (prevents concurrent runs)
- Delete request file immediately after reading
- Full localhost cleanup: SSH keys → API tokens → bind mounts → uninstall
- Remote cleanup still SSH-key-only (can't orchestrate uninstall remotely)
- Better error handling with appropriate log levels
Updated cleanup service unit:
- ExecStart now uses ${CLEANUP_SCRIPT_PATH} variable (new /opt location)
- Changed heredoc from 'SERVICE_EOF' to SERVICE_EOF for variable expansion
Addresses all issues documented in CLEANUP_TODO.md:
- ✅ Read-only filesystem (binaries now in /opt, removable)
- ✅ Process isolation (systemd-run transient unit)
- ✅ Cleanup loops (flock + immediate file deletion)
- ✅ API token parsing (JSON first, filtered table fallback)
The UI message is now accurate: "Removing this proxmox ve node also
scrubs the Pulse footprint on the host — the proxy service, SSH key,
API token, and bind mount are all cleaned up automatically."
Part of: CLEANUP_TODO.md Phase 2-4
Supersedes: ed65fda74 (original cleanup attempt with process issues)
Depends on: b192c60e9 (binary relocation to /opt)
Moves all sensor-proxy binaries and scripts from /usr/local/bin to
/opt/pulse/sensor-proxy/bin to ensure they can be removed during cleanup
even on systems with read-only /usr (hardened Proxmox setups).
Changes:
- INSTALL_ROOT=/opt/pulse/sensor-proxy (new writable location)
- Binary path: /opt/pulse/sensor-proxy/bin/pulse-sensor-proxy
- Wrapper script: /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh
- Cleanup script: /opt/pulse/sensor-proxy/bin/pulse-sensor-cleanup.sh
- Selfheal script: /opt/pulse/sensor-proxy/bin/pulse-sensor-proxy-selfheal.sh
- Installer storage: /opt/pulse/sensor-proxy/install-sensor-proxy.sh
Updated:
- Directory creation to include ${INSTALL_ROOT}/bin
- Systemd service ExecStart paths to use ${BINARY_PATH}
- Self-heal service ExecStart to use ${SELFHEAL_SCRIPT}
- Changed heredoc delimiters from 'EOF' to EOF for variable expansion
Rationale:
Proxmox VE can mount /usr as read-only in hardened configurations.
The previous /usr/local/bin location prevented complete uninstallation
on these systems, violating Pulse's correctness principle. The /opt
location is guaranteed writable and appropriate for third-party software.
This is Phase 1 of implementing full cleanup functionality per
CLEANUP_TODO.md. Subsequent commits will add process isolation,
API token deletion, and bind mount removal.
Part of: #CLEANUP_TODO.md Phase 1
Related: ed65fda74 (original cleanup attempt)
During cluster startup, nodes were temporarily using the primary cluster
endpoint for temperature collection before cluster metadata validation
completed. This caused all nodes to show the same (incorrect) temperature
values for ~4 minutes until validation finished and per-node endpoints
were established.
Example: minipc would show delly's temperature (90°C) instead of its own
(50°C) from startup until cluster validation completed.
Root cause:
- Temperature collection started immediately at startup
- Cluster endpoint validation happened asynchronously
- Code fell back to primary endpoint when ClusterEndpoints was empty
- All nodes used same endpoint, got same temperature data
Fix: Skip temperature collection for cluster nodes until:
1. ClusterEndpoints array is populated (validation complete)
2. Node's specific endpoint is found in the cluster metadata
This ensures correct temperature data from the very first collection,
maintaining data integrity during startup. When persisted config exists,
endpoints are available immediately so no delay occurs. For new clusters,
temperature collection begins once validation completes (~30s).
Preserves Pulse's correctness guarantee: users can trust metrics
immediately after restart without waiting for "warm-up" period.
The update_allowed_nodes function was changing ownership of the temp file
before all writes were complete, causing 'Permission denied' errors when
appending the allowed_nodes section.
Root cause:
- mktemp creates file owned by script runner (root)
- chown changed ownership to pulse-sensor-proxy:pulse-sensor-proxy
- Subsequent append (>>) failed because root can't write to the file
Fix: Defer chown until after all writes complete and file is moved to
final location. Ownership is still correctly set on the final config file.
The full cleanup implementation (ed65fda74) has architectural issues that need addressing:
1. Read-only /usr filesystems prevent binary removal
2. Process isolation issues cause cleanup service to be killed
3. Cleanup loops from improper request file handling
This TODO documents the required changes following Codex recommendations:
- Relocate binaries to /opt/pulse/sensor-proxy/
- Use transient systemd units for uninstall orchestration
- Add flock serialization and proper cleanup-request handling
Current state: SSH keys are removed (critical), full cleanup needs refactoring.
When a Proxmox node is removed from Pulse, the cleanup now performs full uninstallation:
- SSH keys removal (existing functionality)
- Uninstalls pulse-sensor-proxy service
- Removes LXC bind mounts from container configs
- Deletes Proxmox API tokens
- Removes pulse-monitor@pam user
This aligns with security best practices and user expectations - "remove node"
should completely sever trust with that machine, not leave credentials and
privileged services behind.
The cleanup script now calls the uninstaller (--uninstall) and uses pveum
to remove API tokens. This prevents leftover artifacts if the host is
repurposed or compromised.
Related: config_handlers.go triggerPVEHostCleanup() at node deletion
- Change mktemp to use /tmp/pulse-config.XXXXXXXXXX template
- Prevents conflicts with stale temp files from previous runs
- Fixes 'Permission denied' errors when script re-runs
- Add trap to remove temp file on function return (success or failure)
- Add error check for mv command with descriptive message
- Ensure config file has proper permissions after update
This prevents orphaned temp files when errors occur and provides
better diagnostics when file operations fail.
The all_nodes arrays were declared with 'local' keyword outside of
functions, causing bash syntax error:
'local: can only be used in a function'
Fixed by removing 'local' keyword - arrays in main script scope don't
need it and it's actually invalid syntax.
The awk logic was removing allowed_nodes sections but leaving their
comment headers behind. When multiple sections existed, comments would
accumulate.
New approach:
- Buffer all comment lines encountered outside sections
- When a non-comment line is found, flush buffered comments
- When allowed_nodes is found, discard buffered comments (they belonged
to the section we're removing)
- This cleanly removes section headers like:
'# Cluster nodes (auto-discovered during installation)'
'# These nodes are allowed to request...'
Tested with config containing duplicate allowed_nodes sections - now
correctly produces clean output with all duplicates and headers removed.
The installer was only creating base config.yaml in standalone mode,
but update_allowed_nodes() is also called in LXC mode. When the config
didn't exist, update_allowed_nodes() would create an empty file and only
add the allowed_nodes section, missing required fields like
allowed_peer_uids, metrics_address, rate_limit, etc.
This caused the proxy to fail when it tried to parse the incomplete config.
Now creates a proper base config with all required fields if the file
doesn't exist, before any mode-specific configuration is added.
The install-sensor-proxy.sh script was blindly appending allowed_nodes
sections to the config file without checking if they already existed.
When the script was re-run or if the initial config already had an
allowed_nodes section, this created duplicate YAML keys that caused
the proxy service to fail with parse errors.
Changes:
- Add update_allowed_nodes() helper function that safely updates the
allowed_nodes section by removing any existing ones first
- Replace all three cat >> config.yaml heredocs with calls to the
helper function (cluster nodes, standalone mode, pvecm fallback)
- Uses awk to properly parse and remove multi-line YAML sections
This makes the installer idempotent and prevents config corruption on
re-runs.
Fixes issue where proxy service crashed with:
'mapping key "allowed_nodes" already defined at line X'
