# Node Cleanup Implementation - COMPLETED

**Status**: ✅ Full implementation completed in commits b192c60e9 and 6692228e0.

**Previous Status**: Partial implementation attempted (ed65fda74), reverted due to process isolation issues.

**Current Status**: All phases complete, ready for testing.

## Implementation Complete

**Commit b192c60e9**: Relocate binaries to /opt/pulse/sensor-proxy/
- ✅ Binary path moved from /usr/local/bin to /opt/pulse/sensor-proxy/bin
- ✅ All systemd ExecStart paths updated
- ✅ Guarantees cleanup works on read-only /usr systems

**Commit 6692228e0**: Full cleanup script implementation
- ✅ SSH key removal (working)
- ✅ Service uninstallation (via isolated systemd-run transient unit)
- ✅ API token deletion (JSON first, filtered table fallback)
- ✅ Bind mount removal (scans all LXC configs)
- ✅ pulse-monitor user deletion
- ✅ flock serialization (prevents concurrent runs)
- ✅ Immediate request file deletion (prevents loops)

## Issues Discovered During Testing

### 1. Read-Only Filesystem
**Problem**: Proxmox VE can mount `/usr` as read-only (hardened setups, boot-from-snapshots, appliance builds). The binary at `/usr/local/bin/pulse-sensor-proxy` cannot be removed.

**Error**: `rm: cannot remove '/usr/local/bin/pulse-sensor-proxy': Read-only file system`

**Solution**: Relocate all Pulse artifacts to `/opt/pulse/sensor-proxy/` where we control permissions.

### 2. Process Isolation During Uninstall
**Problem**: Cleanup script runs as systemd service. When it calls the uninstaller which stops the proxy service, systemd kills the cleanup service with SIGTERM.

**Attempted fixes** (all failed):
- `systemd-run --scope`
- `at now` scheduling
- `setsid` with double-fork

**Root cause**: Cleanup service and proxy service share dependency tree.

**Solution**: Use transient systemd unit that's independent:
```bash
systemd-run --unit=pulse-sensor-proxy-uninstall@$(uuidgen) \
    /opt/pulse/scripts/uninstall.sh
```

The transient unit should:
- Not use `--scope`
- Have `Conflicts=pulse-sensor-proxy.service`
- Run in its own cgroup

### 3. Cleanup Loop
**Problem**: Cleanup script ran multiple times because cleanup-request file persisted.

**Solution**:
- Delete request file BEFORE starting long-running work
- Use `flock` for serialization
- Ensure `.path` unit triggers on file creation, not existence

### 4. API Token Parsing
**Problem**: Token list includes table formatting characters (│, ┌, └, ╞)

**Current workaround**: Filter with grep, but brittle.

**Better solution**: Use `pveum user token list --output-format json-pretty pulse-monitor@pam` if available.

## Implementation Status

### Phase 1: Relocate Binaries ✅ COMPLETE
- ✅ Update installer to use `/opt/pulse/sensor-proxy/bin/`
- ✅ Update systemd unit ExecStart path
- ✅ Update cleanup script to use new paths
- ⏭️ No symlinks needed (PATH not required for systemd ExecStart)
- ⏸️ Test on read-only `/usr` (deferred to integration testing)

### Phase 2: Fix Uninstall Orchestration ✅ COMPLETE
- ✅ Cleanup script spawns transient systemd unit via systemd-run
- ✅ Added `Conflicts=pulse-sensor-proxy.service` to transient unit
- ✅ Cleanup service exits immediately after spawning uninstaller
- ✅ Uninstaller runs via installer's --uninstall flag (reuses existing code)
- ✅ Process isolation prevents SIGTERM to cleanup service

### Phase 3: Prevent Cleanup Loops ✅ COMPLETE
- ✅ Added `flock` serialization via exec 200>lockfile
- ✅ Delete cleanup-request file immediately after reading (before operations)
- ✅ `.path` unit uses PathChanged/PathModified (correct semantics)
- ✅ Comprehensive logging at info/warn/error levels

### Phase 4: Improve API Token Handling ✅ COMPLETE
- ✅ Try JSON output first (--output-format json-pretty)
- ✅ Fall back to table parsing with proper filtering (removes │┌└╞)
- ✅ Error handling for token deletion failures (logs warnings, continues)
- ✅ Logs each token removal attempt

### Phase 5: Testing & Validation ⏸️ PENDING
- ⏸️ Test on fresh Proxmox VE install
- ⏸️ Test on hardened PVE with read-only `/usr`
- ⏸️ Test cluster vs standalone scenarios
- ⏸️ Test LXC bind mount removal
- ⏸️ Verify no `pulse-*` artifacts remain after cleanup:
  - ⏸️ No systemd units
  - ⏸️ No binaries
  - ⏸️ No bind mounts in LXC configs
  - ⏸️ No API tokens or pulse-monitor user
  - ⏸️ No SSH keys in authorized_keys

## Alternative Approaches Considered

**Option A**: Remove only SSH keys and API tokens, skip service/binary removal
- ❌ Rejected: Leaves privileged services on unmanaged hosts

**Option B**: Make cleanup manual with documented commands
- ❌ Rejected: Shifts security responsibility to users

**Option C**: Run cleanup from Pulse controller via SSH instead of path unit
- ✅ Viable alternative: Controller has no dependency on proxy service
- Consider for future iteration

## References

- Commit: ed65fda74 "Extend node cleanup to fully remove Pulse footprint"
- Codex conversation: conv-1763161052746-956
- Related files:
  - `scripts/pulse-sensor-cleanup.sh`
  - `scripts/install-sensor-proxy.sh`
  - `internal/api/config_handlers.go` (triggerPVEHostCleanup)
  - `cmd/pulse-sensor-proxy/cleanup.go` (handleRequestCleanup)

## Priority

**Medium-High**: Current implementation removes SSH keys (most critical security piece). Full cleanup would be nice-to-have for operational cleanliness and aligns with "remove node = sever trust" principle, but the additional complexity requires careful implementation and testing.