diff --git a/CLEANUP_TODO.md b/CLEANUP_TODO.md deleted file mode 100644 index 2b66877..0000000 --- a/CLEANUP_TODO.md +++ /dev/null @@ -1,209 +0,0 @@ -# Node Cleanup Implementation - CODEX REVIEW COMPLETE - -**Status**: ✅ Implementation complete, Codex review passed, ready for deployment testing. - -**Latest Update**: Addressed all 8 critical issues found by Codex review (conv-1763166192078-1076). - -## Implementation History - -**Commit b192c60e9**: Relocate binaries to /opt/pulse/sensor-proxy/ -- ✅ Binary path moved from /usr/local/bin to /opt/pulse/sensor-proxy/bin -- ✅ All systemd ExecStart paths updated -- ✅ Guarantees cleanup works on read-only /usr systems - -**Commit 6692228e0**: Full cleanup script implementation -- ✅ SSH key removal (working) -- ✅ Service uninstallation (via isolated systemd-run transient unit) -- ✅ API token deletion (JSON first, filtered table fallback) -- ✅ Bind mount removal (scans all LXC configs) -- ✅ pulse-monitor user deletion -- ✅ flock serialization (prevents concurrent runs) -- ✅ Immediate request file deletion (prevents loops) - -**Commits ed48d7555, 17d2e6876**: Bug fixes during testing -- ✅ Fixed directory creation order (create before binary download) -- ✅ Fixed SHARE_DIR unbound variable - -**Commit bcd8d4e0f**: Critical Codex review fixes #1-4 -- ✅ Host detection now includes hostname/FQDN (not just IP) -- ✅ Systemd sandbox relaxed (/etc/pve and /etc/systemd/system writable) -- ✅ Uninstaller called with --purge flag for complete removal -- ✅ All /usr/local references migrated to /opt paths -- ✅ UUID used for transient unit names (prevents collisions) -- ✅ --wait and --collect flags capture uninstaller exit code - -**Commit fe53d6473**: Remaining Codex review fixes #5-8 -- ✅ LXC bind mounts removed via `pct set -delete` (not sed) -- ✅ API token parsing: three-tier fallback (pveum JSON → pvesh JSON → table) -- ✅ Retry logic: rename to .processing, delete only on success - -## Issues Discovered and Resolved - -### 1. Host Detection Failure (CRITICAL) - ✅ FIXED -**Problem**: Cleanup script only compared against IPs from `hostname -I`, missing nodes configured as `https://hostname:8006`. This caused localhost cleanup (API tokens, bind mounts, service removal) to be skipped entirely. - -**Fix**: Check against `hostname`, `hostname -f`, and all IPs. Now catches all localhost variations. - -**Impact**: Without this fix, cleanup would only remove remote SSH keys, leaving services/binaries/tokens intact. - -### 2. Systemd Sandbox Blocked Critical Operations - ✅ FIXED -**Problem**: Cleanup service ran with `ProtectSystem=strict` and `ReadWritePaths=/var/lib/pulse-sensor-proxy /root/.ssh`, blocking writes to `/etc/pve` (Proxmox configs) and `/etc/systemd/system`. - -**Fix**: Added `/etc/pve` and `/etc/systemd/system` to `ReadWritePaths`. - -**Impact**: `pveum` token deletion and `pct set -delete` would fail with "read-only file system". - -### 3. Incomplete Purging - ✅ FIXED -**Problem**: Uninstaller called without `--purge`, leaving `/var/lib/pulse-sensor-proxy`, service user, and SSH private keys on disk. Request file deleted before work completed, preventing retry on failure. - -**Fix**: Added `--purge` flag, added `--wait --collect` to capture exit code, fail cleanup if uninstaller fails. - -**Impact**: Claimed "cleanup completed successfully" even when artifacts remained. - -### 4. Incomplete Path Migration - ✅ FIXED -**Problem**: After relocating binaries to `/opt`, three references still pointed to `/usr/local`: -- Forced command in SSH authorized_keys: `/usr/local/bin/pulse-sensor-wrapper.sh` -- Self-heal script: `/usr/local/share/pulse/install-sensor-proxy.sh` -- Backend removal helpers: `/usr/local/bin/pulse-sensor-cleanup.sh` - -**Fix**: Updated all three locations. Go backend now checks both paths (new + legacy). - -**Impact**: New installs would lose telemetry immediately (SSH command not found). UI-triggered cleanup wouldn't find helpers. - -### 5. Transient Unit Name Collisions - ✅ FIXED -**Problem**: Used `date +%s` for unit names. If two cleanup requests fired within same second, second would fail (unit already exists). Error suppressed, logged as "started" anyway. - -**Fix**: Use `/proc/sys/kernel/random/uuid` for unique names. - -**Impact**: Multiple concurrent cleanups could race, with silent failures. - -### 6. Bind Mount Removal Too Broad - ✅ FIXED -**Problem**: Used `sed -i '/pulse-sensor-proxy/d'` which would delete ANY line mentioning the substring (including unrelated comments/hooks). Also couldn't run inside systemd sandbox. - -**Fix**: Use `pct set -delete mp` which validates syntax and is sandbox-compatible. - -**Impact**: Could break container configs. Sed approach would fail anyway due to sandbox. - -### 7. API Token Parsing Fragile - ✅ FIXED -**Problem**: Table parser filtered only `│┌└╞`, failing on other Unicode borders or locales. "User not found" vs "feature unsupported" both treated as "no tokens". - -**Fix**: Three-tier fallback: -1. `pveum --output-format json` with python3 parsing -2. `pvesh get /access/users/pulse-monitor@pam/token` (always JSON) -3. Improved table parser with better filtering - -**Impact**: Non-English locales or Proxmox versions with different table formatting would silently skip token cleanup. - -### 8. No Retry on Failure - ✅ FIXED -**Problem**: Request file deleted immediately. Any crash left no way to retry. - -**Fix**: Rename to `.processing`, delete only on success. Failures leave `.processing` file for manual investigation/retry. - -**Impact**: Transient failures (network issues, systemd hiccups) couldn't be retried automatically. - -## Read-Only Filesystem - -**Problem**: Proxmox VE can mount `/usr` as read-only (hardened setups, boot-from-snapshots, appliance builds). - -**Solution**: All binaries relocated to `/opt/pulse/sensor-proxy/` where we control permissions. - -## Process Isolation During Uninstall - -**Problem**: Cleanup script runs as systemd service. When it calls the uninstaller which stops the proxy service, systemd kills the cleanup service with SIGTERM. - -**Solution**: Use transient systemd unit with: -- UUID-based unique name -- `Conflicts=pulse-sensor-proxy.service` -- `--wait --collect` to capture exit code -- `--purge` flag for complete removal - -## Implementation Status - -### Phase 1: Relocate Binaries ✅ COMPLETE -- ✅ Update installer to use `/opt/pulse/sensor-proxy/bin/` -- ✅ Update systemd unit ExecStart path -- ✅ Update cleanup script to use new paths -- ✅ Update forced command in SSH authorized_keys -- ✅ Update self-heal script paths -- ✅ Update Go backend removal helpers (supports both new and legacy paths) - -### Phase 2: Fix Uninstall Orchestration ✅ COMPLETE -- ✅ Cleanup script spawns transient systemd unit via systemd-run -- ✅ Added `Conflicts=pulse-sensor-proxy.service` to transient unit -- ✅ UUID-based unit names prevent collisions -- ✅ Added `--wait --collect` to capture exit code -- ✅ Uninstaller runs with `--purge --quiet` flags -- ✅ Cleanup fails if uninstaller exits non-zero - -### Phase 3: Prevent Cleanup Loops ✅ COMPLETE -- ✅ Added `flock` serialization via exec 200>lockfile -- ✅ Rename request file to `.processing` (allows retry on failure) -- ✅ Delete `.processing` only on successful completion -- ✅ `.path` unit uses PathChanged/PathModified (correct semantics) -- ✅ Comprehensive logging at info/warn/error levels - -### Phase 4: Improve API Token Handling ✅ COMPLETE -- ✅ Three-tier fallback: pveum JSON → pvesh JSON → table parsing -- ✅ Python3 JSON parsing for robustness -- ✅ Better table filtering (handles more Unicode characters) -- ✅ Error handling for token deletion failures (logs warnings, continues) -- ✅ Logs each token removal attempt - -### Phase 5: LXC Bind Mount Removal ✅ COMPLETE -- ✅ Use `pct set -delete` instead of sed (validates syntax) -- ✅ Sandbox-compatible (works with ProtectSystem=strict + /etc/pve writable) -- ✅ Only removes mounts containing "pulse-sensor-proxy" - -### Phase 6: Host Detection ✅ COMPLETE -- ✅ Check against hostname, FQDN, and all local IPs -- ✅ Localhost detection works for `https://hostname:8006` URLs -- ✅ Ensures full cleanup runs for all localhost variations - -### Phase 7: Testing & Validation ⏸️ PENDING -- ⏸️ Deploy updated installer to test host -- ⏸️ Test node removal via Pulse UI -- ⏸️ Verify complete cleanup: - - ⏸️ No systemd units - - ⏸️ No binaries in /opt or /usr/local - - ⏸️ No bind mounts in LXC configs - - ⏸️ No API tokens or pulse-monitor user - - ⏸️ No SSH keys in authorized_keys - - ⏸️ No /var/lib/pulse-sensor-proxy directory -- ⏸️ Test retry logic (simulate failure, verify .processing file persists) -- ⏸️ Test on fresh Proxmox VE install -- ⏸️ Test on hardened PVE with read-only `/usr` -- ⏸️ Test cluster vs standalone scenarios - -## Codex Review Summary - -**Review Session**: conv-1763166192078-1076 - -**Findings**: 8 critical issues identified: -1. ❌ Host detection only checked IPs, skipped cleanup for hostname-based URLs -2. ❌ Systemd sandbox blocked /etc/pve and /etc/systemd writes -3. ❌ Uninstaller missing --purge, request file deleted too early -4. ❌ Incomplete /usr/local → /opt migration (SSH forced command, self-heal, backend) -5. ❌ Timestamp-based unit names caused collisions -6. ❌ Brittle sed-based bind mount removal, Unicode table parsing -7. ❌ API token parsing failed on locales, couldn't distinguish error types -8. ❌ No retry mechanism for transient failures - -**Resolution**: All 8 issues fixed in commits bcd8d4e0f and fe53d6473. - -## References - -- Initial commit: ed65fda74 "Extend node cleanup to fully remove Pulse footprint" -- Binary relocation: b192c60e9 "Relocate binaries to /opt/pulse/sensor-proxy/" -- Full implementation: 6692228e0 "Full cleanup script implementation" -- Bug fixes: ed48d7555, 17d2e6876 -- Codex review fixes #1-4: bcd8d4e0f -- Codex review fixes #5-8: fe53d6473 -- Codex review: conv-1763166192078-1076 -- Related files: - - `scripts/install-sensor-proxy.sh` (installer + cleanup script generation) - - `internal/api/config_handlers.go` (triggerPVEHostCleanup + manual removal) - - `cmd/pulse-sensor-proxy/cleanup.go` (handleRequestCleanup) - -## Priority - -**High**: Implementation complete and Codex-reviewed. Ready for deployment testing. All critical security and correctness issues addressed. diff --git a/MIGRATION_SCAFFOLDING.md b/MIGRATION_SCAFFOLDING.md deleted file mode 100644 index ba7c0f2..0000000 --- a/MIGRATION_SCAFFOLDING.md +++ /dev/null @@ -1,58 +0,0 @@ -# Migration & Rollout Scaffolding - -This document tracks temporary code paths, feature flags, and kill switches that -let us roll out risky changes safely. Keep it current so on-call engineers know -how to disable new behavior without spelunking through the codebase. - -## How to Use This Document - -1. **Add an entry** whenever you introduce code that needs a manual cleanup - window (feature flags, compatibility layers, installers with rollback paths). -2. **State the kill switch** (environment variable, UI toggle, config file, or - systemd override) and who owns the rollout. -3. **Define removal criteria**. Once criteria are met, delete the scaffolding - and update this file. - -### Entry Template - -``` -## -- Introduced: vX.Y.Z (date/commit, if known) -- Owner: -- Kill switch: how to disable + default state -- Monitoring: metrics/logs to watch -- Cleanup criteria: what must be true before removing scaffolding -- Notes: cross-links to docs or runbooks -``` - -## Active Scaffolding - -### Adaptive Polling Scheduler -- **Introduced**: v4.24.0 (documented in `docs/monitoring/ADAPTIVE_POLLING.md`) -- **Owner**: Monitoring subsystem (rcourtman) -- **Kill switch**: Toggle **Settings → System → Monitoring → Adaptive - Polling**, or set `ADAPTIVE_POLLING_ENABLED=false` before starting Pulse - (Docker/Kubernetes). Helm users can override `adaptivePollingEnabled: false`. -- **Monitoring**: `pulse_monitor_poll_queue_depth`, - `pulse_monitor_poll_staleness_seconds`, and - `/api/monitoring/scheduler/health` for breaker/dead-letter status. -- **Cleanup criteria**: Remove flag and legacy scheduler code after three stable - releases with `queue.depth < 50` during peak hours and no dead-letter growth. -- **Notes**: Operational guidance lives in - `docs/operations/ADAPTIVE_POLLING_ROLLOUT.md`. - -### Automatic Updates Service -- **Introduced**: commit `f46ff1792` (2025-10-11) added `scripts/pulse-auto-update.sh` -- **Owner**: Installer/Update subsystem (rcourtman) -- **Kill switch**: Disable **Settings → System → Updates → Automatic Updates** or - set `AUTO_UPDATE_ENABLED=false` in `/var/lib/pulse/system.json`. Installers - accept `--no-auto-update` to force the flag off. -- **Monitoring**: `journalctl -u pulse-update.service`, UI update history - (Settings → System → Updates), and webhook notifications tagged with - `event_id=system.update`. -- **Cleanup criteria**: Remove legacy opt-out plumbing once the opt-in rate is - ≥90 % across supported installers and we have at least two release cycles with - zero rollback events triggered by auto-update. -- **Notes**: Keep release-specific caveats in `docs/RELEASE_NOTES.md`. The - update scheduler runs only on systemd installs; container builds ignore the - flag. diff --git a/copy_and_run.sh b/copy_and_run.sh deleted file mode 100755 index 1463137..0000000 --- a/copy_and_run.sh +++ /dev/null @@ -1,8 +0,0 @@ -cd / -if [ -d /tmp/pulse-copy ]; then - rm -rf /tmp/pulse-copy -fi -mkdir -p /tmp/pulse-copy -cp -R /opt/pulse /tmp/pulse-copy/repo -cd /tmp/pulse-copy/repo -GOCACHE=/tmp/go-cache HOME=/tmp/pulse-copy/repo go test ./... diff --git a/scripts/backup-claude-md.sh b/scripts/backup-claude-md.sh deleted file mode 100755 index 6e0f465..0000000 --- a/scripts/backup-claude-md.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/bin/bash -# Daily LOCAL backup script for CLAUDE.md -# Keeps last 30 days of backups in a LOCAL directory (never committed) - -set -e - -CLAUDE_MD="/opt/pulse/CLAUDE.md" -BACKUP_DIR="$HOME/.claude-md-backups" # User's home directory, not in repo -DATE=$(date +%Y-%m-%d) -BACKUP_FILE="${BACKUP_DIR}/CLAUDE.md.${DATE}" - -# Create backup directory if it doesn't exist -mkdir -p "$BACKUP_DIR" - -# Only create backup if CLAUDE.md exists -if [ ! -f "$CLAUDE_MD" ]; then - echo "Error: $CLAUDE_MD not found" - exit 1 -fi - -# Only create backup if one doesn't exist for today -if [ -f "$BACKUP_FILE" ]; then - echo "Backup already exists for today: $BACKUP_FILE" - exit 0 -fi - -# Create backup -cp "$CLAUDE_MD" "$BACKUP_FILE" -echo "Created backup: $BACKUP_FILE" - -# Remove backups older than 30 days -find "$BACKUP_DIR" -name "CLAUDE.md.*" -type f -mtime +30 -delete -echo "Cleaned up backups older than 30 days" - -# Show current backups -echo "" -echo "Current backups in $BACKUP_DIR:" -ls -lh "$BACKUP_DIR" | tail -n +2 | wc -l -echo "backups found" diff --git a/scripts/codex-router.sh b/scripts/codex-router.sh deleted file mode 100755 index 55cc33e..0000000 --- a/scripts/codex-router.sh +++ /dev/null @@ -1,50 +0,0 @@ -#!/bin/bash -# Heuristic router for selecting a Codex reasoning tier. -# Usage: codex-router.sh "Fix the typo" or echo "..." | codex-router.sh - -set -euo pipefail - -WORKDIR="/opt/pulse" - -# Collect a prompt from stdin and/or arguments. -PROMPT_INPUT="" -if [[ ! -t 0 ]]; then - PROMPT_INPUT="$(cat)" -fi - -if [[ $# -gt 0 ]]; then - if [[ -n "${PROMPT_INPUT}" ]]; then - PROMPT="${PROMPT_INPUT}"$'\n'"$*" - else - PROMPT="$*" - fi -else - PROMPT="${PROMPT_INPUT}" -fi - -PROMPT="$(printf '%s' "${PROMPT}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')" - -if [[ -z "${PROMPT}" ]]; then - echo "Usage: $0 " >&2 - exit 1 -fi - -reasoning_profile="low" - -# Escalate when the task hints at analysis, design, or larger edits. -if [[ ${#PROMPT} -gt 400 ]] \ - || grep -qiE '(analysis|investigate|explain|design|architecture|strategy|refactor|spec|diagnos|trade[- ]?off|postmortem)' <<<"${PROMPT}"; then - reasoning_profile="medium" -fi - -case "${reasoning_profile}" in - medium) - profile_arg=(--profile medium) - ;; - *) - profile_arg=(--profile low) - ;; -esac - -echo "Routing prompt to Codex profile '${reasoning_profile}'" >&2 -exec codex exec "${profile_arg[@]}" -C "${WORKDIR}" "${PROMPT}" diff --git a/work.sh b/work.sh deleted file mode 100644 index 4e331a1..0000000 --- a/work.sh +++ /dev/null @@ -1,3 +0,0 @@ -#!/bin/bash -cd /opt/pulse -git checkout cleanup-auto-uninstall