HandleHostRemoved (0%→100%):
- Empty host ID is no-op
- Clears host offline alert and confirmations
- Clears host metric alerts (CPU, memory)
- Clears host disk alerts
- Clears all alert types together
ReevaluateGuestAlert (0%→100%):
- No active alerts is no-op
- Clears alert when threshold disabled (nil)
- Clears alert when trigger is zero
- Clears alert when value below clear threshold
- Clears alert when value below trigger threshold
- Keeps alert when value above both thresholds
- Processes all metric types (7 types)
- Clears pending alert when threshold disabled
- Uses clear equals trigger when clear is zero
- Ignores alerts for different guests
NormalizeMetricTimeThresholds (0%→100%):
- Updated existing test to call public wrapper instead of internal function
Coverage: 54.6% → 55.6%
- TestClearActiveAlertsEmptyMaps: tests early return when both maps empty
- sanitizeAlertKey: test for input with only slashes/backslashes -> root
Coverage improvement:
- sanitizeAlertKey: 96.4% → 100%
- alerts package: 52.8%
- calls callback synchronously with alert ID
- calls callback asynchronously
- noop when callback is nil
- recovers from panic in sync and async callbacks
- safeCallResolvedCallback 90%→100%
- calls callback with alert and level
- noop when callback is nil
- recovers from panic in callback
- clones alert to prevent concurrent modification
- safeCallEscalateCallback 0%→100%
- clears alerts and state tracking not in seen set
- skips alerts from other hosts
- handles empty seen set (clears all)
- cleanupDockerContainerAlerts 75%→100%
- clears alerts not in seen set (stale disk alerts)
- empty host ID is noop
- skips nil alerts without panic
- skips non-matching prefix (non-disk alerts)
- cleanupHostDiskAlerts 76.9%→100%
- clears all disk alerts for specified host
- empty hostID is noop
- skips nil alerts without panic
- noop when no matching alerts
- clearHostDiskAlerts 72.7%→100%
- with/without host ID (prefix selection)
- derives ID from service name with sanitization
- special chars replaced with dashes
- preserves alphanumeric, underscore, hyphen
- trims leading/trailing dashes/underscores
- truncates long IDs to 32 chars
- fallback to 'service' when empty
- dockerServiceResourceID 23.8%→100%
- returns name when present
- returns trimmed name
- returns truncated ID when name empty (first 12 chars)
- returns full short ID when < 12 chars
- returns 'service' when both empty/whitespace
- prefers name over ID
- dockerServiceDisplayName 33.3%→100%
- nil thresholds get defaults (CPU 95, Memory 92, Disk 95)
- low thresholds raised to minimum relaxed values
- high thresholds unchanged
- clear adjusted when too close to trigger
- original config unchanged (no mutation)
- alerts package coverage improvement: applyRelaxedGuestThresholds 75%→93.8%
Cover all branches: disabled rate limit (zero/negative MaxAlertsHour),
under limit, at limit blocking, separate limits per alertID, old entry
cleanup, and mixed old/recent entries. Coverage improved from 71.4% to 100%.
Cover all branches: nil/empty TimeThresholds, direct match, canonical key
matching (vm→guest, container→guest), "all" fallback, specific match
precedence, and global threshold fallback. Coverage improved from 85.7% to 100%.
Cover all branches: disabled state, invalid timezone fallback, day not
enabled, invalid start/end time parsing, overnight window spanning
midnight, and normal daytime window. Coverage improved from 61.5% to 96.2%.
Cover all branches of flapping detection: disabled state, below/at/above
threshold, already-flapping state (no suppression update), and window
expiry pruning. Coverage improved from 61.9% to 95.2%.
- Test different instance and node (full format)
- Test same instance and node (short format)
- Test empty instance defaults to node
- Test whitespace trimming
- Improves BuildGuestKey coverage from 71.4% to 100%
Tests for alert filter evaluation logic:
- evaluateVMCondition: metric/text/raw conditions for VMs
- evaluateContainerCondition: same for containers
- evaluateFilterStack: AND/OR logic with multiple conditions
- evaluateFilterCondition: type dispatch
74 test cases covering:
- All metric types (cpu, memory, disk, diskread, diskwrite, network)
- All comparison operators (>, <, >=, <=, =, ==)
- Text search fields (name, node, vmid) with case insensitivity
- Raw text search across multiple fields
- Value type conversions (int to float64)
- Edge cases (zero, empty, invalid types)
Coverage improved from 46.8% to 48.9%.
Move guest threshold resolution and filter evaluation logic from
alerts.go to filter_evaluation.go for better code organization.
Extracted functions:
- evaluateFilterCondition
- evaluateVMCondition
- evaluateContainerCondition
- evaluateFilterStack
- getGuestThresholds
This reduces alerts.go by ~370 lines and isolates the rule evaluation
engine that determines which thresholds apply to guests.
Test coverage for isMonitorOnlyAlert, quietHoursCategoryForAlert, and
ensureValidHysteresis functions that were previously untested. Improves
internal/alerts coverage from 46.4% to 46.8%.
Test coverage for pure utility functions:
- sanitizeAlertKey: string normalization for alert IDs
- abs: absolute value for float64
- isQueueOutlier: queue anomaly detection logic
- scaleThreshold: threshold scaling with ceiling
- calculateMedianInt: integer median calculation
78 test cases added, coverage improved from 45.3% to 46.2%.
Tests for:
- normalizePoweredOffSeverity: Verifies severity normalization to warning/critical
- cloneMetadata: Deep copy functionality for metadata maps
- cloneMetadataValue: Type-specific cloning for various slice and map types
- Alert.Clone: Full alert cloning with pointer field independence
46 test cases covering nil handling, deep copy verification, slice/map
independence, and all supported metadata value types.
Coverage improved from 44.8% to 45.3%.
Mark intentionally unused parameters with underscore to:
- Silence unparam warnings for legitimate unused parameters
- Keep function signatures intact for API compatibility
- Remove unused req from serveChecksum helper
- Merge variable declaration with assignment (S1021)
- Use unconditional strings.TrimPrefix (S1017)
- Remove unnecessary nil checks around range (S1031)
- Remove unnecessary fmt.Sprintf (S1039)
- Use copy() instead of manual loop (S1001)
- Use time.Until instead of t.Sub(time.Now()) (S1024)
- Use buf.String() instead of string(buf.Bytes()) (S1030)
Implements comprehensive mdadm RAID array monitoring for Linux hosts
via pulse-host-agent. Arrays are automatically detected and monitored
with real-time status updates, rebuild progress tracking, and automatic
alerting for degraded or failed arrays.
Key changes:
**Backend:**
- Add mdadm package for parsing mdadm --detail output
- Extend host agent report structure with RAID array data
- Integrate mdadm collection into host agent (Linux-only, best-effort)
- Add RAID array processing in monitoring system
- Implement automatic alerting:
- Critical alerts for degraded arrays or arrays with failed devices
- Warning alerts for rebuilding/resyncing arrays with progress tracking
- Auto-clear alerts when arrays return to healthy state
**Frontend:**
- Add TypeScript types for RAID arrays and devices
- Display RAID arrays in host details drawer with:
- Array status (clean/degraded/recovering) with color-coded indicators
- Device counts (active/total/failed/spare)
- Rebuild progress percentage and speed when applicable
- Green for healthy, amber for rebuilding, red for degraded
**Documentation:**
- Document mdadm monitoring feature in HOST_AGENT.md
- Explain requirements (Linux, mdadm installed, root access)
- Clarify scope (software RAID only, hardware RAID not supported)
**Testing:**
- Add comprehensive tests for mdadm output parsing
- Test parsing of healthy, degraded, and rebuilding arrays
- Verify proper extraction of device states and rebuild progress
All builds pass successfully. RAID monitoring is automatic and best-effort
- if mdadm is not installed or no arrays exist, host agent continues
reporting other metrics normally.
Related to #676
This change fixes backup-age alert notifications to display VM/CT names
instead of just "VMID XXX" in multi-cluster environments where backups
are stored on PBS.
Changes:
- Store all guests per VMID (not just first match) to handle VMID collisions across clusters
- Persist last-known guest names/types in metadata store for deleted VMs
- Enrich backup correlation with persisted metadata when live inventory is empty
- Update CheckBackups to handle multiple VMID matches intelligently
The fix addresses two scenarios:
1. Multiple PVE clusters with same VMID backing up to one PBS
2. VMs deleted from Proxmox but backups still exist on PBS
Backup-age alerts will now show proper VM/CT names when:
- A unique guest exists with that VMID (live or persisted)
- Multiple guests share a VMID (uses first match, consistent with current behavior)
When truly ambiguous (multiple live VMs, same VMID, no way to determine origin),
the alert gracefully falls back to showing "VMID XXX".
This commit addresses 7 critical issues identified during the alert system audit:
**P0 Critical - Race Conditions Fixed:**
1. **dispatchAlert race in NotifyExistingAlert** (lines 5486-5497)
- Changed from RLock to Lock to hold mutex during dispatchAlert call
- dispatchAlert calls checkFlapping which writes to maps (flappingHistory, flappingActive, suppressedUntil)
- Previous code: grabbed RLock, got alert pointer, released lock, then called dispatchAlert (RACE)
- Fixed: hold Lock through dispatchAlert call
2. **dispatchAlert race in LoadActiveAlerts startup** (lines 8216-8235)
- Startup goroutines called dispatchAlert without holding lock
- Added m.mu.Lock/Unlock around dispatchAlert call in goroutine
- Also added cancellation via escalationStop channel to prevent goroutine leaks on shutdown
3. **checkFlapping documentation** (line 738)
- Added clear comment that checkFlapping requires caller to hold m.mu
- Prevents future race conditions from improper usage
**P1 Important - Data Loss Prevention:**
4. **History save race condition** (lines 177-180 in history.go)
- Added saveMu mutex to serialize disk writes
- Previous: concurrent saves could interleave, causing newer data to be overwritten by older snapshots
- Fixed: saveMu.Lock at start of saveHistoryWithRetry ensures atomic disk writes
- Newer snapshots now always win over older ones
**P2 Memory Leak Prevention:**
5. **PMG anomaly tracker cleanup** (lines 7318-7331)
- Added cleanup for pmgAnomalyTrackers map (24 hour TTL based on LastSampleTime)
- Prevents unbounded growth from decommissioned/transient PMG instances
- Each tracker: ~1-2KB (48 samples + baselines)
6. **PMG quarantine history cleanup** (lines 7333-7354)
- Added cleanup for pmgQuarantineHistory map (7 day TTL based on last snapshot)
- Prevents memory leak for deleted PMG instances
- Removes both empty histories and very old histories
**P2 Goroutine Leak Prevention:**
7. **Startup notification goroutine cancellation** (lines 8218-8234)
- Added select with escalationStop channel to cancel startup notifications
- Prevents goroutines from continuing after Stop() is called
- Scales with number of restored critical alerts
All fixes maintain proper lock ordering and prevent deadlocks by ensuring locks are held when accessing shared maps.
The original fix in c6c0ac63e only handled per-resource overrides when
thresholds were disabled (trigger <= 0 or Disabled=true). It did not
handle global DisableAll* flags (DisableAllStorage, DisableAllNodes,
DisableAllGuests, etc.).
When a user toggled a DisableAll* flag from false to true:
- Check* functions returned early without processing
- Existing active alerts remained in m.activeAlerts map
- Those alerts continued generating webhook notifications
- reevaluateActiveAlertsLocked didn't check DisableAll* flags
This commit fixes the issue by:
1. Updating reevaluateActiveAlertsLocked to check all DisableAll* flags
and resolve alerts for those resource types during config updates
2. Adding alert cleanup to Check* functions before early returns:
- CheckStorage: clears usage and offline alerts
- CheckNode: clears cpu/memory/disk/temperature and offline alerts
- CheckPMG: clears queue/message alerts and offline alerts
- CheckPBS: clears cpu/memory and offline alerts
- CheckHost: calls existing cleanup helpers
3. Adding comprehensive test coverage for DisableAllStorage scenario
Related to #561
