clearNodeOfflineAlert (0%→100%):
- No alert and no count is no-op
- Resets offline count when node comes online
- Clears existing alert and adds to resolved
clearPBSOfflineAlert (0%→100%):
- No alert and no count is no-op
- Resets offline confirmation count
- Clears existing alert and adds to resolved
clearPMGOfflineAlert (0%→100%):
- No alert and no count is no-op
- Resets offline confirmation count
- Clears existing alert and adds to resolved
Coverage: 57.4% → 58.5%
SetMetricHooks (0%→100%):
- Sets all four hooks correctly
- Nil hooks are safely handled
- Properly saves/restores state to avoid test pollution
NotifyExistingAlert (0%→100%):
- Non-existent alert is no-op
- Existing alert dispatches async notification
GetResolvedAlert (0%→100%):
- Returns nil for non-existent alert
- Returns nil for nil resolved entry
- Returns nil when Alert is nil
- Returns cloned alert with resolved time
GetAlertHistory/GetAlertHistorySince (0%→100%):
- Returns history from history manager
- Respects limit parameter
- Zero time returns all history
- Filters by time correctly
ClearAlertHistory (0%→100%):
- Clears all history entries
Coverage: 56.9% → 57.4%
HandleHostRemoved (0%→100%):
- Empty host ID is no-op
- Clears host offline alert and confirmations
- Clears host metric alerts (CPU, memory)
- Clears host disk alerts
- Clears all alert types together
ReevaluateGuestAlert (0%→100%):
- No active alerts is no-op
- Clears alert when threshold disabled (nil)
- Clears alert when trigger is zero
- Clears alert when value below clear threshold
- Clears alert when value below trigger threshold
- Keeps alert when value above both thresholds
- Processes all metric types (7 types)
- Clears pending alert when threshold disabled
- Uses clear equals trigger when clear is zero
- Ignores alerts for different guests
NormalizeMetricTimeThresholds (0%→100%):
- Updated existing test to call public wrapper instead of internal function
Coverage: 54.6% → 55.6%
- TestClearActiveAlertsEmptyMaps: tests early return when both maps empty
- sanitizeAlertKey: test for input with only slashes/backslashes -> root
Coverage improvement:
- sanitizeAlertKey: 96.4% → 100%
- alerts package: 52.8%
- calls callback synchronously with alert ID
- calls callback asynchronously
- noop when callback is nil
- recovers from panic in sync and async callbacks
- safeCallResolvedCallback 90%→100%
- calls callback with alert and level
- noop when callback is nil
- recovers from panic in callback
- clones alert to prevent concurrent modification
- safeCallEscalateCallback 0%→100%
- clears alerts and state tracking not in seen set
- skips alerts from other hosts
- handles empty seen set (clears all)
- cleanupDockerContainerAlerts 75%→100%
- clears alerts not in seen set (stale disk alerts)
- empty host ID is noop
- skips nil alerts without panic
- skips non-matching prefix (non-disk alerts)
- cleanupHostDiskAlerts 76.9%→100%
- clears all disk alerts for specified host
- empty hostID is noop
- skips nil alerts without panic
- noop when no matching alerts
- clearHostDiskAlerts 72.7%→100%
- with/without host ID (prefix selection)
- derives ID from service name with sanitization
- special chars replaced with dashes
- preserves alphanumeric, underscore, hyphen
- trims leading/trailing dashes/underscores
- truncates long IDs to 32 chars
- fallback to 'service' when empty
- dockerServiceResourceID 23.8%→100%
- returns name when present
- returns trimmed name
- returns truncated ID when name empty (first 12 chars)
- returns full short ID when < 12 chars
- returns 'service' when both empty/whitespace
- prefers name over ID
- dockerServiceDisplayName 33.3%→100%
- nil thresholds get defaults (CPU 95, Memory 92, Disk 95)
- low thresholds raised to minimum relaxed values
- high thresholds unchanged
- clear adjusted when too close to trigger
- original config unchanged (no mutation)
- alerts package coverage improvement: applyRelaxedGuestThresholds 75%→93.8%
Cover all branches: disabled rate limit (zero/negative MaxAlertsHour),
under limit, at limit blocking, separate limits per alertID, old entry
cleanup, and mixed old/recent entries. Coverage improved from 71.4% to 100%.
Cover all branches: nil/empty TimeThresholds, direct match, canonical key
matching (vm→guest, container→guest), "all" fallback, specific match
precedence, and global threshold fallback. Coverage improved from 85.7% to 100%.
Cover all branches of flapping detection: disabled state, below/at/above
threshold, already-flapping state (no suppression update), and window
expiry pruning. Coverage improved from 61.9% to 95.2%.
- Test different instance and node (full format)
- Test same instance and node (short format)
- Test empty instance defaults to node
- Test whitespace trimming
- Improves BuildGuestKey coverage from 71.4% to 100%
This change fixes backup-age alert notifications to display VM/CT names
instead of just "VMID XXX" in multi-cluster environments where backups
are stored on PBS.
Changes:
- Store all guests per VMID (not just first match) to handle VMID collisions across clusters
- Persist last-known guest names/types in metadata store for deleted VMs
- Enrich backup correlation with persisted metadata when live inventory is empty
- Update CheckBackups to handle multiple VMID matches intelligently
The fix addresses two scenarios:
1. Multiple PVE clusters with same VMID backing up to one PBS
2. VMs deleted from Proxmox but backups still exist on PBS
Backup-age alerts will now show proper VM/CT names when:
- A unique guest exists with that VMID (live or persisted)
- Multiple guests share a VMID (uses first match, consistent with current behavior)
When truly ambiguous (multiple live VMs, same VMID, no way to determine origin),
the alert gracefully falls back to showing "VMID XXX".
The original fix in c6c0ac63e only handled per-resource overrides when
thresholds were disabled (trigger <= 0 or Disabled=true). It did not
handle global DisableAll* flags (DisableAllStorage, DisableAllNodes,
DisableAllGuests, etc.).
When a user toggled a DisableAll* flag from false to true:
- Check* functions returned early without processing
- Existing active alerts remained in m.activeAlerts map
- Those alerts continued generating webhook notifications
- reevaluateActiveAlertsLocked didn't check DisableAll* flags
This commit fixes the issue by:
1. Updating reevaluateActiveAlertsLocked to check all DisableAll* flags
and resolve alerts for those resource types during config updates
2. Adding alert cleanup to Check* functions before early returns:
- CheckStorage: clears usage and offline alerts
- CheckNode: clears cpu/memory/disk/temperature and offline alerts
- CheckPMG: clears queue/message alerts and offline alerts
- CheckPBS: clears cpu/memory and offline alerts
- CheckHost: calls existing cleanup helpers
3. Adding comprehensive test coverage for DisableAllStorage scenario
Related to #561
Related to #547 and #622
## Samsung SSD Fix (#547)
Samsung 980 and 990 series SSDs have known firmware bugs that cause them to
report incorrect health status (typically FAILED or critical warnings) even
when the drives are actually healthy. This is commonly due to incorrect
temperature threshold reporting in the firmware.
This change adds special handling to detect these drives and skip health
status alerts while still monitoring wearout metrics, which remain reliable.
The fix also clears any existing false alerts for these drives.
Users experiencing these false alerts should update their Samsung SSD firmware
to the latest version from Samsung, which typically resolves the issue.
## Docker Agent CPU Fix (#622)
Addresses issue where Docker container CPU usage shows 0%. The Docker
agent uses ContainerStatsOneShot which typically doesn't populate
PreCPUStats, requiring manual delta tracking between collection cycles.
Changes:
- Fix logic bug where prevContainerCPU was updated before checking if
previous sample existed, causing incorrect delta calculations
- Add comprehensive debug logging showing which calculation method
succeeded (PreCPUStats, system delta, or time-based fallback)
- Add warning after 10 PreCPUStats failures to inform about manual
tracking mode (normal for one-shot stats)
- Add detailed failure logging when CPU calculation cannot complete
Expected behavior: First collection cycle returns 0% (no previous
sample), subsequent cycles show accurate CPU metrics.
Extends the Docker monitoring and alerting system to track writable layer
usage as a percentage of the container's root filesystem. This helps
identify containers with bloated copy-on-write layers before they
consume excessive disk space.
- Add disk threshold to DockerThresholdConfig (default: 85% trigger, 80% clear)
- Evaluate disk alerts for running containers when RootFilesystemBytes > 0
- Include disk metadata (writable layer, total filesystem, block I/O stats)
- Update frontend to display and configure disk thresholds
- Add test coverage for disk usage alert hysteresis
- Document disk monitoring in DOCKER_MONITORING.md
Per-container and per-host overrides apply to disk thresholds the same
way they do for CPU and memory.
