Commit graph

1365 commits

Author SHA1 Message Date
rcourtman
3cc7bfe934 fix: match storage timeout errors without trailing slash
The error pattern `/storage/` only matched storage content endpoints
(`/storage/{name}/content`) but not the main storage list endpoint
(`/nodes/{node}/storage`).

This caused storage timeout errors like:
  Get ".../nodes/pve-100-224/storage": context deadline exceeded

to incorrectly mark cluster nodes as unhealthy, even though the timeout
was due to a slow cross-node storage query, not actual node connectivity
issues.

Fixes #754
2025-12-01 22:48:01 +00:00
rcourtman
9d77de4bf1 test: Add edge case tests for parseReplicationTime function
Add 13 new test cases covering previously untested branches:
- float32 timestamp with valid value (using smaller value for precision)
- float32/float64 zero and negative values
- json.Number zero and negative values
- int32 and uint32 timestamp handling
- Invalid date format strings (no matching layout)
- Partial date strings
- Unsupported types (bool, slice)

Coverage improved from 93.8% to 100%.
2025-12-01 22:44:23 +00:00
rcourtman
84f224cc03 test: Add float32 NaN/Inf tests for intFromAny and floatFromAny
Add 6 test cases covering float32 special values:
- intFromAny: float32 NaN, +Inf, -Inf (all return 0, false)
- floatFromAny: float32 NaN, +Inf, -Inf (all return 0, false)

Coverage improved:
- intFromAny: 96.7% -> 100%
- floatFromAny: 95.0% -> 100%
2025-12-01 22:40:08 +00:00
rcourtman
0b0d43c638 test: Add edge case tests for coerceUint64 function
Add 6 new test cases covering previously untested branches:
- float64 at MaxUint64 boundary (clamping behavior)
- float64 exceeding MaxUint64 (overflow protection)
- String with quoted "null" value
- String with quoted empty value ("")
- String with single quoted empty value ('')
- Invalid float parsing in scientific notation

Coverage improved from 92.3% to 97.4%.
2025-12-01 22:36:03 +00:00
rcourtman
0527f151a6 test: Add tests for CPUInfo.GetMHzString method
Add 11 test cases covering:
- Nil MHz returns empty string
- String MHz returned as-is
- Empty string handling
- Float64 formatted without decimals
- Float64 zero handling
- Float64 rounding for large values
- Int formatting
- Int zero handling
- Default formatting for other types (int64, bool, slice)

Coverage: GetMHzString 0% -> 100%
2025-12-01 22:29:30 +00:00
rcourtman
7c27a74c42 test: Add comprehensive tests for PBS snapshot conversion
Add 13 test cases for convertPBSSnapshots covering:
- Empty/nil input handling
- Basic snapshot field mapping
- Files as string array
- Files as object array with filename field
- Invalid file objects ignored gracefully
- Verification status as string ("ok" vs other)
- Verification status as object with state field
- Verification object without state field
- Multiple snapshots with unique IDs
- Empty namespace handling in ID generation
- Container (ct) backup type

Coverage: convertPBSSnapshots 0% -> 100%
2025-12-01 22:23:47 +00:00
rcourtman
4019b3006b test: Add tests for instance lookup and setup token validation
- Add TestFindInstanceNameByHost (12 test cases):
  - PVE instance lookup by host
  - PBS instance lookup by host
  - Unknown node types return empty
  - Empty config handling

- Add TestValidateSetupToken (8 test cases):
  - Empty token returns false
  - Valid/expired/used setup codes
  - Valid/expired recent setup tokens
  - Non-existent token handling
  - Precedence of setup codes over recent tokens

Both functions now have 100% coverage of their logic paths.
2025-12-01 22:19:30 +00:00
rcourtman
b403cdd7b7 test: Add comprehensive tests for convertDockerTasks function
- Test nil input returns nil
- Test empty slice returns nil
- Test single task with all fields populated
- Test task with error state (failed tasks)
- Test multiple tasks conversion
- Test nil time pointers preserved as nil
- Test zero time values converted to nil (defensive handling)
- Test all optional time fields (UpdatedAt, StartedAt, CompletedAt)

Improves monitoring package coverage for Docker Swarm task handling.
2025-12-01 22:13:43 +00:00
rcourtman
67a351fbd9 test: Add tests for webhook template generation and bootstrap token functions
- Add comprehensive tests for generatePayloadFromTemplateWithService covering:
  - Valid JSON template rendering
  - Invalid template syntax error handling
  - Template execution errors
  - ntfy service (plain text, skips JSON validation)
  - Various services (discord, telegram, pagerduty) JSON validation
  - Template function usage (upper, title)
  - Numeric and boolean values in templates
  - Special characters and empty template edge cases

- Add tests for bootstrapTokenValid covering:
  - Nil router handling
  - Empty hash and empty token cases
  - Valid token validation
  - Invalid token rejection
  - Whitespace trimming

- Add tests for clearBootstrapToken covering:
  - Nil router safety
  - Token file deletion
  - Missing file handling
  - Empty path handling
2025-12-01 22:07:40 +00:00
rcourtman
cb683597c6 test: Add tests for GPU temperature parsing functions
- parseGPUTemps: AMD GPU edge/junction/mem temperatures, sensor mapping
- parseNouveauGPUTemps: NVIDIA nouveau GPU core temperature parsing
- Case insensitive sensor name matching
- Invalid/zero/negative temperature handling
- Append behavior for multiple GPUs
2025-12-01 21:56:33 +00:00
rcourtman
8ba37e2ca6 fix: recover unhealthy cluster nodes even when some nodes are healthy
Previously, recovery of unhealthy nodes only triggered when ALL nodes
were unhealthy. This caused individual degraded nodes to stay degraded
forever since operations would succeed on healthy nodes and never
trigger the recovery path.

Now recovery is attempted whenever any unhealthy nodes exist, allowing
clusters to recover individual nodes over time.

Also added:
- Panic-safe unlock/lock pattern using anonymous function
- Refresh of both healthy and cooling endpoints after recovery
- Updated timestamp for accurate cooldown checks

Related to #754
2025-12-01 21:47:26 +00:00
rcourtman
c9376fa86e test: Add tests for VMAgentField JSON unmarshaling
Covers both Proxmox API formats:
- Integer format (older versions): direct int value
- Object format (Proxmox 8.3+): {enabled, available} fields
- Preference order: available > enabled > 0
- Invalid input handling defaults to 0
- Integration with VMStatus struct
2025-12-01 21:40:47 +00:00
rcourtman
3233c48986 test: Add tests for API helper functions
- isRecentlyAutoRegistered: expiry logic, cleanup behavior
- clearAutoRegistered: key deletion, empty input handling
- normalizeDockerAgentArch: all arch variants, case handling, trimming
2025-12-01 21:33:45 +00:00
rcourtman
80702a51bf test: Add tests for prepareWebhookData function
Covers webhook payload data preparation:
- publicURL handling with trailing slash trimming
- alert.Instance URL detection (http/https)
- Metadata extraction and copying
- Value/Threshold rounding to 1 decimal
- AckTime formatting
- CustomFields passthrough
2025-12-01 21:27:29 +00:00
rcourtman
833ec0933f test: Add tests for notification rate limiting and content building
- checkWebhookRateLimit: rate limit window, counter, per-URL isolation
- buildResolvedNotificationContent: single/multiple alerts, nil filtering,
  HTML escaping, timestamp handling, optional fields
2025-12-01 21:25:02 +00:00
rcourtman
334cffb1c7 test: Add tests for monitor helper and metrics functions
- clearGuestMetadataCache: nil safety, cache clearing, key isolation
- shouldRunBackupPoll: interval-based, cycle-based, config validation
- storeNodeLastSuccess/lastNodeSuccessFor: store/retrieve, overwrite, missing keys

Improves coverage for backup polling logic and node metrics tracking.
2025-12-01 21:18:33 +00:00
rcourtman
8198e1d839 test: Add comprehensive tests for guest metadata functions
Tests for rate limiting, retry, and slot management:
- retryGuestAgentCall: timeout retry, non-timeout break, context cancellation
- acquireGuestMetadataSlot: nil safety, slot acquisition, context cancellation
- releaseGuestMetadataSlot: nil safety, non-blocking release
- tryReserveGuestMetadataFetch: reservation logic, default hold duration
- scheduleNextGuestMetadataFetch: interval, jitter, RNG handling
- deferGuestMetadataRetry: backoff logic, defaults

Covers error handling and edge cases in guest agent metadata fetching.
2025-12-01 21:10:28 +00:00
rcourtman
a74e4d046a fix: add timeout to pmxcfs operations in install-sensor-proxy.sh
Reading and writing container config from /etc/pve/lxc/ can hang
indefinitely if the Proxmox cluster filesystem (pmxcfs) is slow or
unresponsive. This causes the installer to appear to hang after
"Configuring socket bind mount..." with no further output.

Add 10-second timeouts to both cp operations and provide helpful error
messages suggesting the user check cluster health with 'pvecm status'.

Related to #738
2025-12-01 21:04:01 +00:00
rcourtman
a3eb81778f test: Add table-driven tests for parseSensorsJSON
Comprehensive coverage for temperature sensor JSON parsing:
- Empty/whitespace input handling
- Invalid JSON error handling
- Legacy and wrapper format support
- Multiple chip types: coretemp, k10temp, acpitz, nvme, amdgpu, nouveau, zenpower, nct6795, it87, rp1_adc
- Edge cases: non-map chip data, CPUMax calculation from cores
2025-12-01 21:00:46 +00:00
rcourtman
1af38e3f99 test: Add tests for parseContainerMountMetadata, convertContainerDiskInfo, StalenessScore
- parseContainerMountMetadata: parts without equals, mountpoint key variant,
  whitespace handling, single source value (96.4% -> 100%)
- convertContainerDiskInfo: nil metadata, device from metadata, negative free
  clamping, whitespace label handling (95.1% -> 97.6%, remaining is dead code)
- StalenessScore: negative maxStale default, metrics lookup failure, score
  clamping edge cases (95.7% - remaining is defensive dead code)
2025-12-01 20:44:00 +00:00
rcourtman
ed0644035f test: Add tests for sanitizeGuestAddressStrings, parseContainerConfigNetworks, mergeNVMeTempsIntoDisks
- sanitizeGuestAddressStrings: IPv6 loopback with CIDR (::1/128) edge case
  (96.2% -> 100%)
- parseContainerConfigNetworks: parts without equals, mac/ips/ip6addr/ip6prefix
  key variants, whitespace-only values, all-empty results (97.2% -> 100%)
- mergeNVMeTempsIntoDisks: NVMe disk with no legacy temps (continue branch),
  more disks than temps (break branch) (97% -> 100%)
2025-12-01 20:34:48 +00:00
rcourtman
4e33b396e0 test: Add tests for normalizeEndpointHost, SelectInterval, generateDockerHostIdentifier
- normalizeEndpointHost: port-only URL host (parsed.Host fallback),
  edge cases for URL parsing (94.4% -> 100%)
- SelectInterval: negative penalty, instance key derivation, clamping
  edge cases (96% - remaining is unreachable dead code guards)
- generateDockerHostIdentifier: empty base fallbacks, suffix candidates,
  hash suffix, numeric suffix increments (91.7% -> 100%)
2025-12-01 20:26:21 +00:00
rcourtman
4c91dce2f8 test: Add tests for allow, ensureContainerRootDiskEntry; remove dead code
- allow (circuitBreaker): all states including unknown/default branch,
  open→half-open transition, half-open window timing (93.8% -> 100%)
- ensureContainerRootDiskEntry: nil container, existing disks, used>total
  clamping, negative free clamping, usage calculation (92.3% -> 100%)
- convertPoolInfoToModel: removed unreachable nil check (dead code since
  ConvertToModelZFSPool only returns nil for nil receiver) (88.9% -> 100%)
2025-12-01 20:15:32 +00:00
rcourtman
0a9177f161 test: Add tests for lookupClusterEndpointLabel, GetNodeMetrics, stateDetails
- lookupClusterEndpointLabel: nil instance, no match, Host vs IP fallback,
  case-insensitive NodeName, whitespace handling (84.6% -> 100%)
- GetNodeMetrics: unknown metric types, empty nodeID, zero duration,
  empty metrics data, disk type (94.1% -> 100%)
- stateDetails: all breaker states including unknown/invalid state,
  retryAt calculations, timestamps (90% -> 100%)
2025-12-01 20:07:22 +00:00
rcourtman
2fdc051c11 fix: Make parseNVMeTemps deterministic by checking Composite before Sensor 1
Go map iteration order is non-deterministic, causing flaky test failures.
This fix ensures "Composite" sensor is always preferred over "Sensor 1"
by checking them in separate loops rather than relying on iteration order.
2025-12-01 20:03:05 +00:00
rcourtman
f79d20296b test: Add tests for GetGuestMetrics, mergeSnapshot, getDockerCommandPayload
- GetGuestMetrics: all metric types, duration filtering, guest not found,
  unknown metric type, empty data (81% -> 100%)
- mergeSnapshot: timestamp merging for LastSuccess/LastError/LastMutated,
  ChangeHash preservation (86.7% -> 100%)
- getDockerCommandPayload: empty hostID, whitespace normalization, host
  not found, expired command cleanup, queued->dispatched marking,
  already-dispatched preservation (91.3% -> 100%)
2025-12-01 19:58:43 +00:00
rcourtman
196389853e test: Add tests for StalenessScore, parseCPUTemps, queueDockerStopCommand
- StalenessScore: 78.3%→95.7% (4 cases for metrics, future timestamp, defaults)
- parseCPUTemps: 98.1%→100% (core temp exceeding package case)
- queueDockerStopCommand: 76.9%→100% (12 cases for validation, status checks)
2025-12-01 19:50:42 +00:00
rcourtman
2594a6c124 test: Add tests for handleProxyHostFailure, recordNodeSnapshot, evaluateHostAgents
- handleProxyHostFailure: 76.5%→100% (9 cases for per-host failure tracking)
- recordNodeSnapshot: 75%→100% (6 cases for diagnostic snapshot storage)
- evaluateHostAgents: 87%→100% (10 cases for host health evaluation)
2025-12-01 19:40:32 +00:00
rcourtman
c6a34c4412 test: Add tests for handleProxyFailure, parseRPiTemperature, RecordResult
- handleProxyFailure: 86.7%→100% (6 cases for proxy disable threshold logic)
- parseRPiTemperature: 77.8%→100% (7 cases for RPi temp parsing edge cases)
- RecordResult: 95.8%→100% (negative staleness clamping case)
2025-12-01 19:33:06 +00:00
rcourtman
1f380eaaf3 test: Add tests for isProxyEnabled, taskHeap.Pop, NewAdaptiveScheduler
- isProxyEnabled: 94.4%→100% (7 cases for proxy cooldown/restore logic)
- taskHeap.Pop: 87.5%→100% (3 cases for empty/single/multi-element heap)
- NewAdaptiveScheduler: 84.6%→100% (14 cases for default value handling)
2025-12-01 19:26:37 +00:00
rcourtman
0ed84f3179 test: Add tests for disableLegacySSHOnAuthFailure, parseNVMeTemps, updateDeadLetterMetrics
- disableLegacySSHOnAuthFailure: 87.5%→100% (13 cases for auth error detection)
- parseNVMeTemps: 88.9%→100% (16 cases for NVMe temp parsing)
- updateDeadLetterMetrics: 75%→100% (6 cases for nil handling, queue states)
2025-12-01 19:20:09 +00:00
rcourtman
7bab2f86af test: Add tests for AllowDockerHostReenroll, SelectInterval, ensureBreaker
- AllowDockerHostReenroll: 95%→100% (7 cases for empty/blocked/cleanup paths)
- ensureBreaker: 94.4%→100% (6 cases for nil map, existing, custom config)
- SelectInterval: 94%→96% (20+ cases for edge cases, remaining guards unreachable)
2025-12-01 19:13:40 +00:00
rcourtman
e7bc06033a test: Add tests for convertDockerSwarmInfo, namespacePathsForDatastore, preserveFailedStorageBackups
- convertDockerSwarmInfo: 66.7%→100% (4 cases for nil, empty, populated structs)
- namespacePathsForDatastore: 92.3%→100% (removed unreachable dead code)
- preserveFailedStorageBackups: 91.3%→100% (6 cases for filtering, deduplication)
2025-12-01 19:04:23 +00:00
rcourtman
0bfc9ed447 test: Add tests for storageNamesForNode, GetDockerHost, and SchedulerHealth edge cases
- storageNamesForNode: 83.3%→100% (8 cases for filtering logic)
- GetDockerHost: 75%→100% (5 cases for lookup and whitespace handling)
- SchedulerHealth: 85.3%→89.1% (13 cases for nil handling, breaker key parsing)
2025-12-01 18:57:09 +00:00
rcourtman
2d775e9409 test: Add tests for polling interval, proxy success handlers, and failed instance removal
- effectivePVEPollingInterval: 80%→100% (6 cases for nil/clamping)
- handleProxySuccess: 80%→100% (3 cases for nil client, reset)
- handleProxyHostSuccess: →100% (6 cases for empty/whitespace/removal)
- removeFailedPBSNode: 75%→100% (5 cases for removal, backups, health)
- removeFailedPMGInstance: 75%→100% (5 cases for removal, backups, health)
2025-12-01 18:49:09 +00:00
rcourtman
acb7f4cf3a test: Add tests for normalizeEndpointHost, taskExecutionTimeout, decrementPending
- normalizeEndpointHost: 89%→94.4% (19 cases covering schemes, ports, paths, edge cases)
- taskExecutionTimeout: 83%→100% (4 cases covering defaults and custom timeouts)
- decrementPending: 88%→100% (5 cases covering decrement logic and no-op scenarios)
2025-12-01 18:42:57 +00:00
rcourtman
6183727368 test: Add tests for getNodeDisplayName, lookupClusterEndpointLabel, extractSnapshotName
- getNodeDisplayName: 71%→100% (12 tests covering cluster/non-cluster fallbacks)
- lookupClusterEndpointLabel: 77%→85% (14 tests covering endpoint matching)
- extractSnapshotName: 78%→100% (8 tests covering volid parsing)

Monitoring package 46.2%→46.4%
2025-12-01 18:37:27 +00:00
rcourtman
330cec7829 test: Add tests for clampUint64ToInt64 and resetAuthFailures
- clampUint64ToInt64: 67%→100% (5 tests covering boundary conditions)
- resetAuthFailures: 67%→100% (6 tests covering map cleanup logic)

Note: cloneStringFloatMap and cloneStringMap tests already existed.
2025-12-01 18:31:34 +00:00
rcourtman
7672cad174 test: Add tests for SetBreakerState, RecordResult, ResetQueueDepth, IncInFlight, DecInFlight
- SetBreakerState: 67%→100% (8 tests covering state conversion, retryAt clamping)
- RecordResult: 75%→96% (11 tests covering success/failure paths, staleness)
- ResetQueueDepth: 75%→100% (4 tests covering negative clamping)
- IncInFlight/DecInFlight: 67%→100% (4 tests)

Monitoring package 45.9%→46.2%
2025-12-01 18:27:05 +00:00
rcourtman
44b6745741 test: Add tests for recordAuthFailure, shouldSkipProxyHost, recoverFromPanic
- recordAuthFailure: 53%→100% (8 tests covering failure counting, node removal)
- shouldSkipProxyHost: 44%→100% (9 tests covering cooldown logic, state cleanup)
- recoverFromPanic: 50%→100% (7 tests covering various panic value types)

Monitoring package 45.3%→45.9%
2025-12-01 18:20:12 +00:00
rcourtman
b4fb5d2a6a test: Add tests for mergeNVMeTempsIntoDisks, UpdateQueueSnapshot, UpdateDeadLetterCounts, templateFuncMap
- mergeNVMeTempsIntoDisks: 57%→97% (14 tests covering WWN/serial/path matching, NVMe fallback)
- UpdateQueueSnapshot: 57%→100% (5 tests covering nil, gauges, stale key cleanup)
- UpdateDeadLetterCounts: 52%→100% (7 tests covering aggregation, normalization)
- templateFuncMap: 20%→100% (8 tests covering all template helper functions)

Monitoring 44.6%→45.3%, notifications 45.7%→45.9%
2025-12-01 18:14:43 +00:00
rcourtman
0803951854 test: Add tests for getInstanceConfig, baseIntervalForInstanceType, LoadOIDCConfig
- getInstanceConfig: 33%→100% (nil handling, case-insensitive matching)
- baseIntervalForInstanceType: 50%→100% (all instance types, clamping)
- LoadOIDCConfig: 35%→94% (file not exist, read errors, decryption)
2025-12-01 18:06:15 +00:00
rcourtman
4defced1a6 test: Add monitoring tests for parseDurationEnv, parseIntEnv, markFailed, logNodeMemorySource
Add comprehensive test coverage for:
- parseDurationEnv: empty/unset env, valid durations, invalid inputs
- parseIntEnv: empty/unset env, valid integers, invalid inputs
- markFailed: status setting, timestamps, failure reason
- logNodeMemorySource: nil handling, source change detection, log levels

All four functions now at 100% coverage (up from 0-78%).
2025-12-01 17:59:20 +00:00
rcourtman
7314510280 test: Add tests for SSH knownhosts error and path methods
- HostKeyChangeError.Error(): message formatting
- HostKeyChangeError.Unwrap(): error chain verification
- manager.Path(): path retrieval
- HostFieldMatches(): host matching patterns

ssh/knownhosts coverage: 72.2% → 75.0%
2025-12-01 17:50:13 +00:00
rcourtman
5d592bf4e6 test: Add tests for diagnostic snapshot recording and retrieval
- recordGuestSnapshot: nil handling, field setting, map initialization
- GetDiagnosticSnapshots: nil/empty handling, sorting verification

Both functions now at 100% coverage.
2025-12-01 17:48:36 +00:00
rcourtman
3de1eafd65 test: Add error path tests for config persistence functions
- LoadAPITokens: invalid JSON, empty file, file not exist
- LoadEmailConfig: invalid JSON, file not exist
- LoadWebhooks: invalid JSON, legacy migration
- LoadNodesConfig: empty arrays, missing fields, corruption recovery
- cleanupOldBackups: non-existent dir, multiple files cleanup
- Bonus: LoadAppriseConfig, LoadAlertConfig, LoadSystemSettings error paths

config package coverage: 46.3% → 48.2%
2025-12-01 17:45:50 +00:00
rcourtman
2158c14cce test: Add comprehensive tests for API token management
- Clone: deep copy verification for pointers and slices
- NewAPITokenRecord/NewHashedAPITokenRecord: creation and validation
- Config methods: HasAPITokens, APITokenCount, ActiveAPITokenHashes
- Config methods: HasAPITokenHash, PrimaryAPITokenHash, PrimaryAPITokenHint
- Config methods: ValidateAPIToken, UpsertAPIToken, RemoveAPIToken, SortAPITokens

config package coverage: 43.5% → 46.3%
2025-12-01 17:37:27 +00:00
rcourtman
d6359276ed test: Add tests for monitoring and notifications functions
- buildCephClusterModel: 0% → 100% (11 test cases)
- collectContainerRootUsage: 0% → 100% (18 test cases)
- NotificationManager getters/setters: 8 functions now tested

Overall coverage: 45.5% → 45.8%
2025-12-01 17:33:36 +00:00
rcourtman
caa33b3140 test: Add tests for checkPMGNodeQueues
- 12 test cases covering per-node queue monitoring
- Tests total/deferred/hold queue thresholds at warning/critical levels
- Tests oldest message age per node
- Tests outlier detection, threshold clearing, existing alert updates
- Tests empty nodes and nil QueueStatus handling

Coverage: checkPMGNodeQueues 0%→85.1%
Package coverage: 70.3%→72.3%
2025-12-01 17:16:32 +00:00
rcourtman
5f85f4e5ac test: Add tests for checkZFSPoolHealth
- 14 test cases covering pool state alerts (ONLINE, DEGRADED, FAULTED,
  UNAVAIL), pool error tracking, device-level alerts, state transitions
- Tests pool state clearing on recovery, error count updates,
  SPARE device handling, FAULTED device critical alerts

Coverage: checkZFSPoolHealth 0%→100%
Package coverage: 68.6%→70.3%
2025-12-01 17:14:12 +00:00