Commit graph

56 commits

Author SHA1 Message Date
rcourtman
071ccc44ec test: Add ProcessQueuedNotification error path tests
Cover invalid config JSON for email, webhook, and apprise types,
plus unknown notification type handling.
Coverage: 63.3% → 83.3%
2025-12-02 02:26:25 +00:00
rcourtman
7a63cf3be9 test: Add sendNotificationsDirect tests
Cover disabled paths: all channels disabled, webhook disabled,
multiple webhooks with mixed enabled state.
Coverage: 44.4% → 66.7%
2025-12-02 02:21:46 +00:00
rcourtman
ea04e4faae test: Add sendHTMLEmailWithError tests
Cover email sending paths: empty To uses From as recipient,
nil email manager creates new one, existing manager config updates.
Coverage: 57.9% → 89.5%
2025-12-02 02:19:49 +00:00
rcourtman
9768316e07 test: Add sendGroupedApprise error path tests
Add 3 tests for Apprise notification error handling:
- No alerts returns error
- Not enabled returns error
- Empty payload (all nil alerts) returns error

Coverage: 56.2% → 75.0%
2025-12-02 02:14:13 +00:00
rcourtman
c080a6beb6 test: Add DNS resolution failure test for ValidateWebhookURL
Tests the error path when webhook hostname cannot be resolved
(78.1% to 81.2% coverage).
2025-12-02 01:09:03 +00:00
rcourtman
44b835ad15 test: Add critical-only grouped alert template case
Test multiple critical alerts without warnings to cover the
"critical > 0" only branch in subject line generation.

Coverage: 97.7% → 100%
2025-12-02 00:40:02 +00:00
rcourtman
96bf0b0ca6 test: Add edge cases for buildApprisePayload
- Test nil alerts in list are filtered out
- Test all-nil alert list returns empty
- Test multiple alerts changes title format

Coverage: 88.9% → 100%
2025-12-02 00:37:28 +00:00
rcourtman
296a520d81 test: Add edge case for extractTelegramChatID URL parse error
Test URL with control characters triggers url.Parse error path.

Coverage: 92.9% → 100%
2025-12-02 00:31:50 +00:00
rcourtman
890a2f792d test: Add edge case for NormalizeAppriseConfig low timeout
Test TimeoutSeconds value between 1-4 clamps to minimum of 5.

Coverage: 97.4% → 100%
2025-12-02 00:29:12 +00:00
rcourtman
67a351fbd9 test: Add tests for webhook template generation and bootstrap token functions
- Add comprehensive tests for generatePayloadFromTemplateWithService covering:
  - Valid JSON template rendering
  - Invalid template syntax error handling
  - Template execution errors
  - ntfy service (plain text, skips JSON validation)
  - Various services (discord, telegram, pagerduty) JSON validation
  - Template function usage (upper, title)
  - Numeric and boolean values in templates
  - Special characters and empty template edge cases

- Add tests for bootstrapTokenValid covering:
  - Nil router handling
  - Empty hash and empty token cases
  - Valid token validation
  - Invalid token rejection
  - Whitespace trimming

- Add tests for clearBootstrapToken covering:
  - Nil router safety
  - Token file deletion
  - Missing file handling
  - Empty path handling
2025-12-01 22:07:40 +00:00
rcourtman
80702a51bf test: Add tests for prepareWebhookData function
Covers webhook payload data preparation:
- publicURL handling with trailing slash trimming
- alert.Instance URL detection (http/https)
- Metadata extraction and copying
- Value/Threshold rounding to 1 decimal
- AckTime formatting
- CustomFields passthrough
2025-12-01 21:27:29 +00:00
rcourtman
833ec0933f test: Add tests for notification rate limiting and content building
- checkWebhookRateLimit: rate limit window, counter, per-URL isolation
- buildResolvedNotificationContent: single/multiple alerts, nil filtering,
  HTML escaping, timestamp handling, optional fields
2025-12-01 21:25:02 +00:00
rcourtman
b4fb5d2a6a test: Add tests for mergeNVMeTempsIntoDisks, UpdateQueueSnapshot, UpdateDeadLetterCounts, templateFuncMap
- mergeNVMeTempsIntoDisks: 57%→97% (14 tests covering WWN/serial/path matching, NVMe fallback)
- UpdateQueueSnapshot: 57%→100% (5 tests covering nil, gauges, stale key cleanup)
- UpdateDeadLetterCounts: 52%→100% (7 tests covering aggregation, normalization)
- templateFuncMap: 20%→100% (8 tests covering all template helper functions)

Monitoring 44.6%→45.3%, notifications 45.7%→45.9%
2025-12-01 18:14:43 +00:00
rcourtman
d6359276ed test: Add tests for monitoring and notifications functions
- buildCephClusterModel: 0% → 100% (11 test cases)
- collectContainerRootUsage: 0% → 100% (18 test cases)
- NotificationManager getters/setters: 8 functions now tested

Overall coverage: 45.5% → 45.8%
2025-12-01 17:33:36 +00:00
rcourtman
a9fe80e1cd test: Add comprehensive tests for EmailTemplate function
Cover both branches: single alert template (isSingle=true with 1 alert)
and grouped alert template (isSingle=false or multiple alerts). Tests
verify subject format, HTML structure, and text body generation.
Coverage improved from 66.7% to 100%.
2025-12-01 15:29:31 +00:00
rcourtman
9461afc0be test: Add renderWebhookURL URL parsing error test
Add test case for malformed IPv6 URL that triggers url.Parse error.
Improves coverage from 90% to 95%. The remaining 5% is a template
Execute error path that's effectively unreachable with current types.
2025-12-01 14:57:25 +00:00
rcourtman
737c02983f test: Improve coverage for security utility functions
- Add fmt.Stringer test cases for isEmptyInterface (80% -> 100%)
- Expand isTrustedNetwork tests with edge cases (80% -> 100%)
  - Empty/invalid IP strings
  - Invalid CIDR handling
  - Whitespace trimming in CIDRs
  - Multiple network matching
2025-12-01 14:33:31 +00:00
rcourtman
03d41fb71b test: Add tests for notification config copy functions
Tests for defensive copy functions that ensure data isolation:
- copyEmailConfig: copies email config with slice independence
- copyWebhookConfigs: deep copies webhooks including maps
- copyAppriseConfig: copies apprise config with slice independence
- buildNotificationTestAlert: validates test alert structure

Tests verify that modifying copies doesn't affect originals,
critical for concurrent access safety.
2025-12-01 13:43:52 +00:00
rcourtman
e6b9bd501d test: Add error path tests for renderWebhookURL and UpdateAllowedPrivateCIDRs
Add comprehensive error handling tests for two pure functions:

renderWebhookURL (8 new test cases):
- Empty/whitespace URL template validation
- Invalid template syntax (unclosed braces, undefined functions)
- Template producing empty URL
- Missing scheme or host in rendered URL

UpdateAllowedPrivateCIDRs (expanded from 8 to 29 cases):
- Invalid IP addresses (garbage, out of range, malformed)
- Invalid CIDR notation (prefix too large, negative, non-numeric)
- Malformed strings (double slash, invalid IP with valid prefix)
- Success cases for valid IPv4/IPv6 CIDRs and bare IPs
2025-12-01 12:58:15 +00:00
rcourtman
827a8b4d37 test: Add unit tests for email_enhanced.go error handling
Add comprehensive tests for EnhancedEmailManager covering:
- Rate limiting (exceeds limit, resets after minute, concurrency, negative values)
- Provider username defaults (SendGrid, SparkPost, Resend, Postmark)
- TLS routing (TLS, StartTLS, plain connections)
- Retry logic (retries on failure, rate limit prevents retry)
- Connection error handling for sendTLS, sendStartTLS, sendPlain

This is the first test file for email_enhanced.go which previously had
0% coverage on sendTLS, sendStartTLS, and TestConnection functions.
2025-12-01 11:46:44 +00:00
rcourtman
0d786d42c9 Add unit tests for notification utility functions
Tests for annotateResolvedMetadata, resolveAppriseNotificationType,
normalizeQueueType, and resolvedTimeFromAlerts (44 test cases).
Coverage 38.0% → 39.0%.
2025-12-01 02:04:27 +00:00
rcourtman
a7b1cf2ae9 Add unit tests for isRetryableWebhookError (internal/notifications)
47 test cases covering webhook retry logic classification:
- Network errors: timeout, connection refused/reset, DNS lookup, network unreachable
- HTTP 5xx server errors: all status codes 500-599 (retryable)
- HTTP 4xx client errors: all status codes 400-499 except 429 (not retryable)
- HTTP 429 rate limiting: special case (retryable)
- Case insensitivity verification
- Real-world error message patterns from Go net/http
- All status code boundary tests

Coverage 37.4% -> 38.0%.
2025-11-30 22:03:26 +00:00
rcourtman
f9a64b3251 Add unit tests for notification security utilities (notifications)
Test 7 security-related functions with 118 test cases:
- formatWebhookDuration: webhook payload duration formatting
- isNumeric: Telegram chat_id validation helper
- extractTelegramChatID: Telegram webhook URL parsing
- isPrivateIP: SSRF protection (private IP detection)
- isNumericIP: HTTPS IP address detection
- isEmptyInterface: custom field validation
- ensurePushoverCustomFieldAliases: Pushover legacy field aliasing
2025-11-30 08:05:04 +00:00
rcourtman
132dae92a9 Add unit tests for notification template functions (notifications)
Test GetEmailProviders and GetWebhookTemplates functions covering:
- Provider/template uniqueness, required fields, valid values
- Known provider settings (Gmail, SendGrid, custom)
- Known webhook services (Discord, Slack, Telegram, PagerDuty)
- Payload template validation for required alert fields
2025-11-30 07:48:53 +00:00
rcourtman
d2a8184833 Add unit tests for notification queue utility functions
Test coverage for calculateBackoff (exponential backoff calculation),
NotificationQueueStatus constants, and QueuedNotification struct fields.
15 test cases covering backoff timing, cap behavior, and struct defaults.
2025-11-30 03:19:23 +00:00
rcourtman
ed3773b324 Add unit tests for email template utility functions
Test coverage for titleCase, formatDuration, pluralize, formatMetricValue,
and formatMetricThreshold functions. 71 test cases covering edge cases,
boundary conditions, and all metric type variants.
2025-11-30 01:03:05 +00:00
rcourtman
95a258ac32 refactor: replace deprecated strings.Title with custom titleCase
Add simple titleCase functions that handle ASCII strings without
the Unicode punctuation issues of strings.Title.
2025-11-27 09:22:52 +00:00
rcourtman
7dad9c7a17 style: fix additional staticcheck warnings
- Lowercase error messages (ST1005)
- Use context.Background() instead of nil (SA1012)
- Fix rand.Intn(1) which always returns 0 (SA4030)
- Remove unnecessary nil check before len() (S1009)
2025-11-27 09:21:11 +00:00
rcourtman
8a1ee3b05e style: fix staticcheck style warnings
- Merge variable declaration with assignment (S1021)
- Use unconditional strings.TrimPrefix (S1017)
- Remove unnecessary nil checks around range (S1031)
- Remove unnecessary fmt.Sprintf (S1039)
- Use copy() instead of manual loop (S1001)
- Use time.Until instead of t.Sub(time.Now()) (S1024)
- Use buf.String() instead of string(buf.Bytes()) (S1030)
2025-11-27 09:19:33 +00:00
rcourtman
25542ae51d chore: remove more dead code
Remove 330 lines of unreachable code:
- internal/monitoring/temperature_service.go: unused temperature service abstraction
- internal/monitoring/temperature.go: unused NewTemperatureCollector wrapper
- internal/mock/generator.go: unused GenerateAlerts function
- internal/mock/integration.go: unused ToggleMockMode wrapper
- internal/notifications/notifications.go: unused sendEmailWithContent,
  generatePayloadFromTemplate, isPrivateRange172, groupAlerts
- internal/notifications/email_providers.go: unused GetProviderDefaults
2025-11-27 00:10:55 +00:00
rcourtman
2fe7bb6141 style: fix gofmt formatting inconsistencies
Run gofmt -w to fix tab/space inconsistencies across 33 files.
2025-11-26 23:44:36 +00:00
rcourtman
4e68675e0b Add recovery notifications and grouping controls 2025-11-21 22:07:00 +00:00
rcourtman
d779328878 Add Apprise test support for notifications
Related to #584
2025-11-20 17:54:20 +00:00
rcourtman
3b079eeddb Add release dry run workflow and API update integration test 2025-11-12 21:02:52 +00:00
rcourtman
d9e0505394 Fix notification manager deadlock in Stop()
Critical deadlock fix:
- Stop() was holding n.mu lock while calling queue.Stop()
- queue.Stop() waits for worker goroutines to finish
- Worker goroutines call ProcessQueuedNotification() which needs n.mu lock
- This created a classic lock-order deadlock

Fix:
- Unlock n.mu before calling queue.Stop()
- Relock after queue shutdown completes
- Workers can now finish and acquire lock as needed

This resolves 30-second test timeouts in notifications package.

Tests now complete in <1s instead of timing out at 30s.
2025-11-11 23:58:18 +00:00
rcourtman
b41f8a2ac4 Fix backend test failures blocking release workflow
Three categories of fixes:

1. Goroutine leak causing 10-minute timeout:
   - Add defer mon.notificationMgr.Stop() in monitor_memory_test.go
   - Background goroutines from notification manager weren't being stopped

2. Database NULL column scanning errors:
   - Change LastError from string to *string in queue.go
   - Change PayloadBytes from int to *int in queue.go
   - SQL NULL values require pointer types in Go

3. SSRF protection blocking test servers:
   - Check allowlist for localhost before rejecting in notifications.go
   - Set PULSE_DATA_DIR to temp directory in tests
   - Add defer nm.Stop() calls to prevent goroutine leaks

Fixes for preflight test failures in workflow run 19280879903.
2025-11-11 23:27:03 +00:00
rcourtman
515987cc8b Fix failing backend tests in preflight checks
Fixes three test failures that were blocking release workflow:

1. TestApplyDockerReportGeneratesUniqueIDsForCollidingHosts:
   - Initialize dockerTokenBindings and dockerMetadataStore in test helper
   - These maps were nil causing panic on first access

2. TestSendGroupedAppriseHTTP & TestSendTestNotificationAppriseHTTP:
   - Configure allowlist to permit localhost (127.0.0.1) for test servers
   - SSRF protection was blocking httptest.NewServer() URLs
   - Tests need to allowlist the test server IP to bypass security checks

Related to workflow fix in 5fa78c3e3.
2025-11-11 23:02:45 +00:00
rcourtman
a82a345cd6 Improve table column widths and sparkline visibility 2025-11-09 23:36:52 +00:00
rcourtman
1b221cca71 feat: Add configurable allowlist for webhook private IP targets (addresses #673)
Allow homelab users to send webhooks to internal services while maintaining security defaults.

Changes:
- Add webhookAllowedPrivateCIDRs field to SystemSettings (persistent config)
- Implement CIDR parsing and validation in NotificationManager
- Convert ValidateWebhookURL to instance method to access allowlist
- Add UI controls in System Settings for configuring trusted CIDR ranges
- Maintain strict security by default (block all private IPs)
- Keep localhost, link-local, and cloud metadata services blocked regardless of allowlist
- Re-validate on both config save and webhook delivery (DNS rebinding protection)
- Add comprehensive tests for CIDR parsing and IP matching

Backend:
- UpdateAllowedPrivateCIDRs() parses comma-separated CIDRs with validation
- Support for bare IPs (auto-converts to /32 or /128)
- Thread-safe allowlist updates with RWMutex
- Logging when allowlist is updated or used
- Validation errors prevent invalid CIDRs from being saved

Frontend:
- New "Webhook Security" section in System Settings
- Input field with examples and helpful placeholder text
- Real-time unsaved changes tracking
- Loads and saves allowlist via system settings API

Security:
- Default behavior unchanged (all private IPs blocked)
- Explicit opt-in required via configuration
- Localhost (127/8) always blocked
- Link-local (169.254/16) always blocked
- Cloud metadata services always blocked
- DNS resolution checked at both save and send time

Testing:
- Tests for CIDR parsing (valid/invalid inputs)
- Tests for IP allowlist matching
- Tests for bare IP address handling
- Tests for security boundaries (localhost, link-local remain blocked)

Related to #673

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 08:31:12 +00:00
rcourtman
5ec2947d86 Fix Pushover webhook custom field overrides (related to #665)
The Pushover webhook template now honors user-defined custom fields
for sound, priority, and device. Previously, these fields were
hardcoded based on alert level, ignoring any custom values set by
users in the UI.

Changes:
- sound: Uses CustomFields.sound if provided, otherwise falls back to
  level-based default (critical=siren, warning=tugboat, else=pushover)
- priority: Uses CustomFields.priority if provided, otherwise falls back
  to level-based default (critical=1, warning=0, else=-1)
- device: Uses CustomFields.device if provided, otherwise falls back to
  ResourceName

Updated setup instructions to document optional custom fields for sound,
priority, and device configuration.

This allows users to customize Pushover notification behavior without
editing webhook templates, consistent with Pulse's maintainability goals.
2025-11-08 10:32:27 +00:00
rcourtman
99e5a38534 Fix critical monitoring system issues and add robustness improvements
This commit addresses 9 critical issues identified during the monitoring system audit:

**Race Conditions Fixed:**
- PBS backup pollers: Moved lock earlier to eliminate check-then-act race (lines 7316-7378)
- PVE backup poll timing: Fixed double write to lastPVEBackupPoll with proper synchronization (lines 5927-5977)
- Docker hosts cleanup: Refactored to avoid holding both m.mu and s.mu locks simultaneously (lines 1911-1937)

**Context Propagation Fixed:**
- Replaced all context.Background() calls with parent context for proper cancellation chain:
  - PBS backup poller (line 7367)
  - PVE backup poller (line 5955)
  - PBS fallback check (line 7154)

**Memory Leak Prevention:**
- Added cleanup for guest metadata cache (10 minute TTL, lines 1942-1957)
- Added cleanup for diagnostic snapshots (1 hour TTL, lines 1959-1987)
- Added cleanup for RRD cache (1 minute TTL, lines 1989-2007)
- All cleanup methods called on 10-second ticker (lines 3791-3793)

**Panic Recovery:**
- Added recoverFromPanic helper to log panics with stack traces (lines 1910-1920)
- Protected all critical goroutines:
  - poll (line 4020)
  - taskWorker (line 4200)
  - retryFailedConnections (line 3851)
  - checkMockAlerts (line 8896)
  - pollPVEInstance (line 4886)
  - pollPBSInstance (line 7164)
  - pollPMGInstance (line 7498)

**Import Fixes:**
- Added missing sync import to email_enhanced.go
- Added missing os import to queue.go

All fixes maintain proper lock ordering and release locks before calling methods that acquire other locks to prevent deadlocks.
2025-11-07 08:52:37 +00:00
rcourtman
b70dc3d00d Document layered retry semantics (P2 documentation)
Add documentation to explain how transport-level and queue-level retries interact:
- Email: MaxRetries (transport) * MaxAttempts (queue) = total SMTP attempts
- Webhooks: RetryCount (transport) * MaxAttempts (queue) = total HTTP attempts
- Example: 3 * 3 = 9 total delivery attempts for a single notification

This clarifies the multiplicative retry behavior and helps operators understand
the actual retry counts when using the persistent queue.
2025-11-07 08:35:00 +00:00
rcourtman
7ee11105f5 Implement queue cancellation and atomic DB operations (P1 fixes)
Queue cancellation mechanism:
- Add CancelByAlertIDs method to mark queued notifications as cancelled when alerts resolve
- Update CancelAlert to cancel queued notifications containing resolved alert IDs
- Skip cancelled notifications in queue processor
- Prevents resolved alerts from triggering notifications after they clear

Atomic DB operations:
- Add IncrementAttemptAndSetStatus to atomically update attempt counter and status
- Replace separate IncrementAttempt + UpdateStatus calls with single atomic operation
- Prevents orphaned queue entries when crashes occur between operations
- Eliminates race condition where rows get stuck in "pending" or "sending" status

These fixes ensure queued notifications are properly cancelled when alerts resolve
and prevent database inconsistencies during crash scenarios.
2025-11-07 08:33:09 +00:00
rcourtman
c6a69e525c Fix critical notification system bugs and security issues
Critical fixes (P0):
- Fix cooldown timing: Mark cooldown only after successful delivery, not before enqueue
- Add os.MkdirAll to queue initialization to prevent silent failures on fresh installs
- Add DNS re-validation at webhook send time to prevent DNS rebinding SSRF attacks
- Add SSRF validation for Apprise HTTP URLs
- Remove secret logging (bot tokens, routing keys) from debug logs
- Implement lastNotified cleanup to prevent unbounded memory growth
- Use shared HTTP client for webhooks to enable TLS connection reuse
- Add fallback to direct sending when queue enqueue fails
- Make queue worker concurrent (5 workers with semaphore) to prevent head-of-line blocking
- Fix webhook rate limiter race condition with separate mutex
- Fix email manager thread safety with mutex on rate limiter
- Fix grouping timer leak by adding stopCleanup signal
- Fix webhook 429 double sleep (use Retry-After OR backoff, not both)

Frontend improvements:
- Add queue/DLQ management API methods (getQueueStats, getDLQ, retryDLQItem, deleteDLQItem)
- Add getNotificationHealth and getWebhookHistory endpoints
- Add Apprise test support to NotificationTestRequest type

Related to notification system audit
2025-11-07 08:29:13 +00:00
rcourtman
febce91145 Remove internal development documentation files
Remove 4 LLM-generated internal development docs that don't belong in the repository:
- MIGRATION_SCAFFOLDING.md
- NOTIFICATION_AUDIT.md
- NOTIFICATION_QUICK_REFERENCE.md
- NOTIFICATION_SYSTEM_MAP.md

These were internal development notes, not user-facing documentation.
2025-11-07 08:23:19 +00:00
rcourtman
6a48c759e8 Fix critical notification system bugs and security issues
This commit addresses multiple critical issues identified in the notification
system audit conducted with Codex:

**Critical Fixes:**

1. **Queue Retry Logic (Critical #1)**
   - Fixed broken retry/DLQ system where send functions never returned errors
   - Made sendGroupedEmail(), sendGroupedWebhook(), sendGroupedApprise() return errors
   - Made sendWebhookRequest() return errors
   - ProcessQueuedNotification() now properly propagates errors to queue
   - Retry logic and DLQ now function correctly

2. **Attempt Counter Bug (Critical #2)**
   - Fixed double-increment bug in queue processing
   - Separated UpdateStatus() from attempt tracking
   - Added IncrementAttempt() method
   - Notifications now get correct number of retry attempts

3. **Secret Exposure (Critical #3 & #4)**
   - Masked webhook headers and customFields in GET /api/notifications/webhooks
   - Added redactSecretsFromURL() to sanitize webhook URLs in history
   - Truncated/redacted response bodies in webhook history
   - Protected against credential harvesting via API

4. **Email Rate Limiting (Critical #5)**
   - Added emailManager field to NotificationManager
   - Shared EnhancedEmailManager instance across sends
   - Rate limiter now accumulates across multiple emails
   - SMTP rate limits are now enforced correctly

5. **SSRF Protection (High #6)**
   - Added DNS resolution of webhook URLs
   - Added isPrivateIP() check using CIDR ranges
   - Blocks all private IP ranges (10/8, 172.16/12, 192.168/16, 127/8, 169.254/16)
   - Blocks IPv6 private ranges (::1, fe80::/10, fc00::/7)
   - Prevents DNS rebinding attacks
   - Returns error instead of warning for private IPs

**New Features:**

6. **Health Endpoint (High #8)**
   - Added GET /api/notifications/health
   - Returns queue stats (pending, sending, sent, failed, dlq)
   - Shows email/webhook configuration status
   - Provides overall health indicator

**Related to notification system audit**

Files changed:
- internal/notifications/notifications.go: Error returns, rate limiting, SSRF hardening
- internal/notifications/queue.go: Attempt tracking fix
- internal/api/notifications.go: Secret masking, health endpoint
2025-11-06 23:26:03 +00:00
rcourtman
c8e0281953 Add comprehensive alert system reliability improvements
This commit implements critical reliability features to prevent data loss
and improve alert system robustness:

**Persistent Notification Queue:**
- SQLite-backed queue with WAL journaling for crash recovery
- Dead Letter Queue (DLQ) for notifications that exhaust retries
- Exponential backoff retry logic (100ms → 200ms → 400ms)
- Full audit trail for all notification delivery attempts
- New file: internal/notifications/queue.go (661 lines)

**DLQ Management API:**
- GET /api/notifications/dlq - Retrieve DLQ items
- GET /api/notifications/queue/stats - Queue statistics
- POST /api/notifications/dlq/retry - Retry failed notifications
- POST /api/notifications/dlq/delete - Delete DLQ items
- New file: internal/api/notification_queue.go (145 lines)

**Prometheus Metrics:**
- 18 comprehensive metrics for alerts and notifications
- Metric hooks integrated via function pointers to avoid import cycles
- /metrics endpoint exposed for Prometheus scraping
- New file: internal/metrics/alert_metrics.go (193 lines)

**Alert History Reliability:**
- Exponential backoff retry for history saves (3 attempts)
- Automatic backup restoration on write failure
- Modified: internal/alerts/history.go

**Flapping Detection:**
- Detects and suppresses rapidly oscillating alerts
- Configurable window (default: 5 minutes)
- Configurable threshold (default: 5 state changes)
- Configurable cooldown (default: 15 minutes)
- Automatic cleanup of inactive flapping history

**Alert TTL & Auto-Cleanup:**
- MaxAlertAgeDays: Auto-cleanup old alerts (default: 7 days)
- MaxAcknowledgedAgeDays: Faster cleanup for acked alerts (default: 1 day)
- AutoAcknowledgeAfterHours: Auto-ack long-running alerts (default: 24 hours)
- Prevents memory leaks from long-running alerts

**WebSocket Broadcast Sequencer:**
- Channel-based sequencing ensures ordered message delivery
- 100ms coalescing window for rapid state updates
- Prevents race conditions in WebSocket broadcasts
- Modified: internal/websocket/hub.go

**Configuration Fields Added:**
- FlappingEnabled, FlappingWindowSeconds, FlappingThreshold, FlappingCooldownMinutes
- MaxAlertAgeDays, MaxAcknowledgedAgeDays, AutoAcknowledgeAfterHours

All features are production-ready and build successfully.
2025-11-06 16:46:30 +00:00
rcourtman
20099549c6 Add comprehensive release validation to prevent missing artifacts
Adds automated validation script to prevent the pattern of patch
releases caused by missing files/artifacts.

scripts/validate-release.sh validates all 40+ artifacts including:
- Docker image scripts (8 install/uninstall scripts)
- Docker image binaries (17 across all platforms)
- Release tarballs (5 including universal and macOS)
- Standalone binaries (12+)
- Checksums for all distributable assets
- Version embedding in every binary type
- Tarball contents (binaries + scripts + VERSION)
- Binary architectures and file types

The script catches 100% of issues from the last 3 patch releases
(missing scripts, missing install.sh, missing binaries, broken
version embedding).

Updated RELEASE_CHECKLIST.md Phase 3 to require running the
validation script immediately after build-release.sh and before
proceeding to Docker build/publish phases.

Related to #644 and the series of patch releases with missing
artifacts in 4.26.x.
2025-11-06 16:33:49 +00:00
rcourtman
ddc787418b Round float values in webhook payloads to 1 decimal place
Webhook alert payloads now round Value and Threshold fields to 1 decimal
place before template rendering. This eliminates excessive precision in
webhook messages (e.g., 62.27451680630036 becomes 62.3).

The fix is applied in prepareWebhookData() so all webhook templates
benefit automatically, including Google Space webhooks, generic JSON
webhooks, and custom templates.

Related to #619
2025-11-05 19:19:10 +00:00
rcourtman
02864f54dd Add test notification functionality for Apprise
- Add support for testing Apprise notifications via /api/notifications/test endpoint
- Users can now test their Apprise configuration (both CLI and HTTP modes) using method="apprise"
- Added comprehensive unit tests for both CLI and HTTP modes
- Tests verify correct behavior when Apprise is enabled/disabled
- Tests validate that notifications are properly sent through Apprise channels

Related to #584
2025-11-05 18:54:18 +00:00