danvics/Pulse
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Pulse/internal
History
rcourtman c8e0281953 Add comprehensive alert system reliability improvements 
This commit implements critical reliability features to prevent data loss
and improve alert system robustness:

**Persistent Notification Queue:**
- SQLite-backed queue with WAL journaling for crash recovery
- Dead Letter Queue (DLQ) for notifications that exhaust retries
- Exponential backoff retry logic (100ms → 200ms → 400ms)
- Full audit trail for all notification delivery attempts
- New file: internal/notifications/queue.go (661 lines)

**DLQ Management API:**
- GET /api/notifications/dlq - Retrieve DLQ items
- GET /api/notifications/queue/stats - Queue statistics
- POST /api/notifications/dlq/retry - Retry failed notifications
- POST /api/notifications/dlq/delete - Delete DLQ items
- New file: internal/api/notification_queue.go (145 lines)

**Prometheus Metrics:**
- 18 comprehensive metrics for alerts and notifications
- Metric hooks integrated via function pointers to avoid import cycles
- /metrics endpoint exposed for Prometheus scraping
- New file: internal/metrics/alert_metrics.go (193 lines)

**Alert History Reliability:**
- Exponential backoff retry for history saves (3 attempts)
- Automatic backup restoration on write failure
- Modified: internal/alerts/history.go

**Flapping Detection:**
- Detects and suppresses rapidly oscillating alerts
- Configurable window (default: 5 minutes)
- Configurable threshold (default: 5 state changes)
- Configurable cooldown (default: 15 minutes)
- Automatic cleanup of inactive flapping history

**Alert TTL & Auto-Cleanup:**
- MaxAlertAgeDays: Auto-cleanup old alerts (default: 7 days)
- MaxAcknowledgedAgeDays: Faster cleanup for acked alerts (default: 1 day)
- AutoAcknowledgeAfterHours: Auto-ack long-running alerts (default: 24 hours)
- Prevents memory leaks from long-running alerts

**WebSocket Broadcast Sequencer:**
- Channel-based sequencing ensures ordered message delivery
- 100ms coalescing window for rapid state updates
- Prevents race conditions in WebSocket broadcasts
- Modified: internal/websocket/hub.go

**Configuration Fields Added:**
- FlappingEnabled, FlappingWindowSeconds, FlappingThreshold, FlappingCooldownMinutes
- MaxAlertAgeDays, MaxAcknowledgedAgeDays, AutoAcknowledgeAfterHours

All features are production-ready and build successfully.
2025-11-06 16:46:30 +00:00
..
alerts Add comprehensive alert system reliability improvements 2025-11-06 16:46:30 +00:00
api Add comprehensive alert system reliability improvements 2025-11-06 16:46:30 +00:00
auth Refactor: Code cleanup and localStorage consolidation 2025-11-04 21:50:46 +00:00
config Fix VM migration issue where custom alert thresholds are lost 2025-11-06 10:27:15 +00:00
crypto Harden setup token flow and enforce encrypted persistence 2025-10-25 16:00:37 +00:00
discovery Refactor: Code cleanup and localStorage consolidation 2025-11-04 21:50:46 +00:00
dockeragent Fix critical version embedding issues for 4.26 release 2025-11-06 11:42:52 +00:00
errors Fix settings security tab navigation 2025-10-11 23:29:47 +00:00
hostagent Normalize docker agent version handling 2025-10-28 08:42:58 +00:00
hostmetrics Normalize docker agent version handling 2025-10-28 08:42:58 +00:00
logging feat: comprehensive diagnostics and observability improvements 2025-10-21 12:37:39 +00:00
metrics Add comprehensive alert system reliability improvements 2025-11-06 16:46:30 +00:00
mock Refactor: Code cleanup and localStorage consolidation 2025-11-04 21:50:46 +00:00
models Add AMD GPU temperature monitoring support 2025-11-06 00:19:04 +00:00
monitoring Fix VM migration issue where custom alert thresholds are lost 2025-11-06 10:27:15 +00:00
notifications Add comprehensive alert system reliability improvements 2025-11-06 16:46:30 +00:00
ssh/knownhosts Add configurable SSH port for temperature monitoring 2025-11-05 20:03:29 +00:00
system Enhance container detection for temperature SSH safeguards (refs #601) 2025-11-04 22:30:35 +00:00
tempproxy Fix temperature data intermittency caused by proxy rate limit retries 2025-11-05 10:20:15 +00:00
types Fix settings security tab navigation 2025-10-11 23:29:47 +00:00
updates Fix critical rollback download URL bug and doc inconsistencies 2025-11-06 14:25:32 +00:00
utils Refactor: Code cleanup and localStorage consolidation 2025-11-04 21:50:46 +00:00
websocket Add comprehensive alert system reliability improvements 2025-11-06 16:46:30 +00:00