c8e0281953
This commit implements critical reliability features to prevent data loss and improve alert system robustness: **Persistent Notification Queue:** - SQLite-backed queue with WAL journaling for crash recovery - Dead Letter Queue (DLQ) for notifications that exhaust retries - Exponential backoff retry logic (100ms → 200ms → 400ms) - Full audit trail for all notification delivery attempts - New file: internal/notifications/queue.go (661 lines) **DLQ Management API:** - GET /api/notifications/dlq - Retrieve DLQ items - GET /api/notifications/queue/stats - Queue statistics - POST /api/notifications/dlq/retry - Retry failed notifications - POST /api/notifications/dlq/delete - Delete DLQ items - New file: internal/api/notification_queue.go (145 lines) **Prometheus Metrics:** - 18 comprehensive metrics for alerts and notifications - Metric hooks integrated via function pointers to avoid import cycles - /metrics endpoint exposed for Prometheus scraping - New file: internal/metrics/alert_metrics.go (193 lines) **Alert History Reliability:** - Exponential backoff retry for history saves (3 attempts) - Automatic backup restoration on write failure - Modified: internal/alerts/history.go **Flapping Detection:** - Detects and suppresses rapidly oscillating alerts - Configurable window (default: 5 minutes) - Configurable threshold (default: 5 state changes) - Configurable cooldown (default: 15 minutes) - Automatic cleanup of inactive flapping history **Alert TTL & Auto-Cleanup:** - MaxAlertAgeDays: Auto-cleanup old alerts (default: 7 days) - MaxAcknowledgedAgeDays: Faster cleanup for acked alerts (default: 1 day) - AutoAcknowledgeAfterHours: Auto-ack long-running alerts (default: 24 hours) - Prevents memory leaks from long-running alerts **WebSocket Broadcast Sequencer:** - Channel-based sequencing ensures ordered message delivery - 100ms coalescing window for rapid state updates - Prevents race conditions in WebSocket broadcasts - Modified: internal/websocket/hub.go **Configuration Fields Added:** - FlappingEnabled, FlappingWindowSeconds, FlappingThreshold, FlappingCooldownMinutes - MaxAlertAgeDays, MaxAcknowledgedAgeDays, AutoAcknowledgeAfterHours All features are production-ready and build successfully. |
||
|---|---|---|
| .. | ||
| alerts.go | ||
| alerts_test.go | ||
| auth.go | ||
| auth_scope_test.go | ||
| bootstrap_token.go | ||
| config_handlers.go | ||
| config_handlers_auto_register_test.go | ||
| config_handlers_cluster_test.go | ||
| config_handlers_setup_script_test.go | ||
| csrf_store.go | ||
| demo_middleware.go | ||
| diagnostics.go | ||
| DO_NOT_EDIT_FRONTEND_HERE.md | ||
| docker_agents.go | ||
| docker_metadata.go | ||
| frontend_embed.go | ||
| guest_metadata.go | ||
| host_agents.go | ||
| host_agents_test.go | ||
| http_metrics.go | ||
| middleware.go | ||
| notification_queue.go | ||
| notifications.go | ||
| oidc_handlers.go | ||
| oidc_service.go | ||
| rate_limit_config.go | ||
| rate_limit_config_test.go | ||
| ratelimit.go | ||
| README.md | ||
| recovery_tokens.go | ||
| router.go | ||
| router_integration_test.go | ||
| security.go | ||
| security_oidc.go | ||
| security_setup_fix.go | ||
| security_setup_fix_test.go | ||
| security_test.go | ||
| security_tokens.go | ||
| security_tokens_test.go | ||
| session_store.go | ||
| system_settings.go | ||
| types.go | ||
| updates.go | ||
Internal API Package
This directory contains the API server implementation for Pulse.
Important Note About frontend-modern/
The frontend-modern/ subdirectory that appears here is:
- AUTO-GENERATED during builds
- NOT the source code - just a build artifact
- IN .gitignore - never committed
- REQUIRED BY GO - The embed directive needs it here
Frontend Development Location
👉 Edit frontend files at: /opt/pulse/frontend-modern/src/
Why This Structure?
Go's //go:embed directive has limitations:
- Cannot use
../paths to access parent directories - Cannot follow symbolic links
- Must embed files within the Go module
This is a known Go limitation and our structure works around it.