Pulse/NOTIFICATION_SYSTEM_MAP.md

Pulse Notification System Architecture - Complete Map

Overview

Pulse has a sophisticated multi-channel notification system supporting Email, Webhooks, and Apprise. It features persistent queuing, retry logic, rate limiting, and security controls.

---

BACKEND IMPLEMENTATION

1. Core Notification Manager

File: /opt/pulse/internal/notifications/notifications.go (2,358 lines)

Key Structures:

• NotificationManager (lines 107-126)

  • Orchestrates email, webhook, and Apprise notifications
  • Manages alert grouping with time-based windows
  • Implements cooldown and rate limiting
  • Maintains webhook delivery history
  • Integrates with persistent notification queue

• EmailConfig (lines 266-278)

  type EmailConfig struct {
    Enabled  bool
    Provider string  // Gmail, SendGrid, etc.
    SMTPHost string  // "server" in JSON
    SMTPPort int     // "port" in JSON
    Username string
    Password string
    From     string
    To       []string
    TLS      bool
    StartTLS bool   // STARTTLS support
  }
  

• WebhookConfig (lines 280-291)

  type WebhookConfig struct {
    ID           string
    Name         string
    URL          string
    Method       string
    Headers      map[string]string
    Enabled      bool
    Service      string  // discord, slack, teams, etc.
    Template     string  // Custom payload template
    CustomFields map[string]string
  }
  

• AppriseConfig (lines 301-313)

  type AppriseConfig struct {
    Enabled        bool
    Mode           AppriseMode  // "cli" or "http"
    Targets        []string
    CLIPath        string
    TimeoutSeconds int
    ServerURL      string
    ConfigKey      string
    APIKey         string
    APIKeyHeader   string
    SkipTLSVerify  bool
  }
  

Key Methods:

Initialization & Configuration:

• NewNotificationManager(publicURL string) (lines 315-361)

  • Creates persistent queue if available
  • Initializes alert grouping system
  • Wires up queue processor

• SetEmailConfig(config) (lines 388-406)

  • Creates new email manager with provider config
  • Updates configuration in-memory

• SetAppriseConfig(config) (lines 408-413)

  • Normalizes Apprise configuration
  • Validates mode and timeout settings

• GetEmailConfig() / GetWebhooks() / GetAppriseConfig() (lines 502-420)

  • Safe accessor methods with locking

Webhook Management:

• AddWebhook(webhook) (lines 453-458)
• UpdateWebhook(id, webhook) (lines 460-472)
• DeleteWebhook(id) (lines 474-486)
• GetWebhooks() (lines 488-500)

Alert Sending:

• SendAlert(alert) (lines 516-569)

  • Implements cooldown check
  • Adds alert to pending alerts queue
  • Starts alert grouping timer
  • Checks for rate limits

• CancelAlert(alertID) (lines 571-614)

  • Removes resolved alert from pending list
  • Stops grouping timer if no more pending alerts

• sendGroupedAlerts() (lines 616-657)

  • Sends all pending alerts as single batch
  • Updates last notified timestamps
  • Enqueues to persistent queue OR sends directly

Email Sending:

• sendGroupedEmail(config, alertList) (lines 753-764)

  • Uses email template
  • Calls enhanced email manager with retry

• sendHTMLEmailWithError(subject, body, config) (lines 1010-1076)

  • Handles HTML + text multipart emails
  • Returns error for caller
  • Uses shared email manager for rate limiting

• sendHTMLEmail(subject, body, config) (lines 1078-1133)

  • Creates EnhancedEmailManager instance
  • Sends with retries (2 attempts, 3 second delay)

Webhook Sending:

• sendGroupedWebhook(webhook, alertList) (lines 1141-1354)

  • Handles single/grouped alert scenarios
  • Applies service-specific templates (Discord, Slack, Teams, etc.)
  • Supports custom templates
  • Service-specific data enrichment (Telegram chat_id, PagerDuty routing key)

• sendWebhookRequest(webhook, jsonData, alertType) (lines 1396-1533)

  • Performs rate limit check
  • Creates HTTP request with headers
  • Sends using secure webhook client
  • Tracks delivery history

• prepareWebhookData(alert, customFields) (lines 1708-1771)

  • Builds WebhookPayloadData struct
  • Formats metrics and duration
  • Includes metadata and custom fields

Apprise Integration:

• sendGroupedApprise(config, alertList) (lines 766-803)

  • Routes to CLI or HTTP mode
  • Validates configuration before sending

• sendAppriseViaCLI(config, title, body) (lines 866-902)

  • Executes apprise CLI with timeout
  • Passes targets as arguments

• sendAppriseViaHTTP(config, title, body, type) (lines 904-992)

  • Makes HTTP POST to Apprise server
  • Supports custom endpoints with configKey
  • Handles API key header
  • Optional TLS verification skip

Webhook Validation & Security:

• ValidateWebhookURL(webhookURL) (lines 1921-1988)

  • Prevents SSRF attacks
  • Blocks localhost/loopback (127.0.0.1, ::1)
  • Blocks link-local addresses (169.254.*, fe80::)
  • Blocks private IP ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)
  • Blocks cloud metadata services (169.254.169.254, metadata.google.internal)
  • DNS resolution required for security
  • Warns on numeric IPs with HTTPS

• isPrivateIP(ip) (lines 1990-2014)

  • Checks IPv4 and IPv6 private ranges

Webhook URL Templating:

• renderWebhookURL(urlTemplate, data) (lines 1829-1865)
  • Parses Go template syntax
  • Validates rendered URL is valid
  • Supports {{.}} field references

Template Support:

• generatePayloadFromTemplate(templateStr, data) (lines 1793-1796)
• generatePayloadFromTemplateWithService(templateStr, data, service) (lines 1798-1827)
  • Parses and executes Go text templates
  • Custom function map: title, upper, lower, printf, urlquery, urlencode, urlpath, pathescape
  • JSON validation (except ntfy service)

Telegram-specific:

• extractTelegramChatID(webhookURL) (lines 1882-1909)
  • Extracts and validates chat_id parameter
  • Handles negative IDs (group chats)
  • Validates numeric format

Test Notifications:

• SendTestNotification(method) (lines 2117-2195)

  • Creates test alert with predefined values
  • Routes to email/webhook/apprise

• SendTestWebhook(webhook) (lines 2197-2229)

  • Tests specific webhook with sample alert

• SendTestNotificationWithConfig(method, config, nodeInfo) (lines 2231-2300)

  • Tests email with provided config (without saving)
  • Uses actual node info from monitor state

Queue Integration:

• enqueueNotifications(emailConfig, webhooks, appriseConfig, alerts) (lines 659-722)

  • Marshals configs to JSON
  • Creates QueuedNotification structs
  • Adds to persistent queue

• ProcessQueuedNotification(notif) (lines 2302-2335)

  • Unmarshals config based on type
  • Routes to appropriate sender
  • Called by queue processor

Utility Functions:

• checkWebhookRateLimit(webhookURL) (lines 1356-1394)

  • Per-webhook rate limiting (10 per minute)
  • Tracks sent count per window

• formatWebhookDuration(duration) (lines 1867-1880)

  • Human-readable duration: "5s", "2m", "1h 30m", "2d 3h"

• NormalizeAppriseConfig(config) (lines 176-240)

  • Cleans whitespace
  • Validates mode and server URL
  • Removes duplicate targets
  • Enforces timeout bounds (5-120 seconds)

---

2. Enhanced Webhook Support

File: /opt/pulse/internal/notifications/webhook_enhanced.go (605 lines)

Key Structures:

• EnhancedWebhookConfig (lines 17-27)

  type EnhancedWebhookConfig struct {
    WebhookConfig
    Service         string
    PayloadTemplate string
    RetryEnabled    bool
    RetryCount      int
    FilterRules     WebhookFilterRules
    CustomFields    map[string]interface{}
    ResponseLogging bool
  }
  

• WebhookFilterRules (lines 29-35)

  • Filter by alert level, type, node, resource type

• WebhookPayloadData (lines 37-66)

  • Complete template data structure
  • Includes alerts array for grouped notifications
  • ChatID for Telegram webhooks

Key Methods:

• SendEnhancedWebhook(webhook, alert) (lines 68-124)

• shouldSendWebhook(webhook, alert) (lines 129-194)

• sendWebhookWithRetry(webhook, payload) (lines 196-326)

  • Exponential backoff: 1s → 2s → 4s → 8s → 16s → 30s (max)
  • Respects Retry-After header for 429 responses
  • Max 3 retries by default
  • Logs attempt counts

• isRetryableWebhookError(err) (lines 328-365)

  • Network errors: timeout, connection refused/reset, no such host
  • Retryable HTTP: 429, 502, 503, 504, 5xx
  • Non-retryable: 4xx client errors

• sendWebhookOnceWithResponse(webhook, payload) (lines 367-428)

  • Returns response for inspection
  • Size-limited response reading

• sendWebhookOnce(webhook, payload) (lines 430-434)

  • Wrapper for compatibility

• TestEnhancedWebhook(webhook) (lines 438-604)

  • Tests webhook with realistic sample alert
  • Handles Telegram chat_id extraction
  • PagerDuty routing key injection
  • ntfy service special header handling
  • Returns (statusCode, responseBody, error)

---

3. Email Implementation

File: /opt/pulse/internal/notifications/email_enhanced.go (415 lines)

Key Structures:

• EnhancedEmailManager (lines 15-19)

  type EnhancedEmailManager struct {
    config    EmailProviderConfig
    rateLimit *RateLimiter
  }
  

• RateLimiter (lines 21-26)

  • Simple per-minute rate limiting

Key Methods:

• NewEnhancedEmailManager(config) (lines 28-36)

• SendEmailWithRetry(subject, htmlBody, textBody) (lines 38-78)

  • Up to 3 retry attempts (configurable)
  • 5 second delay between retries (configurable)
  • Checks rate limit before each attempt

• checkRateLimit() (lines 80-99)

  • Per-minute limiter
  • Default 60 emails/minute

• sendEmailOnce(subject, htmlBody, textBody) (lines 101-138)

  • Builds multipart MIME message
  • Sets standard headers (Date, Message-ID, MIME-Version)
  • Creates text + HTML parts
  • Calls provider-specific send

• sendViaProvider(msg) (lines 140-183)

  • Provider-specific username/password handling
  • SendGrid: "apikey" as username
  • Postmark: API token for both
  • SparkPost: "SMTP_Injection" username
  • Resend: "resend" username

• sendTLS(addr, auth, msg) (lines 185-245)

  • TLS from start (port 465)
  • Uses tls.DialWithDialer with timeout
  • Sets connection deadline (30s)

• sendStartTLS(addr, auth, msg) (lines 247-309)

  • Plain TCP then STARTTLS upgrade
  • 10s dial timeout, 30s connection deadline

• sendPlain(addr, auth, msg) (lines 362-414)

  • Plain SMTP without encryption
  • 10s dial timeout, 30s connection deadline

• TestConnection() (lines 311-360)

  • Tests SMTP connectivity
  • Supports TLS and STARTTLS modes
  • Tests authentication if configured

---

4. Email Configuration & Templates

Files:

• /opt/pulse/internal/notifications/email_providers.go (217 lines)
• /opt/pulse/internal/notifications/email_template.go (200+ lines)

Email Providers (email_providers.go):

• GetEmailProviders() returns templates for:

  1. Gmail / Google Workspace (STARTTLS, port 587)
  2. SendGrid (STARTTLS, port 587)
  3. Mailgun (STARTTLS, port 587)
  4. Amazon SES (STARTTLS, port 587)
  5. Microsoft 365 / Outlook (STARTTLS, port 587, App Password required)
  6. Brevo / Sendinblue (STARTTLS, port 587)
  7. Postmark (STARTTLS, port 587)
  8. SparkPost (STARTTLS, port 587)
  9. Resend (STARTTLS, port 587)
  10. SMTP2GO (STARTTLS, port 587)
  11. Custom SMTP Server

• Each provider includes:

  • SMTP host and port
  • TLS/STARTTLS settings
  • Authentication requirements
  • Setup instructions with links

Email Templates (email_template.go):

• EmailTemplate(alertList, isSingle) (lines 11-17)

  • Routes to single or grouped template

• singleAlertTemplate(alert) (lines 19-40+)

  • Professional HTML email with:
    • Responsive design
    • Level-specific colors (red for critical, yellow for warning)
    • Alert metrics display
    • Details section with duration, node, type
    • Links to Pulse dashboard
    • Footer with Pulse logo and branding

• groupedAlertTemplate(alertList) (lines 100+)

  • Multiple alert summary
  • All alerts listed with values
  • Single cohesive email

---

5. Webhook Templates

File: /opt/pulse/internal/notifications/webhook_templates.go (300+ lines)

GetWebhookTemplates() provides for:

1. Discord Webhook

   • Uses embed format with color coding
   • Fields: Resource, Node, Type, Value, Threshold, Duration
   • Includes timestamp and footer

2. Telegram Bot

   • Markdown formatted message
   • Inline link to Pulse dashboard
   • Requires chat_id parameter

3. Slack Incoming Webhook

   • Header block with alert level
   • Section with message
   • 6-field layout (Resource, Node, Type, Value, Threshold, Duration)
   • Links to Proxmox and Pulse

4. Microsoft Teams

   • Adaptive Card format
   • Facts section with details
   • Color-coded by alert level

5. PagerDuty

   • Event API format
   • Routing key from headers

6. Gotify

   • Title + message format
   • Priority based on level

7. Pushover

   • Custom field aliases (app_token, user_token)
   • Priority encoding

8. ntfy.sh

   • Plain text content
   • Title and Priority headers
   • Tags header with emoji/type

Each template includes:

• Go template syntax for field substitution
• Service-specific format requirements
• Instructions for setup

---

6. Persistent Notification Queue

File: /opt/pulse/internal/notifications/queue.go (600+ lines)

Key Structures:

• QueuedNotification (lines 29-45)

  type QueuedNotification struct {
    ID          string
    Type        string  // email, webhook, apprise
    Method      string
    Status      NotificationQueueStatus  // pending, sending, sent, failed, dlq, cancelled
    Alerts      []*alerts.Alert
    Config      json.RawMessage
    Attempts    int
    MaxAttempts int
    LastAttempt *time.Time
    LastError   string
    CreatedAt   time.Time
    NextRetryAt *time.Time
    CompletedAt *time.Time
    PayloadBytes int
  }
  

• NotificationQueue (lines 47-58)

  • SQLite-backed persistent queue
  • Background processors:
    • processQueue(): Periodic retry processor (5s interval)
    • cleanupOldEntries(): Cleanup job (1 hour interval)
  • Notification channel for signaling new items

Database Schema (lines 115-150):

• notification_queue table

  • Primary key: id (TEXT)
  • Statuses: pending, sending, sent, failed, dlq, cancelled
  • Indexes: status, next_retry_at (for pending), created_at
  • WAL mode for concurrency
  • NORMAL synchronous for durability

• notification_audit table (lines 139-150)

  • Audit trail for all notifications
  • Alert IDs and count tracking
  • Success/failure metrics

Key Methods:

• NewNotificationQueue(dataDir) (lines 60-113)

  • Opens/creates SQLite database
  • Configures pragmas:
    • WAL mode for better concurrency
    • NORMAL synchronous for balance
    • 5s busy timeout
    • 64MB cache
  • Initializes schema
  • Starts background processors

• Enqueue(notif) (enqueues notifications)

• GetQueueStats() (returns queue statistics)

• GetDLQ(limit) (retrieves dead letter queue)

• RetryDLQItem(id) (retries failed notification)

• DeleteDLQItem(id) (removes from DLQ)

• SetProcessor(func) (sets notification processor)

---

API LAYER

1. Notification Endpoints Handler

File: /opt/pulse/internal/api/notifications.go (753 lines)

NotificationHandlers Structure (lines 17-27):

• Reference to Monitor for accessing notification manager and config persistence

GET Endpoints:

Path	Method	Handler	Auth Scope	Purpose
/api/notifications/email	GET	GetEmailConfig	SettingsRead	Retrieve current email config (password masked)
/api/notifications/webhooks	GET	GetWebhooks	SettingsRead	List all webhooks (sensitive fields masked)
/api/notifications/webhook-templates	GET	GetWebhookTemplates	SettingsRead	Get available webhook templates
/api/notifications/webhook-history	GET	GetWebhookHistory	SettingsRead	Get recent webhook delivery history
/api/notifications/email-providers	GET	GetEmailProviders	SettingsRead	Get email provider templates
/api/notifications/apprise	GET	GetAppriseConfig	SettingsRead	Get Apprise configuration
/api/notifications/health	GET	GetNotificationHealth	SettingsRead	Get queue and channel health stats

PUT/UPDATE Endpoints:

Path	Method	Handler	Auth Scope	Purpose
/api/notifications/email	PUT	UpdateEmailConfig	SettingsWrite	Update email config (preserves password)
/api/notifications/apprise	PUT	UpdateAppriseConfig	SettingsWrite	Update Apprise config
/api/notifications/webhooks/{id}	PUT	UpdateWebhook	SettingsWrite	Update specific webhook

POST Endpoints:

Path	Method	Handler	Auth Scope	Purpose
/api/notifications/webhooks	POST	CreateWebhook	SettingsWrite	Create new webhook
/api/notifications/webhooks/test	POST	TestWebhook	SettingsWrite	Test webhook with sample alert
/api/notifications/test	POST	TestNotification	SettingsWrite	Test email/webhook/apprise

DELETE Endpoints:

Path	Method	Handler	Auth Scope	Purpose
/api/notifications/webhooks/{id}	DELETE	DeleteWebhook	SettingsWrite	Delete webhook

Key Handler Methods:

GetEmailConfig (lines 34-43):

• Returns masked config (password redacted)
• JSON response

UpdateEmailConfig (lines 45-89):

• Reads raw body
• Preserves existing password if new one empty
• Updates in-memory config
• Persists to storage via ConfigPersistence

GetWebhooks (lines 142-186):

• Masks header and customField values
• Shows only keys, not secrets
• Preserves template if present

CreateWebhook (lines 188-233):

• Generates ID if not provided
• Validates URL (SSRF protection)
• Saves all webhooks to persistent storage
• Returns full webhook data

UpdateWebhook (lines 235-289):

• Validates URL before update
• Updates webhook in manager
• Persists all webhooks to storage
• Returns updated webhook

DeleteWebhook (lines 291-317):

• Removes webhook from manager
• Persists remaining webhooks

TestNotification (lines 319-421):

• Supports "email" or "webhook" method
• Accepts optional config for testing without saving
• Gets actual node info from monitor state
• Routes to appropriate test sender

TestWebhook (lines 487-601):

• Decodes webhook configuration
• Applies service-specific templates
• Calls enhanced webhook test
• Returns (statusCode, response body)

GetNotificationHealth (lines 603-648):

• Returns queue statistics:
  • pending, sending, sent, failed, dlq counts
  • Healthy flag
• Email status: enabled, configured
• Webhook status: total, enabled count
• Overall health status

GetWebhookHistory (lines 431-444):

• Returns recent webhook deliveries (last 100)
• Redacts secrets from URLs
• Redacts Telegram bot tokens
• Redacts query parameter secrets (token, apikey, key, secret, password)

redactSecretsFromURL (lines 446-477):

• Telegram: /botXXXX:REDACTED/
• Query params: token=REDACTED, apikey=REDACTED, etc.

Security Features:

• All handlers require authentication (RequireAdmin wrapper)
• Sensitive fields masked/redacted
• SSRF protection on webhook URLs
• Password preservation on updates
• Scope-based access control (SettingsRead/SettingsWrite)

---

2. Notification Queue Endpoints Handler

File: /opt/pulse/internal/api/notification_queue.go (80+ lines)

NotificationQueueHandlers Structure:

• Reference to Monitor

Endpoints:

Path	Method	Handler	Auth Scope
/api/notifications/dlq	GET	GetDLQ	MonitoringRead
/api/notifications/queue/stats	GET	GetQueueStats	MonitoringRead
/api/notifications/dlq/retry	POST	RetryDLQItem	MonitoringRead
/api/notifications/dlq/delete	POST	DeleteDLQItem	MonitoringRead

Key Methods:

• GetDLQ (lines 27-56):

  • Returns dead letter queue items
  • Supports limit parameter (default 100, max 1000)
  • Returns JSON array

• GetQueueStats (lines 58-80):

  • Returns dictionary of queue statistics
  • Pending, sending, sent, failed, dlq counts
  • Healthy flag

---

3. Configuration Persistence

Integration with ConfigPersistence (mentioned in notifications.go):

• SaveEmailConfig(config) - Persists email settings
• SaveWebhooks(webhooks) - Persists all webhook configurations
• SaveAppriseConfig(config) - Persists Apprise settings

These are called after every modification to ensure durability.

---

4. Router Integration

File: /opt/pulse/internal/api/router.go

Notification Handler Registration (lines 147-148, 900-926):

r.notificationHandlers = NewNotificationHandlers(r.monitor)
r.notificationQueueHandlers = NewNotificationQueueHandlers(r.monitor)

// Main notification endpoints
r.mux.HandleFunc("/api/notifications/", RequireAdmin(r.config, r.notificationHandlers.HandleNotifications))

// Queue management endpoints
r.mux.HandleFunc("/api/notifications/dlq", RequireAdmin(r.config, r.notificationQueueHandlers.GetDLQ))
r.mux.HandleFunc("/api/notifications/queue/stats", RequireAdmin(r.config, r.notificationQueueHandlers.GetQueueStats))
r.mux.HandleFunc("/api/notifications/dlq/retry", RequireAdmin(r.config, r.notificationQueueHandlers.RetryDLQItem))
r.mux.HandleFunc("/api/notifications/dlq/delete", RequireAdmin(r.config, r.notificationQueueHandlers.DeleteDLQItem))

Monitor Reference Update (line 1125-1126):

• Notification handlers updated when Monitor reference changes
• Ensures handlers always use latest monitor instance

Public URL Detection (line 1516):

• Public URL detected from inbound requests for webhook payload generation

---

FRONTEND IMPLEMENTATION

1. Notification API Client

File: /opt/pulse/frontend-modern/src/api/notifications.ts (201 lines)

Interface Types:

EmailConfig:

interface EmailConfig {
  enabled: boolean;
  provider: string;
  server: string;
  port: number;
  username: string;
  password?: string;
  from: string;
  to: string[];
  tls: boolean;
  startTLS: boolean;
}

Webhook:

interface Webhook {
  id: string;
  name: string;
  url: string;
  method: string;
  headers: Record<string, string>;
  template?: string;
  enabled: boolean;
  service?: string;
  customFields?: Record<string, string>;
}

AppriseConfig:

interface AppriseConfig {
  enabled: boolean;
  mode?: 'cli' | 'http';
  targets?: string[];
  cliPath?: string;
  timeoutSeconds?: number;
  serverUrl?: string;
  configKey?: string;
  apiKey?: string;
  apiKeyHeader?: string;
  skipTlsVerify?: boolean;
}

API Methods (NotificationsAPI class):

Email Configuration:

• getEmailConfig() → EmailConfig
  • Returns config with password field
• updateEmailConfig(config) → { success: boolean }
  • Sends: server, port, tls, startTLS (not smtpHost/smtpPort)
  • Sends: enabled, provider, username, password, from, to

Webhook Management:

• getWebhooks() → Webhook[]
• createWebhook(webhook) → Webhook
• updateWebhook(id, webhook) → Webhook
• deleteWebhook(id) → { success: boolean }

Apprise Configuration:

• getAppriseConfig() → AppriseConfig
• updateAppriseConfig(config) → AppriseConfig

Templates & Providers:

• getEmailProviders() → EmailProvider[]
• getWebhookTemplates() → WebhookTemplate[]

Testing:

• testNotification(request) → { success: boolean; message?: string }
  • Request: { type: 'email'|'webhook', config?: object, webhookId?: string }
• testWebhook(webhook) → { success: boolean; message?: string }

---

2. Frontend Components

Email Provider Selector

File: /opt/pulse/frontend-modern/src/components/Alerts/EmailProviderSelect.tsx (200+ lines)

Features:

• Loads provider templates dynamically
• Dropdown select with provider options
• Auto-fills SMTP host, port, TLS/STARTTLS based on selection
• Displays setup instructions for selected provider
• Advanced settings toggle (Reply-To, Max Retries, Rate Limit)
• Test connection button

Key States:

• providers - Loaded email provider templates
• showAdvanced - Toggle advanced settings display
• showInstructions - Show provider-specific setup instructions

Key Functions:

• applyProvider(provider) - Updates form with provider defaults
• handleProviderChange(value) - Routes provider selection
• currentProvider() - Gets currently selected provider

Webhook Configuration

File: /opt/pulse/frontend-modern/src/components/Alerts/WebhookConfig.tsx (300+ lines)

Features:

• Create/update/delete webhooks
• Service-specific configuration (Discord, Slack, Teams, Telegram, etc.)
• Custom headers management
• Custom fields management (e.g., Pushover app_token, user_token)
• Template selection from provided templates
• Custom payload template support
• Test webhook functionality
• URL template rendering support

Key States:

• webhooks - List of configured webhooks
• editingId - Currently editing webhook ID
• selectedService - Selected service type
• customHeaders - Header key-value inputs
• customFields - Custom field inputs based on service
• templates - Available webhook templates

Key Features:

• Service-specific custom field presets (Pushover fields, etc.)
• Template preview with field substitution
• URL template support with {{.}} placeholders
• Payload validation
• Header value masking (shows "*REDACTED" for sensitive values)

Notification Stores

File: /opt/pulse/frontend-modern/src/stores/notifications.ts

Reactive state management for:

• Email configuration
• Webhook list
• Apprise configuration
• UI state (loading, errors, testing)

Settings Integration

File: /opt/pulse/frontend-modern/src/components/Settings/Settings.tsx

Integration point for:

• Email notification settings page
• Webhook management interface
• Apprise configuration
• Notification health monitoring

---

DATA FLOW & SEQUENCES

Alert Notification Flow

Alert Triggered (in monitoring)
    ↓
monitor.GetNotificationManager().SendAlert(alert)
    ↓
CheckCooldown → Skip if in cooldown
    ↓
AddToPendingAlerts → Start GroupingTimer
    ↓
GroupingWindow Timer Expires (default 30s)
    ↓
sendGroupedAlerts()
    ├→ Snapshot configurations (email, webhooks, apprise)
    ├→ If PersistentQueue available:
    │   └→ enqueueNotifications() → Queue.Enqueue()
    │
    └→ If No Queue:
        ├→ sendNotificationsDirect()
        ├→ For each enabled webhook:
        │   └→ sendGroupedWebhook() → async goroutine
        ├→ If email enabled:
        │   └→ sendGroupedEmail() → async goroutine
        └→ If apprise enabled:
            └→ sendGroupedApprise() → async goroutine

Webhook Sending Flow

sendGroupedWebhook(webhook, alertList)
    ↓
Check for custom template
    ├→ If custom template exists:
    │   └→ generatePayloadFromTemplateWithService()
    │
    └→ If service-specific (Discord, Slack, etc.):
        ├→ Lookup service template
        ├→ Apply service-specific data enrichment
        ├→ generatePayloadFromTemplateWithService()
        │
        └→ If no template found:
            └→ Use generic JSON payload
    ↓
sendWebhookRequest(webhook, payload)
    ├→ checkWebhookRateLimit()
    ├→ Create HTTP request
    ├→ Set headers
    ├→ Send with secure client (SSRF protection)
    ├→ Read response (max 1MB)
    └→ addWebhookDelivery() to history

Email Sending Flow

sendGroupedEmail(config, alertList)
    ↓
EmailTemplate(alertList, isSingle)
    ├→ Grouped vs single template
    └→ Generate HTML + text body
    ↓
sendHTMLEmailWithError(subject, htmlBody, textBody, config)
    ├→ Use From as To if To empty
    ├→ Get/create EnhancedEmailManager
    └→ SendEmailWithRetry()
        ├→ Attempt 1: sendEmailOnce()
        ├→ Check rate limit
        ├→ Build MIME message
        ├→ sendViaProvider()
        │   ├→ If TLS: sendTLS() [port 465]
        │   ├→ If STARTTLS: sendStartTLS() [port 587]
        │   └→ Else: sendPlain()
        └→ Retry up to 3 times with 5s delays

Persistent Queue Flow

NotificationQueue starts (2 background goroutines)
    ├→ processQueue() - runs every 5 seconds
    │   ├→ SELECT WHERE status='pending' AND next_retry_at <= now
    │   ├→ For each notification:
    │   │   ├→ Update status to 'sending'
    │   │   ├→ Call processor.ProcessQueuedNotification()
    │   │   ├→ If success → Update status to 'sent', set completed_at
    │   │   └→ If failed → Update status to 'failed', set next_retry_at
    │   └→ After max attempts → Move to 'dlq' status
    │
    └→ cleanupOldEntries() - runs hourly
        └→ Delete entries older than retention period

---

KEY SECURITY FEATURES

1. SSRF Prevention

• DNS resolution required for all webhook URLs
• Blocks localhost: 127.0.0.1, ::1, "localhost", 127.*
• Blocks private ranges: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
• Blocks link-local: 169.254., fe80::
• Blocks cloud metadata: 169.254.169.254, metadata.google.internal, metadata.goog
• Redirect validation: max 3 redirects allowed
• Numeric IP warning for HTTPS connections

2. Webhook Security

• Secure client with redirect limits
• Response size limit (1MB max)
• Timeout: 30s default, 10s for tests
• Rate limiting: 10 per minute per webhook URL
• URL validation before creation/update

3. Email Security

• Password not logged or returned in responses
• Password preserved on config update if empty
• TLS/STARTTLS enforcement
• Optional TLS verification skip (for self-signed certs)
• SMTP authentication with proper auth methods

4. API Security

• Scope-based access control (SettingsRead/SettingsWrite)
• Admin requirement on all notification endpoints
• Sensitive field masking in API responses
• Token/secret redaction in webhook history
• Raw body reading (not automatic JSON decode) for sensitive endpoints

5. Queue Security

• Persistent database in user data directory
• SQLite WAL mode for atomicity
• Foreign key constraints
• Audit trail table for all notifications

---

CONFIGURATION PERSISTENCE

Storage Mechanism

Configurations are persisted via ConfigPersistence interface:

• Implementations handle storage (likely YAML/JSON files or database)
• Called after every modification
• Survive service restarts

Persisted Configs

1. Email configuration (EmailConfig struct)
2. All webhook configurations ([]WebhookConfig)
3. Apprise configuration (AppriseConfig struct)
4. Alert notification settings (cooldown, grouping, etc.)

Startup Sequence

1. Load configs from persistent storage
2. Initialize NotificationManager with loaded configs
3. Initialize EnhancedEmailManager with email config
4. Initialize persistent queue (if enabled)
5. Start background processors

---

NOTIFICATION TYPES & TRIGGERS

Alert Levels

• Critical (red, #ff6b6b)
• Warning (yellow, #ffd93d)
• Info (green)

Alert Types

• cpu
• memory
• disk
• io
• diskRead
• diskWrite
• temperature (from sensor proxy)

Metrics Formatting

• CPU/Memory: percentages (e.g., 95.5%)
• Disk I/O: MB/s (e.g., 150.2 MB/s)
• Temperature: degrees (e.g., 78.5°C)

Grouping Configuration

• Cooldown: Per-alert grace period (default 5 minutes)
• Grouping window: Alert batching window (default 30 seconds)
• Group by node: Organize alerts by node
• Group by guest type: Organize alerts by VM/LXC/host

---

ERROR HANDLING & RETRY LOGIC

Email Retries

• Max 3 attempts (configurable)
• 5 second delay between attempts (configurable)
• Rate limiting: 60 per minute (configurable)

Webhook Retries

• Max 3 attempts (configurable per webhook)
• Exponential backoff: 1s, 2s, 4s, 8s, 16s, 30s (max)
• Respects Retry-After header for 429 responses
• Retryable errors: timeouts, network errors, 5xx, 429
• Non-retryable: 4xx client errors (except 429)

Queue Retries

• Configurable max attempts per notification
• Periodic retry processor (5 second intervals)
• Dead letter queue for permanently failed items
• Manual DLQ retry/delete via API

Logging

• Structured logging with zerolog
• All critical operations logged with context:
  • Alert ID, resource name, type
  • Webhook name, service, URL (redacted)
  • Email recipients and SMTP server
  • Retry counts and errors

---

CONSTANTS & LIMITS

Webhook Settings

• Timeout: 30 seconds
• Test timeout: 10 seconds
• Max response size: 1 MB
• Max redirects: 3
• Rate limit: 10 per minute per webhook URL
• Initial backoff: 1 second
• Max backoff: 30 seconds
• Default retries: 3
• History size: Last 100 deliveries

Email Settings

• Dial timeout: 10 seconds
• Connection deadline: 30 seconds
• Default rate limit: 60 per minute
• Default retries: 3
• Default retry delay: 5 seconds

Queue Settings

• Processor interval: 5 seconds
• Cleanup interval: 1 hour
• Database journal: WAL mode
• Cache: 64 MB
• Busy timeout: 5 seconds

Apprise Settings

• Timeout bounds: 5-120 seconds (default 15)
• Max targets: Unlimited (but must have >0)
• CLI path default: "apprise"
• API key header default: "X-API-KEY"

---

FILES SUMMARY TABLE

File	Lines	Purpose
notifications.go	2,358	Core notification manager, alert grouping, webhook/email/apprise sending
webhook_enhanced.go	605	Enhanced webhook support, retry logic, filtering
email_enhanced.go	415	Email manager, SMTP connections, provider support
email_providers.go	217	Email provider templates and instructions
email_template.go	200+	HTML email templates for single/grouped alerts
webhook_templates.go	300+	Templates for Discord, Slack, Teams, Telegram, etc.
queue.go	600+	Persistent notification queue, SQLite backend
notifications.go (API)	753	HTTP handlers for all notification endpoints
notification_queue.go (API)	80+	HTTP handlers for queue management
notifications.ts (Frontend)	201	TypeScript API client for notification endpoints
EmailProviderSelect.tsx	200+	Email provider configuration UI component
WebhookConfig.tsx	300+	Webhook configuration UI component

---

INTEGRATION POINTS WITH OTHER SYSTEMS

Monitor Integration

• monitor.GetNotificationManager() - Access notification manager
• monitor.GetConfigPersistence() - Persist configurations
• monitor.GetState() - Get node and instance info for test notifications

Alert System Integration

• Receives Alert objects with:
  • ID, Type, Level, ResourceName, ResourceID
  • Node, Instance, Message
  • Value, Threshold, StartTime, LastSeen
  • Metadata (resourceType, etc.)
  • Acknowledged, AckTime, AckUser

Router Integration

• /api/notifications/ main handler routes to sub-endpoints
• All endpoints require admin auth + scope
• Integrated in main API router initialization

Config System

• Notifications loaded/saved via ConfigPersistence
• Can survive service restarts
• Loaded at startup into NotificationManager

---

WORKFLOW EXAMPLES

Example 1: Configure Gmail Notifications

User → Settings → Email Notifications
  ↓
Select "Gmail / Google Workspace" from provider dropdown
  ↓
Auto-filled:
  - Server: smtp.gmail.com
  - Port: 587
  - STARTTLS: true
  - Display: Setup instructions
  ↓
User enters:
  - From: user@gmail.com
  - To: [alerts@company.com]
  - Username: user@gmail.com
  - Password: <app-password>
  ↓
PUT /api/notifications/email
  ↓
Backend: UpdateEmailConfig() → SetEmailConfig() → SaveEmailConfig()
  ↓
Test → POST /api/notifications/test {method: "email"}
  ↓
SendTestNotification() → sendEmail() → EnhancedEmailManager.SendEmailWithRetry()
  ↓
SMTP → gmail via STARTTLS → Success/Error logged
  ↓
Frontend: Display success/error

Example 2: Add Discord Webhook

User → Settings → Webhooks → Add New
  ↓
Select Service: Discord
  ↓
Copy webhook URL from Discord server
  ↓
Enter URL: https://discord.com/api/webhooks/...
  ↓
Backend: ValidateWebhookURL() → DNS check → Private IP check → SSRF check
  ↓
POST /api/notifications/webhooks
  ↓
Backend: GenerateID() → AddWebhook() → SaveWebhooks()
  ↓
Test: POST /api/notifications/webhooks/test
  ↓
Backend: sendWebhook() → Apply Discord template
  ↓
Generate payload with embed format → Send HTTP POST
  ↓
Discord returns 204 → Success logged to webhook history
  ↓
Frontend: Display success + sample embed preview

Example 3: Alert Triggering Grouped Notifications

Monitoring Detects: CPU > 95% on node1
  ↓
Create Alert → monitor.GetNotificationManager().SendAlert(alert)
  ↓
NotificationManager.SendAlert():
  - Check cooldown → Not in cooldown
  - Add to pendingAlerts list
  - Start groupingTimer (30s)
  ↓
Wait 5s → Another alert: Memory > 90% on node1
  - Add to pendingAlerts list
  - Timer still running
  ↓
Wait 25s → Grouping timer expires
  ↓
sendGroupedAlerts() with 2 alerts:
  - Snapshot config (2 webhooks, email enabled)
  - If queue: Enqueue 3 notifications (email + 2 webhooks)
  - If no queue: Send directly (3 async goroutines)
  ↓
For each webhook:
  - sendGroupedWebhook(webhook, [alert1, alert2])
  - Prepare template data with AlertCount=2
  - Generate combined message
  - Send single HTTP POST
  ↓
For email:
  - sendGroupedEmail(emailConfig, [alert1, alert2])
  - Template generates grouped email
  - Send via SMTP with retries
  ↓
Queue (if enabled):
  - Background processor retries every 5s
  - Updates status: pending → sending → sent/failed
  - Failed items retry with backoff
  - After 3 attempts, move to DLQ
  ↓
Update last notified timestamps