Commit graph

940 commits

Author SHA1 Message Date
rcourtman
286edee8aa feat(ai): Add operational memory (Phase 3) - change detection and remediation logging
Phase 3 of Pulse AI differentiation:

Create internal/ai/memory package with:

1. Change Detection (changes.go):
   - Tracks infrastructure changes: creation, deletion, config changes
   - Detects status changes (started, stopped)
   - Detects VM/container migrations between nodes
   - Detects CPU/memory configuration changes
   - Detects backup completions
   - Persists change history to ai_changes.json
   - GetChangesSummary for AI context

2. Remediation Logging (remediation.go):
   - Records actions taken to fix problems
   - Tracks command, output, and outcome
   - Links to AI findings via findingID
   - GetSimilar finds past similar problems
   - GetSuccessfulRemediations for learning
   - Persists to ai_remediations.json

3. Type exports (memory_exports.go):
   - Clean re-exports from ai package

This enables the AI to say things like:
- 'This VM was migrated 2 hours ago'
- 'Memory was increased from 4GB to 8GB yesterday'
- 'Last time this happened, restarting nginx resolved it'

All tests passing.
2025-12-12 13:49:37 +00:00
rcourtman
5f2c240480 Clarify AI cost estimates with pricing coverage 2025-12-12 13:19:03 +00:00
rcourtman
11ebc0e91c Unify provider/model normalization for AI cost export 2025-12-12 13:04:42 +00:00
rcourtman
5c9ebf8d3a Add AI cost export and top target rollups 2025-12-12 12:55:39 +00:00
rcourtman
1294a66f56 Backup AI usage history on reset 2025-12-12 12:14:13 +00:00
rcourtman
4df04970ea Persist AI cost budget and allow history reset 2025-12-12 12:10:58 +00:00
rcourtman
5ba4e6a84c Improve AI cost dashboard ranges and breakdowns 2025-12-12 11:35:41 +00:00
rcourtman
47eefe6763 feat(ai): Wire baseline learning loop into router startup
Complete Phase 2 baseline integration:

- Add baseline_exports.go for clean type aliasing
- Wire baseline store initialization into StartPatrol
- Implement startBaselineLearning background loop
  - Runs initial learning after 5 min delay
  - Updates baselines every hour from metrics history
  - Learns from 7 days of data for nodes, VMs, containers
- Add SetBaselineStore methods throughout the chain
  (Router -> AIHandler -> Service -> PatrolService)
- Persists baselines to data directory as JSON

The baseline learning loop:
1. Starts automatically when AI patrol starts
2. Queries metrics history for all resources
3. Computes mean, stddev, percentiles for cpu/memory/disk
4. Saves baselines to disk for durability
5. Anomaly detection uses these baselines in context builder

All tests passing.
2025-12-12 11:29:47 +00:00
rcourtman
f3e95c24ae feat(ai): Add baseline learning and anomaly detection (Phase 2)
Phase 2 of Pulse AI differentiation:

- Create internal/ai/baseline package for learned baselines
- Implement statistical baseline learning with mean, stddev, percentiles
- Add z-score based anomaly detection with severity classification
  (low, medium, high, critical based on standard deviations)
- Integrate baseline provider into context builder
- Wire baseline store into patrol service with adapters
- Add anomaly enrichment to resource contexts

Key features:
- Learn computes baseline from historical metric data points
- IsAnomaly and CheckAnomaly detect deviations from normal
- Persists baselines to disk as JSON for durability
- Formatted anomaly descriptions for AI consumption
  Example: 'Memory is high above normal (85.2% vs typical 42.1% ± 8.3%)'

The baseline store needs to be initialized and triggered to learn
from metrics history. Next step is adding the learning loop.

All tests passing.
2025-12-12 11:26:31 +00:00
rcourtman
3ea6c1be5d Show AI cost refresh errors and harden log redaction 2025-12-12 11:05:24 +00:00
rcourtman
877a4f08e7 Fix DeepSeek cost attribution and pricing 2025-12-12 10:49:56 +00:00
rcourtman
71b6a2ae12 Add estimated USD to AI cost dashboard 2025-12-12 10:43:07 +00:00
rcourtman
96af101c98 feat(ai): Add enriched context with historical trends and predictions
Phase 1 of Pulse AI differentiation:

- Create internal/ai/context package with types, trends, builder, formatter
- Implement linear regression for trend computation (growing/declining/stable/volatile)
- Add storage capacity predictions (predicts days until 90% and 100%)
- Wire MetricsHistory from monitor to patrol service
- Update patrol to use buildEnrichedContext instead of basic summary
- Update patrol prompt to reference trend indicators and predictions

This gives the AI awareness of historical patterns, enabling it to:
- Identify resources with concerning growth rates
- Predict capacity exhaustion before it happens
- Distinguish between stable high usage vs growing problems
- Provide more actionable, time-aware insights

All tests passing. Falls back to basic summary if metrics history unavailable.
2025-12-12 09:45:57 +00:00
rcourtman
1a78e8846a feat(ai): Replace patrol frequency dropdown with custom minutes input
- Changed patrol schedule from preset dropdown to freeform number input
- Users can now set any interval (min 10 minutes, max 7 days, or 0 to disable)
- Added patrol_interval_minutes to API request/response (preset is now deprecated)
- Backend validates: min 10 minutes when enabled, max 10080 (7 days)
- Frontend shows human-readable duration next to input (e.g., '6h', '2h 30m')

Also improved Auto-Fix Mode safety:
- Removed '(recommended)' from preset options (was subjective)
- Added 'I understand the risks' acknowledgement checkbox
- Toggle is disabled until user explicitly acknowledges the risks
- Shows prominent warning when Auto-Fix is enabled
- Acknowledgement is session-based (must re-acknowledge on page reload)
2025-12-11 23:24:33 +00:00
rcourtman
d03b3d644c fix(ai-chat): Display messages chronologically in AI chatbot
- Add 'content' type to StreamDisplayEvent for tracking text chunks
- Track content events in streamEvents array for chronological display
- Update render to use Switch/Match for cleaner conditional rendering
- Interleave thinking, tool calls, and content as they stream in
- Add fallback for old messages without streamEvents for backwards compat

Previously, tool/command outputs stayed at top while AI text responses
accumulated at the bottom. Now all events appear in order like a
normal chatbot.
2025-12-11 23:02:59 +00:00
rcourtman
bf92c0510a feat: Add clear credentials button for each AI provider
- Add clear_anthropic_key, clear_openai_key, clear_deepseek_key, clear_ollama_url flags to API
- Backend handles clearing with confirmation prompt
- Each provider accordion shows Test and Clear buttons when configured
- Clear button requires confirmation before removing credentials
- Frontend automatically refreshes settings after clearing
2025-12-11 18:24:25 +00:00
rcourtman
597527fc04 feat: Add per-provider test buttons and documentation links
- Add /api/ai/test/{provider} endpoint for testing individual providers
- Add 'Test' button to each provider accordion (visible when configured)
- Shows test result inline (success/error message)
- Update help links with direct URLs to API key pages:
  - Anthropic: console.anthropic.com/settings/keys
  - OpenAI: platform.openai.com/api-keys
  - DeepSeek: platform.deepseek.com/api_keys
  - Ollama: ollama.ai
2025-12-11 18:11:31 +00:00
rcourtman
8b261f9ec2 fix: Ollama should only show as configured when URL is explicitly set
Previously Ollama always showed as 'Available' even if not set up.
Now it only shows as configured when user has entered an OllamaBaseURL.
2025-12-11 17:12:01 +00:00
rcourtman
0c3dcf353a feat: Implement multi-provider AI support
Backend:
- Add per-provider API key fields to AIConfig (AnthropicAPIKey, OpenAIAPIKey, DeepSeekAPIKey, OllamaBaseURL, OpenAIBaseURL)
- Add NewForProvider() and NewForModel() factory functions for multi-provider instantiation
- Update ListModels() to aggregate models from all configured providers with provider:model format
- Update Execute/ExecuteStream to dynamically create provider based on selected model
- Update TestConnection to use multi-provider aware provider creation
- Add helper functions: HasProvider(), GetConfiguredProviders(), GetAPIKeyForProvider(), GetBaseURLForProvider(), ParseModelString(), FormatModelString()

Frontend:
- Remove legacy single-provider UI (provider grid, single API key input, single base URL)
- Add accordion-style UI for configuring all providers independently
- Add model grouping by provider in selectors using optgroup
- Update AIChat model dropdown with grouped provider sections
- Add helper functions for parsing provider from model ID and grouping models

API:
- Add multi-provider fields to AISettingsResponse and AISettingsUpdateRequest
- Add /api/ai/models endpoint for dynamic model listing
- Update settings handlers for per-provider credential management
2025-12-11 16:00:45 +00:00
rcourtman
a8d0b15346 feat(ai): Add suppression rules management API and UI
Users can now:
1. View all suppression rules (both from dismissed findings and manually created)
2. Create manual rules like 'ignore performance issues on debian-go'
3. Delete rules when they want alerts to come back

Backend:
- Added SuppressionRule type for user-defined rules
- Added suppressionRules storage to FindingsStore
- Added AddSuppressionRule/GetSuppressionRules/DeleteSuppressionRule methods
- Added isSuppressedInternal check for manual rules
- Added API handlers and routes for /api/ai/patrol/suppressions

Frontend:
- Added SuppressionRule interface
- Added getSuppressionRules/addSuppressionRule/deleteSuppressionRule API functions
- Added getDismissedFindings for viewing dismissed findings

Example usage:
POST /api/ai/patrol/suppressions
{
  'resource_id': 'debian-go',
  'category': 'performance',
  'description': 'Dev container runs hot - expected'
}
2025-12-11 00:12:18 +00:00
rcourtman
7ef96919d3 fix(ai): Make LLM finding IDs stable across patrol runs
The main issue was that finding IDs included the title, which the LLM
generates differently each time. 'High CPU on minipc' vs 'Node minipc
experiencing high CPU load' got different IDs, making dismissals useless.

Changes:
1. LLM findings now get IDs based on resource+category only, not title
2. Add() now checks if finding is suppressed before adding as new
3. Add() now checks dismissed findings and only reactivates on severity escalation
4. IsSuppressed() now matches by resource+category only, not title
5. Added isSuppressedInternal() for use when lock is already held

Now when you dismiss 'performance issues on minipc', any future patrol finding
about performance on minipc will be recognized as the same issue and stay dismissed.
2025-12-11 00:03:17 +00:00
rcourtman
b1199b3cbf fix(ai): Use context.Background() for forced patrol runs
The ForcePatrol() function was using the HTTP request context, which gets
cancelled immediately when the API response is sent. This caused LLM analysis
to fail with 'context canceled' before it could complete.

Now uses context.Background() so the goroutine runs independently of the
HTTP request lifecycle.

Also fixed dropdown hover gap issue in the dismiss menu.
2025-12-10 23:31:21 +00:00
rcourtman
fd8cc4a32e feat(ai): Add per-resource notes to patrol context
Knowledge store notes are now included in the patrol LLM prompt. When users
save notes about resources (e.g., 'This VM intentionally runs hot'), the patrol
AI will see these notes and avoid flagging documented behavior as issues.

Changes:
- Added knowledge store reference to PatrolService
- Added SetKnowledgeStore() method to configure the store
- Enhanced buildPatrolPrompt() to include knowledge context
- Connected knowledge store to patrol in service.go SetStateProvider()

This complements the dismissed findings context to give the LLM a complete
picture of what the user considers normal/expected behavior.
2025-12-10 23:03:01 +00:00
rcourtman
a3d953172c feat(ai): Add LLM memory system for patrol findings
Implements a comprehensive feedback system that allows the LLM to 'remember'
user decisions about findings, preventing repetitive/annoying alerts.

Backend changes:
- Extended Finding struct with dismissed_reason, user_note, times_raised, suppressed
- Added Dismiss(), Suppress(), SetUserNote(), IsSuppressed() methods to FindingsStore
- Added GetDismissedForContext() to format dismissed findings for LLM context
- Enhanced buildPatrolPrompt() to inject user feedback context
- Added POST /api/ai/patrol/dismiss and /api/ai/patrol/suppress endpoints
- Updated IsActive() to exclude suppressed findings

Frontend changes:
- Added Dismiss dropdown with options: Not an Issue, Expected Behavior, Will Fix Later
- Added Never Alert Again option for permanent suppression
- Expected Behavior prompts for optional note to help LLM understand context
- Added visual badges: recurrence count (×N), dismissed status, suppressed indicator
- Display user notes in expanded finding view

Also fixes:
- Fixed 403 error on Run Patrol (compilation errors from partial refactoring)
- Removed non-LLM patrol checks - patrol now uses LLM analysis only
- Fixed function signature mismatches in alert_triggered.go

The LLM now receives context about previously dismissed findings and is
instructed not to re-raise them unless severity has significantly worsened.
2025-12-10 22:55:34 +00:00
rcourtman
c88e2db7b4 feat(ai): Enhanced AI patrol system with alert triggers and history persistence
- Add alert-triggered AI analysis for real-time incident response
- Implement patrol history persistence across restarts
- Add patrol schedule configuration UI in AI Settings
- Enhance AIChat with patrol status and manual trigger controls
- Add resource store improvements for AI context building
- Expand Alerts page with AI-powered analysis integration
- Add Vite proxy config for AI API endpoints
- Support both Anthropic and OpenAI providers with streaming
2025-12-10 21:08:22 +00:00
rcourtman
d2330cf405 refactor(ai): Remove over-engineered URL discovery service
Keep only the simple AI-powered approach:
- set_resource_url tool lets AI save discovered URLs
- Users ask AI directly: 'Find URLs for my containers'
- AI uses its intelligence to discover and set URLs

Removed:
- URLDiscoveryService (rigid port scanning)
- Bulk discovery API endpoints
- Frontend discovery button

The AI itself is smart enough to iterate through resources
and discover URLs when asked.
2025-12-10 08:35:24 +00:00
rcourtman
0ae4daca8d feat(ai): Add bulk URL discovery service
- Add URLDiscoveryService for scanning all resources at once
- Scans common web ports (80, 443, 8080, 8096, 3000, etc.)
- Automatically saves discovered URLs to resource metadata
- Add API endpoints for start/status/cancel discovery
- Progress tracking with results reporting

Endpoints:
- POST /api/ai/discover-urls/start - Start bulk discovery
- GET /api/ai/discover-urls/status - Check progress
- POST /api/ai/discover-urls/cancel - Cancel discovery
2025-12-10 08:30:56 +00:00
rcourtman
fac2bf91d9 feat(ai): Add URL discovery tool - AI can find and set resource URLs
- Add MetadataProvider interface for AI to update resource URLs
- Add set_resource_url tool to AI service
- Wire up metadata stores to AI service via router
- Add URL discovery guidance to AI system prompt
- AI can now inspect guests/containers/hosts for web services
  and automatically save discovered URLs to Pulse metadata

Usage: Ask the AI 'Find the web URL for this container' and it will:
1. Check for listening ports and web servers
2. Get the IP address
3. Verify the URL works
4. Save it to Pulse for quick dashboard access
2025-12-10 00:29:07 +00:00
rcourtman
a6656381c8 Add AI monitoring enhancements and host metadata features
- Add host metadata API for custom URL editing on hosts page
- Enhance AI routing with unified resource provider lookup
- Add encryption key watcher script for debugging key issues
- Improve AI service with better command timeout handling
- Update dev environment workflow with key monitoring docs
- Fix resource store deduplication logic
2025-12-09 16:27:46 +00:00
rcourtman
67351bf58c feat: AI integration, Docker metrics, RAID display, and infrastructure improvements
- Add Claude OAuth authentication support with hybrid API key/OAuth flow
- Implement Docker container historical metrics in backend and charts API
- Add CEPH cluster data collection and new Ceph page
- Enhance RAID status display with detailed tooltips and visual indicators
- Fix host deduplication logic with Docker bridge IP filtering
- Fix NVMe temperature collection in host agent
- Add comprehensive test coverage for new features
- Improve frontend sparklines and metrics history handling
- Fix navigation issues and frontend reload loops
2025-12-09 09:29:27 +00:00
rcourtman
f7eed7b052 cleanup: remove legacy AI context fallback (buildInfrastructureContext)
The AI service now uses only buildUnifiedResourceContext() for
infrastructure context, since the resourceProvider is always set
during router initialization.

Removed:
- buildInfrastructureContext() function (~288 lines of dead code)
- Legacy fallback path in buildSystemPrompt()

The unified resource context provides a cleaner, deduplicated view
of infrastructure that includes:
- All resources grouped by platform and type
- Top CPU/Memory/Disk consumers
- Active alerts on resources
- Infrastructure summary statistics

This completes the AI service migration to unified resources.
2025-12-08 09:21:03 +00:00
rcourtman
7e8611e0ae cleanup: consolidate unified resources adapters into useResourcesAsLegacy hook
This cleanup addresses transition debt from the unified resources migration:

Frontend cleanup:
- Move all Resource→Legacy type conversions to useResourcesAsLegacy() hook
- Add asNodes() and asDockerHosts() adapter functions to the hook
- Simplify DockerRoute, HostsRoute, DashboardView to use the centralized hook
- Remove ~300 lines of duplicated adapter code from App.tsx
- Remove debug console.log statements from Dashboard.tsx
- Fix CPU value conversion (divide by 100) for Dashboard compatibility

Backend fixes (from previous session):
- Fix parentID format in converters (VM, Container, Storage) to match Node.ID
- Format changed from 'instance/node/nodename' to 'instance-nodename'
- Update tests to match new parentID format

This consolidates all legacy type conversion logic in one place,
making future cleanup easier when components are migrated to use
unified resources directly.
2025-12-08 09:16:26 +00:00
rcourtman
cc4b32ad4a fix: complete unified resources WebSocket integration
Backend:
- Call SetMonitor after router creation to inject resource store
- Add debug logging for resource population and broadcast

Frontend:
- Add resources array to WebSocket store initial state
- Handle resources in WebSocket message processing
- Use reconcile for efficient state updates

The unified resources are now properly:
1. Populated from StateSnapshot on each broadcast cycle
2. Converted to frontend format (ResourceFrontend)
3. Included in WebSocket state messages
4. Received and stored in frontend state
5. Consumed by migrated route components

Console now shows '[DashboardView] Using unified resources: VMs: X'
confirming the migration is working end-to-end.
2025-12-07 23:52:00 +00:00
rcourtman
a10c13dfe9 test: add PopulateFromSnapshot test to verify resource conversion
Test confirms that StateSnapshot data is correctly converted to unified
resources. Validates conversion for nodes, VMs, containers, and hosts.
2025-12-07 23:38:15 +00:00
rcourtman
881f00a5d9 ensure resource store is populated before WebSocket broadcasts
- Added PopulateFromSnapshot method to resources.Store
- Extended ResourceStoreInterface to include PopulateFromSnapshot
- Monitor now calls updateResourceStore before broadcasts
- This ensures resources are fresh on every WebSocket broadcast

Without this, the store would only be populated when /api/resources or
/api/state endpoints are hit, leaving WebSocket broadcasts empty.
2025-12-07 23:15:02 +00:00
rcourtman
cdd2249e00 phase 1: add unified resources to WebSocket state broadcasts
- Extended StateFrontend with Resources field containing unified resource data
- Added ResourceFrontend and related types for frontend-compatible resource data
- Extended ResourceStoreInterface to include GetAll() method
- Monitor now injects resources into WebSocket broadcasts
- Added helper method getResourcesForBroadcast() to convert resources to frontend format
- All existing tests pass

This enables the frontend to access unified resources via WebSocket state.
2025-12-07 23:02:56 +00:00
rcourtman
ef35120afb checkpoint: remove unified resources UI, keep backend model
- Removed /resources page and associated frontend components
- Removed ResourcesOverview.tsx, UnifiedResourceRow.tsx, columns.ts
- Removed frontend types/resource.ts
- Updated unified-resource-architecture.md to mark Phase 4 as ABANDONED
- Removed unified-view-migration-plan.md
- Backend unified resource model remains for AI context

This is a checkpoint before attempting full frontend migration to unified model.
2025-12-07 22:54:27 +00:00
rcourtman
cac109b497 fix: Populate resources on-demand when /api/resources is called
The Resources page was showing 0 resources because the store was only
populated when /api/state was called (from the dashboard). Now the
resources are populated on-demand when /api/resources is accessed.

Changes:
- Added StateProvider interface to ResourceHandlers
- SetStateProvider() method for injecting the monitor
- HandleGetResources now calls PopulateFromSnapshot before querying
- Router injects monitor as state provider during SetMonitor()

This ensures the /resources page works even when accessed directly
without visiting the main dashboard first.
2025-12-07 14:47:29 +00:00
rcourtman
5f000b7974 feat: Complete Unified Resource Architecture (Phases 1-3)
This commit implements the Unified Resource Architecture for AI-first
infrastructure management. Key features:

Phase 1 - Backend Unification:
- New unified Resource type with 9 resource types, 7 platforms, 7 statuses
- Resource store with identity-based deduplication (hostname, machineID, IP)
- 8 converter functions (FromNode, FromVM, FromContainer, etc.)
- REST API endpoints: /api/resources, /api/resources/stats, /api/resources/{id}
- 28 comprehensive unit tests

Phase 2 - AI Context Enhancement:
- Unified context builder for AI system prompts
- Cross-platform query methods: GetTopByCPU, GetTopByMemory, GetTopByDisk
- Resource correlation: GetRelated (parent, children, siblings, cluster)
- Infrastructure summary: GetResourceSummary with health status counts
- AI context now includes top consumers and infrastructure overview

Phase 3 - Agent Preference & Hybrid Mode:
- Polling optimization methods in resource store
- ResourceStoreInterface added to Monitor
- SetResourceStore() and shouldSkipNodeMetrics() helper methods
- Store automatically wired into Monitor via Router.SetMonitor()
- Foundation ready for reduced API polling when agents are active

Files added:
- internal/resources/resource.go - Core Resource type
- internal/resources/store.go - Store with deduplication
- internal/resources/converters.go - Type converters
- internal/resources/platform_data.go - Platform-specific data
- internal/resources/store_test.go - 28 tests
- internal/resources/converters_test.go - Converter tests
- internal/api/resource_handlers.go - REST API handlers
- internal/ai/resource_context.go - AI context builder
- .gemini/docs/unified-resource-architecture.md - Architecture docs

All tests pass.
2025-12-07 13:49:00 +00:00
rcourtman
190e02fbbc Add host alert deduplication with tests
- Modified alerts.go for improved host alert handling
- Added host_dedup_test.go for deduplication test coverage
2025-12-07 12:38:38 +00:00
rcourtman
721f973271 AI features checkpoint: Host selection, memory sparklines, UI refinements
- Extended AI context selection to host rows in HostsOverview
- Added resourceId prop to StackedMemoryBar for sparkline support
- Relocated guest URL editing from GuestRow name click
- Added GuestNotes component with URL field in AI sidebar
- Refined host routing in AI service backend
- Minor animation and styling improvements
2025-12-07 12:25:26 +00:00
rcourtman
f2e6927436 Additional updates 2025-12-07 10:22:42 +00:00
rcourtman
90c45968e7 AI Problem Solver implementation and various fixes
- Implement 'Show Problems Only' toggle combining degraded status, high CPU/memory alerts, and needs backup filters
- Add 'Investigate with AI' button to filter bar for problematic guests
- Fix dashboard column sizing inconsistencies between bars and sparklines view modes
- Fix PBS backups display and polling
- Refine AI prompt for general-purpose usage
- Fix frontend flickering and reload loops during initial load
- Integrate persistent SQLite metrics store with Monitor
- Fortify AI command routing with improved validation and logging
- Fix CSRF token handling for note deletion
- Debug and fix AI command execution issues
- Various AI reliability improvements and command safety enhancements
2025-12-06 23:46:08 +00:00
rcourtman
c9915ead8b feat: Add OS type display for LXC containers
- Extract ostype from LXC container config (debian, ubuntu, alpine, etc.)
- Map ostype values to human-readable names (e.g., "debian" -> "Debian")
- Add OSName field to Container model and ContainerFrontend
- Add icons for NixOS, openSUSE, and Gentoo in frontend
- LXC containers now show OS icons alongside VMs in the dashboard

Supported LXC OS types: alpine, archlinux, centos, debian, devuan,
fedora, gentoo, nixos, opensuse, ubuntu, unmanaged
2025-12-05 12:43:32 +00:00
rcourtman
fcb258ac2d wip: AI provider improvements and chat store enhancements
- Improve Anthropic provider error handling
- Add AI service enhancements
- Update AI chat store with additional state management
2025-12-05 12:14:00 +00:00
rcourtman
cc3c0187a0 feat: AI features, agent improvements, and host monitoring enhancements
AI Chat Integration:
- Multi-provider support (Anthropic, OpenAI, Ollama)
- Streaming responses with markdown rendering
- Agent command execution for remote troubleshooting
- Context-aware conversations with host/container metadata

Agent Updates:
- Add --enable-proxmox flag for automatic PVE/PBS token setup
- Improve auto-update with semver comparison (prevents downgrades)
- Add updatedFrom tracking to report previous version after update
- Reduce initial update check delay from 30s to 5s
- Add agent version column to Hosts page table

Host Metrics:
- Add DiskIO stats collection (read/write bytes, ops, time)
- Improve disk filtering to exclude Docker overlay mounts
- Add RAID array monitoring via mdadm
- Enhanced temperature sensor parsing

Frontend:
- New Agent Version column on Hosts overview table
- Improved node modal with agent-first installation flow
- Add DiskIO display in host drawer
- Better responsive handling for metric bars
2025-12-05 10:37:02 +00:00
rcourtman
0cd66da8f6 fix: Race condition in TestLoadActiveAlerts causing flaky test
ClearActiveAlerts triggers an async save to disk, which can race with
LoadActiveAlerts reading the file. The test now clears the in-memory
map directly without triggering the async save.
2025-12-04 20:41:00 +00:00
rcourtman
3f08bfe8d0 wip: AI chat integration with multi-provider support
- Add AI service with Anthropic, OpenAI, and Ollama providers
- Add AI chat UI component with streaming responses
- Add AI settings page for configuration
- Add agent exec framework for command execution
- Add API endpoints for AI chat and configuration
2025-12-04 20:16:53 +00:00
rcourtman
29e4f54243 fix: Exclude TrueNAS Docker overlay mounts from disk stats
Host agent was including Docker overlay2 mounts from TrueNAS SCALE's
.ix-apps directory in disk totals. These mounts inherit the ZFS pool's
AVAIL space, causing massively inflated storage numbers (e.g., 173 TB
per container overlay instead of actual usage).

Changes:
- Add /mnt/.ix-apps/docker/ to container overlay path exclusions
- Use ShouldSkipFilesystem() in host agent disk collection (was only
  using ShouldIgnoreReadOnlyFilesystem() which missed container paths)
- Add test cases for TrueNAS overlay paths

Related to #718
2025-12-04 03:03:04 +00:00
rcourtman
8fca8c6eb4 Refactor alert system: fix race conditions, memory leaks, and improve code quality
- Rename checkFlapping to checkFlappingLocked to clarify lock contract
- Replace goto statements with structured control flow
- Wire up unused recordAlertFired/recordAlertResolved metric hooks
- Add trackingMapCleanup goroutine to prevent memory leaks from stale entries
- Tighten alert ID validation to alphanumeric + safe punctuation
- Fix history save error handling to properly manage backup lifecycle
- Add auto-migration for deprecated GroupingWindow field
- Refactor 300+ line UpdateConfig into focused helper functions
- Unify duplicate evaluateVMCondition/evaluateContainerCondition
- Add constants for magic numbers (thresholds, timing, flapping)
- Update tests to match new backup behavior
2025-12-02 23:31:36 +00:00
rcourtman
7ab68cf682 Fix offline buffering: add tests, remove unused config, fix flaky test
- Add unit tests for internal/buffer package
- Fix misleading "ring buffer" comment (it's a bounded FIFO queue)
- Remove unused BufferCapacity config field from both agents
- Rewrite flaky integration test to use polling instead of fixed sleeps
2025-12-02 22:31:44 +00:00
courtmanr@gmail.com
b2d434b4c2 feat: Implement offline buffering for host and docker agents
- Add internal/buffer package with generic ring buffer
- Add buffering logic to host agent for failed reports
- Add buffering logic to docker agent for failed reports
- Add BufferCapacity configuration option
- Add integration tests for buffering logic
2025-12-02 22:12:47 +00:00
rcourtman
93223d8283 Add refresh-cluster button to detect new Proxmox cluster members
When new nodes are added to a Proxmox cluster after Pulse was
initially configured, they weren't showing up in Settings. The
existing "Refresh" button only triggered network discovery, not
cluster membership re-detection.

Changes:
- Add POST /api/config/nodes/{id}/refresh-cluster endpoint
- Add "Refresh" button in cluster node panel in Settings
- Re-detect cluster membership and update stored endpoints

Related to #799
2025-12-02 22:01:00 +00:00
rcourtman
72af7bed5e style: Apply gofmt to 37 files
Standardize code formatting across test files and monitor.go.
No functional changes.
2025-12-02 17:21:48 +00:00
rcourtman
ec6133afd8 refactor: Remove duplicate IsPasswordHashed from auth package
The config package has a more robust IsPasswordHashed function that
handles truncated hashes. The auth package had a simpler duplicate
that was only used in tests. Removed the duplicate and its test
(already covered by config/config_utils_test.go).

Reduces deadcode findings from 7 to 6.
2025-12-02 17:19:07 +00:00
rcourtman
185e7098bd refactor: Remove duplicate isLoopback function from hostagent
The isLoopback function in internal/hostagent/agent.go was unused in
production code - it was a duplicate of the same function in
internal/hostmetrics/collector.go which is actively used.

Removed the dead code along with its associated tests to reduce
maintenance burden and improve code clarity.
2025-12-02 16:52:55 +00:00
rcourtman
c108cba8bb security: Add request body size limits to remaining API handlers
Add http.MaxBytesReader to 8 additional handlers to complete API
hardening against memory exhaustion attacks:

- guest_metadata.go: HandleUpdateMetadata (16KB)
- notification_queue.go: RetryDLQItem, DeleteDLQItem (8KB each)
- temperature_proxy.go: HandleRegister (8KB)
- host_agents.go: HandleReport (256KB)
- updates.go: HandleApplyUpdate (8KB)
- docker_metadata.go: HandleUpdateMetadata (16KB)
- system_settings.go: UpdateSystemSettings (64KB)

All API handlers that decode JSON request bodies now have size limits.
2025-12-02 16:47:13 +00:00
rcourtman
7b55564b62 security: Add request body size limits to API handlers
Add http.MaxBytesReader to 16 additional handlers to prevent memory
exhaustion attacks via oversized request bodies:

- docker_agents.go: HandleReport (512KB), HandleCommandAck (8KB),
  HandleSetCustomDisplayName (8KB)
- alerts.go: UpdateAlertConfig (64KB), BulkAcknowledgeAlerts (32KB),
  BulkClearAlerts (32KB)
- config_handlers.go: HandleAddNode, HandleTestConnection,
  HandleUpdateNode, HandleTestNodeConfig (32KB each),
  HandleVerifyTemperatureSSH, HandleExportConfig, HandleDiscoverServers,
  HandleSetupScriptURL (8KB each), HandleImportConfig (1MB),
  HandleUpdateMockMode (16KB)
2025-12-02 16:43:13 +00:00
rcourtman
ccc26624e8 security: Add request body size limits to notification handlers
Add http.MaxBytesReader limits to prevent memory exhaustion attacks:
- UpdateEmailConfig: 32KB limit
- UpdateAppriseConfig: 64KB limit
- CreateWebhook: 64KB limit
- UpdateWebhook: 64KB limit

This follows the pattern already used in system_settings.go for
SSH config validation.
2025-12-02 16:37:30 +00:00
rcourtman
0bcaa41499 test: Fix unreachable code warning in panic recovery test
Refactor the test to avoid unreachable code after panic by
checking a flag set before the panic instead of after. This
resolves the go vet warning while maintaining test coverage.
2025-12-02 16:30:31 +00:00
rcourtman
2bcdeea537 refactor: Use zerolog instead of fmt.Printf in config export
Replace raw fmt.Printf calls with structured zerolog logging for
consistency with the rest of the codebase. This improves log
formatting and enables proper log level filtering.
2025-12-02 16:27:54 +00:00
rcourtman
12f133f6f8 test: Add tests for SyncGuestBackupTimes and UpdateStorageBackupsForInstance
Add tests for the remaining uncovered State methods:
- SyncGuestBackupTimes: Tests backup time sync from storage and PBS backups
- UpdateStorageBackupsForInstance: Tests storage backup updates with instance isolation

Improves internal/models coverage from 89.6% to 95.6%.
2025-12-02 16:22:38 +00:00
rcourtman
b5a5fddaa3 test: Add tests for Ceph, backup, replication, and snapshot methods
Add tests for previously uncovered State methods:
- UpdateCephClustersForInstance
- UpdateBackupTasksForInstance
- UpdateReplicationJobsForInstance
- UpdateGuestSnapshotsForInstance

Improves internal/models coverage from 80.4% to 89.6%.
2025-12-02 16:20:11 +00:00
rcourtman
cbbb790284 test: Add more State method tests for coverage
Add tests for previously uncovered State methods:
- UpdatePhysicalDisks
- UpdateStorageForInstance
- UpdatePBSInstances / UpdatePBSInstance
- UpdatePMGInstances / UpdatePMGInstance

Improves internal/models coverage from 71.9% to 80.4%.
2025-12-02 16:17:04 +00:00
rcourtman
5b7e7a3133 test: Add tests for state connection health and backup methods
Add tests for previously uncovered State methods:
- SetConnectionHealth / RemoveConnectionHealth
- UpdatePBSBackups
- UpdatePMGBackups

Improves internal/models coverage from 67.0% to 71.9%.
2025-12-02 16:14:40 +00:00
rcourtman
36740bffe3 docs: Add godoc comments to more exported functions
Add missing godoc comments to:
- BuildGuestKey in alerts/alerts.go
- GenerateMockData in mock/generator.go
- NewDockerUpdater, NewAURUpdater in updates/adapter_installsh.go
- NewMockUpdater in updates/mock_updater.go
2025-12-02 16:03:57 +00:00
rcourtman
c019e4cd0e docs: Add godoc comments to exported functions
Add missing godoc comments to:
- NewRateLimiter and Allow in ratelimit.go
- SnapshotSyncStatus in temperature_proxy.go
- NewClient and GetVersion in pkg/pmg/client.go
2025-12-02 15:58:59 +00:00
rcourtman
25294da5d5 perf: Cache hostname lowercase in temperature proxy lookups
Pre-compute strings.ToLower(hostname) before loops that search for
matching PVE instances. Avoids repeated lowercasing in two functions.
2025-12-02 15:43:41 +00:00
rcourtman
6931fa34cd perf: Cache err.Error() in cluster node validation
Cache err.Error() result in two locations in config_handlers.go:
- TLS mismatch detection (3x calls to 1)
- Standalone node detection (2x calls to 1)
2025-12-02 15:41:18 +00:00
rcourtman
6a29e20b5e perf: Cache err.Error() in storage timeout error handling
Cache err.Error() result in two locations:
- monitor.go: storage query retry logic (2x calls to 1)
- monitor_polling.go: storage timeout handling (2x calls to 1)
2025-12-02 15:39:37 +00:00
rcourtman
5d2eca3660 perf: Cache err.Error() in disk monitoring error handling
Compute err.Error() once and reuse errStr instead of calling Error()
four times when checking disk monitoring error types.
2025-12-02 15:37:48 +00:00
rcourtman
9fc567dc2e perf: Cache lowercase RAID state in alerts processing
Compute strings.ToLower(array.State) once and reuse stateLower instead
of calling ToLower three times when checking RAID array status.
2025-12-02 15:35:32 +00:00
rcourtman
128dccf270 chore: Remove debug logging that exposed config JSON
Removed two DEBUG log statements that were logging full nodes config
JSON at Info level. This was verbose and potentially exposed sensitive
configuration data (credentials, tokens) in logs.
2025-12-02 15:32:02 +00:00
rcourtman
8f5b258424 perf: Cache lowercase error string in guest agent error handling
Compute strings.ToLower(errStr) once and reuse errStrLower instead of
calling ToLower three times on the same string in permission check.
2025-12-02 15:26:40 +00:00
rcourtman
f383cf8ddf perf: Remove redundant fmt.Sprintf in tempproxy contains function
The fmt.Sprintf("%v", s) call was pointless since s is already a string.
Removing this unnecessary conversion.
2025-12-02 15:24:40 +00:00
rcourtman
b47a04aa97 perf: Use strconv.Itoa instead of fmt.Sprintf for int conversion
strconv.Itoa is faster than fmt.Sprintf("%d", ...) because it doesn't
need to parse a format string. Changed 4 occurrences in monitoring
package where integers are converted to strings.
2025-12-02 15:21:41 +00:00
rcourtman
a965285738 perf: Pre-compile regexes in updates/version package
Move regex compilation from function bodies to package-level variables
to avoid recompilation when parsing version strings.

Affected regexes:
- semverRe: Matches semantic version format (X.Y.Z-prerelease+build)
- rcNumRe: Extracts RC number from prerelease strings

These are called multiple times during version comparison and update checks.
2025-12-02 15:14:15 +00:00
rcourtman
fd3c57377c perf: Pre-compile regexes in mdadm package
Move regex compilation from function bodies to package-level variables
to avoid recompilation on each RAID array check during polling cycles.

Affected regexes:
- mdDeviceRe: Matches md device names in /proc/mdstat
- slotRe: Matches device slot lines in mdadm --detail output
- speedRe: Extracts rebuild speed from /proc/mdstat
2025-12-02 15:11:04 +00:00
rcourtman
ac4e48fa4a fix: Properly show disabled storage status from Proxmox
Storage that is disabled in Proxmox (Datacenter > Storage > enabled=no)
was incorrectly showing as "available" in Pulse. The issue was that
Enabled/Active fields defaulted to true and were never set to false
from the per-node API response.

Now the model correctly initializes Enabled/Active from the Proxmox
per-node storage API response, and the status determination prioritizes
checking the disabled state first.

Related to #796
2025-12-02 15:03:40 +00:00
rcourtman
be23624592 refactor: Remove unreachable dead code branches
- parseDetail: Remove len(parts) != 2 check after SplitN(line, ":", 2)
  when we already verified line contains ":" (SplitN always returns 2)

Function improved from 96.4% to 98.1% coverage.
2025-12-02 15:00:26 +00:00
rcourtman
a0a473ada8 refactor: Remove unreachable dead code branches
- checkOrigin: Remove redundant same-origin check at line 155 that was
  already handled at line 116 (origin == requestOrigin)

Function improved from 95.1% to 97.4% coverage.
2025-12-02 14:56:35 +00:00
rcourtman
a647ffb219 refactor: Remove unreachable dead code branches
- Host.ToFrontend: Remove redundant h.DisplayName check after struct
  initialization already set host.DisplayName = h.DisplayName

Function now at 100% coverage.
2025-12-02 14:53:07 +00:00
rcourtman
c1d98e6aa8 refactor: Remove unreachable dead code branches
- errors.isRetryable: Convert final case to default (all ErrorType values covered)
- scheduler.SelectInterval: Remove bounds checks after mathematical computation
  that guarantees target ∈ [min, max]

Both functions now at 100% coverage.
2025-12-02 14:48:57 +00:00
rcourtman
2453198f62 refactor: Remove unreachable dead code branches
- firstForwardedValue: strings.Split always returns at least one element
- shouldRunBackupPoll: remaining is always >= 1 by math
- convertContainerDiskInfo: lowerLabel is never empty for non-rootfs

All three functions now at 100% coverage.
2025-12-02 14:41:53 +00:00
rcourtman
3d9c4f6da1 test: Add UpdateStatus not found test for notifications package 2025-12-02 14:26:17 +00:00
rcourtman
3f8455dfa9 test: Add DispatchDue enqueue error test for monitoring package 2025-12-02 14:21:25 +00:00
rcourtman
a5f263cf79 test: Add NewNotificationQueue invalid path test for notifications package 2025-12-02 14:17:05 +00:00
rcourtman
eb302e6d4e test: Add SessionStore.load legacy format tests for API package 2025-12-02 14:12:32 +00:00
rcourtman
e7fa996e03 test: Add CSRFTokenStore.load format tests for API package
Cover legacy JSON format migration and current format with nil/expired
entries. Improves load function coverage from 67.9% to 100%.
2025-12-02 14:07:00 +00:00
rcourtman
daafb51493 test: Add CheckAuth tests for API package
Add tests for sliding expiration session validation and no-auth
configured scenarios. These test explicit paths for better coverage
documentation even though they were already exercised indirectly.
2025-12-02 13:57:23 +00:00
rcourtman
2f203bf2d9 test: Add CheckCSRF valid token test for 100% coverage
Test the success path where a valid CSRF token is provided with a
matching session. This covers the final branch in CheckCSRF.
2025-12-02 13:51:27 +00:00
rcourtman
e1ff07fd9f test: Add singleAlertTemplate type coverage tests
Cover io type (formats as "I/O") and custom type (uses titleCase)
branches that were previously untested in the email template.
2025-12-02 13:45:49 +00:00
rcourtman
4797e8d76a test: Add ValidateWebhookURL edge case tests for notifications package
Cover empty URL, invalid scheme, missing hostname, cloud metadata
endpoints, loopback variants, and IPv6 link-local addresses.
2025-12-02 13:41:34 +00:00
rcourtman
50769e12ea test: Add HandleSetupScript PBS path tests for API package
Cover the PBS script generation branch that was previously untested.
Verifies PBS-specific content, auth token handling, and placeholder host.
2025-12-02 13:36:23 +00:00
rcourtman
28533975f6 test: Add CheckAuth tests for API package
Cover proxy auth headers, OIDC session validation, and session cookie
paths that were previously untested.
2025-12-02 13:32:21 +00:00
rcourtman
d058b2ab96 test: Add CheckProxyAuth tests for API package
Add comprehensive direct tests for the CheckProxyAuth function covering:
- Not configured (returns false)
- Invalid secret (returns false)
- Missing secret header (returns false)
- Valid secret without user header configured (returns true, admin)
- Missing user header when configured (returns false)
- Valid auth with username (returns true with username)
- Role checking with empty roles header (defaults to admin)
- Role checking with admin role present (returns admin=true)
- Role checking without admin role (returns admin=false)
- Custom role separator (comma instead of pipe)
- Role with whitespace (trimmed correctly)

Coverage: CheckProxyAuth 89.3% → 100%
2025-12-02 13:28:02 +00:00
rcourtman
d219d9965d test: Add ValidateSession tests for API package
Add comprehensive tests for the ValidateSession wrapper function covering:
- Non-existent token (returns false)
- Empty token (returns false)
- Valid token (returns true)
- Expired token (returns false)

The ValidateSession function is a simple wrapper around the SessionStore's
ValidateSession method, but having direct tests ensures the wrapper is
exercised and documents its expected behavior.

Coverage: ValidateSession 0% → 100%
2025-12-02 13:22:45 +00:00
rcourtman
ef6a370144 test: Add ensureScope tests for API package
Add comprehensive tests for the ensureScope function covering:
- Empty scope parameter (always allows access)
- No token in context (session-based request, allows access)
- Token with matching scope (allows access)
- Token with multiple scopes including required one (allows access)
- Token missing required scope (rejects with 403)
- Token with empty scopes (defaults to wildcard, allows access)
- Rejection returns proper JSON response format

Coverage: ensureScope 0% → 100%
Coverage: API package 32.1% → 32.2%
2025-12-02 13:19:11 +00:00
rcourtman
301338d800 test: Add sendResolvedEmail tests for notifications package
Add comprehensive tests for the sendResolvedEmail function covering:
- Empty alert list (returns error)
- Nil alert list (returns error)
- All nil alerts (returns error from content builder)
- Single alert (exercises email sending path)
- Multiple alerts (tests grouped notification)
- Mixed nil and valid alerts (filters correctly)
- Zero resolved time (handles gracefully)

Also improves buildResolvedNotificationContent coverage as a collateral
benefit since sendResolvedEmail calls it internally.

Coverage: sendResolvedEmail 0% → 100%
Coverage: buildResolvedNotificationContent → 100%
Coverage: notifications package 58.3% → 58.6%
2025-12-02 13:14:22 +00:00
rcourtman
d970055532 test: Add RequireAuth tests for API package
Add comprehensive tests for the RequireAuth middleware covering:
- No auth configured (allows access by design)
- API-only mode (rejects requests without token)
- API-only mode (accepts valid X-API-Token)
- Basic auth with invalid credentials
- Basic auth JSON vs plain text error responses
- Valid basic auth (allowed)
- Proxy auth (allowed)
- Proxy auth with invalid secret (rejected)
- Bearer token with basic auth configured (allowed)
- Invalid Bearer token (rejected)

Coverage: RequireAuth 7.1% → 78.6%
Coverage: CheckAuth 66.9% → 69.1%
Coverage: API package 31.9% → 32.1%
2025-12-02 13:09:48 +00:00
rcourtman
1073c511b3 test: Add RequireAdmin tests for API package
Add comprehensive tests for the RequireAdmin middleware covering:
- No auth configured (allows access by design)
- API-only mode (rejects requests without token)
- Basic auth with invalid credentials
- Proxy auth with admin role (allowed)
- Proxy auth with non-admin role (forbidden)
- Proxy auth with invalid secret (unauthorized)
- Proxy auth without role header (defaults to admin)
- Proxy auth with custom role separator
- Proxy auth with spaces in roles (trimmed)
- Basic auth authenticated users (allowed as admin)
- JSON vs plain text error responses based on path/Accept header

Also improves CheckProxyAuth coverage as a side effect.

Coverage: RequireAdmin 20.8% → 87.5%
Coverage: CheckProxyAuth 0.0% → 89.3%
Coverage: API package 30.9% → 31.9%
2025-12-02 13:06:06 +00:00
rcourtman
71e15886ed test: Add isRequestAuthenticated tests for API package
Add comprehensive tests for the isRequestAuthenticated function covering:
- Nil inputs (config, request, both)
- Basic auth (valid, invalid password, invalid username, malformed base64, missing colon)
- API token via X-API-Token header
- API token via Bearer authorization header (case insensitive)
- Invalid/empty/whitespace API tokens
- No auth configured scenarios
- Empty session cookie handling

Coverage: isRequestAuthenticated 26.1% → 82.6%
Coverage: API package 30.7% → 30.9%
2025-12-02 12:59:18 +00:00
rcourtman
cf6149e884 test: Add CheckCSRF tests for API package
Add comprehensive tests for the CheckCSRF function covering:
- Safe methods (GET, HEAD, OPTIONS) bypass
- API token authentication bypass
- Basic auth bypass
- No session cookie handling
- Missing CSRF token rejection with new token issuance
- Invalid CSRF token rejection with new token issuance
- CSRF token from FormValue
- Unsafe methods (POST, PUT, DELETE, PATCH) enforcement

Coverage: CheckCSRF 32.0% → 96.0%
Coverage: API package 30.5% → 30.7%
2025-12-02 12:53:32 +00:00
rcourtman
5deca850d4 test: Add saveHistoryWithRetry tests for alerts package
Add comprehensive tests for the saveHistoryWithRetry function covering:
- Backup file creation from existing history
- Empty history serialization
- Single retry success
- Read-only directory failure with retries
- Concurrent saves with serialization via saveMu
- Snapshot isolation during save

Coverage: saveHistoryWithRetry 58.6% → 86.2%
Coverage: alerts package 87.4% → 87.8%
2025-12-02 12:49:27 +00:00
rcourtman
cd33c577f8 test: Add resolvePublicURL tests for API package
Add comprehensive tests for the resolvePublicURL function covering:
- Configured PublicURL (simple, trailing slashes, ports, whitespace)
- Request-derived URL (HTTP, HTTPS via TLS, X-Forwarded-Proto)
- No host fallback (with/without frontend port)
- Nil request handling

Coverage: resolvePublicURL 12.5% → 100%
Coverage: API package 30.3% → 30.5%
2025-12-02 12:45:04 +00:00
rcourtman
80739cce94 test: Add rescheduleTask tests for monitoring package
Add comprehensive tests for the rescheduleTask function covering:
- Nil taskQueue handling (early return)
- Successful task outcome (regular rescheduling)
- Transient failure with backoff
- Non-transient failure routing to dead letter queue
- Exceeded retry attempts routing to dead letter queue
- No outcome uses default interval
- PBS and PMG instance type intervals
- Adaptive polling max interval capping
- Existing interval preservation

Coverage: rescheduleTask 32.1% → 58.9%
Coverage: monitoring package 52.8% → 53.5%
2025-12-02 12:38:50 +00:00
rcourtman
2842dded28 test: Add getGuestThresholds tests for alerts package
Add comprehensive tests for the getGuestThresholds function covering:
- Default threshold application
- Guest-specific overrides
- Custom rule filter matching
- Override precedence over custom rules
- Priority-based rule selection
- Disabled rules handling
- Disabled override handling
- DisableConnectivity propagation from overrides and rules
- Legacy CPU threshold conversion
- Legacy ID migration (clustered and standalone VMs)
- Container type support
- All metric thresholds application

Coverage: getGuestThresholds 40.2% → 77.6%
2025-12-02 12:35:07 +00:00
rcourtman
a4e3fe65b8 test: Add preserveAlertState, checkPMGQuarantineBacklog, LoadActiveAlerts tests
Add comprehensive tests for three low-coverage functions:
- preserveAlertState: nil handling, state preservation from existing alerts,
  ackState fallback, new alert handling
- checkPMGQuarantineBacklog: nil quarantine handling, warning/critical
  thresholds, growth rate alerts, alert updates, virus quarantine
- LoadActiveAlerts: missing file, valid file loading, old alert filtering,
  old acknowledged alert filtering, ack state restoration, invalid JSON,
  duplicate alert handling

Coverage improvements:
- preserveAlertState: 63.6% → 100%
- checkPMGQuarantineBacklog: 12.9% → 100%
- checkQuarantineMetric: 0% → 93.1%
- LoadActiveAlerts: 26.2% → 80.0%
- Alerts package: 83.5% → 86.6%
2025-12-02 12:22:14 +00:00
rcourtman
2133b25bb7 test: Add performCleanup tests for notifications queue
Add 4 tests covering the performCleanup function:
- cleanup removes old completed entries (>7 days)
- cleanup removes old DLQ entries (>30 days)
- cleanup removes old audit logs (>30 days)
- cleanup with empty database (no panic)

performCleanup coverage: 0% → 87.0%
Notifications package: 57.3% → 58.3%
2025-12-02 12:16:55 +00:00
rcourtman
a6f4fc7bc0 test: Add GetQueue, addWebhookDelivery, GetWebhookHistory tests
Tests for NotificationManager accessor and helper functions.
Covers queue retrieval, webhook delivery tracking, history trimming
to max 100 entries, and copy-on-read semantics. Notifications 56.6%→57.3%.
2025-12-02 12:10:29 +00:00
rcourtman
f184174f2b test: Add HistoryManager tests for alerts package
New history_test.go with 24 tests covering GetStats, getFileSize,
AddAlert, GetHistory, GetAllHistory, RemoveAlert, ClearAllHistory,
cleanOldEntries, saveHistory, loadHistory. Alerts package 83.6%→84.0%.
2025-12-02 12:06:53 +00:00
rcourtman
fc96ba7b69 test: Add WaitNext and key() tests for TaskQueue
Tests cover context cancellation, immediately due tasks, waiting for
future tasks, empty queue timeout, and multiple task priority ordering.
WaitNext coverage 0%→92.3%, key() 0%→100%. Monitoring package 52.3%→52.9%.
2025-12-02 12:02:13 +00:00
rcourtman
e63c297ca3 test: Add BuildPlan, FilterDue, DispatchDue, LastScheduled tests
Comprehensive tests for AdaptiveScheduler methods covering empty inventory,
single/multiple instances, priority ordering, interval clamping, task caching,
nil receiver handling, and task filtering. Monitoring package 51.4%→52.3%.
2025-12-02 11:59:37 +00:00
rcourtman
ea120b78d7 test: Add RecordNodeResult, RecordQueueWait, SetQueueDepth tests
Improves metrics.go coverage to 100%. Added comprehensive tests for
node-level poll metrics, queue wait time recording, and queue depth
setting with edge cases (nil receiver, negative values, label normalization).
2025-12-02 11:57:05 +00:00
rcourtman
47d0fb5a44 test: Add dispatchAlert tests (55.6%→77.8%)
Add TestDispatchAlert with 8 test cases covering:
- Returns false when onAlert callback is nil
- Returns false when alert is nil
- Returns false when activation state is pending
- Returns false when activation state is snoozed
- Returns false for monitor-only alerts
- Dispatches synchronously when async is false
- Dispatches asynchronously when async is true
- Clones alert before dispatch

Alerts package coverage: 83.4%→83.5%
2025-12-02 11:47:24 +00:00
rcourtman
b3bf63d3c7 test: Add CheckStorage tests (52.4%→92.9%)
Add comprehensive TestCheckStorageComprehensive with 11 test cases covering:
- Returns early when alerts disabled
- DisableAllStorage clears existing usage and offline alerts
- Override with Disabled clears alerts
- Usage threshold checking
- Override threshold applied correctly
- Skips usage check when offline/unavailable/zero usage
- Offline status creates alert after confirmations
- Unavailable status creates alert
- Clears offline alert when back online

Alerts package coverage: 82.4%→83.4%
2025-12-02 11:43:56 +00:00
rcourtman
9127b27f4c test: Add CheckPMG tests (0%→100%)
Add comprehensive TestCheckPMGComprehensive with 9 test cases covering:
- Returns early when alerts disabled
- DisableAllPMG clears all PMG alert types (queue-total, queue-deferred,
  queue-hold, oldest-message, offline)
- Override with Disabled clears alerts
- DisableAllPMGOffline clears offline alert
- Offline status creates alert after confirmations
- Connection health error triggers offline alert
- Connection health unhealthy triggers offline alert
- Clears offline alert when back online
- Skips metrics when PMG is offline

Alerts package coverage: 81.5%→82.4%
2025-12-02 11:40:53 +00:00
rcourtman
073d5ffa55 fix: Use --ctid instead of --standalone --http-mode in quick-setup command
The quick-setup command for temperature monitoring was generating
--standalone --http-mode which is meant for Docker deployments. This
confused users trying to set up multi-server Proxmox monitoring.

Now uses --ctid which works for both local and remote Proxmox hosts.
The installer detects when the container doesn't exist locally and
installs in "host monitoring only" mode automatically.

If we can determine the actual CTID from the host proxy summary,
we use it; otherwise we show <PULSE_CTID> for the user to replace.

Related to #785
2025-12-02 11:38:47 +00:00
rcourtman
9f174be20b test: Add CheckPBS tests (0%→98.3%)
Add comprehensive TestCheckPBSComprehensive with 12 test cases covering:
- Returns early when alerts disabled
- DisableAllPBS clears existing CPU/memory/offline alerts
- Override with Disabled clears alerts
- DisableAllPBSOffline clears offline alert
- CPU threshold checking when online
- Memory threshold checking when online
- Skips metrics when PBS is offline
- Override thresholds applied correctly
- Offline status creates alert after confirmations
- Connection health error triggers offline alert
- Connection health unhealthy triggers offline alert
- Clears offline alert when back online

Alerts package coverage: 80.0%→81.5%
2025-12-02 11:38:36 +00:00
rcourtman
13f5ca4018 test: Add CheckHost tests (49.6%→98.3%)
Add comprehensive TestCheckHostComprehensive with 17 test cases covering:
- Empty host ID early return
- Alerts disabled early return
- DisableAllHosts clears existing alerts
- Override with Disabled clears alerts
- CPU/Memory/Disk threshold nil clears alerts
- RAID degraded/rebuilding/healthy states
- RAID with failed devices triggers critical alert
- RAID resync triggers rebuilding alert
- Existing RAID alert not duplicated (preserves start time)
- Override thresholds applied correctly
- Multiple disks handling
- Offline alert cleared when host comes online
- Tags included in metadata

Alerts package coverage: 78.6%→80.0%
2025-12-02 11:34:20 +00:00
rcourtman
f4e3f625e7 test: Add CheckGuest tests (41.4%→97.4%)
Cover all CheckGuest branches:
- Early return when alerts disabled
- Early return when all guests disabled
- VM and Container type handling
- Unsupported guest type returns early
- pulse-no-alerts tag suppresses alerts
- Stopped guest triggers powered-off check
- DisableAllGuestsOffline clears tracking
- Paused guest clears powered-off alert
- Non-running guest clears metric alerts
- Running guest clears powered-off alert
- Disabled thresholds clear existing alerts
- CPU, memory, disk metric checks
- Individual disk checks (mountpoint, device, index fallback)
- Skips disk with zero total or negative usage
- I/O metrics (diskRead, diskWrite, networkIn, networkOut)
- pulse-relaxed tag applies relaxed thresholds

Alerts package coverage: 76.0%→78.6%
2025-12-02 11:27:32 +00:00
rcourtman
4ceb4f9a35 test: Add CheckNode tests (31%→100%)
Cover all CheckNode branches:
- Early return when alerts disabled
- DisableAllNodes clears existing alerts
- DisableNodesOffline clears tracking
- Offline/connection error/failed triggers offline check
- Online node clears offline alert
- Online node triggers metric checks
- Offline node skips metric checks
- Override thresholds applied correctly
- Temperature with package temp and max fallback
- Temperature skipped when unavailable/nil/no threshold
- Memory and disk metric checks

Alerts package coverage: 75.2%→76.0%
2025-12-02 11:24:04 +00:00
rcourtman
9791fc07cb test: Add suppressGuestAlerts and guestHasMonitorOnlyAlerts tests
Coverage improvements:
- suppressGuestAlerts: 37% -> 96.3%
- guestHasMonitorOnlyAlerts: 40% -> 90%

Tests cover:
- No alerts returns false
- Exact ResourceID match clears
- Prefix match (e.g., "vm100/disk1") clears
- All auxiliary maps cleared (pending, recent, suppressed, rateLimit)
- Multiple alerts cleared
- Monitor-only detection via metadata (bool and string types)
2025-12-02 11:14:30 +00:00
rcourtman
f661c5a6a8 test: Add applyThresholdOverride tests
Coverage for applyThresholdOverride: 50% -> 93.2%

Tests cover:
- Empty override returns base unchanged
- Disabled/DisableConnectivity flag overrides
- Modern CPU threshold override
- Legacy CPU threshold conversion
- Modern takes precedence over legacy
- Multiple metrics override
- Note override, clearing, trimming
- All legacy metric types (Memory, Disk, etc.)
- Temperature and Usage override
- ensureHysteresisThreshold Clear value filling
2025-12-02 11:08:58 +00:00
rcourtman
ed118c973f test: Add ClearActiveAlerts test with existing alerts
Coverage for ClearActiveAlerts: 16% -> 92%

Tests verify all internal maps are properly cleared when alerts exist:
- activeAlerts, pendingAlerts, recentAlerts
- suppressedUntil, alertRateLimit
- nodeOfflineCount, offlineConfirmations
- dockerOfflineCount, dockerStateConfirm
- ackState, recentlyResolved
2025-12-02 11:01:54 +00:00
rcourtman
219c568649 test: Add CheckDiskHealth normal path tests
Coverage for CheckDiskHealth: 51% -> 98%

Tests cover:
- Healthy disk (PASSED/OK) creates no alert
- Failed non-Samsung disk creates critical alert
- Alert cleared when disk health recovers
- Low wearout (<10%) creates warning alert
- Wearout alert updates on subsequent checks
- Wearout alert cleared when wearout >= 10%
- Empty/UNKNOWN health creates no alert
2025-12-02 10:59:48 +00:00
rcourtman
c04d559700 test: Add Docker container restart loop alert tests
Coverage for checkDockerContainerRestartLoop: 53.5% -> 95.3%

Tests cover:
- First check initializes tracking without alert
- Stable restart count doesn't alert
- Restarts under threshold (<=3) don't alert
- Restart loop threshold (>3) triggers critical alert
- Recovery after window expires clears alert
- Incremental restart accumulation
- Alert StartTime preservation on updates
2025-12-02 10:54:51 +00:00
rcourtman
b6624e32cf test: Add Docker container health and OOM kill alert tests
Coverage improvements:
- checkDockerContainerHealth: 21.1% -> 94.7%
- checkDockerContainerOOMKill: 19.2% -> 96.2%

Tests cover:
- Health states (healthy, empty, none, starting, unhealthy, degraded)
- Health alert recovery when container becomes healthy
- OOM kill detection (exit code 137)
- OOM alert deduplication (repeated 137 doesn't re-alert)
- OOM alert clearing when container recovers or exits with different code
2025-12-02 10:49:42 +00:00
rcourtman
1ed8b18c12 fix: Deadlock in CancelByAlertIDs and add tests
Fixed deadlock where CancelByAlertIDs held nq.mu.Lock() and then called
UpdateStatus() which also tried to acquire the same lock. Now uses
direct SQL while holding the lock.

Tests added for CancelByAlertIDs:
- No matching notifications (notification stays pending)
- Matching notification cancelled
- Multiple alerts with partial match (any match cancels)

Coverage: CancelByAlertIDs 65.7% -> 81.1%
2025-12-02 10:40:07 +00:00
rcourtman
5ef4587767 test: Add queue IncrementAttempt and GetQueueStats tests
Coverage improvements:
- IncrementAttempt: 0% -> 85.7%
- GetQueueStats: 0% -> 87.5%

Tests verify attempt counter increments correctly and queue stats
aggregate notification counts by status.
2025-12-02 10:31:00 +00:00
rcourtman
e27717f1a2 test: Add secure webhook client tests for redirect handling
Tests SSRF protection in webhook client:
- Redirect limit enforcement (max 3)
- Blocking redirects to private networks (10.x, 192.168.x, 172.16.x)
- Blocking redirects to link-local addresses (169.254.x)
- Allowing valid redirects to allowlisted servers

Coverage: createSecureWebhookClient 18.2% -> 100%
2025-12-02 10:26:34 +00:00
rcourtman
9118344355 fix: Isolate alerts tests with temp directories to prevent flaky failures
Tests using NewManager() were sharing /etc/pulse/alerts, causing race
conditions when running in parallel. Added newTestManager(t) helper that
creates isolated temp directories for each test.
2025-12-02 10:21:03 +00:00
rcourtman
350d21c80d test: Add diagnostics function tests for error handling
Add comprehensive tests for untested diagnostics functions:

- fingerprintPublicKey: 14 test cases covering empty/whitespace input,
  invalid key formats, truncated/malformed keys, and valid ED25519 keys

- countLegacySSHKeys: 8 test cases covering non-existent directories,
  empty directories, files without id_ prefix, multiple key types,
  and directory filtering (subdirectories not counted)

- resolveUserName: 4 test cases covering UID 0 (root), current user,
  non-existent UID fallback, and max uint32 boundary

- resolveGroupName: 4 test cases covering GID 0 (root/wheel), current
  group, non-existent GID fallback, and max uint32 boundary

Coverage: fingerprintPublicKey 0% -> 100%, countLegacySSHKeys 0% -> 100%,
resolveUserName 0% -> 100%, resolveGroupName 0% -> 100%
2025-12-02 03:39:52 +00:00
rcourtman
b053a2848b fix: Update TestPublicURLDetectionUsesForwardedHeaders for proxy hardening
The test was failing after commit d6cbfc23 added security hardening
that requires authentication and trusted proxy configuration for
X-Forwarded-* headers to be read during public URL detection.

- Add API token authentication to the test request
- Configure 127.0.0.1 as trusted proxy for the test
- Add export_test.go with ResetTrustedProxyConfigForTests() to allow
  external tests to reset the trusted proxy configuration
2025-12-02 03:16:52 +00:00
rcourtman
201f613ac3 test: Add sendNotificationsDirect email and apprise tests
Test email enabled and apprise enabled code paths.
Coverage: 66.7% → 100.0%
2025-12-02 03:08:35 +00:00
rcourtman
37ec6c3a93 test: Add scanNotification DLQ timestamp test
Test scanNotification with CompletedAt and LastAttempt populated
via DLQ path. Coverage: 61.9% → 81.0%
2025-12-02 03:04:37 +00:00
rcourtman
ae54cfad8c test: Add processNotification tests for all code paths
Test cancelled notification, no processor, success, and failure paths.
Coverage: 50.0% → 80.0%
2025-12-02 03:01:17 +00:00
rcourtman
ca7da7c18f test: Add CancelByAlertIDs empty input and no-match tests
Test early return for empty alertIDs and happy path with empty queue.
Coverage: 62.9% → 65.7%
2025-12-02 02:58:10 +00:00
rcourtman
7a1c9ff53f test: Add SaveNodesConfigAllowEmpty test and document deadlock
Add test for SaveNodesConfigAllowEmpty which permits explicit
deletion of all nodes. Document deadlock bug in saveNodesConfig
where empty config protection tries to call LoadNodesConfig
while holding write lock.
2025-12-02 02:54:49 +00:00
rcourtman
d0fc62d45f test: Add LoadNodesConfig PBS/PMG migration tests
Test PBS and PMG token clearing and host normalization migrations.
Coverage: 61.4% → 76.3%
2025-12-02 02:46:47 +00:00
rcourtman
6d1eb44f7b test: Add LoadEmailConfig encrypted round-trip test
Test Save+Load with encryption enabled to cover success path
after decryption. Coverage: 58.8% → 88.2%
2025-12-02 02:43:49 +00:00
rcourtman
3c1ae26166 test: Add LoadWebhooks migration and error path tests
Cover unencrypted .enc file migration fallback and legacy file
with invalid JSON graceful handling.
Coverage: 67.9% → 75.0%
2025-12-02 02:39:04 +00:00
rcourtman
9eb7c59a4d test: Add SaveAlertConfig host defaults normalization tests
Cover nil HostDefaults initialization (CPU/Memory/Disk) and clear
threshold computation when clear=0 but trigger is set.
Coverage: 68.2% → 75.3%
2025-12-02 02:37:01 +00:00
rcourtman
9027ab359b test: Add StageFile transaction tests
Cover success path, already committed error, replacing existing staged
file, and empty basename edge case.
2025-12-02 02:34:20 +00:00
rcourtman
6cb12bd048 test: Add ImportConfig error path tests
Cover empty passphrase, invalid base64, wrong passphrase decryption
failure, and invalid JSON content error paths.
Coverage: 66.7% → 73.7%
2025-12-02 02:32:06 +00:00
rcourtman
0d482f9d63 test: Add DiscoveryConfig UnmarshalJSON tests
Cover invalid JSON error path, modern field parsing, legacy field
parsing, and empty object default handling.
Coverage: 60% → 88.9%
2025-12-02 02:29:38 +00:00
rcourtman
071ccc44ec test: Add ProcessQueuedNotification error path tests
Cover invalid config JSON for email, webhook, and apprise types,
plus unknown notification type handling.
Coverage: 63.3% → 83.3%
2025-12-02 02:26:25 +00:00
rcourtman
7a63cf3be9 test: Add sendNotificationsDirect tests
Cover disabled paths: all channels disabled, webhook disabled,
multiple webhooks with mixed enabled state.
Coverage: 44.4% → 66.7%
2025-12-02 02:21:46 +00:00
rcourtman
ea04e4faae test: Add sendHTMLEmailWithError tests
Cover email sending paths: empty To uses From as recipient,
nil email manager creates new one, existing manager config updates.
Coverage: 57.9% → 89.5%
2025-12-02 02:19:49 +00:00
rcourtman
9768316e07 test: Add sendGroupedApprise error path tests
Add 3 tests for Apprise notification error handling:
- No alerts returns error
- Not enabled returns error
- Empty payload (all nil alerts) returns error

Coverage: 56.2% → 75.0%
2025-12-02 02:14:13 +00:00