- Add cluster-aware guest ID generation (clusterName-VMID instead of instanceName-VMID)
to prevent duplicate VMs/containers when multiple cluster nodes are monitored
- Add cluster deduplication at registration time - when a node is added that belongs
to an already-configured cluster, merge as endpoint instead of creating duplicate
- Add startup consolidation to automatically merge duplicate cluster instances
- Change host agent token binding from agent GUID to hostname, allowing:
- Multiple host agents to share a token (each bound by hostname)
- Agent reinstalls on same host without token conflicts
- Remove 12-character password minimum requirement
- Remove emoji from auto-registration success message
- Fix grouped view node lookup to support both cluster-aware node IDs
(clusterName-nodeName) and legacy guest grouping keys (instance-nodeName)
Fixes duplicate guests appearing when agents are installed on multiple
cluster nodes. Also improves multi-agent UX by allowing shared tokens.
When a host agent registers, it now searches for a PVE node with a
matching hostname and links them together. Similarly, when PVE nodes
are discovered, they check for existing host agents with matching hostnames.
This prevents the confusion of seeing duplicate entries when users install
agents on PVE cluster nodes that were already discovered via the cluster API.
- Added LinkedHostAgentID field to Node struct
- Added LinkedNodeID/LinkedVMID/LinkedContainerID fields to Host struct
- Added findLinkedProxmoxEntity() to match by hostname (with domain stripping)
- Updated UpdateNodesForInstance() to preserve and auto-set links
When no auth is configured (fresh install), CheckAuth allows all requests.
This creates a race condition where existing agents from a previous setup
can report data before the wizard completes security configuration.
This fix clears all host agents and docker hosts when /api/security/quick-setup
is called, ensuring the wizard shows a clean state after security is configured.
Added:
- State.ClearAllHosts() - removes all host agents
- State.ClearAllDockerHosts() - removes all docker hosts
- Monitor.ClearUnauthenticatedAgents() - clears both and resets token bindings
- Call to ClearUnauthenticatedAgents() in handleQuickSecuritySetupFixed()
- Add GET /api/metrics-store/history endpoint for querying SQLite-backed metrics
- Support flexible time ranges: 1h, 6h, 12h, 24h, 7d, 30d, 90d
- Return aggregated data with min/max values for longer time ranges
- Add TypeScript types and ChartsAPI.getMetricsHistory() client method
This enables frontend charts to visualize long-term trends using the
tiered retention system (raw → minute → hourly → daily averages).
- Add MetricsRetentionRawHours, MetricsRetentionMinuteHours, MetricsRetentionHourlyDays, MetricsRetentionDailyDays to SystemSettings
- Wire settings from system.json through Config to metrics store initialization
- Set sensible defaults: Raw=2h, Minute=24h, Hourly=7d, Daily=90d
- Log active retention values on startup for transparency
Users can now customize how long metrics are stored at each aggregation tier.
1. resources/store.go: Implement sorting in Query.Execute()
- Added sortResources function with support for common fields
- Supports: name, type, status, cpu, memory, disk, last_seen
- Both ascending and descending order supported
2. ai/service.go: Implement hasAgentForTarget properly
- Now maps target to specific agent based on hostname/node
- Uses ResourceProvider lookup for container→host mapping
- Supports cluster peer routing for Proxmox clusters
- Properly handles single-agent vs multi-agent scenarios
- Add sortable table headers for Pod and Deployment views
- Click column headers to toggle sort direction
- Sort state persists across sessions
- Add namespace dropdown filter for Pods/Deployments views
- Auto-populates from available namespaces
- Include namespace filter in reset and active filters check
The set_resource_url tool had an incorrect example ID format ('pve1-delly-101')
which caused the AI to save URLs with wrong IDs that didn't match the actual
guest IDs used by Pulse ('instance-VMID' format like 'delly-150').
This fix updates the tool description to clearly document the correct format,
so URLs saved by the AI will now properly appear in the dashboard.
Backend:
- Seed OCI classification from previous state so containers never
'downgrade' to LXC if config fetching intermittently fails
- Prevent type regression in recordGuestSnapshot when OCI was previously detected
- Move metrics zeroing before snapshot recording for cleaner flow
Frontend:
- Add isOCIContainer() memo that checks both type and isOci flag
- Use isOCI helper in Dashboard.tsx for AI context building
- Include oci-container type in useResources container conversion
- Preserve isOci and osTemplate fields through legacy conversion
This ensures OCI containers retain their classification even when
Proxmox API permissions or transient errors prevent config reads.
- Refactored enrichContainerMetadata to not return early when container is stopped
- Status API calls are still skipped for stopped containers (as expected)
- Config fetch now runs regardless of status, enabling OCI detection
- Added test for OCI detection on stopped containers
Discovered: Proxmox 9.1 requires VM.Config.Options permission to read
OCI container configs (not just VM.Audit). Document this in setup guides.
- Added isOCIContainerByConfig() to detect OCI containers by:
- Presence of 'entrypoint' field (only OCI containers have this)
- Combination of ostype=unmanaged, cmode=console, and lxc.signal.halt
- This is needed because Proxmox doesn't persist ostemplate after creation
- Now supports detection of already-created OCI containers (like the test alpine container)
- Frontend: Add ociImage memo to extract clean image name from osTemplate
- Frontend: Show OCI image name in type badge tooltip
- Frontend: Display OCI image in OS column when no guest agent info available
- Frontend: Include ociImage in AI context data for selected OCI containers
- Backend: Differentiate OCI containers as 'oci_container' type in AI context
- Backend: Add Metadata field to ResourceContext for extensibility
- Backend: Include oci_image in container metadata for AI analysis
- Backend: Update section heading to 'LXC/OCI Containers' in AI context
This follows Docker container patterns to avoid duplicating work.
- Backend: Add IsOCI and OSTemplate fields to Container model
- Backend: Add extractContainerOSTemplate() and isOCITemplate() detection functions
- Backend: Detect OCI containers via ostemplate config and set type to 'oci'
- Frontend: Add isOci and osTemplate to Container interface
- Frontend: Add 'oci-container' to ResourceType with distinct purple badge
- Frontend: Update Dashboard filters to include OCI containers with LXC
- Tests: Add comprehensive unit tests for OCI detection logic
OCI containers are detected by checking the ostemplate for patterns like:
- oci: prefix (e.g., oci:docker.io/library/alpine:latest)
- docker: prefix (e.g., docker:nginx:latest)
- Known registry URLs (docker.io, ghcr.io, gcr.io, quay.io, etc.)
- Local templates with oci- or oci_ filename patterns
- Add DOMPurify sanitization for AI chat markdown rendering (XSS fix)
- Configure DOMPurify to add target=_blank and rel=noopener to links
- Update system prompt to align with command approval policy
- Clarify safe vs destructive commands in prompt
- Improve patrol auto-fix mode guidance with safe operation list
- Add verification requirements for auto-fix actions
- Update observe-only mode to be clearer about read-only restrictions
Add configurable model specifically for automatic remediation actions:
Backend (internal/config/ai.go):
- Add AutoFixModel field to AIConfig
- Add GetAutoFixModel() getter with fallback chain:
AutoFixModel -> PatrolModel -> Model
Frontend (AISettings.tsx, types/ai.ts):
- Add auto_fix_model to AISettings types
- Add Auto-Fix Model dropdown (only shows when patrol_auto_fix enabled)
- Falls back to patrol model if not set
API (ai_handlers.go):
- Add auto_fix_model to response and update request
- Handle saving/loading the new field
Rationale:
- Auto-fix takes real actions, may warrant a more capable model
- Patrol observation can use cheaper models for cost savings
- Gives users granular control over model costs vs reliability
- Model hierarchy: Chat > AutoFix > Patrol > Default
Create internal/ai/correlation package:
1. Correlation Detector (detector.go):
- Tracks events across resources
- Detects when events on one resource follow events on another
- Calculates average delay between correlated events
- Confidence scoring based on occurrence count
- Persists to ai_correlations.json
2. Features:
- GetCorrelations() - All detected relationships
- GetCorrelationsForResource() - Relationships for one resource
- GetDependencies() - What resources depend on this one
- GetDependsOn() - What this resource depends on
- PredictCascade() - Predict what will be affected
- FormatForContext() - AI-consumable summary
3. Integration:
- Wire to alert history in router startup
- Map alert types to correlation event types
- Add correlation context to enriched AI context
Example AI context now includes:
'When local-zfs experiences high usage, database often follows within 5 minutes'
This enables the AI to understand infrastructure dependencies
and predict cascade failures.
All tests passing.
Connect alert system to failure prediction:
1. Add AlertCallback to HistoryManager:
- OnAlert() method to register callbacks
- Callbacks invoked when alerts are added
- Called outside lock to prevent deadlocks
2. Expose OnAlertHistory() on alerts.Manager:
- Pass-through to HistoryManager.OnAlert()
- Enables external systems to track alerts
3. Wire pattern detector in router startup:
- Register callback when pattern detector is created
- Convert alert types to trackable events
- Pattern detector now learns from production alerts
Now every alert (memory_warning, cpu_critical, etc.) is recorded as
a historical event for pattern analysis. The AI can predict:
'High memory usage typically occurs every ~3 days (next expected in ~1 day)'
All tests passing.
Create internal/ai/patterns package:
1. Pattern Detector (detector.go):
- Records historical events (high memory, OOM, restarts, etc.)
- Detects recurring failure patterns
- Calculates average interval between occurrences
- Computes confidence based on pattern consistency
- Predicts when failures will occur again
- Persists to ai_patterns.json
2. Event types tracked:
- high_memory, high_cpu, disk_full
- oom, restart, unresponsive
- backup_failed
3. Integration:
- Wire PatternDetector into router startup
- Add to AI context in buildEnrichedContext
- FormatForContext generates failure predictions
Example AI context now includes:
'OOM events typically occurs every ~10 days (next expected in ~3 days)'
This enables proactive alerts before problems recur.
All tests passing.
Phase 4 - Remediation logging integration:
1. logRemediation hook after tool execution:
- Only logs run_command tools (main remediation action)
- Records resourceID, resourceType, findingID
- Extracts problem summary from user prompt
- Truncates output for storage (max 1000 chars)
- Distinguishes automatic (patrol) vs manual (chat) actions
2. buildRemediationContext for system prompts:
- Shows 'Past Successful Fixes for Similar Issues' section
- Uses keyword matching to find relevant past fixes
- Shows 'Remediation History for This Resource' section
- Includes timestamps and outcomes
This enables the AI to say things like:
- 'This worked before: apt clean to free 6GB (resolved)'
- 'Last time on this resource: restarted nginx (resolved)'
All tests passing.
Complete Phase 3 integration:
- Initialize ChangeDetector and RemediationLog in StartPatrol
- Add SetChangeDetector/SetRemediationLog to handler chain:
Router -> AISettingsHandler -> Service -> PatrolService
- Persist change history to ai_changes.json
- Persist remediation log to ai_remediations.json
- Both use the Pulse config directory for storage
Operational memory is now fully integrated:
- Change detector tracks infrastructure changes on each patrol
- Recent changes (24h) are appended to AI context
- Remediation log ready for command execution logging
All tests passing.
Integrate operational memory into patrol context:
- Add changeDetector and remediationLog fields to PatrolService
- Add SetChangeDetector and SetRemediationLog methods
- Integrate change detection into buildEnrichedContext
- Convert state to ResourceSnapshots for change tracking
- Append recent changes summary to AI context
The AI now sees a 'Recent Infrastructure Changes (24h)' section
showing events like:
- VM 'web-server' status changed: running → stopped (2h ago)
- 'db-server' migrated from node1 to node2 (4h ago)
- 'web-server' memory increased: 4 GB → 8 GB (1d ago)
All tests passing.
Phase 3 of Pulse AI differentiation:
Create internal/ai/memory package with:
1. Change Detection (changes.go):
- Tracks infrastructure changes: creation, deletion, config changes
- Detects status changes (started, stopped)
- Detects VM/container migrations between nodes
- Detects CPU/memory configuration changes
- Detects backup completions
- Persists change history to ai_changes.json
- GetChangesSummary for AI context
2. Remediation Logging (remediation.go):
- Records actions taken to fix problems
- Tracks command, output, and outcome
- Links to AI findings via findingID
- GetSimilar finds past similar problems
- GetSuccessfulRemediations for learning
- Persists to ai_remediations.json
3. Type exports (memory_exports.go):
- Clean re-exports from ai package
This enables the AI to say things like:
- 'This VM was migrated 2 hours ago'
- 'Memory was increased from 4GB to 8GB yesterday'
- 'Last time this happened, restarting nginx resolved it'
All tests passing.
Complete Phase 2 baseline integration:
- Add baseline_exports.go for clean type aliasing
- Wire baseline store initialization into StartPatrol
- Implement startBaselineLearning background loop
- Runs initial learning after 5 min delay
- Updates baselines every hour from metrics history
- Learns from 7 days of data for nodes, VMs, containers
- Add SetBaselineStore methods throughout the chain
(Router -> AIHandler -> Service -> PatrolService)
- Persists baselines to data directory as JSON
The baseline learning loop:
1. Starts automatically when AI patrol starts
2. Queries metrics history for all resources
3. Computes mean, stddev, percentiles for cpu/memory/disk
4. Saves baselines to disk for durability
5. Anomaly detection uses these baselines in context builder
All tests passing.
Phase 2 of Pulse AI differentiation:
- Create internal/ai/baseline package for learned baselines
- Implement statistical baseline learning with mean, stddev, percentiles
- Add z-score based anomaly detection with severity classification
(low, medium, high, critical based on standard deviations)
- Integrate baseline provider into context builder
- Wire baseline store into patrol service with adapters
- Add anomaly enrichment to resource contexts
Key features:
- Learn computes baseline from historical metric data points
- IsAnomaly and CheckAnomaly detect deviations from normal
- Persists baselines to disk as JSON for durability
- Formatted anomaly descriptions for AI consumption
Example: 'Memory is high above normal (85.2% vs typical 42.1% ± 8.3%)'
The baseline store needs to be initialized and triggered to learn
from metrics history. Next step is adding the learning loop.
All tests passing.
Phase 1 of Pulse AI differentiation:
- Create internal/ai/context package with types, trends, builder, formatter
- Implement linear regression for trend computation (growing/declining/stable/volatile)
- Add storage capacity predictions (predicts days until 90% and 100%)
- Wire MetricsHistory from monitor to patrol service
- Update patrol to use buildEnrichedContext instead of basic summary
- Update patrol prompt to reference trend indicators and predictions
This gives the AI awareness of historical patterns, enabling it to:
- Identify resources with concerning growth rates
- Predict capacity exhaustion before it happens
- Distinguish between stable high usage vs growing problems
- Provide more actionable, time-aware insights
All tests passing. Falls back to basic summary if metrics history unavailable.
- Changed patrol schedule from preset dropdown to freeform number input
- Users can now set any interval (min 10 minutes, max 7 days, or 0 to disable)
- Added patrol_interval_minutes to API request/response (preset is now deprecated)
- Backend validates: min 10 minutes when enabled, max 10080 (7 days)
- Frontend shows human-readable duration next to input (e.g., '6h', '2h 30m')
Also improved Auto-Fix Mode safety:
- Removed '(recommended)' from preset options (was subjective)
- Added 'I understand the risks' acknowledgement checkbox
- Toggle is disabled until user explicitly acknowledges the risks
- Shows prominent warning when Auto-Fix is enabled
- Acknowledgement is session-based (must re-acknowledge on page reload)
- Add 'content' type to StreamDisplayEvent for tracking text chunks
- Track content events in streamEvents array for chronological display
- Update render to use Switch/Match for cleaner conditional rendering
- Interleave thinking, tool calls, and content as they stream in
- Add fallback for old messages without streamEvents for backwards compat
Previously, tool/command outputs stayed at top while AI text responses
accumulated at the bottom. Now all events appear in order like a
normal chatbot.
- Add clear_anthropic_key, clear_openai_key, clear_deepseek_key, clear_ollama_url flags to API
- Backend handles clearing with confirmation prompt
- Each provider accordion shows Test and Clear buttons when configured
- Clear button requires confirmation before removing credentials
- Frontend automatically refreshes settings after clearing
- Add /api/ai/test/{provider} endpoint for testing individual providers
- Add 'Test' button to each provider accordion (visible when configured)
- Shows test result inline (success/error message)
- Update help links with direct URLs to API key pages:
- Anthropic: console.anthropic.com/settings/keys
- OpenAI: platform.openai.com/api-keys
- DeepSeek: platform.deepseek.com/api_keys
- Ollama: ollama.ai
Backend:
- Add per-provider API key fields to AIConfig (AnthropicAPIKey, OpenAIAPIKey, DeepSeekAPIKey, OllamaBaseURL, OpenAIBaseURL)
- Add NewForProvider() and NewForModel() factory functions for multi-provider instantiation
- Update ListModels() to aggregate models from all configured providers with provider:model format
- Update Execute/ExecuteStream to dynamically create provider based on selected model
- Update TestConnection to use multi-provider aware provider creation
- Add helper functions: HasProvider(), GetConfiguredProviders(), GetAPIKeyForProvider(), GetBaseURLForProvider(), ParseModelString(), FormatModelString()
Frontend:
- Remove legacy single-provider UI (provider grid, single API key input, single base URL)
- Add accordion-style UI for configuring all providers independently
- Add model grouping by provider in selectors using optgroup
- Update AIChat model dropdown with grouped provider sections
- Add helper functions for parsing provider from model ID and grouping models
API:
- Add multi-provider fields to AISettingsResponse and AISettingsUpdateRequest
- Add /api/ai/models endpoint for dynamic model listing
- Update settings handlers for per-provider credential management
Users can now:
1. View all suppression rules (both from dismissed findings and manually created)
2. Create manual rules like 'ignore performance issues on debian-go'
3. Delete rules when they want alerts to come back
Backend:
- Added SuppressionRule type for user-defined rules
- Added suppressionRules storage to FindingsStore
- Added AddSuppressionRule/GetSuppressionRules/DeleteSuppressionRule methods
- Added isSuppressedInternal check for manual rules
- Added API handlers and routes for /api/ai/patrol/suppressions
Frontend:
- Added SuppressionRule interface
- Added getSuppressionRules/addSuppressionRule/deleteSuppressionRule API functions
- Added getDismissedFindings for viewing dismissed findings
Example usage:
POST /api/ai/patrol/suppressions
{
'resource_id': 'debian-go',
'category': 'performance',
'description': 'Dev container runs hot - expected'
}
The main issue was that finding IDs included the title, which the LLM
generates differently each time. 'High CPU on minipc' vs 'Node minipc
experiencing high CPU load' got different IDs, making dismissals useless.
Changes:
1. LLM findings now get IDs based on resource+category only, not title
2. Add() now checks if finding is suppressed before adding as new
3. Add() now checks dismissed findings and only reactivates on severity escalation
4. IsSuppressed() now matches by resource+category only, not title
5. Added isSuppressedInternal() for use when lock is already held
Now when you dismiss 'performance issues on minipc', any future patrol finding
about performance on minipc will be recognized as the same issue and stay dismissed.
