# Unified Resource Architecture ## Status: Draft ## Author: AI Assistant + rcourtman ## Created: 2025-12-07 ## Last Updated: 2025-12-07 --- ## Executive Summary Pulse is evolving from a traditional "stare at dashboards" monitoring tool to an **AI-first infrastructure management platform** where AI is the primary operator and humans are supervisors. This document outlines the architectural changes needed to support this vision while preserving the excellent UI that 20,000+ users love. The core change is introducing a **Unified Resource Model** - a common data abstraction that all platforms (Proxmox, Docker, Hosts, and future platforms like Kubernetes and TrueNAS) normalize to. This enables: 1. **AI intelligence across all platforms** - AI can reason about your entire infrastructure 2. **Elimination of duplicate monitoring** - One machine = one set of alerts 3. **Extensibility for new platforms** - Adding Kubernetes is adding a new resource type, not a new architecture 4. **Foundation for unified views** - Optional consolidated UI without breaking existing pages --- ## Problem Statement ### Current Architecture Issues #### 1. Multiple Data Sources for Same Machine When a Proxmox node also has a host agent running, Pulse monitors it twice: - Via Proxmox API → "Node cpu at 97.9%" - Via Host Agent → "Host cpu at 99.7%" Result: Duplicate alerts, user confusion. #### 2. Siloed Data Models Each platform has its own types with no common abstraction: ``` Node, VM, Container (Proxmox) DockerHost, DockerContainer (Docker) Host (Host Agent) PBSInstance, PBSDatastore (PBS) PMGInstance (PMG) ``` AI can't easily answer: "What's using the most CPU across my infrastructure?" #### 3. Platform-Specific Pages The frontend has separate components for each platform: - Dashboard (Proxmox VMs/Containers) - Docker page - Hosts page - Storage page Adding a new platform (Kubernetes) means building an entirely new page. #### 4. Agent Capabilities Underutilized The unified pulse-agent can: - Report richer metrics than Proxmox API - Execute commands for AI - Monitor temperature, RAID, disk I/O at granular level But the architecture doesn't fully leverage this. --- ## Vision: AI-First Monitoring ``` ┌────────────────────────────────────────────────────────────────────────┐ │ TRADITIONAL MONITORING │ │ │ │ Human 👁️ ──watches──▶ Dashboard 📊 ──shows──▶ Metrics/Alerts │ │ │ │ │ └──────────────────── investigates ──▶ fixes manually │ └────────────────────────────────────────────────────────────────────────┘ ⬇️ ┌────────────────────────────────────────────────────────────────────────┐ │ PULSE: AI-FIRST │ │ │ │ Infrastructure 🖥️ ──reports to──▶ Pulse AI 🤖 │ │ │ │ │ ├── Detects anomalies │ │ ├── Correlates across systems │ │ ├── Diagnoses root causes │ │ ├── Suggests fixes │ │ └── Executes fixes (approved) │ │ │ │ │ ▼ │ │ Human 👤 ──reviews/approves──▶ Dashboard = CONTEXT, not PRIMARY │ └────────────────────────────────────────────────────────────────────────┘ ``` The dashboard remains beautiful and functional, but AI becomes the primary interface for investigation and remediation. --- ## Solution: Unified Resource Model ### Core Abstraction Every monitored entity becomes a `Resource` with common fields: ```go // Resource is the universal abstraction for any monitored entity type Resource struct { // Identity ID string `json:"id"` // Globally unique ID Type ResourceType `json:"type"` // vm, container, docker-container, pod, host, etc. Name string `json:"name"` // Human-readable name DisplayName string `json:"displayName"` // Custom display name (if set) // Platform/Source PlatformID string `json:"platformId"` // Which platform instance PlatformType PlatformType `json:"platformType"` // proxmox-pve, docker, kubernetes, etc. SourceType SourceType `json:"sourceType"` // api, agent, hybrid // Hierarchy ParentID string `json:"parentId,omitempty"` // VM → Node, Pod → K8s Node ClusterID string `json:"clusterId,omitempty"` // Cluster membership // Universal Metrics (nullable - not all resources have all metrics) Status ResourceStatus `json:"status"` // online, offline, running, stopped, degraded CPU *MetricValue `json:"cpu,omitempty"` Memory *MetricValue `json:"memory,omitempty"` Disk *MetricValue `json:"disk,omitempty"` Network *NetworkMetric `json:"network,omitempty"` Temperature *float64 `json:"temperature,omitempty"` Uptime *int64 `json:"uptime,omitempty"` // Universal Metadata Tags []string `json:"tags,omitempty"` Labels map[string]string `json:"labels,omitempty"` LastSeen time.Time `json:"lastSeen"` Alerts []ResourceAlert `json:"alerts,omitempty"` // Platform-Specific Data (discriminated by Type) // This preserves all the rich data while allowing common handling PlatformData json.RawMessage `json:"platformData,omitempty"` } type ResourceType string const ( // Infrastructure ResourceTypeNode ResourceType = "node" ResourceTypeHost ResourceType = "host" ResourceTypeDockerHost ResourceType = "docker-host" ResourceTypeK8sNode ResourceType = "k8s-node" ResourceTypeTrueNASSystem ResourceType = "truenas-system" // Compute Workloads ResourceTypeVM ResourceType = "vm" ResourceTypeContainer ResourceType = "container" // LXC ResourceTypeDockerContainer ResourceType = "docker-container" ResourceTypePod ResourceType = "pod" ResourceTypeJail ResourceType = "jail" // Services ResourceTypeDockerService ResourceType = "docker-service" ResourceTypeK8sDeployment ResourceType = "k8s-deployment" ResourceTypeK8sService ResourceType = "k8s-service" // Storage ResourceTypeStorage ResourceType = "storage" ResourceTypeDatastore ResourceType = "datastore" ResourceTypePool ResourceType = "pool" ResourceTypeDataset ResourceType = "dataset" // Backup Systems ResourceTypePBS ResourceType = "pbs" ResourceTypePMG ResourceType = "pmg" ) type PlatformType string const ( PlatformProxmoxPVE PlatformType = "proxmox-pve" PlatformProxmoxPBS PlatformType = "proxmox-pbs" PlatformProxmoxPMG PlatformType = "proxmox-pmg" PlatformDocker PlatformType = "docker" PlatformKubernetes PlatformType = "kubernetes" PlatformTrueNAS PlatformType = "truenas" PlatformHostAgent PlatformType = "host-agent" ) type SourceType string const ( SourceAPI SourceType = "api" // Data from polling an API SourceAgent SourceType = "agent" // Data pushed from agent SourceHybrid SourceType = "hybrid" // Both sources, agent preferred ) ``` ### Deduplication Strategy When multiple sources report on the same physical machine: ```go type ResourceIdentity struct { Hostname string // Primary identifier MachineID string // /etc/machine-id or equivalent IPs []string // Network addresses } // IdentityMatcher determines if two resources are the same machine func (r *ResourceStore) IdentityMatcher(a, b Resource) bool { // 1. Machine ID match (most reliable) if a.MachineID != "" && a.MachineID == b.MachineID { return true } // 2. Hostname match (case-insensitive) if strings.EqualFold(a.Hostname, b.Hostname) { return true } // 3. IP overlap (if same IP, likely same machine) for _, ipA := range a.IPs { for _, ipB := range b.IPs { if ipA == ipB && !isLocalhost(ipA) { return true } } } return false } ``` When duplicates are detected, prefer **Agent > API**: | Scenario | Result | |----------|--------| | Node only (no agent) | Use Node data | | Host agent only (no Proxmox) | Use Host data | | Both Node + Host agent | Use Host agent data, suppress Node alerts | | Docker agent on Proxmox node | Combine: Docker data + Host metrics from agent | --- ## Implementation Phases ### Phase 0: Document & Design (This Document) - [x] Capture the vision - [x] Define the data model - [ ] Review with stakeholders - [ ] Create GitHub issues/milestones ### Phase 1: Backend Unification (Invisible to Users) **Goal:** Create the Resource abstraction without changing any frontend behavior. **Status:** ✅ Core implementation complete **Completed Tasks:** 1. ✅ Create `internal/resources/resource.go` with types - Core Resource type, enums, and helper methods 2. ✅ Create `internal/resources/platform_data.go` - Platform-specific data types for PlatformData field 3. ✅ Create `internal/resources/store.go` for unified storage - In-memory store with indexes 4. ✅ Create `internal/resources/converters.go` - Converters: FromNode(), FromHost(), FromDockerHost(), FromVM(), FromContainer(), FromDockerContainer(), FromPBSInstance(), FromStorage() 5. ✅ Add deduplication logic using identity matching (hostname, machineID, IP) 6. ✅ Create `internal/resources/converters_test.go` and `store_test.go` - 25 passing tests 7. ✅ Create `internal/api/resource_handlers.go` - HTTP handlers for unified resources API 8. ✅ Add `/api/resources` endpoint with filtering (type, platform, status, parent, infrastructure, workloads) 9. ✅ Add `/api/resources/stats` endpoint for store statistics 10. ✅ Add `/api/resources/{id}` endpoint for individual resource lookup 11. ✅ Add deduplication helper methods for alert manager: `IsSuppressed()`, `GetPreferredResourceFor()`, `IsSamePhysicalMachine()`, `HasPreferredSourceForHostname()` **Integration Points for Alert Manager:** The resource store provides these methods that the existing alert manager can optionally use: - `store.IsSuppressed(resourceID)` - Check if a resource was deduplicated - `store.GetPreferredResourceFor(resourceID)` - Get the preferred resource - `store.HasPreferredSourceForHostname(hostname)` - Check if an agent is preferred for a hostname **Remaining Tasks:** - [ ] *(Optional)* Enhance alert manager to use resource store methods instead of hostname-only dedup - [ ] *(Deferred to Phase 2)* Add optional `resources` array to WebSocket state **Backward Compatibility:** - WebSocket still sends `nodes`, `vms`, `containers`, `hosts`, `dockerHosts` as separate arrays - Unified resources available via REST API at `/api/resources` - All existing frontend code continues to work unchanged - Existing hostname-based deduplication in alert manager still works **Metrics:** - Zero frontend changes - All existing tests pass - New resource tests added (25 tests) - New REST API endpoints available ### Phase 2: AI Context Enhancement **Goal:** AI chat can query and act across all resource types. **Status:** ✅ Fully complete **Completed Tasks:** 1. ✅ Create `internal/ai/resource_context.go` - Unified context builder for AI 2. ✅ Add `ResourceProvider` interface to AI service 3. ✅ Modify `buildSystemPrompt` to use unified model when available 4. ✅ Wire up resource provider in router to AI handlers 5. ✅ AI now uses deduplicated view of infrastructure (falls back to legacy if not available) 6. ✅ Add cross-platform query methods: `GetTopByCPU()`, `GetTopByMemory()`, `GetTopByDisk()` 7. ✅ Add resource correlation: `GetRelated()` for parent/children/siblings/cluster members 8. ✅ Add infrastructure summary: `GetResourceSummary()` with status counts and averages 9. ✅ AI context includes "Top CPU Consumers", "Top Memory Consumers", "Top Disk Usage" 10. ✅ AI context includes infrastructure summary with health status **How It Works:** - When `ResourceProvider` is set, AI gets a cleaner "Unified Infrastructure View" - Resources grouped by platform (Proxmox nodes, Standalone hosts, Docker hosts) - Workloads grouped by parent infrastructure - Agent status shown inline with infrastructure - Resources with alerts highlighted - **Top consumers** shown for CPU, Memory, and Disk - **Infrastructure summary** with healthy/degraded/offline counts **Cross-Platform Query Methods:** ```go // Find top resource consumers across all platforms store.GetTopByCPU(10, nil) // Top 10 by CPU, any type store.GetTopByMemory(5, []ResourceType{ResourceTypeVM}) // Top 5 VMs by memory // Find related resources store.GetRelated("vm-123") // Returns parent, children, siblings, cluster_members // Get infrastructure overview store.GetResourceSummary() // TotalResources, Healthy, Degraded, Offline, ByType, ByPlatform ``` **User Experience:** - AI can now answer "What's using the most CPU?" across all platforms - AI knows about resource relationships (parent nodes, sibling VMs, cluster members) - AI has infrastructure summary context for better analysis ### Phase 3: Agent Preference & Hybrid Mode **Goal:** When agents exist, prefer their data over API polling. **Status:** ✅ Core implementation complete **Completed Tasks:** 1. ✅ Polling optimization methods added to resource store: - `ShouldSkipAPIPolling(hostname)` - Check if API polling should be skipped - `GetAgentMonitoredHostnames()` - Get list of agent-monitored hosts - `GetPollingRecommendations()` - Get per-hostname polling multipliers 2. ✅ Hybrid source type defined (`SourceHybrid`) 3. ✅ Agent data automatically preferred over API data (store deduplication) 4. ✅ `ResourceStoreInterface` added to Monitor 5. ✅ `SetResourceStore()` method added to inject store into Monitor 6. ✅ `shouldSkipNodeMetrics()` helper method added to Monitor 7. ✅ Resource store wired into Monitor via `Router.SetMonitor()` **How It Works:** ```go // In Monitor struct resourceStore ResourceStoreInterface // Router injects the store when setting the monitor func (r *Router) SetMonitor(m *monitoring.Monitor) { // ... other setup ... if r.resourceHandlers != nil { m.SetResourceStore(r.resourceHandlers.Store()) } } // Monitor can now check if polling should be skipped func (m *Monitor) shouldSkipNodeMetrics(nodeName string) bool { if store := m.resourceStore; store != nil { return store.ShouldSkipAPIPolling(nodeName) } return false } ``` **How to Enable Polling Optimization:** To enable polling optimization in the actual polling loops, add this check in `pollPVENode()`: ```go // In monitor_polling.go, at the start of pollPVENode(): func (m *Monitor) pollPVENode(...) (models.Node, string, error) { // Skip detailed metric polling if host agent provides data if m.shouldSkipNodeMetrics(node.Node) { // Still return basic node info but skip expensive API calls // like GetNodeStatus, GetStorage, etc. } // ... rest of function } ``` **Remaining Tasks (Future Enhancement):** - [ ] Add config flag: `EnableAgentPollingOptimization bool` - [ ] Actually integrate skip logic into `pollPVENode()` and related functions - [ ] Add Prometheus metrics for skipped polls - [ ] Add logging for poll optimization decisions **Benefits:** - Better data quality (agent metrics are more accurate) - Reduced API load (skip redundant polling) - More AI capabilities (command execution via agents) ### Phase 4: Unified Monitoring Page (ABANDONED) **Goal:** Replace separate platform pages with ONE unified monitoring experience. **Status:** ❌ **ABANDONED** - Prototyped but reverted in favor of existing pages **Decision (2025-12-07):** After building and testing the unified resources view, we determined that the **existing platform-specific pages are superior** in terms of: - User experience and layout - Feature richness - Familiarity for existing users The backend unified resource model (Phases 1-3) remains valuable for: - AI context enhancement (cross-platform intelligence) - Deduplication logic (avoiding duplicate alerts) - Future API consumers **What was removed:** - `src/components/Resources/ResourcesOverview.tsx` - `src/components/Resources/UnifiedResourceRow.tsx` - `src/components/Resources/columns.ts` - `src/types/resource.ts` - `/resources` route **What remains:** - Backend `/api/resources` endpoint (for AI and API consumers) - Backend unified resource store (`internal/resources/`) - Platform-specific frontend pages (Dashboard, Docker, Hosts) **Lesson Learned:** Sometimes the best architecture is to keep what works. The existing pages have years of refinement and user-driven features. A "unified" view lost much of that polish. ### Phase 5: New Platform Support (Ongoing) Each new platform follows the pattern: 1. **Collector:** Poll API or receive agent telemetry 2. **Converter:** `PlatformDataToResource()` function 3. **Platform Data:** Type-specific struct stored in `PlatformData` field 4. **UI Components:** (Optional) Platform-specific detail views Example for Kubernetes: ```go func K8sNodeToResource(node k8s.Node) Resource { return Resource{ ID: fmt.Sprintf("k8s/%s/%s", clusterID, node.Name), Type: ResourceTypeK8sNode, Name: node.Name, PlatformType: PlatformKubernetes, SourceType: SourceAPI, Status: mapK8sNodeStatus(node.Status), CPU: extractK8sCPU(node), Memory: extractK8sMemory(node), PlatformData: marshalK8sNodeData(node), } } ``` --- ## Migration Strategy ### Database Considerations Current state is in-memory with JSON persistence for alerts. The unified model should: 1. **Short-term:** Continue in-memory, Resource is just an abstraction layer 2. **Medium-term:** SQLite metrics store (already implemented) extended for resources 3. **Long-term:** Consider time-series DB for metrics history ### API Compatibility **Existing endpoints remain unchanged:** - `/api/state` - Returns current format - WebSocket `state` message - Returns current format **New optional endpoints:** - `/api/resources` - Returns unified resource list - `/api/resources/{id}` - Returns single resource with full detail - WebSocket `resources` message - Unified resource updates ### Frontend Migration **No forced migration.** Existing components continue to work. New unified view is additive. If we eventually want to migrate existing components: 1. Create `useResources()` hook that abstracts data source 2. Components can switch when ready 3. Old and new can coexist --- ## Success Metrics ### Phase 1 Complete When: - [x] Resource type defined and tested - [x] All current types have converters - [x] Deduplication prevents duplicate alerts (store logic complete) - [x] Zero frontend changes required - [x] All existing tests pass - [x] REST API endpoints available (`/api/resources`, `/api/resources/stats`, `/api/resources/{id}`) - [x] Deduplication helper methods available for alert manager integration ### Phase 2 Complete When: - [x] AI context includes unified resource view - [x] AI can answer cross-platform questions (via GetTopByCPU/Memory/Disk) - [x] Correlation across platforms works (via GetRelated) ### Phase 3 Complete When: - [x] Agent data preferred when available (via store deduplication) - [x] Polling optimization methods available (`ShouldSkipAPIPolling`, `GetPollingRecommendations`) - [x] Resource store wired into Monitor (via `SetResourceStore`) - [x] `shouldSkipNodeMetrics()` helper available for polling loops - [x] AI can execute commands via agents (already implemented) - [ ] Polling optimization actually used in live polling loops (optional enhancement) ### Phase 4 Complete When: - [x] Unified view accessible from UI (route: `/resources`) - [x] Filtering and grouping works - [x] Existing pages still work - [ ] Navigation link added to main UI (optional) --- ## Open Questions 1. **Should Resource replace existing types entirely, or wrap them?** - Recommendation: Wrap initially, replace gradually 2. **How to handle platform-specific features in unified view?** - Recommendation: Show common fields, expand for platform-specific 3. **Should we version the Resource schema?** - Recommendation: Yes, include version field for future evolution 4. **How to handle offline/stale data in unified model?** - Recommendation: Include `lastSeen` and `staleness` fields --- ## Appendix: Resource Type Mapping | Current Type | Resource Type | Platform Type | Notes | |--------------|---------------|---------------|-------| | Node | node | proxmox-pve | Proxmox VE node | | VM | vm | proxmox-pve | Proxmox VM | | Container | container | proxmox-pve | LXC container | | Host | host | host-agent | Standalone host | | DockerHost | docker-host | docker | Docker/Podman host | | DockerContainer | docker-container | docker | Docker container | | PBSInstance | pbs | proxmox-pbs | Backup server | | PBSDatastore | datastore | proxmox-pbs | PBS datastore | | PMGInstance | pmg | proxmox-pmg | Mail gateway | | Storage | storage | proxmox-pve | PVE storage | | CephCluster | ceph-cluster | proxmox-pve | Ceph cluster | --- ## References - [Conversation that sparked this design](internal discussion 2025-12-07) - [Host Agent Deduplication Fix](commit implementing hostname-based dedup) - [Pulse AI Features](current AI implementation)