Pulse/.gemini/docs/unified-resource-architecture.md

Unified Resource Architecture

Status: Draft

Author: AI Assistant + rcourtman

Created: 2025-12-07

Last Updated: 2025-12-07

---

Executive Summary

Pulse is evolving from a traditional "stare at dashboards" monitoring tool to an AI-first infrastructure management platform where AI is the primary operator and humans are supervisors. This document outlines the architectural changes needed to support this vision while preserving the excellent UI that 20,000+ users love.

The core change is introducing a Unified Resource Model - a common data abstraction that all platforms (Proxmox, Docker, Hosts, and future platforms like Kubernetes and TrueNAS) normalize to. This enables:

1. AI intelligence across all platforms - AI can reason about your entire infrastructure
2. Elimination of duplicate monitoring - One machine = one set of alerts
3. Extensibility for new platforms - Adding Kubernetes is adding a new resource type, not a new architecture
4. Foundation for unified views - Optional consolidated UI without breaking existing pages

---

Problem Statement

Current Architecture Issues

1. Multiple Data Sources for Same Machine

When a Proxmox node also has a host agent running, Pulse monitors it twice:

• Via Proxmox API → "Node cpu at 97.9%"
• Via Host Agent → "Host cpu at 99.7%"

Result: Duplicate alerts, user confusion.

2. Siloed Data Models

Each platform has its own types with no common abstraction:

Node, VM, Container           (Proxmox)
DockerHost, DockerContainer   (Docker)
Host                          (Host Agent)
PBSInstance, PBSDatastore     (PBS)
PMGInstance                   (PMG)

AI can't easily answer: "What's using the most CPU across my infrastructure?"

3. Platform-Specific Pages

The frontend has separate components for each platform:

• Dashboard (Proxmox VMs/Containers)
• Docker page
• Hosts page
• Storage page

Adding a new platform (Kubernetes) means building an entirely new page.

4. Agent Capabilities Underutilized

The unified pulse-agent can:

• Report richer metrics than Proxmox API
• Execute commands for AI
• Monitor temperature, RAID, disk I/O at granular level

But the architecture doesn't fully leverage this.

---

Vision: AI-First Monitoring

┌────────────────────────────────────────────────────────────────────────┐
│                         TRADITIONAL MONITORING                         │
│                                                                        │
│   Human 👁️ ──watches──▶ Dashboard 📊 ──shows──▶ Metrics/Alerts        │
│      │                                                                 │
│      └──────────────────── investigates ──▶ fixes manually            │
└────────────────────────────────────────────────────────────────────────┘

                                 ⬇️

┌────────────────────────────────────────────────────────────────────────┐
│                         PULSE: AI-FIRST                                │
│                                                                        │
│   Infrastructure 🖥️ ──reports to──▶ Pulse AI 🤖                        │
│                                          │                             │
│                                          ├── Detects anomalies         │
│                                          ├── Correlates across systems │
│                                          ├── Diagnoses root causes     │
│                                          ├── Suggests fixes            │
│                                          └── Executes fixes (approved) │
│                                                    │                   │
│                                                    ▼                   │
│   Human 👤 ──reviews/approves──▶ Dashboard = CONTEXT, not PRIMARY     │
└────────────────────────────────────────────────────────────────────────┘

The dashboard remains beautiful and functional, but AI becomes the primary interface for investigation and remediation.

---

Solution: Unified Resource Model

Core Abstraction

Every monitored entity becomes a Resource with common fields:

// Resource is the universal abstraction for any monitored entity
type Resource struct {
    // Identity
    ID           string       `json:"id"`           // Globally unique ID
    Type         ResourceType `json:"type"`         // vm, container, docker-container, pod, host, etc.
    Name         string       `json:"name"`         // Human-readable name
    DisplayName  string       `json:"displayName"`  // Custom display name (if set)
    
    // Platform/Source
    PlatformID   string       `json:"platformId"`   // Which platform instance
    PlatformType PlatformType `json:"platformType"` // proxmox-pve, docker, kubernetes, etc.
    SourceType   SourceType   `json:"sourceType"`   // api, agent, hybrid
    
    // Hierarchy
    ParentID     string       `json:"parentId,omitempty"` // VM → Node, Pod → K8s Node
    ClusterID    string       `json:"clusterId,omitempty"` // Cluster membership
    
    // Universal Metrics (nullable - not all resources have all metrics)
    Status       ResourceStatus `json:"status"`       // online, offline, running, stopped, degraded
    CPU          *MetricValue   `json:"cpu,omitempty"`
    Memory       *MetricValue   `json:"memory,omitempty"`
    Disk         *MetricValue   `json:"disk,omitempty"`
    Network      *NetworkMetric `json:"network,omitempty"`
    Temperature  *float64       `json:"temperature,omitempty"`
    Uptime       *int64         `json:"uptime,omitempty"`
    
    // Universal Metadata
    Tags         []string               `json:"tags,omitempty"`
    Labels       map[string]string      `json:"labels,omitempty"`
    LastSeen     time.Time              `json:"lastSeen"`
    Alerts       []ResourceAlert        `json:"alerts,omitempty"`
    
    // Platform-Specific Data (discriminated by Type)
    // This preserves all the rich data while allowing common handling
    PlatformData json.RawMessage `json:"platformData,omitempty"`
}

type ResourceType string

const (
    // Infrastructure
    ResourceTypeNode            ResourceType = "node"
    ResourceTypeHost            ResourceType = "host"
    ResourceTypeDockerHost      ResourceType = "docker-host"
    ResourceTypeK8sNode         ResourceType = "k8s-node"
    ResourceTypeTrueNASSystem   ResourceType = "truenas-system"
    
    // Compute Workloads
    ResourceTypeVM              ResourceType = "vm"
    ResourceTypeContainer       ResourceType = "container"       // LXC
    ResourceTypeDockerContainer ResourceType = "docker-container"
    ResourceTypePod             ResourceType = "pod"
    ResourceTypeJail            ResourceType = "jail"
    
    // Services
    ResourceTypeDockerService   ResourceType = "docker-service"
    ResourceTypeK8sDeployment   ResourceType = "k8s-deployment"
    ResourceTypeK8sService      ResourceType = "k8s-service"
    
    // Storage
    ResourceTypeStorage         ResourceType = "storage"
    ResourceTypeDatastore       ResourceType = "datastore"
    ResourceTypePool            ResourceType = "pool"
    ResourceTypeDataset         ResourceType = "dataset"
    
    // Backup Systems
    ResourceTypePBS             ResourceType = "pbs"
    ResourceTypePMG             ResourceType = "pmg"
)

type PlatformType string

const (
    PlatformProxmoxPVE  PlatformType = "proxmox-pve"
    PlatformProxmoxPBS  PlatformType = "proxmox-pbs"
    PlatformProxmoxPMG  PlatformType = "proxmox-pmg"
    PlatformDocker      PlatformType = "docker"
    PlatformKubernetes  PlatformType = "kubernetes"
    PlatformTrueNAS     PlatformType = "truenas"
    PlatformHostAgent   PlatformType = "host-agent"
)

type SourceType string

const (
    SourceAPI    SourceType = "api"    // Data from polling an API
    SourceAgent  SourceType = "agent"  // Data pushed from agent
    SourceHybrid SourceType = "hybrid" // Both sources, agent preferred
)

Deduplication Strategy

When multiple sources report on the same physical machine:

type ResourceIdentity struct {
    Hostname  string   // Primary identifier
    MachineID string   // /etc/machine-id or equivalent
    IPs       []string // Network addresses
}

// IdentityMatcher determines if two resources are the same machine
func (r *ResourceStore) IdentityMatcher(a, b Resource) bool {
    // 1. Machine ID match (most reliable)
    if a.MachineID != "" && a.MachineID == b.MachineID {
        return true
    }
    
    // 2. Hostname match (case-insensitive)
    if strings.EqualFold(a.Hostname, b.Hostname) {
        return true
    }
    
    // 3. IP overlap (if same IP, likely same machine)
    for _, ipA := range a.IPs {
        for _, ipB := range b.IPs {
            if ipA == ipB && !isLocalhost(ipA) {
                return true
            }
        }
    }
    
    return false
}

When duplicates are detected, prefer Agent > API:

Scenario	Result
Node only (no agent)	Use Node data
Host agent only (no Proxmox)	Use Host data
Both Node + Host agent	Use Host agent data, suppress Node alerts
Docker agent on Proxmox node	Combine: Docker data + Host metrics from agent

---

Implementation Phases

Phase 0: Document & Design (This Document)

• Capture the vision
• Define the data model
• Review with stakeholders
• Create GitHub issues/milestones

Phase 1: Backend Unification (Invisible to Users)

Goal: Create the Resource abstraction without changing any frontend behavior.

Status: ✅ Core implementation complete

Completed Tasks:

1. ✅ Create internal/resources/resource.go with types - Core Resource type, enums, and helper methods
2. ✅ Create internal/resources/platform_data.go - Platform-specific data types for PlatformData field
3. ✅ Create internal/resources/store.go for unified storage - In-memory store with indexes
4. ✅ Create internal/resources/converters.go - Converters: FromNode(), FromHost(), FromDockerHost(), FromVM(), FromContainer(), FromDockerContainer(), FromPBSInstance(), FromStorage()
5. ✅ Add deduplication logic using identity matching (hostname, machineID, IP)
6. ✅ Create internal/resources/converters_test.go and store_test.go - 25 passing tests
7. ✅ Create internal/api/resource_handlers.go - HTTP handlers for unified resources API
8. ✅ Add /api/resources endpoint with filtering (type, platform, status, parent, infrastructure, workloads)
9. ✅ Add /api/resources/stats endpoint for store statistics
10. ✅ Add /api/resources/{id} endpoint for individual resource lookup
11. ✅ Add deduplication helper methods for alert manager: IsSuppressed(), GetPreferredResourceFor(), IsSamePhysicalMachine(), HasPreferredSourceForHostname()

Integration Points for Alert Manager: The resource store provides these methods that the existing alert manager can optionally use:

• store.IsSuppressed(resourceID) - Check if a resource was deduplicated
• store.GetPreferredResourceFor(resourceID) - Get the preferred resource
• store.HasPreferredSourceForHostname(hostname) - Check if an agent is preferred for a hostname

Remaining Tasks:

• (Optional) Enhance alert manager to use resource store methods instead of hostname-only dedup
• (Deferred to Phase 2) Add optional resources array to WebSocket state

Backward Compatibility:

• WebSocket still sends nodes, vms, containers, hosts, dockerHosts as separate arrays
• Unified resources available via REST API at /api/resources
• All existing frontend code continues to work unchanged
• Existing hostname-based deduplication in alert manager still works

Metrics:

• Zero frontend changes
• All existing tests pass
• New resource tests added (25 tests)
• New REST API endpoints available

Phase 2: AI Context Enhancement

Goal: AI chat can query and act across all resource types.

Status: ✅ Fully complete

Completed Tasks:

1. ✅ Create internal/ai/resource_context.go - Unified context builder for AI
2. ✅ Add ResourceProvider interface to AI service
3. ✅ Modify buildSystemPrompt to use unified model when available
4. ✅ Wire up resource provider in router to AI handlers
5. ✅ AI now uses deduplicated view of infrastructure (falls back to legacy if not available)
6. ✅ Add cross-platform query methods: GetTopByCPU(), GetTopByMemory(), GetTopByDisk()
7. ✅ Add resource correlation: GetRelated() for parent/children/siblings/cluster members
8. ✅ Add infrastructure summary: GetResourceSummary() with status counts and averages
9. ✅ AI context includes "Top CPU Consumers", "Top Memory Consumers", "Top Disk Usage"
10. ✅ AI context includes infrastructure summary with health status

How It Works:

• When ResourceProvider is set, AI gets a cleaner "Unified Infrastructure View"
• Resources grouped by platform (Proxmox nodes, Standalone hosts, Docker hosts)
• Workloads grouped by parent infrastructure
• Agent status shown inline with infrastructure
• Resources with alerts highlighted
• Top consumers shown for CPU, Memory, and Disk
• Infrastructure summary with healthy/degraded/offline counts

Cross-Platform Query Methods:

// Find top resource consumers across all platforms
store.GetTopByCPU(10, nil)    // Top 10 by CPU, any type
store.GetTopByMemory(5, []ResourceType{ResourceTypeVM}) // Top 5 VMs by memory

// Find related resources
store.GetRelated("vm-123")    // Returns parent, children, siblings, cluster_members

// Get infrastructure overview
store.GetResourceSummary()    // TotalResources, Healthy, Degraded, Offline, ByType, ByPlatform

User Experience:

• AI can now answer "What's using the most CPU?" across all platforms
• AI knows about resource relationships (parent nodes, sibling VMs, cluster members)
• AI has infrastructure summary context for better analysis

Phase 3: Agent Preference & Hybrid Mode

Goal: When agents exist, prefer their data over API polling.

Status: ✅ Core implementation complete

Completed Tasks:

1. ✅ Polling optimization methods added to resource store:
   • ShouldSkipAPIPolling(hostname) - Check if API polling should be skipped
   • GetAgentMonitoredHostnames() - Get list of agent-monitored hosts
   • GetPollingRecommendations() - Get per-hostname polling multipliers
2. ✅ Hybrid source type defined (SourceHybrid)
3. ✅ Agent data automatically preferred over API data (store deduplication)
4. ✅ ResourceStoreInterface added to Monitor
5. ✅ SetResourceStore() method added to inject store into Monitor
6. ✅ shouldSkipNodeMetrics() helper method added to Monitor
7. ✅ Resource store wired into Monitor via Router.SetMonitor()

How It Works:

// In Monitor struct
resourceStore ResourceStoreInterface

// Router injects the store when setting the monitor
func (r *Router) SetMonitor(m *monitoring.Monitor) {
    // ... other setup ...
    if r.resourceHandlers != nil {
        m.SetResourceStore(r.resourceHandlers.Store())
    }
}

// Monitor can now check if polling should be skipped
func (m *Monitor) shouldSkipNodeMetrics(nodeName string) bool {
    if store := m.resourceStore; store != nil {
        return store.ShouldSkipAPIPolling(nodeName)
    }
    return false
}

How to Enable Polling Optimization:

To enable polling optimization in the actual polling loops, add this check in pollPVENode():

// In monitor_polling.go, at the start of pollPVENode():
func (m *Monitor) pollPVENode(...) (models.Node, string, error) {
    // Skip detailed metric polling if host agent provides data
    if m.shouldSkipNodeMetrics(node.Node) {
        // Still return basic node info but skip expensive API calls
        // like GetNodeStatus, GetStorage, etc.
    }
    // ... rest of function
}

Remaining Tasks (Future Enhancement):

• Add config flag: EnableAgentPollingOptimization bool
• Actually integrate skip logic into pollPVENode() and related functions
• Add Prometheus metrics for skipped polls
• Add logging for poll optimization decisions

Benefits:

• Better data quality (agent metrics are more accurate)
• Reduced API load (skip redundant polling)
• More AI capabilities (command execution via agents)

Phase 4: Unified Monitoring Page (ABANDONED)

Goal: Replace separate platform pages with ONE unified monitoring experience.

Status: ❌ ABANDONED - Prototyped but reverted in favor of existing pages

Decision (2025-12-07): After building and testing the unified resources view, we determined that the existing platform-specific pages are superior in terms of:

• User experience and layout
• Feature richness
• Familiarity for existing users

The backend unified resource model (Phases 1-3) remains valuable for:

• AI context enhancement (cross-platform intelligence)
• Deduplication logic (avoiding duplicate alerts)
• Future API consumers

What was removed:

• src/components/Resources/ResourcesOverview.tsx
• src/components/Resources/UnifiedResourceRow.tsx
• src/components/Resources/columns.ts
• src/types/resource.ts
• /resources route

What remains:

• Backend /api/resources endpoint (for AI and API consumers)
• Backend unified resource store (internal/resources/)
• Platform-specific frontend pages (Dashboard, Docker, Hosts)

Lesson Learned: Sometimes the best architecture is to keep what works. The existing pages have years of refinement and user-driven features. A "unified" view lost much of that polish.

Phase 5: New Platform Support (Ongoing)

Each new platform follows the pattern:

1. Collector: Poll API or receive agent telemetry
2. Converter: PlatformDataToResource() function
3. Platform Data: Type-specific struct stored in PlatformData field
4. UI Components: (Optional) Platform-specific detail views

Example for Kubernetes:

func K8sNodeToResource(node k8s.Node) Resource {
    return Resource{
        ID:           fmt.Sprintf("k8s/%s/%s", clusterID, node.Name),
        Type:         ResourceTypeK8sNode,
        Name:         node.Name,
        PlatformType: PlatformKubernetes,
        SourceType:   SourceAPI,
        Status:       mapK8sNodeStatus(node.Status),
        CPU:          extractK8sCPU(node),
        Memory:       extractK8sMemory(node),
        PlatformData: marshalK8sNodeData(node),
    }
}

---

Migration Strategy

Database Considerations

Current state is in-memory with JSON persistence for alerts. The unified model should:

1. Short-term: Continue in-memory, Resource is just an abstraction layer
2. Medium-term: SQLite metrics store (already implemented) extended for resources
3. Long-term: Consider time-series DB for metrics history

API Compatibility

Existing endpoints remain unchanged:

• /api/state - Returns current format
• WebSocket state message - Returns current format

New optional endpoints:

• /api/resources - Returns unified resource list
• /api/resources/{id} - Returns single resource with full detail
• WebSocket resources message - Unified resource updates

Frontend Migration

No forced migration. Existing components continue to work. New unified view is additive.

If we eventually want to migrate existing components:

1. Create useResources() hook that abstracts data source
2. Components can switch when ready
3. Old and new can coexist

---

Success Metrics

Phase 1 Complete When:

• Resource type defined and tested
• All current types have converters
• Deduplication prevents duplicate alerts (store logic complete)
• Zero frontend changes required
• All existing tests pass
• REST API endpoints available (/api/resources, /api/resources/stats, /api/resources/{id})
• Deduplication helper methods available for alert manager integration

Phase 2 Complete When:

• AI context includes unified resource view
• AI can answer cross-platform questions (via GetTopByCPU/Memory/Disk)
• Correlation across platforms works (via GetRelated)

Phase 3 Complete When:

• Agent data preferred when available (via store deduplication)
• Polling optimization methods available (ShouldSkipAPIPolling, GetPollingRecommendations)
• Resource store wired into Monitor (via SetResourceStore)
• shouldSkipNodeMetrics() helper available for polling loops
• AI can execute commands via agents (already implemented)
• Polling optimization actually used in live polling loops (optional enhancement)

Phase 4 Complete When:

• Unified view accessible from UI (route: /resources) - ABANDONED (see below)
• Filtering and grouping works
• Existing pages still work
• Navigation link added to main UI (optional)

Note: Phase 4 Direction Changed (2025-12-07)

The dedicated /resources unified view was abandoned in favor of migrating existing pages:

1. WebSocket State Migration - COMPLETED

   • Backend broadcasts unified resources via state.resources[]
   • Frontend types defined (resource.ts)
   • useResources() hook available for components
   • Hosts page uses unified resources (with fallback)
   • Docker page uses unified resources (with fallback)
   • Dashboard page uses unified resources (with fallback)

2. What Was Removed:

   • /resources route and ResourcesOverview component
   • Frontend-only resource types (replaced by types matching backend)

3. What Remains:

   • Backend unified resource model (for AI context)
   • WebSocket state now includes resources array
   • All pages can consume from unified model with legacy fallback

4. Cleanup (Phase 6) - COMPLETED:

   • Remove debug console.log statements from frontend routes
   • Remove legacy fallback code from route components (Docker, Hosts, Dashboard)
   • Remove unused legacy arrays from backend StateFrontend (optional - still broadcast for API compatibility)
   • Remove legacy AI context fallback (optional - verify AI uses unified model first)
   • Simplify route components to use resources directly (remove adapter layer)

---

Open Questions

1. Should Resource replace existing types entirely, or wrap them?

   • Recommendation: Wrap initially, replace gradually

2. How to handle platform-specific features in unified view?

   • Recommendation: Show common fields, expand for platform-specific

3. Should we version the Resource schema?

   • Recommendation: Yes, include version field for future evolution

4. How to handle offline/stale data in unified model?

   • Recommendation: Include lastSeen and staleness fields

---

Appendix: Resource Type Mapping

Current Type	Resource Type	Platform Type	Notes
Node	node	proxmox-pve	Proxmox VE node
VM	vm	proxmox-pve	Proxmox VM
Container	container	proxmox-pve	LXC container
Host	host	host-agent	Standalone host
DockerHost	docker-host	docker	Docker/Podman host
DockerContainer	docker-container	docker	Docker container
PBSInstance	pbs	proxmox-pbs	Backup server
PBSDatastore	datastore	proxmox-pbs	PBS datastore
PMGInstance	pmg	proxmox-pmg	Mail gateway
Storage	storage	proxmox-pve	PVE storage
CephCluster	ceph-cluster	proxmox-pve	Ceph cluster

---

References

• [Conversation that sparked this design](internal discussion 2025-12-07)
• [Host Agent Deduplication Fix](commit implementing hostname-based dedup)
• [Pulse AI Features](current AI implementation)