Pulse/.gemini/docs/unified-resource-architecture.md
rcourtman ef35120afb checkpoint: remove unified resources UI, keep backend model
- Removed /resources page and associated frontend components
- Removed ResourcesOverview.tsx, UnifiedResourceRow.tsx, columns.ts
- Removed frontend types/resource.ts
- Updated unified-resource-architecture.md to mark Phase 4 as ABANDONED
- Removed unified-view-migration-plan.md
- Backend unified resource model remains for AI context

This is a checkpoint before attempting full frontend migration to unified model.
2025-12-07 22:54:27 +00:00

566 lines
23 KiB
Markdown

# Unified Resource Architecture
## Status: Draft
## Author: AI Assistant + rcourtman
## Created: 2025-12-07
## Last Updated: 2025-12-07
---
## Executive Summary
Pulse is evolving from a traditional "stare at dashboards" monitoring tool to an **AI-first infrastructure management platform** where AI is the primary operator and humans are supervisors. This document outlines the architectural changes needed to support this vision while preserving the excellent UI that 20,000+ users love.
The core change is introducing a **Unified Resource Model** - a common data abstraction that all platforms (Proxmox, Docker, Hosts, and future platforms like Kubernetes and TrueNAS) normalize to. This enables:
1. **AI intelligence across all platforms** - AI can reason about your entire infrastructure
2. **Elimination of duplicate monitoring** - One machine = one set of alerts
3. **Extensibility for new platforms** - Adding Kubernetes is adding a new resource type, not a new architecture
4. **Foundation for unified views** - Optional consolidated UI without breaking existing pages
---
## Problem Statement
### Current Architecture Issues
#### 1. Multiple Data Sources for Same Machine
When a Proxmox node also has a host agent running, Pulse monitors it twice:
- Via Proxmox API → "Node cpu at 97.9%"
- Via Host Agent → "Host cpu at 99.7%"
Result: Duplicate alerts, user confusion.
#### 2. Siloed Data Models
Each platform has its own types with no common abstraction:
```
Node, VM, Container (Proxmox)
DockerHost, DockerContainer (Docker)
Host (Host Agent)
PBSInstance, PBSDatastore (PBS)
PMGInstance (PMG)
```
AI can't easily answer: "What's using the most CPU across my infrastructure?"
#### 3. Platform-Specific Pages
The frontend has separate components for each platform:
- Dashboard (Proxmox VMs/Containers)
- Docker page
- Hosts page
- Storage page
Adding a new platform (Kubernetes) means building an entirely new page.
#### 4. Agent Capabilities Underutilized
The unified pulse-agent can:
- Report richer metrics than Proxmox API
- Execute commands for AI
- Monitor temperature, RAID, disk I/O at granular level
But the architecture doesn't fully leverage this.
---
## Vision: AI-First Monitoring
```
┌────────────────────────────────────────────────────────────────────────┐
│ TRADITIONAL MONITORING │
│ │
│ Human 👁️ ──watches──▶ Dashboard 📊 ──shows──▶ Metrics/Alerts │
│ │ │
│ └──────────────────── investigates ──▶ fixes manually │
└────────────────────────────────────────────────────────────────────────┘
⬇️
┌────────────────────────────────────────────────────────────────────────┐
│ PULSE: AI-FIRST │
│ │
│ Infrastructure 🖥️ ──reports to──▶ Pulse AI 🤖 │
│ │ │
│ ├── Detects anomalies │
│ ├── Correlates across systems │
│ ├── Diagnoses root causes │
│ ├── Suggests fixes │
│ └── Executes fixes (approved) │
│ │ │
│ ▼ │
│ Human 👤 ──reviews/approves──▶ Dashboard = CONTEXT, not PRIMARY │
└────────────────────────────────────────────────────────────────────────┘
```
The dashboard remains beautiful and functional, but AI becomes the primary interface for investigation and remediation.
---
## Solution: Unified Resource Model
### Core Abstraction
Every monitored entity becomes a `Resource` with common fields:
```go
// Resource is the universal abstraction for any monitored entity
type Resource struct {
// Identity
ID string `json:"id"` // Globally unique ID
Type ResourceType `json:"type"` // vm, container, docker-container, pod, host, etc.
Name string `json:"name"` // Human-readable name
DisplayName string `json:"displayName"` // Custom display name (if set)
// Platform/Source
PlatformID string `json:"platformId"` // Which platform instance
PlatformType PlatformType `json:"platformType"` // proxmox-pve, docker, kubernetes, etc.
SourceType SourceType `json:"sourceType"` // api, agent, hybrid
// Hierarchy
ParentID string `json:"parentId,omitempty"` // VM → Node, Pod → K8s Node
ClusterID string `json:"clusterId,omitempty"` // Cluster membership
// Universal Metrics (nullable - not all resources have all metrics)
Status ResourceStatus `json:"status"` // online, offline, running, stopped, degraded
CPU *MetricValue `json:"cpu,omitempty"`
Memory *MetricValue `json:"memory,omitempty"`
Disk *MetricValue `json:"disk,omitempty"`
Network *NetworkMetric `json:"network,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
Uptime *int64 `json:"uptime,omitempty"`
// Universal Metadata
Tags []string `json:"tags,omitempty"`
Labels map[string]string `json:"labels,omitempty"`
LastSeen time.Time `json:"lastSeen"`
Alerts []ResourceAlert `json:"alerts,omitempty"`
// Platform-Specific Data (discriminated by Type)
// This preserves all the rich data while allowing common handling
PlatformData json.RawMessage `json:"platformData,omitempty"`
}
type ResourceType string
const (
// Infrastructure
ResourceTypeNode ResourceType = "node"
ResourceTypeHost ResourceType = "host"
ResourceTypeDockerHost ResourceType = "docker-host"
ResourceTypeK8sNode ResourceType = "k8s-node"
ResourceTypeTrueNASSystem ResourceType = "truenas-system"
// Compute Workloads
ResourceTypeVM ResourceType = "vm"
ResourceTypeContainer ResourceType = "container" // LXC
ResourceTypeDockerContainer ResourceType = "docker-container"
ResourceTypePod ResourceType = "pod"
ResourceTypeJail ResourceType = "jail"
// Services
ResourceTypeDockerService ResourceType = "docker-service"
ResourceTypeK8sDeployment ResourceType = "k8s-deployment"
ResourceTypeK8sService ResourceType = "k8s-service"
// Storage
ResourceTypeStorage ResourceType = "storage"
ResourceTypeDatastore ResourceType = "datastore"
ResourceTypePool ResourceType = "pool"
ResourceTypeDataset ResourceType = "dataset"
// Backup Systems
ResourceTypePBS ResourceType = "pbs"
ResourceTypePMG ResourceType = "pmg"
)
type PlatformType string
const (
PlatformProxmoxPVE PlatformType = "proxmox-pve"
PlatformProxmoxPBS PlatformType = "proxmox-pbs"
PlatformProxmoxPMG PlatformType = "proxmox-pmg"
PlatformDocker PlatformType = "docker"
PlatformKubernetes PlatformType = "kubernetes"
PlatformTrueNAS PlatformType = "truenas"
PlatformHostAgent PlatformType = "host-agent"
)
type SourceType string
const (
SourceAPI SourceType = "api" // Data from polling an API
SourceAgent SourceType = "agent" // Data pushed from agent
SourceHybrid SourceType = "hybrid" // Both sources, agent preferred
)
```
### Deduplication Strategy
When multiple sources report on the same physical machine:
```go
type ResourceIdentity struct {
Hostname string // Primary identifier
MachineID string // /etc/machine-id or equivalent
IPs []string // Network addresses
}
// IdentityMatcher determines if two resources are the same machine
func (r *ResourceStore) IdentityMatcher(a, b Resource) bool {
// 1. Machine ID match (most reliable)
if a.MachineID != "" && a.MachineID == b.MachineID {
return true
}
// 2. Hostname match (case-insensitive)
if strings.EqualFold(a.Hostname, b.Hostname) {
return true
}
// 3. IP overlap (if same IP, likely same machine)
for _, ipA := range a.IPs {
for _, ipB := range b.IPs {
if ipA == ipB && !isLocalhost(ipA) {
return true
}
}
}
return false
}
```
When duplicates are detected, prefer **Agent > API**:
| Scenario | Result |
|----------|--------|
| Node only (no agent) | Use Node data |
| Host agent only (no Proxmox) | Use Host data |
| Both Node + Host agent | Use Host agent data, suppress Node alerts |
| Docker agent on Proxmox node | Combine: Docker data + Host metrics from agent |
---
## Implementation Phases
### Phase 0: Document & Design (This Document)
- [x] Capture the vision
- [x] Define the data model
- [ ] Review with stakeholders
- [ ] Create GitHub issues/milestones
### Phase 1: Backend Unification (Invisible to Users)
**Goal:** Create the Resource abstraction without changing any frontend behavior.
**Status:** ✅ Core implementation complete
**Completed Tasks:**
1. ✅ Create `internal/resources/resource.go` with types - Core Resource type, enums, and helper methods
2. ✅ Create `internal/resources/platform_data.go` - Platform-specific data types for PlatformData field
3. ✅ Create `internal/resources/store.go` for unified storage - In-memory store with indexes
4. ✅ Create `internal/resources/converters.go` - Converters: FromNode(), FromHost(), FromDockerHost(), FromVM(), FromContainer(), FromDockerContainer(), FromPBSInstance(), FromStorage()
5. ✅ Add deduplication logic using identity matching (hostname, machineID, IP)
6. ✅ Create `internal/resources/converters_test.go` and `store_test.go` - 25 passing tests
7. ✅ Create `internal/api/resource_handlers.go` - HTTP handlers for unified resources API
8. ✅ Add `/api/resources` endpoint with filtering (type, platform, status, parent, infrastructure, workloads)
9. ✅ Add `/api/resources/stats` endpoint for store statistics
10. ✅ Add `/api/resources/{id}` endpoint for individual resource lookup
11. ✅ Add deduplication helper methods for alert manager: `IsSuppressed()`, `GetPreferredResourceFor()`, `IsSamePhysicalMachine()`, `HasPreferredSourceForHostname()`
**Integration Points for Alert Manager:**
The resource store provides these methods that the existing alert manager can optionally use:
- `store.IsSuppressed(resourceID)` - Check if a resource was deduplicated
- `store.GetPreferredResourceFor(resourceID)` - Get the preferred resource
- `store.HasPreferredSourceForHostname(hostname)` - Check if an agent is preferred for a hostname
**Remaining Tasks:**
- [ ] *(Optional)* Enhance alert manager to use resource store methods instead of hostname-only dedup
- [ ] *(Deferred to Phase 2)* Add optional `resources` array to WebSocket state
**Backward Compatibility:**
- WebSocket still sends `nodes`, `vms`, `containers`, `hosts`, `dockerHosts` as separate arrays
- Unified resources available via REST API at `/api/resources`
- All existing frontend code continues to work unchanged
- Existing hostname-based deduplication in alert manager still works
**Metrics:**
- Zero frontend changes
- All existing tests pass
- New resource tests added (25 tests)
- New REST API endpoints available
### Phase 2: AI Context Enhancement
**Goal:** AI chat can query and act across all resource types.
**Status:** ✅ Fully complete
**Completed Tasks:**
1. ✅ Create `internal/ai/resource_context.go` - Unified context builder for AI
2. ✅ Add `ResourceProvider` interface to AI service
3. ✅ Modify `buildSystemPrompt` to use unified model when available
4. ✅ Wire up resource provider in router to AI handlers
5. ✅ AI now uses deduplicated view of infrastructure (falls back to legacy if not available)
6. ✅ Add cross-platform query methods: `GetTopByCPU()`, `GetTopByMemory()`, `GetTopByDisk()`
7. ✅ Add resource correlation: `GetRelated()` for parent/children/siblings/cluster members
8. ✅ Add infrastructure summary: `GetResourceSummary()` with status counts and averages
9. ✅ AI context includes "Top CPU Consumers", "Top Memory Consumers", "Top Disk Usage"
10. ✅ AI context includes infrastructure summary with health status
**How It Works:**
- When `ResourceProvider` is set, AI gets a cleaner "Unified Infrastructure View"
- Resources grouped by platform (Proxmox nodes, Standalone hosts, Docker hosts)
- Workloads grouped by parent infrastructure
- Agent status shown inline with infrastructure
- Resources with alerts highlighted
- **Top consumers** shown for CPU, Memory, and Disk
- **Infrastructure summary** with healthy/degraded/offline counts
**Cross-Platform Query Methods:**
```go
// Find top resource consumers across all platforms
store.GetTopByCPU(10, nil) // Top 10 by CPU, any type
store.GetTopByMemory(5, []ResourceType{ResourceTypeVM}) // Top 5 VMs by memory
// Find related resources
store.GetRelated("vm-123") // Returns parent, children, siblings, cluster_members
// Get infrastructure overview
store.GetResourceSummary() // TotalResources, Healthy, Degraded, Offline, ByType, ByPlatform
```
**User Experience:**
- AI can now answer "What's using the most CPU?" across all platforms
- AI knows about resource relationships (parent nodes, sibling VMs, cluster members)
- AI has infrastructure summary context for better analysis
### Phase 3: Agent Preference & Hybrid Mode
**Goal:** When agents exist, prefer their data over API polling.
**Status:** ✅ Core implementation complete
**Completed Tasks:**
1. ✅ Polling optimization methods added to resource store:
- `ShouldSkipAPIPolling(hostname)` - Check if API polling should be skipped
- `GetAgentMonitoredHostnames()` - Get list of agent-monitored hosts
- `GetPollingRecommendations()` - Get per-hostname polling multipliers
2. ✅ Hybrid source type defined (`SourceHybrid`)
3. ✅ Agent data automatically preferred over API data (store deduplication)
4.`ResourceStoreInterface` added to Monitor
5.`SetResourceStore()` method added to inject store into Monitor
6.`shouldSkipNodeMetrics()` helper method added to Monitor
7. ✅ Resource store wired into Monitor via `Router.SetMonitor()`
**How It Works:**
```go
// In Monitor struct
resourceStore ResourceStoreInterface
// Router injects the store when setting the monitor
func (r *Router) SetMonitor(m *monitoring.Monitor) {
// ... other setup ...
if r.resourceHandlers != nil {
m.SetResourceStore(r.resourceHandlers.Store())
}
}
// Monitor can now check if polling should be skipped
func (m *Monitor) shouldSkipNodeMetrics(nodeName string) bool {
if store := m.resourceStore; store != nil {
return store.ShouldSkipAPIPolling(nodeName)
}
return false
}
```
**How to Enable Polling Optimization:**
To enable polling optimization in the actual polling loops, add this check in `pollPVENode()`:
```go
// In monitor_polling.go, at the start of pollPVENode():
func (m *Monitor) pollPVENode(...) (models.Node, string, error) {
// Skip detailed metric polling if host agent provides data
if m.shouldSkipNodeMetrics(node.Node) {
// Still return basic node info but skip expensive API calls
// like GetNodeStatus, GetStorage, etc.
}
// ... rest of function
}
```
**Remaining Tasks (Future Enhancement):**
- [ ] Add config flag: `EnableAgentPollingOptimization bool`
- [ ] Actually integrate skip logic into `pollPVENode()` and related functions
- [ ] Add Prometheus metrics for skipped polls
- [ ] Add logging for poll optimization decisions
**Benefits:**
- Better data quality (agent metrics are more accurate)
- Reduced API load (skip redundant polling)
- More AI capabilities (command execution via agents)
### Phase 4: Unified Monitoring Page (ABANDONED)
**Goal:** Replace separate platform pages with ONE unified monitoring experience.
**Status:****ABANDONED** - Prototyped but reverted in favor of existing pages
**Decision (2025-12-07):**
After building and testing the unified resources view, we determined that the **existing platform-specific pages are superior** in terms of:
- User experience and layout
- Feature richness
- Familiarity for existing users
The backend unified resource model (Phases 1-3) remains valuable for:
- AI context enhancement (cross-platform intelligence)
- Deduplication logic (avoiding duplicate alerts)
- Future API consumers
**What was removed:**
- `src/components/Resources/ResourcesOverview.tsx`
- `src/components/Resources/UnifiedResourceRow.tsx`
- `src/components/Resources/columns.ts`
- `src/types/resource.ts`
- `/resources` route
**What remains:**
- Backend `/api/resources` endpoint (for AI and API consumers)
- Backend unified resource store (`internal/resources/`)
- Platform-specific frontend pages (Dashboard, Docker, Hosts)
**Lesson Learned:**
Sometimes the best architecture is to keep what works. The existing pages have years of refinement and user-driven features. A "unified" view lost much of that polish.
### Phase 5: New Platform Support (Ongoing)
Each new platform follows the pattern:
1. **Collector:** Poll API or receive agent telemetry
2. **Converter:** `PlatformDataToResource()` function
3. **Platform Data:** Type-specific struct stored in `PlatformData` field
4. **UI Components:** (Optional) Platform-specific detail views
Example for Kubernetes:
```go
func K8sNodeToResource(node k8s.Node) Resource {
return Resource{
ID: fmt.Sprintf("k8s/%s/%s", clusterID, node.Name),
Type: ResourceTypeK8sNode,
Name: node.Name,
PlatformType: PlatformKubernetes,
SourceType: SourceAPI,
Status: mapK8sNodeStatus(node.Status),
CPU: extractK8sCPU(node),
Memory: extractK8sMemory(node),
PlatformData: marshalK8sNodeData(node),
}
}
```
---
## Migration Strategy
### Database Considerations
Current state is in-memory with JSON persistence for alerts. The unified model should:
1. **Short-term:** Continue in-memory, Resource is just an abstraction layer
2. **Medium-term:** SQLite metrics store (already implemented) extended for resources
3. **Long-term:** Consider time-series DB for metrics history
### API Compatibility
**Existing endpoints remain unchanged:**
- `/api/state` - Returns current format
- WebSocket `state` message - Returns current format
**New optional endpoints:**
- `/api/resources` - Returns unified resource list
- `/api/resources/{id}` - Returns single resource with full detail
- WebSocket `resources` message - Unified resource updates
### Frontend Migration
**No forced migration.** Existing components continue to work. New unified view is additive.
If we eventually want to migrate existing components:
1. Create `useResources()` hook that abstracts data source
2. Components can switch when ready
3. Old and new can coexist
---
## Success Metrics
### Phase 1 Complete When:
- [x] Resource type defined and tested
- [x] All current types have converters
- [x] Deduplication prevents duplicate alerts (store logic complete)
- [x] Zero frontend changes required
- [x] All existing tests pass
- [x] REST API endpoints available (`/api/resources`, `/api/resources/stats`, `/api/resources/{id}`)
- [x] Deduplication helper methods available for alert manager integration
### Phase 2 Complete When:
- [x] AI context includes unified resource view
- [x] AI can answer cross-platform questions (via GetTopByCPU/Memory/Disk)
- [x] Correlation across platforms works (via GetRelated)
### Phase 3 Complete When:
- [x] Agent data preferred when available (via store deduplication)
- [x] Polling optimization methods available (`ShouldSkipAPIPolling`, `GetPollingRecommendations`)
- [x] Resource store wired into Monitor (via `SetResourceStore`)
- [x] `shouldSkipNodeMetrics()` helper available for polling loops
- [x] AI can execute commands via agents (already implemented)
- [ ] Polling optimization actually used in live polling loops (optional enhancement)
### Phase 4 Complete When:
- [x] Unified view accessible from UI (route: `/resources`)
- [x] Filtering and grouping works
- [x] Existing pages still work
- [ ] Navigation link added to main UI (optional)
---
## Open Questions
1. **Should Resource replace existing types entirely, or wrap them?**
- Recommendation: Wrap initially, replace gradually
2. **How to handle platform-specific features in unified view?**
- Recommendation: Show common fields, expand for platform-specific
3. **Should we version the Resource schema?**
- Recommendation: Yes, include version field for future evolution
4. **How to handle offline/stale data in unified model?**
- Recommendation: Include `lastSeen` and `staleness` fields
---
## Appendix: Resource Type Mapping
| Current Type | Resource Type | Platform Type | Notes |
|--------------|---------------|---------------|-------|
| Node | node | proxmox-pve | Proxmox VE node |
| VM | vm | proxmox-pve | Proxmox VM |
| Container | container | proxmox-pve | LXC container |
| Host | host | host-agent | Standalone host |
| DockerHost | docker-host | docker | Docker/Podman host |
| DockerContainer | docker-container | docker | Docker container |
| PBSInstance | pbs | proxmox-pbs | Backup server |
| PBSDatastore | datastore | proxmox-pbs | PBS datastore |
| PMGInstance | pmg | proxmox-pmg | Mail gateway |
| Storage | storage | proxmox-pve | PVE storage |
| CephCluster | ceph-cluster | proxmox-pve | Ceph cluster |
---
## References
- [Conversation that sparked this design](internal discussion 2025-12-07)
- [Host Agent Deduplication Fix](commit implementing hostname-based dedup)
- [Pulse AI Features](current AI implementation)