diff --git a/.gemini/docs/unified-resource-architecture.md b/.gemini/docs/unified-resource-architecture.md deleted file mode 100644 index dd51e00..0000000 --- a/.gemini/docs/unified-resource-architecture.md +++ /dev/null @@ -1,594 +0,0 @@ -# Unified Resource Architecture - -## Status: Draft -## Author: AI Assistant + rcourtman -## Created: 2025-12-07 -## Last Updated: 2025-12-07 - ---- - -## Executive Summary - -Pulse is evolving from a traditional "stare at dashboards" monitoring tool to an **AI-first infrastructure management platform** where AI is the primary operator and humans are supervisors. This document outlines the architectural changes needed to support this vision while preserving the excellent UI that 20,000+ users love. - -The core change is introducing a **Unified Resource Model** - a common data abstraction that all platforms (Proxmox, Docker, Hosts, and future platforms like Kubernetes and TrueNAS) normalize to. This enables: - -1. **AI intelligence across all platforms** - AI can reason about your entire infrastructure -2. **Elimination of duplicate monitoring** - One machine = one set of alerts -3. **Extensibility for new platforms** - Adding Kubernetes is adding a new resource type, not a new architecture -4. **Foundation for unified views** - Optional consolidated UI without breaking existing pages - ---- - -## Problem Statement - -### Current Architecture Issues - -#### 1. Multiple Data Sources for Same Machine -When a Proxmox node also has a host agent running, Pulse monitors it twice: -- Via Proxmox API → "Node cpu at 97.9%" -- Via Host Agent → "Host cpu at 99.7%" - -Result: Duplicate alerts, user confusion. - -#### 2. Siloed Data Models -Each platform has its own types with no common abstraction: -``` -Node, VM, Container (Proxmox) -DockerHost, DockerContainer (Docker) -Host (Host Agent) -PBSInstance, PBSDatastore (PBS) -PMGInstance (PMG) -``` - -AI can't easily answer: "What's using the most CPU across my infrastructure?" - -#### 3. Platform-Specific Pages -The frontend has separate components for each platform: -- Dashboard (Proxmox VMs/Containers) -- Docker page -- Hosts page -- Storage page - -Adding a new platform (Kubernetes) means building an entirely new page. - -#### 4. Agent Capabilities Underutilized -The unified pulse-agent can: -- Report richer metrics than Proxmox API -- Execute commands for AI -- Monitor temperature, RAID, disk I/O at granular level - -But the architecture doesn't fully leverage this. - ---- - -## Vision: AI-First Monitoring - -``` -┌────────────────────────────────────────────────────────────────────────┐ -│ TRADITIONAL MONITORING │ -│ │ -│ Human 👁️ ──watches──▶ Dashboard 📊 ──shows──▶ Metrics/Alerts │ -│ │ │ -│ └──────────────────── investigates ──▶ fixes manually │ -└────────────────────────────────────────────────────────────────────────┘ - - ⬇️ - -┌────────────────────────────────────────────────────────────────────────┐ -│ PULSE: AI-FIRST │ -│ │ -│ Infrastructure 🖥️ ──reports to──▶ Pulse AI 🤖 │ -│ │ │ -│ ├── Detects anomalies │ -│ ├── Correlates across systems │ -│ ├── Diagnoses root causes │ -│ ├── Suggests fixes │ -│ └── Executes fixes (approved) │ -│ │ │ -│ ▼ │ -│ Human 👤 ──reviews/approves──▶ Dashboard = CONTEXT, not PRIMARY │ -└────────────────────────────────────────────────────────────────────────┘ -``` - -The dashboard remains beautiful and functional, but AI becomes the primary interface for investigation and remediation. - ---- - -## Solution: Unified Resource Model - -### Core Abstraction - -Every monitored entity becomes a `Resource` with common fields: - -```go -// Resource is the universal abstraction for any monitored entity -type Resource struct { - // Identity - ID string `json:"id"` // Globally unique ID - Type ResourceType `json:"type"` // vm, container, docker-container, pod, host, etc. - Name string `json:"name"` // Human-readable name - DisplayName string `json:"displayName"` // Custom display name (if set) - - // Platform/Source - PlatformID string `json:"platformId"` // Which platform instance - PlatformType PlatformType `json:"platformType"` // proxmox-pve, docker, kubernetes, etc. - SourceType SourceType `json:"sourceType"` // api, agent, hybrid - - // Hierarchy - ParentID string `json:"parentId,omitempty"` // VM → Node, Pod → K8s Node - ClusterID string `json:"clusterId,omitempty"` // Cluster membership - - // Universal Metrics (nullable - not all resources have all metrics) - Status ResourceStatus `json:"status"` // online, offline, running, stopped, degraded - CPU *MetricValue `json:"cpu,omitempty"` - Memory *MetricValue `json:"memory,omitempty"` - Disk *MetricValue `json:"disk,omitempty"` - Network *NetworkMetric `json:"network,omitempty"` - Temperature *float64 `json:"temperature,omitempty"` - Uptime *int64 `json:"uptime,omitempty"` - - // Universal Metadata - Tags []string `json:"tags,omitempty"` - Labels map[string]string `json:"labels,omitempty"` - LastSeen time.Time `json:"lastSeen"` - Alerts []ResourceAlert `json:"alerts,omitempty"` - - // Platform-Specific Data (discriminated by Type) - // This preserves all the rich data while allowing common handling - PlatformData json.RawMessage `json:"platformData,omitempty"` -} - -type ResourceType string - -const ( - // Infrastructure - ResourceTypeNode ResourceType = "node" - ResourceTypeHost ResourceType = "host" - ResourceTypeDockerHost ResourceType = "docker-host" - ResourceTypeK8sNode ResourceType = "k8s-node" - ResourceTypeTrueNASSystem ResourceType = "truenas-system" - - // Compute Workloads - ResourceTypeVM ResourceType = "vm" - ResourceTypeContainer ResourceType = "container" // LXC - ResourceTypeDockerContainer ResourceType = "docker-container" - ResourceTypePod ResourceType = "pod" - ResourceTypeJail ResourceType = "jail" - - // Services - ResourceTypeDockerService ResourceType = "docker-service" - ResourceTypeK8sDeployment ResourceType = "k8s-deployment" - ResourceTypeK8sService ResourceType = "k8s-service" - - // Storage - ResourceTypeStorage ResourceType = "storage" - ResourceTypeDatastore ResourceType = "datastore" - ResourceTypePool ResourceType = "pool" - ResourceTypeDataset ResourceType = "dataset" - - // Backup Systems - ResourceTypePBS ResourceType = "pbs" - ResourceTypePMG ResourceType = "pmg" -) - -type PlatformType string - -const ( - PlatformProxmoxPVE PlatformType = "proxmox-pve" - PlatformProxmoxPBS PlatformType = "proxmox-pbs" - PlatformProxmoxPMG PlatformType = "proxmox-pmg" - PlatformDocker PlatformType = "docker" - PlatformKubernetes PlatformType = "kubernetes" - PlatformTrueNAS PlatformType = "truenas" - PlatformHostAgent PlatformType = "host-agent" -) - -type SourceType string - -const ( - SourceAPI SourceType = "api" // Data from polling an API - SourceAgent SourceType = "agent" // Data pushed from agent - SourceHybrid SourceType = "hybrid" // Both sources, agent preferred -) -``` - -### Deduplication Strategy - -When multiple sources report on the same physical machine: - -```go -type ResourceIdentity struct { - Hostname string // Primary identifier - MachineID string // /etc/machine-id or equivalent - IPs []string // Network addresses -} - -// IdentityMatcher determines if two resources are the same machine -func (r *ResourceStore) IdentityMatcher(a, b Resource) bool { - // 1. Machine ID match (most reliable) - if a.MachineID != "" && a.MachineID == b.MachineID { - return true - } - - // 2. Hostname match (case-insensitive) - if strings.EqualFold(a.Hostname, b.Hostname) { - return true - } - - // 3. IP overlap (if same IP, likely same machine) - for _, ipA := range a.IPs { - for _, ipB := range b.IPs { - if ipA == ipB && !isLocalhost(ipA) { - return true - } - } - } - - return false -} -``` - -When duplicates are detected, prefer **Agent > API**: - -| Scenario | Result | -|----------|--------| -| Node only (no agent) | Use Node data | -| Host agent only (no Proxmox) | Use Host data | -| Both Node + Host agent | Use Host agent data, suppress Node alerts | -| Docker agent on Proxmox node | Combine: Docker data + Host metrics from agent | - ---- - -## Implementation Phases - -### Phase 0: Document & Design (This Document) -- [x] Capture the vision -- [x] Define the data model -- [ ] Review with stakeholders -- [ ] Create GitHub issues/milestones - -### Phase 1: Backend Unification (Invisible to Users) - -**Goal:** Create the Resource abstraction without changing any frontend behavior. - -**Status:** ✅ Core implementation complete - -**Completed Tasks:** -1. ✅ Create `internal/resources/resource.go` with types - Core Resource type, enums, and helper methods -2. ✅ Create `internal/resources/platform_data.go` - Platform-specific data types for PlatformData field -3. ✅ Create `internal/resources/store.go` for unified storage - In-memory store with indexes -4. ✅ Create `internal/resources/converters.go` - Converters: FromNode(), FromHost(), FromDockerHost(), FromVM(), FromContainer(), FromDockerContainer(), FromPBSInstance(), FromStorage() -5. ✅ Add deduplication logic using identity matching (hostname, machineID, IP) -6. ✅ Create `internal/resources/converters_test.go` and `store_test.go` - 25 passing tests -7. ✅ Create `internal/api/resource_handlers.go` - HTTP handlers for unified resources API -8. ✅ Add `/api/resources` endpoint with filtering (type, platform, status, parent, infrastructure, workloads) -9. ✅ Add `/api/resources/stats` endpoint for store statistics -10. ✅ Add `/api/resources/{id}` endpoint for individual resource lookup -11. ✅ Add deduplication helper methods for alert manager: `IsSuppressed()`, `GetPreferredResourceFor()`, `IsSamePhysicalMachine()`, `HasPreferredSourceForHostname()` - -**Integration Points for Alert Manager:** -The resource store provides these methods that the existing alert manager can optionally use: -- `store.IsSuppressed(resourceID)` - Check if a resource was deduplicated -- `store.GetPreferredResourceFor(resourceID)` - Get the preferred resource -- `store.HasPreferredSourceForHostname(hostname)` - Check if an agent is preferred for a hostname - -**Remaining Tasks:** -- [ ] *(Optional)* Enhance alert manager to use resource store methods instead of hostname-only dedup -- [ ] *(Deferred to Phase 2)* Add optional `resources` array to WebSocket state - -**Backward Compatibility:** -- WebSocket still sends `nodes`, `vms`, `containers`, `hosts`, `dockerHosts` as separate arrays -- Unified resources available via REST API at `/api/resources` -- All existing frontend code continues to work unchanged -- Existing hostname-based deduplication in alert manager still works - -**Metrics:** -- Zero frontend changes -- All existing tests pass -- New resource tests added (25 tests) -- New REST API endpoints available - -### Phase 2: AI Context Enhancement - -**Goal:** AI chat can query and act across all resource types. - -**Status:** ✅ Fully complete - -**Completed Tasks:** -1. ✅ Create `internal/ai/resource_context.go` - Unified context builder for AI -2. ✅ Add `ResourceProvider` interface to AI service -3. ✅ Modify `buildSystemPrompt` to use unified model when available -4. ✅ Wire up resource provider in router to AI handlers -5. ✅ AI now uses deduplicated view of infrastructure (falls back to legacy if not available) -6. ✅ Add cross-platform query methods: `GetTopByCPU()`, `GetTopByMemory()`, `GetTopByDisk()` -7. ✅ Add resource correlation: `GetRelated()` for parent/children/siblings/cluster members -8. ✅ Add infrastructure summary: `GetResourceSummary()` with status counts and averages -9. ✅ AI context includes "Top CPU Consumers", "Top Memory Consumers", "Top Disk Usage" -10. ✅ AI context includes infrastructure summary with health status - -**How It Works:** -- When `ResourceProvider` is set, AI gets a cleaner "Unified Infrastructure View" -- Resources grouped by platform (Proxmox nodes, Standalone hosts, Docker hosts) -- Workloads grouped by parent infrastructure -- Agent status shown inline with infrastructure -- Resources with alerts highlighted -- **Top consumers** shown for CPU, Memory, and Disk -- **Infrastructure summary** with healthy/degraded/offline counts - -**Cross-Platform Query Methods:** -```go -// Find top resource consumers across all platforms -store.GetTopByCPU(10, nil) // Top 10 by CPU, any type -store.GetTopByMemory(5, []ResourceType{ResourceTypeVM}) // Top 5 VMs by memory - -// Find related resources -store.GetRelated("vm-123") // Returns parent, children, siblings, cluster_members - -// Get infrastructure overview -store.GetResourceSummary() // TotalResources, Healthy, Degraded, Offline, ByType, ByPlatform -``` - -**User Experience:** -- AI can now answer "What's using the most CPU?" across all platforms -- AI knows about resource relationships (parent nodes, sibling VMs, cluster members) -- AI has infrastructure summary context for better analysis - -### Phase 3: Agent Preference & Hybrid Mode - -**Goal:** When agents exist, prefer their data over API polling. - -**Status:** ✅ Core implementation complete - -**Completed Tasks:** -1. ✅ Polling optimization methods added to resource store: - - `ShouldSkipAPIPolling(hostname)` - Check if API polling should be skipped - - `GetAgentMonitoredHostnames()` - Get list of agent-monitored hosts - - `GetPollingRecommendations()` - Get per-hostname polling multipliers -2. ✅ Hybrid source type defined (`SourceHybrid`) -3. ✅ Agent data automatically preferred over API data (store deduplication) -4. ✅ `ResourceStoreInterface` added to Monitor -5. ✅ `SetResourceStore()` method added to inject store into Monitor -6. ✅ `shouldSkipNodeMetrics()` helper method added to Monitor -7. ✅ Resource store wired into Monitor via `Router.SetMonitor()` - -**How It Works:** -```go -// In Monitor struct -resourceStore ResourceStoreInterface - -// Router injects the store when setting the monitor -func (r *Router) SetMonitor(m *monitoring.Monitor) { - // ... other setup ... - if r.resourceHandlers != nil { - m.SetResourceStore(r.resourceHandlers.Store()) - } -} - -// Monitor can now check if polling should be skipped -func (m *Monitor) shouldSkipNodeMetrics(nodeName string) bool { - if store := m.resourceStore; store != nil { - return store.ShouldSkipAPIPolling(nodeName) - } - return false -} -``` - -**How to Enable Polling Optimization:** - -To enable polling optimization in the actual polling loops, add this check in `pollPVENode()`: - -```go -// In monitor_polling.go, at the start of pollPVENode(): -func (m *Monitor) pollPVENode(...) (models.Node, string, error) { - // Skip detailed metric polling if host agent provides data - if m.shouldSkipNodeMetrics(node.Node) { - // Still return basic node info but skip expensive API calls - // like GetNodeStatus, GetStorage, etc. - } - // ... rest of function -} -``` - -**Remaining Tasks (Future Enhancement):** -- [ ] Add config flag: `EnableAgentPollingOptimization bool` -- [ ] Actually integrate skip logic into `pollPVENode()` and related functions -- [ ] Add Prometheus metrics for skipped polls -- [ ] Add logging for poll optimization decisions - -**Benefits:** -- Better data quality (agent metrics are more accurate) -- Reduced API load (skip redundant polling) -- More AI capabilities (command execution via agents) - -### Phase 4: Unified Monitoring Page (ABANDONED) - -**Goal:** Replace separate platform pages with ONE unified monitoring experience. - -**Status:** ❌ **ABANDONED** - Prototyped but reverted in favor of existing pages - -**Decision (2025-12-07):** -After building and testing the unified resources view, we determined that the **existing platform-specific pages are superior** in terms of: -- User experience and layout -- Feature richness -- Familiarity for existing users - -The backend unified resource model (Phases 1-3) remains valuable for: -- AI context enhancement (cross-platform intelligence) -- Deduplication logic (avoiding duplicate alerts) -- Future API consumers - -**What was removed:** -- `src/components/Resources/ResourcesOverview.tsx` -- `src/components/Resources/UnifiedResourceRow.tsx` -- `src/components/Resources/columns.ts` -- `src/types/resource.ts` -- `/resources` route - -**What remains:** -- Backend `/api/resources` endpoint (for AI and API consumers) -- Backend unified resource store (`internal/resources/`) -- Platform-specific frontend pages (Dashboard, Docker, Hosts) - -**Lesson Learned:** -Sometimes the best architecture is to keep what works. The existing pages have years of refinement and user-driven features. A "unified" view lost much of that polish. - -### Phase 5: New Platform Support (Ongoing) - -Each new platform follows the pattern: - -1. **Collector:** Poll API or receive agent telemetry -2. **Converter:** `PlatformDataToResource()` function -3. **Platform Data:** Type-specific struct stored in `PlatformData` field -4. **UI Components:** (Optional) Platform-specific detail views - -Example for Kubernetes: -```go -func K8sNodeToResource(node k8s.Node) Resource { - return Resource{ - ID: fmt.Sprintf("k8s/%s/%s", clusterID, node.Name), - Type: ResourceTypeK8sNode, - Name: node.Name, - PlatformType: PlatformKubernetes, - SourceType: SourceAPI, - Status: mapK8sNodeStatus(node.Status), - CPU: extractK8sCPU(node), - Memory: extractK8sMemory(node), - PlatformData: marshalK8sNodeData(node), - } -} -``` - ---- - -## Migration Strategy - -### Database Considerations - -Current state is in-memory with JSON persistence for alerts. The unified model should: - -1. **Short-term:** Continue in-memory, Resource is just an abstraction layer -2. **Medium-term:** SQLite metrics store (already implemented) extended for resources -3. **Long-term:** Consider time-series DB for metrics history - -### API Compatibility - -**Existing endpoints remain unchanged:** -- `/api/state` - Returns current format -- WebSocket `state` message - Returns current format - -**New optional endpoints:** -- `/api/resources` - Returns unified resource list -- `/api/resources/{id}` - Returns single resource with full detail -- WebSocket `resources` message - Unified resource updates - -### Frontend Migration - -**No forced migration.** Existing components continue to work. New unified view is additive. - -If we eventually want to migrate existing components: -1. Create `useResources()` hook that abstracts data source -2. Components can switch when ready -3. Old and new can coexist - ---- - -## Success Metrics - -### Phase 1 Complete When: -- [x] Resource type defined and tested -- [x] All current types have converters -- [x] Deduplication prevents duplicate alerts (store logic complete) -- [x] Zero frontend changes required -- [x] All existing tests pass -- [x] REST API endpoints available (`/api/resources`, `/api/resources/stats`, `/api/resources/{id}`) -- [x] Deduplication helper methods available for alert manager integration - -### Phase 2 Complete When: -- [x] AI context includes unified resource view -- [x] AI can answer cross-platform questions (via GetTopByCPU/Memory/Disk) -- [x] Correlation across platforms works (via GetRelated) - -### Phase 3 Complete When: -- [x] Agent data preferred when available (via store deduplication) -- [x] Polling optimization methods available (`ShouldSkipAPIPolling`, `GetPollingRecommendations`) -- [x] Resource store wired into Monitor (via `SetResourceStore`) -- [x] `shouldSkipNodeMetrics()` helper available for polling loops -- [x] AI can execute commands via agents (already implemented) -- [ ] Polling optimization actually used in live polling loops (optional enhancement) - -### Phase 4 Complete When: -- [x] Unified view accessible from UI (route: `/resources`) - **ABANDONED** (see below) -- [x] Filtering and grouping works -- [x] Existing pages still work -- [ ] Navigation link added to main UI (optional) - -**Note: Phase 4 Direction Changed (2025-12-07)** - -The dedicated `/resources` unified view was abandoned in favor of migrating existing pages: - -1. **WebSocket State Migration** - COMPLETED - - [x] Backend broadcasts unified resources via `state.resources[]` - - [x] Frontend types defined (`resource.ts`) - - [x] `useResources()` hook available for components - - [x] Hosts page uses unified resources (with fallback) - - [x] Docker page uses unified resources (with fallback) - - [x] Dashboard page uses unified resources (with fallback) - -2. **What Was Removed:** - - `/resources` route and ResourcesOverview component - - Frontend-only resource types (replaced by types matching backend) - -3. **What Remains:** - - Backend unified resource model (for AI context) - - WebSocket state now includes resources array - - All pages can consume from unified model with legacy fallback - -4. **Cleanup (Phase 6) - COMPLETED:** - - [x] Remove debug console.log statements from frontend routes - - [x] Remove legacy fallback code from route components (Docker, Hosts, Dashboard) - - [x] Simplify route components to use centralized `useResourcesAsLegacy()` hook - - [x] Remove legacy AI context fallback (`buildInfrastructureContext` removed, 288 lines) - - [ ] Remove unused legacy arrays from backend StateFrontend (optional - still broadcast for API compatibility) - ---- - -## Open Questions - -1. **Should Resource replace existing types entirely, or wrap them?** - - Recommendation: Wrap initially, replace gradually - -2. **How to handle platform-specific features in unified view?** - - Recommendation: Show common fields, expand for platform-specific - -3. **Should we version the Resource schema?** - - Recommendation: Yes, include version field for future evolution - -4. **How to handle offline/stale data in unified model?** - - Recommendation: Include `lastSeen` and `staleness` fields - ---- - -## Appendix: Resource Type Mapping - -| Current Type | Resource Type | Platform Type | Notes | -|--------------|---------------|---------------|-------| -| Node | node | proxmox-pve | Proxmox VE node | -| VM | vm | proxmox-pve | Proxmox VM | -| Container | container | proxmox-pve | LXC container | -| Host | host | host-agent | Standalone host | -| DockerHost | docker-host | docker | Docker/Podman host | -| DockerContainer | docker-container | docker | Docker container | -| PBSInstance | pbs | proxmox-pbs | Backup server | -| PBSDatastore | datastore | proxmox-pbs | PBS datastore | -| PMGInstance | pmg | proxmox-pmg | Mail gateway | -| Storage | storage | proxmox-pve | PVE storage | -| CephCluster | ceph-cluster | proxmox-pve | Ceph cluster | - ---- - -## References - -- [Conversation that sparked this design](internal discussion 2025-12-07) -- [Host Agent Deduplication Fix](commit implementing hostname-based dedup) -- [Pulse AI Features](current AI implementation) diff --git a/.gemini/tasks/persistent-metrics-storage.md b/.gemini/tasks/persistent-metrics-storage.md deleted file mode 100644 index a4dd3c3..0000000 --- a/.gemini/tasks/persistent-metrics-storage.md +++ /dev/null @@ -1,182 +0,0 @@ -# Task: Persistent Metrics Storage for Sparklines - -## Problem -Currently, metrics history for sparklines is stored **in-memory only**. When the Pulse backend restarts, all historical metrics are lost. Users expect to see historical trends even after being away for days. - -## Goal -Implement SQLite-based persistent metrics storage that: -- Survives backend restarts -- Provides historical data for sparklines/trends view -- Supports configurable retention periods -- Minimizes disk I/O and storage footprint - -## Architecture - -### Storage Tiers (Data Rollup) -``` -┌─────────────────────────────────────────────────────────┐ -│ RAW (5s intervals) → Keep 2 hours → ~1,440 pts │ -│ MINUTE (1min avg) → Keep 24 hours → ~1,440 pts │ -│ HOURLY (1hr avg) → Keep 7 days → ~168 pts │ -│ DAILY (1day avg) → Keep 90 days → ~90 pts │ -└─────────────────────────────────────────────────────────┘ -``` - -### Database Schema -```sql --- Main metrics table (partitioned by time for efficient pruning) -CREATE TABLE metrics ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - resource_type TEXT NOT NULL, -- 'node', 'vm', 'container', 'storage' - resource_id TEXT NOT NULL, - metric_type TEXT NOT NULL, -- 'cpu', 'memory', 'disk' - value REAL NOT NULL, - timestamp INTEGER NOT NULL, -- Unix timestamp in seconds - tier TEXT DEFAULT 'raw' -- 'raw', 'minute', 'hourly', 'daily' -); - --- Indexes for efficient queries -CREATE INDEX idx_metrics_lookup ON metrics(resource_type, resource_id, metric_type, tier, timestamp); -CREATE INDEX idx_metrics_timestamp ON metrics(timestamp); -CREATE INDEX idx_metrics_tier_time ON metrics(tier, timestamp); -``` - -### Configuration -```yaml -metrics: - enabled: true - database_path: "${PULSE_DATA_DIR}/metrics.db" - retention: - raw: 2h # 2 hours of raw data - minute: 24h # 24 hours of 1-minute averages - hourly: 168h # 7 days of hourly averages - daily: 2160h # 90 days of daily averages - write_buffer: 100 # Buffer size before batch write - rollup_interval: 5m # How often to run rollup job -``` - -## Implementation Steps - -### Phase 1: SQLite Foundation ✅ COMPLETED -- [x] Add SQLite dependency (`modernc.org/sqlite` - pure Go, no CGO) -- [x] Create `internal/metrics/store.go` with: - - `Store` struct - - `NewStore(config StoreConfig) (*Store, error)` - - `Close() error` - - Schema auto-migration on startup - -### Phase 2: Write Path ✅ COMPLETED -- [x] Create `Write(resourceType, resourceID, metricType string, value float64, timestamp time.Time)` -- [x] Implement write buffering (batch inserts every 100 records or 5 seconds) -- [x] Integrate with existing `AddGuestMetric`, `AddNodeMetric` calls in monitor.go and monitor_polling.go -- [x] Add graceful shutdown to flush buffer - -### Phase 3: Read Path ✅ COMPLETED -- [x] Create `Query(resourceType, resourceID, metricType string, start, end time.Time) ([]MetricPoint, error)` -- [x] Auto-select appropriate tier based on time range: - - < 2 hours → raw data - - 2-24 hours → minute data - - 1-7 days → hourly data - - 7+ days → daily data -- [x] Add `/api/metrics-store/stats` endpoint for monitoring - -### Phase 4: Rollup & Retention ✅ COMPLETED -- [x] Create background rollup job: - - Runs every 5 minutes - - Aggregates raw → minute (AVG, MIN, MAX) - - Aggregates minute → hourly - - Aggregates hourly → daily -- [x] Create retention pruning job: - - Runs every hour - - Deletes data older than configured retention -- [x] Use SQLite transactions for atomic operations - -### Phase 5: Integration -- [ ] Add configuration to `system.json` or new `metrics.json` -- [ ] Add Settings UI for metrics retention config -- [ ] Add database file size monitoring -- [ ] Add vacuum/optimize scheduled job (weekly) - -## Files to Create/Modify - -### New Files -``` -internal/metrics/ -├── store.go # MetricsStore implementation -├── store_test.go # Unit tests -├── rollup.go # Rollup/aggregation logic -├── retention.go # Retention/pruning logic -└── config.go # Metrics configuration -``` - -### Files to Modify -``` -internal/monitoring/monitor.go # Initialize MetricsStore, call Write() -internal/monitoring/metrics_history.go # Keep in-memory as cache, backed by SQLite -internal/api/router.go # Update handleCharts to query from store -internal/config/persistence.go # Add metrics config persistence -``` - -## API Changes - -### `/api/charts` Query Parameters -``` -GET /api/charts?range=1h # Last hour (raw/minute data) -GET /api/charts?range=24h # Last 24 hours (minute data) -GET /api/charts?range=7d # Last 7 days (hourly data) -GET /api/charts?range=30d # Last 30 days (daily data) -GET /api/charts?start=...&end=... # Custom range -``` - -### Response Enhancement -```json -{ - "data": { ... }, - "nodeData": { ... }, - "stats": { - "oldestDataTimestamp": 1699900000000, - "tier": "hourly", - "pointCount": 168 - } -} -``` - -## Performance Considerations - -1. **Write Buffering**: Batch inserts to reduce I/O -2. **WAL Mode**: Enable SQLite WAL for concurrent reads/writes -3. **Prepared Statements**: Reuse for repeated queries -4. **Index Strategy**: Composite index on (resource_type, resource_id, metric_type, tier, timestamp) -5. **Connection Pooling**: Single connection with proper locking for SQLite -6. **Memory Mapping**: Use `PRAGMA mmap_size` for faster reads - -## Storage Estimates -For a typical Pulse installation (5 nodes, 50 VMs, 20 containers, 10 storage): -- 85 resources × 3 metrics = 255 metric series -- Raw (2h at 5s): ~86,400 rows → ~10 MB -- Minute (24h): ~367,200 rows → ~40 MB -- Hourly (7d): ~42,840 rows → ~5 MB -- Daily (90d): ~22,950 rows → ~3 MB -- **Total: ~60-100 MB** for comprehensive historical data - -## Testing Plan -1. Unit tests for store CRUD operations -2. Unit tests for rollup logic -3. Integration tests with mock monitor -4. Performance tests with 100+ resources -5. Restart resilience tests - -## Rollout Plan -1. Implement as opt-in feature (disable by default initially) -2. Add migration path from in-memory to SQLite -3. Test in dev environment for 1 week -4. Enable by default in next minor release - -## Definition of Done -- [ ] SQLite metrics storage implemented -- [ ] Data survives backend restart -- [ ] Rollup/retention working correctly -- [ ] Charts endpoint serves historical data -- [ ] Documentation updated -- [ ] Settings UI for retention config -- [ ] Performance validated (no noticeable slowdown) diff --git a/frontend-modern/PLAN-column-visibility.md b/frontend-modern/PLAN-column-visibility.md deleted file mode 100644 index 78510e8..0000000 --- a/frontend-modern/PLAN-column-visibility.md +++ /dev/null @@ -1,302 +0,0 @@ -# Plan: Toggleable Table Columns - -## Problem -The current drawer pattern hides useful information (IPs, OS, backup status, node, tags) that users need to click to reveal. This goes against Pulse's core philosophy of dense, scannable, comparable data at a glance. - -## Goal -Replace drawer-hidden info with optional table columns that: -1. Show data inline for easy comparison across rows -2. Are toggleable by the user (show/hide) -3. Auto-show based on available horizontal space -4. Persist user preferences - -## Current State - -### Infrastructure Already Exists -- `ColumnPriority` system: `'essential' | 'primary' | 'secondary' | 'supplementary' | 'detailed'` -- `PRIORITY_BREAKPOINTS` maps priorities to responsive breakpoints -- `usePersistentSignal` for localStorage persistence -- `STANDARD_COLUMNS` with predefined column configs -- `useBreakpoint` hook for responsive behavior - -### Current Columns (Dashboard/Proxmox) -All marked `essential` (always visible): -- Name, Type, VMID, Uptime -- CPU, Memory, Disk (progress bars) -- Disk Read, Disk Write, Net In, Net Out - -### Data Available but Hidden in Drawer -- IP addresses (`guest.ipAddresses`) -- OS name/version (`guest.osName`, `guest.osVersion`) -- Node (`guest.node`) -- Backup status (`guest.lastBackup`) -- Tags (`guest.tags`) -- CPUs allocated (`guest.cpus`) -- Agent version (`guest.agentVersion`) - -## Implementation - -### Phase 1: Add New Column Definitions - -Update `GUEST_COLUMNS` in `GuestRow.tsx`: - -```typescript -export const GUEST_COLUMNS: ColumnDef[] = [ - // Essential - always visible - { id: 'name', label: 'Name', priority: 'essential' }, - { id: 'type', label: 'Type', priority: 'essential' }, - { id: 'vmid', label: 'VMID', priority: 'essential' }, - - // Primary - visible on sm+ (640px) - { id: 'cpu', label: 'CPU', priority: 'essential', minWidth: '55px', maxWidth: '156px' }, - { id: 'memory', label: 'Memory', priority: 'essential', minWidth: '75px', maxWidth: '156px' }, - { id: 'disk', label: 'Disk', priority: 'essential', minWidth: '75px', maxWidth: '156px' }, - - // Secondary - visible on md+ (768px), user toggleable - { id: 'ip', label: 'IP', priority: 'secondary', toggleable: true }, - { id: 'uptime', label: 'Uptime', priority: 'secondary', toggleable: true }, - { id: 'node', label: 'Node', priority: 'secondary', toggleable: true }, - - // Supplementary - visible on lg+ (1024px), user toggleable - { id: 'backup', label: 'Backup', priority: 'supplementary', toggleable: true }, - { id: 'os', label: 'OS', priority: 'supplementary', toggleable: true }, - { id: 'tags', label: 'Tags', priority: 'supplementary', toggleable: true }, - - // Detailed - visible on xl+ (1280px), user toggleable - { id: 'diskRead', label: 'D Read', priority: 'detailed', toggleable: true }, - { id: 'diskWrite', label: 'D Write', priority: 'detailed', toggleable: true }, - { id: 'netIn', label: 'Net In', priority: 'detailed', toggleable: true }, - { id: 'netOut', label: 'Net Out', priority: 'detailed', toggleable: true }, -]; -``` - -### Phase 2: Column Visibility State - -Create a hook for managing column visibility: - -```typescript -// hooks/useColumnVisibility.ts -export function useColumnVisibility( - storageKey: string, - columns: ColumnDef[] -) { - // Get toggleable columns - const toggleableIds = columns.filter(c => c.toggleable).map(c => c.id); - - // Default: all toggleable columns visible - const defaultVisible = new Set(toggleableIds); - - // Persist to localStorage - const [hiddenColumns, setHiddenColumns] = usePersistentSignal>( - storageKey, - new Set(), - { - serialize: (set) => JSON.stringify([...set]), - deserialize: (str) => new Set(JSON.parse(str)), - } - ); - - const isVisible = (id: string) => !hiddenColumns().has(id); - const toggle = (id: string) => { - const hidden = new Set(hiddenColumns()); - if (hidden.has(id)) { - hidden.delete(id); - } else { - hidden.add(id); - } - setHiddenColumns(hidden); - }; - - return { isVisible, toggle, hiddenColumns, toggleableIds }; -} -``` - -### Phase 3: Column Picker UI - -Add a column picker dropdown to the filter bar: - -```typescript -// components/shared/ColumnPicker.tsx -
- - - -
- - {(col) => ( - - )} - -
- -
-``` - -### Phase 4: Render New Column Cells - -Add cell renderers in `GuestRow.tsx`: - -```typescript -// IP column - - - - - {guest.ipAddresses?.[0]} - {guest.ipAddresses?.length > 1 && ` +${guest.ipAddresses.length - 1}`} - - - - - -// Backup column - - - - - - -// Node column - - - {guest.node} - - - -// OS column - - - - {guest.osName || '—'} - - - - -// Tags column - - - - - -``` - -### Phase 5: Update Table Header - -Dynamically render headers based on visible columns: - -```typescript - - - - {(col) => ( - col.sortable && handleSort(col.id)} - > - {col.label} - - - - - )} - - - -``` - -### Phase 6: Responsive Behavior - -Combine user preferences with breakpoint-based visibility: - -```typescript -const visibleColumns = createMemo(() => { - const breakpoint = useBreakpoint(); - - return GUEST_COLUMNS.filter(col => { - // Always show essential columns - if (col.priority === 'essential') return true; - - // Check if breakpoint supports this priority - const minBreakpoint = PRIORITY_BREAKPOINTS[col.priority]; - const hasSpace = breakpointIndex(breakpoint()) >= breakpointIndex(minBreakpoint); - - // If toggleable, also check user preference - if (col.toggleable) { - return hasSpace && isVisible(col.id); - } - - return hasSpace; - }); -}); -``` - -## Files to Modify - -1. **`src/components/Dashboard/GuestRow.tsx`** - - Expand `GUEST_COLUMNS` with new columns - - Add cell renderers for IP, backup, node, OS, tags - - Accept `visibleColumns` prop - -2. **`src/components/Dashboard/Dashboard.tsx`** - - Import and use column visibility hook - - Pass visible columns to header and rows - -3. **`src/components/Dashboard/DashboardFilter.tsx`** - - Add ColumnPicker component - -4. **`src/hooks/useColumnVisibility.ts`** (new) - - Create the column visibility management hook - -5. **`src/components/shared/ColumnPicker.tsx`** (new) - - Create the column picker dropdown component - -6. **`src/utils/localStorage.ts`** - - Add `DASHBOARD_COLUMN_VISIBILITY` storage key - -7. **Repeat for Hosts and Docker tabs** - - Similar changes to `HostsOverview.tsx` - - Similar changes to `DockerUnifiedTable.tsx` - -## What Happens to Drawers? - -After columns are implemented: -- **Keep drawers** but make them optional/minimal -- Drawer becomes a place for: - - AI Context annotations (already there) - - Very detailed info (full filesystem list, all network interfaces) - - Actions (future: start/stop/migrate buttons) -- Or **remove drawers entirely** if columns cover everything needed - -## Migration Path - -1. Implement columns first (Phase 1-6) -2. Test with real data -3. Decide what remains valuable in drawers -4. Either slim down drawers or remove them - -## Estimated Scope - -- New hook: ~50 lines -- ColumnPicker component: ~80 lines -- GuestRow changes: ~150 lines (new cells) -- Dashboard changes: ~30 lines (wiring) -- Header changes: ~40 lines -- Repeat for Hosts: ~200 lines -- Repeat for Docker: ~200 lines - -**Total: ~750 lines of new/modified code**