chore: remove internal development documentation

These were internal planning/architecture docs not meant for end users:
- .gemini/docs/unified-resource-architecture.md (design doc)
- .gemini/tasks/persistent-metrics-storage.md (implementation plan)
- frontend-modern/PLAN-column-visibility.md (implementation plan)
This commit is contained in:
rcourtman 2025-12-08 09:27:21 +00:00
parent f7eed7b052
commit 2def8dee09
3 changed files with 0 additions and 1078 deletions

View file

@ -1,594 +0,0 @@
# Unified Resource Architecture
## Status: Draft
## Author: AI Assistant + rcourtman
## Created: 2025-12-07
## Last Updated: 2025-12-07
---
## Executive Summary
Pulse is evolving from a traditional "stare at dashboards" monitoring tool to an **AI-first infrastructure management platform** where AI is the primary operator and humans are supervisors. This document outlines the architectural changes needed to support this vision while preserving the excellent UI that 20,000+ users love.
The core change is introducing a **Unified Resource Model** - a common data abstraction that all platforms (Proxmox, Docker, Hosts, and future platforms like Kubernetes and TrueNAS) normalize to. This enables:
1. **AI intelligence across all platforms** - AI can reason about your entire infrastructure
2. **Elimination of duplicate monitoring** - One machine = one set of alerts
3. **Extensibility for new platforms** - Adding Kubernetes is adding a new resource type, not a new architecture
4. **Foundation for unified views** - Optional consolidated UI without breaking existing pages
---
## Problem Statement
### Current Architecture Issues
#### 1. Multiple Data Sources for Same Machine
When a Proxmox node also has a host agent running, Pulse monitors it twice:
- Via Proxmox API → "Node cpu at 97.9%"
- Via Host Agent → "Host cpu at 99.7%"
Result: Duplicate alerts, user confusion.
#### 2. Siloed Data Models
Each platform has its own types with no common abstraction:
```
Node, VM, Container (Proxmox)
DockerHost, DockerContainer (Docker)
Host (Host Agent)
PBSInstance, PBSDatastore (PBS)
PMGInstance (PMG)
```
AI can't easily answer: "What's using the most CPU across my infrastructure?"
#### 3. Platform-Specific Pages
The frontend has separate components for each platform:
- Dashboard (Proxmox VMs/Containers)
- Docker page
- Hosts page
- Storage page
Adding a new platform (Kubernetes) means building an entirely new page.
#### 4. Agent Capabilities Underutilized
The unified pulse-agent can:
- Report richer metrics than Proxmox API
- Execute commands for AI
- Monitor temperature, RAID, disk I/O at granular level
But the architecture doesn't fully leverage this.
---
## Vision: AI-First Monitoring
```
┌────────────────────────────────────────────────────────────────────────┐
│ TRADITIONAL MONITORING │
│ │
│ Human 👁️ ──watches──▶ Dashboard 📊 ──shows──▶ Metrics/Alerts │
│ │ │
│ └──────────────────── investigates ──▶ fixes manually │
└────────────────────────────────────────────────────────────────────────┘
⬇️
┌────────────────────────────────────────────────────────────────────────┐
│ PULSE: AI-FIRST │
│ │
│ Infrastructure 🖥️ ──reports to──▶ Pulse AI 🤖 │
│ │ │
│ ├── Detects anomalies │
│ ├── Correlates across systems │
│ ├── Diagnoses root causes │
│ ├── Suggests fixes │
│ └── Executes fixes (approved) │
│ │ │
│ ▼ │
│ Human 👤 ──reviews/approves──▶ Dashboard = CONTEXT, not PRIMARY │
└────────────────────────────────────────────────────────────────────────┘
```
The dashboard remains beautiful and functional, but AI becomes the primary interface for investigation and remediation.
---
## Solution: Unified Resource Model
### Core Abstraction
Every monitored entity becomes a `Resource` with common fields:
```go
// Resource is the universal abstraction for any monitored entity
type Resource struct {
// Identity
ID string `json:"id"` // Globally unique ID
Type ResourceType `json:"type"` // vm, container, docker-container, pod, host, etc.
Name string `json:"name"` // Human-readable name
DisplayName string `json:"displayName"` // Custom display name (if set)
// Platform/Source
PlatformID string `json:"platformId"` // Which platform instance
PlatformType PlatformType `json:"platformType"` // proxmox-pve, docker, kubernetes, etc.
SourceType SourceType `json:"sourceType"` // api, agent, hybrid
// Hierarchy
ParentID string `json:"parentId,omitempty"` // VM → Node, Pod → K8s Node
ClusterID string `json:"clusterId,omitempty"` // Cluster membership
// Universal Metrics (nullable - not all resources have all metrics)
Status ResourceStatus `json:"status"` // online, offline, running, stopped, degraded
CPU *MetricValue `json:"cpu,omitempty"`
Memory *MetricValue `json:"memory,omitempty"`
Disk *MetricValue `json:"disk,omitempty"`
Network *NetworkMetric `json:"network,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
Uptime *int64 `json:"uptime,omitempty"`
// Universal Metadata
Tags []string `json:"tags,omitempty"`
Labels map[string]string `json:"labels,omitempty"`
LastSeen time.Time `json:"lastSeen"`
Alerts []ResourceAlert `json:"alerts,omitempty"`
// Platform-Specific Data (discriminated by Type)
// This preserves all the rich data while allowing common handling
PlatformData json.RawMessage `json:"platformData,omitempty"`
}
type ResourceType string
const (
// Infrastructure
ResourceTypeNode ResourceType = "node"
ResourceTypeHost ResourceType = "host"
ResourceTypeDockerHost ResourceType = "docker-host"
ResourceTypeK8sNode ResourceType = "k8s-node"
ResourceTypeTrueNASSystem ResourceType = "truenas-system"
// Compute Workloads
ResourceTypeVM ResourceType = "vm"
ResourceTypeContainer ResourceType = "container" // LXC
ResourceTypeDockerContainer ResourceType = "docker-container"
ResourceTypePod ResourceType = "pod"
ResourceTypeJail ResourceType = "jail"
// Services
ResourceTypeDockerService ResourceType = "docker-service"
ResourceTypeK8sDeployment ResourceType = "k8s-deployment"
ResourceTypeK8sService ResourceType = "k8s-service"
// Storage
ResourceTypeStorage ResourceType = "storage"
ResourceTypeDatastore ResourceType = "datastore"
ResourceTypePool ResourceType = "pool"
ResourceTypeDataset ResourceType = "dataset"
// Backup Systems
ResourceTypePBS ResourceType = "pbs"
ResourceTypePMG ResourceType = "pmg"
)
type PlatformType string
const (
PlatformProxmoxPVE PlatformType = "proxmox-pve"
PlatformProxmoxPBS PlatformType = "proxmox-pbs"
PlatformProxmoxPMG PlatformType = "proxmox-pmg"
PlatformDocker PlatformType = "docker"
PlatformKubernetes PlatformType = "kubernetes"
PlatformTrueNAS PlatformType = "truenas"
PlatformHostAgent PlatformType = "host-agent"
)
type SourceType string
const (
SourceAPI SourceType = "api" // Data from polling an API
SourceAgent SourceType = "agent" // Data pushed from agent
SourceHybrid SourceType = "hybrid" // Both sources, agent preferred
)
```
### Deduplication Strategy
When multiple sources report on the same physical machine:
```go
type ResourceIdentity struct {
Hostname string // Primary identifier
MachineID string // /etc/machine-id or equivalent
IPs []string // Network addresses
}
// IdentityMatcher determines if two resources are the same machine
func (r *ResourceStore) IdentityMatcher(a, b Resource) bool {
// 1. Machine ID match (most reliable)
if a.MachineID != "" && a.MachineID == b.MachineID {
return true
}
// 2. Hostname match (case-insensitive)
if strings.EqualFold(a.Hostname, b.Hostname) {
return true
}
// 3. IP overlap (if same IP, likely same machine)
for _, ipA := range a.IPs {
for _, ipB := range b.IPs {
if ipA == ipB && !isLocalhost(ipA) {
return true
}
}
}
return false
}
```
When duplicates are detected, prefer **Agent > API**:
| Scenario | Result |
|----------|--------|
| Node only (no agent) | Use Node data |
| Host agent only (no Proxmox) | Use Host data |
| Both Node + Host agent | Use Host agent data, suppress Node alerts |
| Docker agent on Proxmox node | Combine: Docker data + Host metrics from agent |
---
## Implementation Phases
### Phase 0: Document & Design (This Document)
- [x] Capture the vision
- [x] Define the data model
- [ ] Review with stakeholders
- [ ] Create GitHub issues/milestones
### Phase 1: Backend Unification (Invisible to Users)
**Goal:** Create the Resource abstraction without changing any frontend behavior.
**Status:** ✅ Core implementation complete
**Completed Tasks:**
1. ✅ Create `internal/resources/resource.go` with types - Core Resource type, enums, and helper methods
2. ✅ Create `internal/resources/platform_data.go` - Platform-specific data types for PlatformData field
3. ✅ Create `internal/resources/store.go` for unified storage - In-memory store with indexes
4. ✅ Create `internal/resources/converters.go` - Converters: FromNode(), FromHost(), FromDockerHost(), FromVM(), FromContainer(), FromDockerContainer(), FromPBSInstance(), FromStorage()
5. ✅ Add deduplication logic using identity matching (hostname, machineID, IP)
6. ✅ Create `internal/resources/converters_test.go` and `store_test.go` - 25 passing tests
7. ✅ Create `internal/api/resource_handlers.go` - HTTP handlers for unified resources API
8. ✅ Add `/api/resources` endpoint with filtering (type, platform, status, parent, infrastructure, workloads)
9. ✅ Add `/api/resources/stats` endpoint for store statistics
10. ✅ Add `/api/resources/{id}` endpoint for individual resource lookup
11. ✅ Add deduplication helper methods for alert manager: `IsSuppressed()`, `GetPreferredResourceFor()`, `IsSamePhysicalMachine()`, `HasPreferredSourceForHostname()`
**Integration Points for Alert Manager:**
The resource store provides these methods that the existing alert manager can optionally use:
- `store.IsSuppressed(resourceID)` - Check if a resource was deduplicated
- `store.GetPreferredResourceFor(resourceID)` - Get the preferred resource
- `store.HasPreferredSourceForHostname(hostname)` - Check if an agent is preferred for a hostname
**Remaining Tasks:**
- [ ] *(Optional)* Enhance alert manager to use resource store methods instead of hostname-only dedup
- [ ] *(Deferred to Phase 2)* Add optional `resources` array to WebSocket state
**Backward Compatibility:**
- WebSocket still sends `nodes`, `vms`, `containers`, `hosts`, `dockerHosts` as separate arrays
- Unified resources available via REST API at `/api/resources`
- All existing frontend code continues to work unchanged
- Existing hostname-based deduplication in alert manager still works
**Metrics:**
- Zero frontend changes
- All existing tests pass
- New resource tests added (25 tests)
- New REST API endpoints available
### Phase 2: AI Context Enhancement
**Goal:** AI chat can query and act across all resource types.
**Status:** ✅ Fully complete
**Completed Tasks:**
1. ✅ Create `internal/ai/resource_context.go` - Unified context builder for AI
2. ✅ Add `ResourceProvider` interface to AI service
3. ✅ Modify `buildSystemPrompt` to use unified model when available
4. ✅ Wire up resource provider in router to AI handlers
5. ✅ AI now uses deduplicated view of infrastructure (falls back to legacy if not available)
6. ✅ Add cross-platform query methods: `GetTopByCPU()`, `GetTopByMemory()`, `GetTopByDisk()`
7. ✅ Add resource correlation: `GetRelated()` for parent/children/siblings/cluster members
8. ✅ Add infrastructure summary: `GetResourceSummary()` with status counts and averages
9. ✅ AI context includes "Top CPU Consumers", "Top Memory Consumers", "Top Disk Usage"
10. ✅ AI context includes infrastructure summary with health status
**How It Works:**
- When `ResourceProvider` is set, AI gets a cleaner "Unified Infrastructure View"
- Resources grouped by platform (Proxmox nodes, Standalone hosts, Docker hosts)
- Workloads grouped by parent infrastructure
- Agent status shown inline with infrastructure
- Resources with alerts highlighted
- **Top consumers** shown for CPU, Memory, and Disk
- **Infrastructure summary** with healthy/degraded/offline counts
**Cross-Platform Query Methods:**
```go
// Find top resource consumers across all platforms
store.GetTopByCPU(10, nil) // Top 10 by CPU, any type
store.GetTopByMemory(5, []ResourceType{ResourceTypeVM}) // Top 5 VMs by memory
// Find related resources
store.GetRelated("vm-123") // Returns parent, children, siblings, cluster_members
// Get infrastructure overview
store.GetResourceSummary() // TotalResources, Healthy, Degraded, Offline, ByType, ByPlatform
```
**User Experience:**
- AI can now answer "What's using the most CPU?" across all platforms
- AI knows about resource relationships (parent nodes, sibling VMs, cluster members)
- AI has infrastructure summary context for better analysis
### Phase 3: Agent Preference & Hybrid Mode
**Goal:** When agents exist, prefer their data over API polling.
**Status:** ✅ Core implementation complete
**Completed Tasks:**
1. ✅ Polling optimization methods added to resource store:
- `ShouldSkipAPIPolling(hostname)` - Check if API polling should be skipped
- `GetAgentMonitoredHostnames()` - Get list of agent-monitored hosts
- `GetPollingRecommendations()` - Get per-hostname polling multipliers
2. ✅ Hybrid source type defined (`SourceHybrid`)
3. ✅ Agent data automatically preferred over API data (store deduplication)
4. ✅ `ResourceStoreInterface` added to Monitor
5. ✅ `SetResourceStore()` method added to inject store into Monitor
6. ✅ `shouldSkipNodeMetrics()` helper method added to Monitor
7. ✅ Resource store wired into Monitor via `Router.SetMonitor()`
**How It Works:**
```go
// In Monitor struct
resourceStore ResourceStoreInterface
// Router injects the store when setting the monitor
func (r *Router) SetMonitor(m *monitoring.Monitor) {
// ... other setup ...
if r.resourceHandlers != nil {
m.SetResourceStore(r.resourceHandlers.Store())
}
}
// Monitor can now check if polling should be skipped
func (m *Monitor) shouldSkipNodeMetrics(nodeName string) bool {
if store := m.resourceStore; store != nil {
return store.ShouldSkipAPIPolling(nodeName)
}
return false
}
```
**How to Enable Polling Optimization:**
To enable polling optimization in the actual polling loops, add this check in `pollPVENode()`:
```go
// In monitor_polling.go, at the start of pollPVENode():
func (m *Monitor) pollPVENode(...) (models.Node, string, error) {
// Skip detailed metric polling if host agent provides data
if m.shouldSkipNodeMetrics(node.Node) {
// Still return basic node info but skip expensive API calls
// like GetNodeStatus, GetStorage, etc.
}
// ... rest of function
}
```
**Remaining Tasks (Future Enhancement):**
- [ ] Add config flag: `EnableAgentPollingOptimization bool`
- [ ] Actually integrate skip logic into `pollPVENode()` and related functions
- [ ] Add Prometheus metrics for skipped polls
- [ ] Add logging for poll optimization decisions
**Benefits:**
- Better data quality (agent metrics are more accurate)
- Reduced API load (skip redundant polling)
- More AI capabilities (command execution via agents)
### Phase 4: Unified Monitoring Page (ABANDONED)
**Goal:** Replace separate platform pages with ONE unified monitoring experience.
**Status:** ❌ **ABANDONED** - Prototyped but reverted in favor of existing pages
**Decision (2025-12-07):**
After building and testing the unified resources view, we determined that the **existing platform-specific pages are superior** in terms of:
- User experience and layout
- Feature richness
- Familiarity for existing users
The backend unified resource model (Phases 1-3) remains valuable for:
- AI context enhancement (cross-platform intelligence)
- Deduplication logic (avoiding duplicate alerts)
- Future API consumers
**What was removed:**
- `src/components/Resources/ResourcesOverview.tsx`
- `src/components/Resources/UnifiedResourceRow.tsx`
- `src/components/Resources/columns.ts`
- `src/types/resource.ts`
- `/resources` route
**What remains:**
- Backend `/api/resources` endpoint (for AI and API consumers)
- Backend unified resource store (`internal/resources/`)
- Platform-specific frontend pages (Dashboard, Docker, Hosts)
**Lesson Learned:**
Sometimes the best architecture is to keep what works. The existing pages have years of refinement and user-driven features. A "unified" view lost much of that polish.
### Phase 5: New Platform Support (Ongoing)
Each new platform follows the pattern:
1. **Collector:** Poll API or receive agent telemetry
2. **Converter:** `PlatformDataToResource()` function
3. **Platform Data:** Type-specific struct stored in `PlatformData` field
4. **UI Components:** (Optional) Platform-specific detail views
Example for Kubernetes:
```go
func K8sNodeToResource(node k8s.Node) Resource {
return Resource{
ID: fmt.Sprintf("k8s/%s/%s", clusterID, node.Name),
Type: ResourceTypeK8sNode,
Name: node.Name,
PlatformType: PlatformKubernetes,
SourceType: SourceAPI,
Status: mapK8sNodeStatus(node.Status),
CPU: extractK8sCPU(node),
Memory: extractK8sMemory(node),
PlatformData: marshalK8sNodeData(node),
}
}
```
---
## Migration Strategy
### Database Considerations
Current state is in-memory with JSON persistence for alerts. The unified model should:
1. **Short-term:** Continue in-memory, Resource is just an abstraction layer
2. **Medium-term:** SQLite metrics store (already implemented) extended for resources
3. **Long-term:** Consider time-series DB for metrics history
### API Compatibility
**Existing endpoints remain unchanged:**
- `/api/state` - Returns current format
- WebSocket `state` message - Returns current format
**New optional endpoints:**
- `/api/resources` - Returns unified resource list
- `/api/resources/{id}` - Returns single resource with full detail
- WebSocket `resources` message - Unified resource updates
### Frontend Migration
**No forced migration.** Existing components continue to work. New unified view is additive.
If we eventually want to migrate existing components:
1. Create `useResources()` hook that abstracts data source
2. Components can switch when ready
3. Old and new can coexist
---
## Success Metrics
### Phase 1 Complete When:
- [x] Resource type defined and tested
- [x] All current types have converters
- [x] Deduplication prevents duplicate alerts (store logic complete)
- [x] Zero frontend changes required
- [x] All existing tests pass
- [x] REST API endpoints available (`/api/resources`, `/api/resources/stats`, `/api/resources/{id}`)
- [x] Deduplication helper methods available for alert manager integration
### Phase 2 Complete When:
- [x] AI context includes unified resource view
- [x] AI can answer cross-platform questions (via GetTopByCPU/Memory/Disk)
- [x] Correlation across platforms works (via GetRelated)
### Phase 3 Complete When:
- [x] Agent data preferred when available (via store deduplication)
- [x] Polling optimization methods available (`ShouldSkipAPIPolling`, `GetPollingRecommendations`)
- [x] Resource store wired into Monitor (via `SetResourceStore`)
- [x] `shouldSkipNodeMetrics()` helper available for polling loops
- [x] AI can execute commands via agents (already implemented)
- [ ] Polling optimization actually used in live polling loops (optional enhancement)
### Phase 4 Complete When:
- [x] Unified view accessible from UI (route: `/resources`) - **ABANDONED** (see below)
- [x] Filtering and grouping works
- [x] Existing pages still work
- [ ] Navigation link added to main UI (optional)
**Note: Phase 4 Direction Changed (2025-12-07)**
The dedicated `/resources` unified view was abandoned in favor of migrating existing pages:
1. **WebSocket State Migration** - COMPLETED
- [x] Backend broadcasts unified resources via `state.resources[]`
- [x] Frontend types defined (`resource.ts`)
- [x] `useResources()` hook available for components
- [x] Hosts page uses unified resources (with fallback)
- [x] Docker page uses unified resources (with fallback)
- [x] Dashboard page uses unified resources (with fallback)
2. **What Was Removed:**
- `/resources` route and ResourcesOverview component
- Frontend-only resource types (replaced by types matching backend)
3. **What Remains:**
- Backend unified resource model (for AI context)
- WebSocket state now includes resources array
- All pages can consume from unified model with legacy fallback
4. **Cleanup (Phase 6) - COMPLETED:**
- [x] Remove debug console.log statements from frontend routes
- [x] Remove legacy fallback code from route components (Docker, Hosts, Dashboard)
- [x] Simplify route components to use centralized `useResourcesAsLegacy()` hook
- [x] Remove legacy AI context fallback (`buildInfrastructureContext` removed, 288 lines)
- [ ] Remove unused legacy arrays from backend StateFrontend (optional - still broadcast for API compatibility)
---
## Open Questions
1. **Should Resource replace existing types entirely, or wrap them?**
- Recommendation: Wrap initially, replace gradually
2. **How to handle platform-specific features in unified view?**
- Recommendation: Show common fields, expand for platform-specific
3. **Should we version the Resource schema?**
- Recommendation: Yes, include version field for future evolution
4. **How to handle offline/stale data in unified model?**
- Recommendation: Include `lastSeen` and `staleness` fields
---
## Appendix: Resource Type Mapping
| Current Type | Resource Type | Platform Type | Notes |
|--------------|---------------|---------------|-------|
| Node | node | proxmox-pve | Proxmox VE node |
| VM | vm | proxmox-pve | Proxmox VM |
| Container | container | proxmox-pve | LXC container |
| Host | host | host-agent | Standalone host |
| DockerHost | docker-host | docker | Docker/Podman host |
| DockerContainer | docker-container | docker | Docker container |
| PBSInstance | pbs | proxmox-pbs | Backup server |
| PBSDatastore | datastore | proxmox-pbs | PBS datastore |
| PMGInstance | pmg | proxmox-pmg | Mail gateway |
| Storage | storage | proxmox-pve | PVE storage |
| CephCluster | ceph-cluster | proxmox-pve | Ceph cluster |
---
## References
- [Conversation that sparked this design](internal discussion 2025-12-07)
- [Host Agent Deduplication Fix](commit implementing hostname-based dedup)
- [Pulse AI Features](current AI implementation)

View file

@ -1,182 +0,0 @@
# Task: Persistent Metrics Storage for Sparklines
## Problem
Currently, metrics history for sparklines is stored **in-memory only**. When the Pulse backend restarts, all historical metrics are lost. Users expect to see historical trends even after being away for days.
## Goal
Implement SQLite-based persistent metrics storage that:
- Survives backend restarts
- Provides historical data for sparklines/trends view
- Supports configurable retention periods
- Minimizes disk I/O and storage footprint
## Architecture
### Storage Tiers (Data Rollup)
```
┌─────────────────────────────────────────────────────────┐
│ RAW (5s intervals) → Keep 2 hours → ~1,440 pts │
│ MINUTE (1min avg) → Keep 24 hours → ~1,440 pts │
│ HOURLY (1hr avg) → Keep 7 days → ~168 pts │
│ DAILY (1day avg) → Keep 90 days → ~90 pts │
└─────────────────────────────────────────────────────────┘
```
### Database Schema
```sql
-- Main metrics table (partitioned by time for efficient pruning)
CREATE TABLE metrics (
id INTEGER PRIMARY KEY AUTOINCREMENT,
resource_type TEXT NOT NULL, -- 'node', 'vm', 'container', 'storage'
resource_id TEXT NOT NULL,
metric_type TEXT NOT NULL, -- 'cpu', 'memory', 'disk'
value REAL NOT NULL,
timestamp INTEGER NOT NULL, -- Unix timestamp in seconds
tier TEXT DEFAULT 'raw' -- 'raw', 'minute', 'hourly', 'daily'
);
-- Indexes for efficient queries
CREATE INDEX idx_metrics_lookup ON metrics(resource_type, resource_id, metric_type, tier, timestamp);
CREATE INDEX idx_metrics_timestamp ON metrics(timestamp);
CREATE INDEX idx_metrics_tier_time ON metrics(tier, timestamp);
```
### Configuration
```yaml
metrics:
enabled: true
database_path: "${PULSE_DATA_DIR}/metrics.db"
retention:
raw: 2h # 2 hours of raw data
minute: 24h # 24 hours of 1-minute averages
hourly: 168h # 7 days of hourly averages
daily: 2160h # 90 days of daily averages
write_buffer: 100 # Buffer size before batch write
rollup_interval: 5m # How often to run rollup job
```
## Implementation Steps
### Phase 1: SQLite Foundation ✅ COMPLETED
- [x] Add SQLite dependency (`modernc.org/sqlite` - pure Go, no CGO)
- [x] Create `internal/metrics/store.go` with:
- `Store` struct
- `NewStore(config StoreConfig) (*Store, error)`
- `Close() error`
- Schema auto-migration on startup
### Phase 2: Write Path ✅ COMPLETED
- [x] Create `Write(resourceType, resourceID, metricType string, value float64, timestamp time.Time)`
- [x] Implement write buffering (batch inserts every 100 records or 5 seconds)
- [x] Integrate with existing `AddGuestMetric`, `AddNodeMetric` calls in monitor.go and monitor_polling.go
- [x] Add graceful shutdown to flush buffer
### Phase 3: Read Path ✅ COMPLETED
- [x] Create `Query(resourceType, resourceID, metricType string, start, end time.Time) ([]MetricPoint, error)`
- [x] Auto-select appropriate tier based on time range:
- < 2 hours raw data
- 2-24 hours → minute data
- 1-7 days → hourly data
- 7+ days → daily data
- [x] Add `/api/metrics-store/stats` endpoint for monitoring
### Phase 4: Rollup & Retention ✅ COMPLETED
- [x] Create background rollup job:
- Runs every 5 minutes
- Aggregates raw → minute (AVG, MIN, MAX)
- Aggregates minute → hourly
- Aggregates hourly → daily
- [x] Create retention pruning job:
- Runs every hour
- Deletes data older than configured retention
- [x] Use SQLite transactions for atomic operations
### Phase 5: Integration
- [ ] Add configuration to `system.json` or new `metrics.json`
- [ ] Add Settings UI for metrics retention config
- [ ] Add database file size monitoring
- [ ] Add vacuum/optimize scheduled job (weekly)
## Files to Create/Modify
### New Files
```
internal/metrics/
├── store.go # MetricsStore implementation
├── store_test.go # Unit tests
├── rollup.go # Rollup/aggregation logic
├── retention.go # Retention/pruning logic
└── config.go # Metrics configuration
```
### Files to Modify
```
internal/monitoring/monitor.go # Initialize MetricsStore, call Write()
internal/monitoring/metrics_history.go # Keep in-memory as cache, backed by SQLite
internal/api/router.go # Update handleCharts to query from store
internal/config/persistence.go # Add metrics config persistence
```
## API Changes
### `/api/charts` Query Parameters
```
GET /api/charts?range=1h # Last hour (raw/minute data)
GET /api/charts?range=24h # Last 24 hours (minute data)
GET /api/charts?range=7d # Last 7 days (hourly data)
GET /api/charts?range=30d # Last 30 days (daily data)
GET /api/charts?start=...&end=... # Custom range
```
### Response Enhancement
```json
{
"data": { ... },
"nodeData": { ... },
"stats": {
"oldestDataTimestamp": 1699900000000,
"tier": "hourly",
"pointCount": 168
}
}
```
## Performance Considerations
1. **Write Buffering**: Batch inserts to reduce I/O
2. **WAL Mode**: Enable SQLite WAL for concurrent reads/writes
3. **Prepared Statements**: Reuse for repeated queries
4. **Index Strategy**: Composite index on (resource_type, resource_id, metric_type, tier, timestamp)
5. **Connection Pooling**: Single connection with proper locking for SQLite
6. **Memory Mapping**: Use `PRAGMA mmap_size` for faster reads
## Storage Estimates
For a typical Pulse installation (5 nodes, 50 VMs, 20 containers, 10 storage):
- 85 resources × 3 metrics = 255 metric series
- Raw (2h at 5s): ~86,400 rows → ~10 MB
- Minute (24h): ~367,200 rows → ~40 MB
- Hourly (7d): ~42,840 rows → ~5 MB
- Daily (90d): ~22,950 rows → ~3 MB
- **Total: ~60-100 MB** for comprehensive historical data
## Testing Plan
1. Unit tests for store CRUD operations
2. Unit tests for rollup logic
3. Integration tests with mock monitor
4. Performance tests with 100+ resources
5. Restart resilience tests
## Rollout Plan
1. Implement as opt-in feature (disable by default initially)
2. Add migration path from in-memory to SQLite
3. Test in dev environment for 1 week
4. Enable by default in next minor release
## Definition of Done
- [ ] SQLite metrics storage implemented
- [ ] Data survives backend restart
- [ ] Rollup/retention working correctly
- [ ] Charts endpoint serves historical data
- [ ] Documentation updated
- [ ] Settings UI for retention config
- [ ] Performance validated (no noticeable slowdown)

View file

@ -1,302 +0,0 @@
# Plan: Toggleable Table Columns
## Problem
The current drawer pattern hides useful information (IPs, OS, backup status, node, tags) that users need to click to reveal. This goes against Pulse's core philosophy of dense, scannable, comparable data at a glance.
## Goal
Replace drawer-hidden info with optional table columns that:
1. Show data inline for easy comparison across rows
2. Are toggleable by the user (show/hide)
3. Auto-show based on available horizontal space
4. Persist user preferences
## Current State
### Infrastructure Already Exists
- `ColumnPriority` system: `'essential' | 'primary' | 'secondary' | 'supplementary' | 'detailed'`
- `PRIORITY_BREAKPOINTS` maps priorities to responsive breakpoints
- `usePersistentSignal` for localStorage persistence
- `STANDARD_COLUMNS` with predefined column configs
- `useBreakpoint` hook for responsive behavior
### Current Columns (Dashboard/Proxmox)
All marked `essential` (always visible):
- Name, Type, VMID, Uptime
- CPU, Memory, Disk (progress bars)
- Disk Read, Disk Write, Net In, Net Out
### Data Available but Hidden in Drawer
- IP addresses (`guest.ipAddresses`)
- OS name/version (`guest.osName`, `guest.osVersion`)
- Node (`guest.node`)
- Backup status (`guest.lastBackup`)
- Tags (`guest.tags`)
- CPUs allocated (`guest.cpus`)
- Agent version (`guest.agentVersion`)
## Implementation
### Phase 1: Add New Column Definitions
Update `GUEST_COLUMNS` in `GuestRow.tsx`:
```typescript
export const GUEST_COLUMNS: ColumnDef[] = [
// Essential - always visible
{ id: 'name', label: 'Name', priority: 'essential' },
{ id: 'type', label: 'Type', priority: 'essential' },
{ id: 'vmid', label: 'VMID', priority: 'essential' },
// Primary - visible on sm+ (640px)
{ id: 'cpu', label: 'CPU', priority: 'essential', minWidth: '55px', maxWidth: '156px' },
{ id: 'memory', label: 'Memory', priority: 'essential', minWidth: '75px', maxWidth: '156px' },
{ id: 'disk', label: 'Disk', priority: 'essential', minWidth: '75px', maxWidth: '156px' },
// Secondary - visible on md+ (768px), user toggleable
{ id: 'ip', label: 'IP', priority: 'secondary', toggleable: true },
{ id: 'uptime', label: 'Uptime', priority: 'secondary', toggleable: true },
{ id: 'node', label: 'Node', priority: 'secondary', toggleable: true },
// Supplementary - visible on lg+ (1024px), user toggleable
{ id: 'backup', label: 'Backup', priority: 'supplementary', toggleable: true },
{ id: 'os', label: 'OS', priority: 'supplementary', toggleable: true },
{ id: 'tags', label: 'Tags', priority: 'supplementary', toggleable: true },
// Detailed - visible on xl+ (1280px), user toggleable
{ id: 'diskRead', label: 'D Read', priority: 'detailed', toggleable: true },
{ id: 'diskWrite', label: 'D Write', priority: 'detailed', toggleable: true },
{ id: 'netIn', label: 'Net In', priority: 'detailed', toggleable: true },
{ id: 'netOut', label: 'Net Out', priority: 'detailed', toggleable: true },
];
```
### Phase 2: Column Visibility State
Create a hook for managing column visibility:
```typescript
// hooks/useColumnVisibility.ts
export function useColumnVisibility(
storageKey: string,
columns: ColumnDef[]
) {
// Get toggleable columns
const toggleableIds = columns.filter(c => c.toggleable).map(c => c.id);
// Default: all toggleable columns visible
const defaultVisible = new Set(toggleableIds);
// Persist to localStorage
const [hiddenColumns, setHiddenColumns] = usePersistentSignal<Set<string>>(
storageKey,
new Set(),
{
serialize: (set) => JSON.stringify([...set]),
deserialize: (str) => new Set(JSON.parse(str)),
}
);
const isVisible = (id: string) => !hiddenColumns().has(id);
const toggle = (id: string) => {
const hidden = new Set(hiddenColumns());
if (hidden.has(id)) {
hidden.delete(id);
} else {
hidden.add(id);
}
setHiddenColumns(hidden);
};
return { isVisible, toggle, hiddenColumns, toggleableIds };
}
```
### Phase 3: Column Picker UI
Add a column picker dropdown to the filter bar:
```typescript
// components/shared/ColumnPicker.tsx
<div class="relative">
<button
onClick={() => setOpen(!open())}
class="inline-flex items-center gap-1.5 px-2.5 py-1.5 text-xs font-medium rounded-lg
bg-gray-100 dark:bg-gray-700 text-gray-600 dark:text-gray-300
hover:bg-gray-200 dark:hover:bg-gray-600"
>
<ColumnsIcon class="w-3.5 h-3.5" />
Columns
</button>
<Show when={open()}>
<div class="absolute right-0 mt-1 w-48 rounded-lg border bg-white dark:bg-gray-800 shadow-lg z-50">
<For each={toggleableColumns}>
{(col) => (
<label class="flex items-center gap-2 px-3 py-2 hover:bg-gray-50 dark:hover:bg-gray-700 cursor-pointer">
<input
type="checkbox"
checked={isVisible(col.id)}
onChange={() => toggle(col.id)}
/>
<span class="text-sm">{col.label}</span>
</label>
)}
</For>
</div>
</Show>
</div>
```
### Phase 4: Render New Column Cells
Add cell renderers in `GuestRow.tsx`:
```typescript
// IP column
<Show when={visibleColumns().includes('ip')}>
<td class="...">
<Show when={guest.ipAddresses?.length}>
<span class="text-xs font-mono truncate" title={guest.ipAddresses?.join(', ')}>
{guest.ipAddresses?.[0]}
{guest.ipAddresses?.length > 1 && ` +${guest.ipAddresses.length - 1}`}
</span>
</Show>
</td>
</Show>
// Backup column
<Show when={visibleColumns().includes('backup')}>
<td class="...">
<BackupStatusBadge lastBackup={guest.lastBackup} />
</td>
</Show>
// Node column
<Show when={visibleColumns().includes('node')}>
<td class="...">
<span class="text-xs truncate">{guest.node}</span>
</td>
</Show>
// OS column
<Show when={visibleColumns().includes('os')}>
<td class="...">
<span class="text-xs truncate" title={`${guest.osName} ${guest.osVersion}`}>
{guest.osName || '—'}
</span>
</td>
</Show>
// Tags column
<Show when={visibleColumns().includes('tags')}>
<td class="...">
<TagBadges tags={guest.tags} compact />
</td>
</Show>
```
### Phase 5: Update Table Header
Dynamically render headers based on visible columns:
```typescript
<thead>
<tr>
<For each={visibleColumns()}>
{(col) => (
<th
class="..."
onClick={() => col.sortable && handleSort(col.id)}
>
{col.label}
<Show when={sortKey() === col.id}>
<SortIndicator direction={sortDirection()} />
</Show>
</th>
)}
</For>
</tr>
</thead>
```
### Phase 6: Responsive Behavior
Combine user preferences with breakpoint-based visibility:
```typescript
const visibleColumns = createMemo(() => {
const breakpoint = useBreakpoint();
return GUEST_COLUMNS.filter(col => {
// Always show essential columns
if (col.priority === 'essential') return true;
// Check if breakpoint supports this priority
const minBreakpoint = PRIORITY_BREAKPOINTS[col.priority];
const hasSpace = breakpointIndex(breakpoint()) >= breakpointIndex(minBreakpoint);
// If toggleable, also check user preference
if (col.toggleable) {
return hasSpace && isVisible(col.id);
}
return hasSpace;
});
});
```
## Files to Modify
1. **`src/components/Dashboard/GuestRow.tsx`**
- Expand `GUEST_COLUMNS` with new columns
- Add cell renderers for IP, backup, node, OS, tags
- Accept `visibleColumns` prop
2. **`src/components/Dashboard/Dashboard.tsx`**
- Import and use column visibility hook
- Pass visible columns to header and rows
3. **`src/components/Dashboard/DashboardFilter.tsx`**
- Add ColumnPicker component
4. **`src/hooks/useColumnVisibility.ts`** (new)
- Create the column visibility management hook
5. **`src/components/shared/ColumnPicker.tsx`** (new)
- Create the column picker dropdown component
6. **`src/utils/localStorage.ts`**
- Add `DASHBOARD_COLUMN_VISIBILITY` storage key
7. **Repeat for Hosts and Docker tabs**
- Similar changes to `HostsOverview.tsx`
- Similar changes to `DockerUnifiedTable.tsx`
## What Happens to Drawers?
After columns are implemented:
- **Keep drawers** but make them optional/minimal
- Drawer becomes a place for:
- AI Context annotations (already there)
- Very detailed info (full filesystem list, all network interfaces)
- Actions (future: start/stop/migrate buttons)
- Or **remove drawers entirely** if columns cover everything needed
## Migration Path
1. Implement columns first (Phase 1-6)
2. Test with real data
3. Decide what remains valuable in drawers
4. Either slim down drawers or remove them
## Estimated Scope
- New hook: ~50 lines
- ColumnPicker component: ~80 lines
- GuestRow changes: ~150 lines (new cells)
- Dashboard changes: ~30 lines (wiring)
- Header changes: ~40 lines
- Repeat for Hosts: ~200 lines
- Repeat for Docker: ~200 lines
**Total: ~750 lines of new/modified code**