chore: remove completed phase summary documents
Removed PHASE1_SUMMARY.md and PHASE2_SUMMARY.md as both phases are complete. All relevant documentation has been integrated into the main docs: - Security hardening docs in SECURITY.md - Adaptive polling architecture in docs/monitoring/ADAPTIVE_POLLING.md
This commit is contained in:
rcourtman
parent
b3f37a798c
commit
fa21e9c69c
2 changed files with 0 additions and 235 deletions
|
|
@ -1,71 +0,0 @@
|
|||
# Pulse Sensor Proxy – Phase 1 Summary
|
||||
|
||||
## Executive Summary
|
||||
Phase 1 delivered a complete hardening and observability overhaul for the Pulse sensor proxy. The service now runs under least privilege, exposes tamper-evident audit trails, forwards logs off-host, enforces adaptive rate caps, and ships with comprehensive validation tests plus documentation for ongoing operations and security posture. These improvements dramatically reduce the proxy's attack surface while giving operators clear visibility and controls.
|
||||
|
||||
## Security Improvements
|
||||
- **Host hardening**
|
||||
- SSH daemon locked down (no passwords, no forwarding, `ForceCommand` wrapper).
|
||||
- Dedicated user `pulse-sensor` with minimal home/project directories.
|
||||
- File permissions tightened (0750 binaries, 0600 private keys, 0640 append-only logs).
|
||||
- Privilege drop via `Setuid/Setgid` post-bind; service confirms running as unprivileged UID 995.
|
||||
- **Command execution guardrails**
|
||||
- Whitelist-based command validator for `sensors`/`ipmitool`; rejects shell metacharacters, subshells, dangerous ipmitool subcommands, null bytes, and path traversal.
|
||||
- Enhanced node-name validation covering unicode, length, and absolute path abuse.
|
||||
- **Logging & audit**
|
||||
- Structured audit logger with hash chain + HMAC-style tamper detection.
|
||||
- Remote forwarding via `rsyslog` (RELP/TLS) and local queue for resilience.
|
||||
- **Sandboxing**
|
||||
- Network segmentation documentation with firewall ACLs.
|
||||
- AppArmor profile restricting filesystem/networking; seccomp profile (classic + OCI JSON).
|
||||
- **Rate limiting**
|
||||
- Per-UID token bucket (0.2 QPS burst 2) + global concurrency cap (8) + penalty sleeps.
|
||||
- Audit + metrics instrumentation for limiter decisions and penalties.
|
||||
|
||||
## Key Metrics & Tests
|
||||
- `go test ./cmd/pulse-sensor-proxy/...` passes (including new unit suites for command validation, sanitizer, limiter penalties, audit logging).
|
||||
- 10 hostile-command attack cases covered (metacharacters, subshells, redirects, homoglyphs, null bytes, long args, path traversal, dangerous ipmitool ops, env prefixes, absolute paths).
|
||||
- Fuzz harness (`FuzzValidateCommand`) executing for 24 h (Task #24).
|
||||
- Prometheus metrics validated:
|
||||
- `pulse_proxy_limiter_rejections_total{reason="rate"}` increments under load.
|
||||
- `pulse_proxy_limiter_penalties_total{reason="invalid_json"}` increments on validation failure.
|
||||
- `pulse_proxy_limiter_active_peers` accurate (UID grouping).
|
||||
- Audit log entries verified: connection acceptance, limiter rejections, validation failures, command start/finish with event hash chaining.
|
||||
|
||||
## Deployment Checklist
|
||||
1. **Scripts**
|
||||
- Run `scripts/create-sensor-user.sh`
|
||||
- Run `scripts/harden-sensor-proxy.sh`
|
||||
- Run `scripts/secure-sensor-files.sh`
|
||||
- Run `scripts/setup-log-forwarding.sh`
|
||||
2. **Binaries**
|
||||
- Build/install `/opt/pulse/sensor-proxy/bin/pulse-sensor-proxy` (0750, root:pulse-sensor).
|
||||
3. **Configuration**
|
||||
- `/etc/pulse-sensor-proxy/config.yaml` updated (allowed subnets, UID/GID list, metrics addr).
|
||||
- Systemd unit exports `PULSE_SENSOR_PROXY_USER`, `PULSE_SENSOR_PROXY_SSH_DIR`, `PULSE_SENSOR_PROXY_AUDIT_LOG`.
|
||||
4. **Profiles**
|
||||
- Deploy AppArmor profile (`security/apparmor/pulse-sensor-proxy.apparmor`).
|
||||
- Apply seccomp (systemd `SystemCallFilter` overrides or container JSON profile).
|
||||
5. **Networking**
|
||||
- Implement firewall ACLs per `docs/security/pulse-sensor-proxy-network.md`.
|
||||
6. **Log Forwarding**
|
||||
- Place TLS certs in `/etc/pulse/log-forwarding`.
|
||||
- Verify rsyslog forwarding to remote collector.
|
||||
7. **Restart & Validate**
|
||||
- `systemctl restart pulse-sensor-proxy`.
|
||||
- Confirm metrics endpoint, audit log creation, limiter behaviour.
|
||||
|
||||
## Verification Steps
|
||||
1. **Privilege Drop**: `ps -o user= -p $(pgrep -f pulse-sensor-proxy)` → `pulse-sensor`.
|
||||
2. **Audit Trail**: Trigger RPC (`get_status`) → verify `audit.log` entries with valid `event_hash`.
|
||||
3. **Rate Limiter**: Fire >10 concurrent requests → confirm `pulse_proxy_limiter_rejections_total{reason="rate"}` and audit `limiter.rejection`.
|
||||
4. **Remote Logging**: `logger` or manual append to proxy log → confirm arrival at remote collector.
|
||||
5. **Security Profiles**: `aa-status | grep pulse-sensor-proxy` (enforced), `systemctl show pulse-sensor-proxy -p SystemCallFilter`.
|
||||
6. **App Functionality**: Run `ensure_cluster_keys`, `get_temperature` RPCs, ensure success and no audit warnings.
|
||||
|
||||
## Known Limitations / Deferred to Phase 2
|
||||
- **Adaptive Polling**: still fixed intervals (Phase 2 focuses on controller, backpressure, staleness SLOs).
|
||||
- **Queue Backpressure**: groundwork in rate limiter; full queue-based collector scheduling to be built next.
|
||||
- **External Sentinels**: cross-check monitoring and metric ingestion planned in Phase 3.
|
||||
- **AppArmor/Seccomp Tuning**: profiles may need refinement after real-world observation.
|
||||
- **Long-run Fuzz Results**: Task #24 fuzz campaign active; incorporate findings post-run.
|
||||
|
|
@ -1,164 +0,0 @@
|
|||
# Pulse Adaptive Polling – Phase 2 Summary
|
||||
|
||||
## Executive Summary
|
||||
Phase 2 delivers adaptive polling infrastructure that dynamically adjusts monitoring intervals based on instance freshness, error rates, and system load. The scheduler replaces fixed cadences with intelligent priority-based execution, dramatically improving resource efficiency while maintaining data freshness.
|
||||
|
||||
## Completed Tasks (9/10 - 90%)
|
||||
|
||||
### ✅ Task 1: Poll Cycle Metrics
|
||||
- 7 new Prometheus metrics (duration, staleness, queue depth, in-flight, errors)
|
||||
- Per-instance tracking with histogram/counter/gauge types
|
||||
- Integrated into all poll functions (PVE/PBS/PMG)
|
||||
- Metrics server on port 9091
|
||||
|
||||
### ✅ Task 2: Adaptive Scheduler Scaffold
|
||||
- Pluggable interfaces (StalenessSource, IntervalSelector, TaskEnqueuer)
|
||||
- BuildPlan generates ordered task lists with NextRun times
|
||||
- FilterDue/DispatchDue for queue management
|
||||
- Default no-op implementations for gradual rollout
|
||||
|
||||
### ✅ Task 3: Configuration & Feature Flags
|
||||
- `ADAPTIVE_POLLING_ENABLED` feature flag (default: false)
|
||||
- Min/max/base interval tuning (5s / 5m / 10s defaults)
|
||||
- Environment variable overrides
|
||||
- Persisted in system.json
|
||||
- Validation logic (min ≤ base ≤ max)
|
||||
|
||||
### ✅ Task 4: Staleness Tracker
|
||||
- Per-instance freshness metadata (last success/error/mutation)
|
||||
- SHA1 change hash detection
|
||||
- Normalized staleness scoring (0-1 scale)
|
||||
- Integration with PollMetrics for authoritative timestamps
|
||||
- Updates from all poll result handlers
|
||||
|
||||
### ✅ Task 5: Adaptive Interval Logic
|
||||
- EMA smoothing (alpha=0.6) to prevent oscillations
|
||||
- Staleness-based interpolation (min-max range)
|
||||
- Error penalty (0.6x per failure) for faster recovery detection
|
||||
- Queue depth stretch (0.1x per task) for backpressure
|
||||
- ±5% jitter to avoid thundering herd
|
||||
|
||||
### ✅ Task 6: Priority Queue Execution
|
||||
- Min-heap (container/heap) ordered by NextRun + Priority
|
||||
- Worker goroutines with WaitNext() blocking
|
||||
- Tasks only execute when due (respects adaptive intervals)
|
||||
- Automatic rescheduling after execution
|
||||
- Upsert semantics prevent duplicates
|
||||
|
||||
### ✅ Task 7: Error Handling & Circuit Breakers
|
||||
- Circuit breaker with closed/open/half-open states
|
||||
- Trips after 3 consecutive failures
|
||||
- Exponential backoff (5s initial, 2x multiplier, 5min max)
|
||||
- ±20% jitter on retry delays
|
||||
- Dead-letter queue after 5 transient failures
|
||||
- Error classification (transient vs permanent)
|
||||
|
||||
### ✅ Task 8: API Surfaces
|
||||
- GET /api/monitoring/scheduler/health endpoint (auth required)
|
||||
- Scheduler health snapshot (queue, dead-letter, breakers, staleness)
|
||||
- Snapshot methods: StalenessTracker.Snapshot(), TaskQueue.Snapshot()
|
||||
- Circuit breaker state export via circuitBreaker.State()
|
||||
- Dead-letter queue inspection (top 25 tasks with error details)
|
||||
|
||||
### ✅ Task 9: Unit Testing (Partial)
|
||||
- Backoff logic tests (13 test cases): exponential growth, jitter, capping
|
||||
- Circuit breaker tests (10 test cases): state machine, retry intervals, half-open window
|
||||
- Staleness tracker tests (17 test cases): scoring, normalization, snapshot API
|
||||
- **Total: 40+ unit tests** covering core scheduling math
|
||||
- ✅ All tests passing (`go test ./internal/monitoring`)
|
||||
|
||||
### ✅ Task 10: Documentation
|
||||
- Architecture guide in `docs/monitoring/ADAPTIVE_POLLING.md`
|
||||
- API endpoint specification with example responses
|
||||
- Configuration reference
|
||||
- Metrics catalog
|
||||
- Operational guidance & troubleshooting
|
||||
- Rollout plan (dev → staged → full)
|
||||
|
||||
## Deferred Tasks (1/10 - 10%)
|
||||
|
||||
### ⏭ Task 9b: Integration & Soak Testing (Future Phase)
|
||||
- Integration tests with mock PVE/PBS clients
|
||||
- Soak tests for queue stability over extended runtime
|
||||
- Regression suite for Phase 1 hardening features
|
||||
|
||||
## Key Metrics Delivered
|
||||
|
||||
| Metric | Purpose |
|
||||
|--------|---------|
|
||||
| `pulse_monitor_poll_total` | Success/error rate tracking |
|
||||
| `pulse_monitor_poll_duration_seconds` | Latency per instance |
|
||||
| `pulse_monitor_poll_staleness_seconds` | Data freshness indicator |
|
||||
| `pulse_monitor_poll_queue_depth` | Backpressure monitoring |
|
||||
| `pulse_monitor_poll_inflight` | Concurrency tracking |
|
||||
| `pulse_monitor_poll_errors_total` | Error type classification |
|
||||
| `pulse_monitor_poll_last_success_timestamp` | Recovery timeline |
|
||||
|
||||
## Technical Achievements
|
||||
|
||||
**Performance:**
|
||||
- Adaptive intervals reduce unnecessary polls on idle instances
|
||||
- Queue-based execution prevents task pile-up
|
||||
- Circuit breakers stop hot loops on failing endpoints
|
||||
|
||||
**Reliability:**
|
||||
- Exponential backoff with jitter prevents thundering herd
|
||||
- Dead-letter queue isolates persistent failures
|
||||
- Transient error retry logic (5 attempts before DLQ)
|
||||
|
||||
**Observability:**
|
||||
- 7 Prometheus metrics for complete visibility
|
||||
- Structured logging for all state transitions
|
||||
- Tamper-evident audit trail (from Phase 1)
|
||||
|
||||
## Deployment Status
|
||||
|
||||
**Current State:** Feature flag disabled by default (`ADAPTIVE_POLLING_ENABLED=false`)
|
||||
|
||||
**Activation Path:**
|
||||
1. Enable flag in dev/QA environment
|
||||
2. Observe metrics for 24-48 hours
|
||||
3. Staged rollout to subset of production clusters
|
||||
4. Full activation after validation
|
||||
|
||||
**Rollback:** Set `ADAPTIVE_POLLING_ENABLED=false` to revert to fixed intervals
|
||||
|
||||
## Git Commits
|
||||
|
||||
1. `c048e7b9b` - Tasks 1-3: Metrics + scheduler + config
|
||||
2. `8ce93c1df` - Task 4: Staleness tracker
|
||||
3. `e8bd79c6c` - Task 5: Adaptive interval logic
|
||||
4. `1d6fa9188` - Task 6: Priority queue execution
|
||||
5. `7d9aaa406` - Task 7: Circuit breakers & backoff
|
||||
6. `fdd2bebe4` - Task 10: Documentation
|
||||
7. `1b969d008` - Task 8: Scheduler health API endpoint
|
||||
8. `938a8153f` - Task 9a: Backoff & circuit breaker tests
|
||||
9. `6d3786041` - Task 9b: Staleness tracker tests
|
||||
|
||||
## Phase 2 Success Criteria ✅
|
||||
|
||||
- [x] Metrics pipeline operational
|
||||
- [x] Scheduler produces valid task plans
|
||||
- [x] Queue respects adaptive intervals
|
||||
- [x] Circuit breakers prevent runaway failures
|
||||
- [x] Documentation enables ops team rollout
|
||||
- [x] Feature flag allows safe activation
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- Dead-letter queue state lost on restart (no persistence yet)
|
||||
- No integration/soak test coverage (deferred from Task 9)
|
||||
- Queue depth metric updated per-cycle (not real-time within cycle)
|
||||
- API endpoint read-only (no circuit breaker reset or DLQ management)
|
||||
|
||||
## Next Steps
|
||||
|
||||
**Immediate (Post-Phase 2):**
|
||||
- Deploy to dev environment with flag enabled
|
||||
- Configure Grafana dashboards for new metrics
|
||||
- Set alerting thresholds (queue depth >50, staleness >60s)
|
||||
|
||||
**Future Phases:**
|
||||
- Task 9b: Integration tests with mock clients + soak testing
|
||||
- Phase 3: External sentinels and cross-cluster coordination
|
||||
- API enhancements: Write endpoints for circuit breaker reset and DLQ management
|
||||
Loading…
Reference in a new issue