docs: add Phase 5 analytics enhancement design document
Tabbed analytics page with coverage heatmap, flow quality scoring, and PSA metrics. Extends existing FlowPilot analytics infrastructure. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
166
docs/plans/2026-03-19-phase5-analytics-enhancement-design.md
Normal file
166
docs/plans/2026-03-19-phase5-analytics-enhancement-design.md
Normal file
@@ -0,0 +1,166 @@
|
||||
# Phase 5: Analytics Enhancement — Design Document
|
||||
|
||||
> **Date:** 2026-03-19
|
||||
> **Status:** Approved
|
||||
> **Audience:** Team leads (primary), individual engineers (secondary)
|
||||
> **Future note:** A dedicated `/insights` dashboard is planned for a later phase — keep analytics queries modular for reuse.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Extend the existing FlowPilot Analytics page with tabbed sections for coverage analysis, flow quality scoring, and PSA metrics. This completes the pivot architecture's Phase 5 ("Analytics + Polish") and provides the data team leads need to justify ROI and identify knowledge gaps.
|
||||
|
||||
## Page Structure
|
||||
|
||||
Add 4 tab sections to `FlowPilotAnalyticsPage`. Each tab fetches data lazily (only when selected).
|
||||
|
||||
| Tab | Content | Data Source |
|
||||
|-----|---------|-------------|
|
||||
| **Overview** | Existing metrics (sessions, resolution rate, MTTR, rating, confidence tiers, domain breakdown) | Existing `GET /analytics/flowpilot` |
|
||||
| **Coverage** | Domain grid heatmap, domain-to-flow mapping, knowledge gap summary | New `GET /analytics/flowpilot/coverage` |
|
||||
| **Flow Quality** | Flow scoring table, top/bottom performers, needs-attention badges | New `GET /analytics/flowpilot/flow-quality` |
|
||||
| **PSA** | Time entries, hours tracked, push success funnel, trend chart | Extended existing endpoint + new tracking |
|
||||
|
||||
---
|
||||
|
||||
## Backend
|
||||
|
||||
### New Endpoint: Coverage Data
|
||||
|
||||
`GET /analytics/flowpilot/coverage?period=7d|30d|90d`
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"domains": [
|
||||
{
|
||||
"domain": "Active Directory",
|
||||
"flow_count": 12,
|
||||
"session_count": 87,
|
||||
"resolution_rate": 0.92,
|
||||
"escalation_rate": 0.05,
|
||||
"guided_rate": 0.78,
|
||||
"avg_resolution_minutes": 12.5
|
||||
}
|
||||
],
|
||||
"unmapped_session_count": 15,
|
||||
"total_domains": 8
|
||||
}
|
||||
```
|
||||
|
||||
Domain-to-flow mapping: match flows to domains via their category (categories already exist on trees). Sessions have `problem_domain`. Cross-reference to compute coverage per domain.
|
||||
|
||||
### New Endpoint: Flow Quality Data
|
||||
|
||||
`GET /analytics/flowpilot/flow-quality?period=7d|30d|90d&sort=quality|usage|success_rate`
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"flows": [
|
||||
{
|
||||
"flow_id": "uuid",
|
||||
"name": "AD Password Reset",
|
||||
"usage_count": 45,
|
||||
"success_rate": 0.91,
|
||||
"last_matched_at": "2026-03-18T...",
|
||||
"avg_confidence": 0.82,
|
||||
"quality_score": 0.87
|
||||
}
|
||||
],
|
||||
"top_performers": [...],
|
||||
"needs_attention": [...]
|
||||
}
|
||||
```
|
||||
|
||||
Quality score formula: `(success_rate * 0.5) + (guided_rate * 0.3) + (recency_score * 0.2)` where recency_score decays from 1.0 (used today) to 0.0 (not used in 90+ days).
|
||||
|
||||
### Flow Model Additions
|
||||
|
||||
Add to `trees` table:
|
||||
- `usage_count` (Integer, default 0) — incremented when a session matches this flow
|
||||
- `success_rate` (Float, nullable) — recalculated periodically by analytics query
|
||||
- `last_matched_at` (DateTime(timezone=True), nullable) — updated on flow match
|
||||
|
||||
### PSA Activity Tracking
|
||||
|
||||
Add `psa_activity_log` table:
|
||||
- `id` (UUID PK)
|
||||
- `account_id` (UUID FK)
|
||||
- `session_id` (UUID FK, nullable)
|
||||
- `activity_type` (String) — "time_entry_posted", "note_posted", "status_updated"
|
||||
- `hours_logged` (Float, nullable) — for time entries
|
||||
- `psa_ticket_id` (String)
|
||||
- `created_at` (DateTime)
|
||||
|
||||
Wire into the ConnectWise provider: when a time entry or note is successfully pushed, log to this table.
|
||||
|
||||
Extended PSA analytics response:
|
||||
```json
|
||||
{
|
||||
"total_time_entries": 142,
|
||||
"total_hours_logged": 356.5,
|
||||
"avg_hours_per_session": 2.51,
|
||||
"push_funnel": {
|
||||
"total_sessions": 500,
|
||||
"linked_to_ticket": 380,
|
||||
"doc_pushed": 350,
|
||||
"time_entry_logged": 142
|
||||
},
|
||||
"daily_trend": [
|
||||
{"date": "2026-03-18", "entries": 8, "hours": 19.5}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Frontend
|
||||
|
||||
### Coverage Tab — Domain Grid Heatmap
|
||||
|
||||
A table with colored cells:
|
||||
|
||||
| Domain | Flows | Sessions | Resolution % | Escalation % | Guided % |
|
||||
|--------|-------|----------|-------------|-------------|---------|
|
||||
| Active Directory | 12 | 87 | 92% (green) | 5% (green) | 78% (green) |
|
||||
| Microsoft 365 | 3 | 45 | 58% (red) | 32% (red) | 28% (red) |
|
||||
|
||||
Cell coloring:
|
||||
- Resolution rate: `bg-emerald-400/10` (>75%), `bg-amber-400/10` (50-75%), `bg-rose-500/10` (<50%)
|
||||
- Escalation rate: green (<10%), amber (10-25%), red (>25%)
|
||||
- Guided rate: green (>60%), amber (30-60%), red (<30%)
|
||||
- Flow count: green (5+), amber (1-4), red (0)
|
||||
|
||||
Domains with red cells show "Create Flow" suggestion.
|
||||
|
||||
### Flow Quality Tab — Sortable Table
|
||||
|
||||
`.glass-card-static` table:
|
||||
- Columns: Flow Name, Usage, Success Rate, Last Used, Avg Confidence, Quality Score
|
||||
- Top 5 highlighted in emerald, bottom 5 in rose
|
||||
- "Needs attention" badge: success_rate < 50% or unused 30+ days
|
||||
- Click flow name → navigate to editor
|
||||
|
||||
### PSA Tab
|
||||
|
||||
- Metric cards: total entries, total hours, avg hours/session
|
||||
- Push success funnel: horizontal funnel visualization with conversion rates
|
||||
- Trend chart (Recharts area chart): daily time entries and hours
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
Backend:
|
||||
- Coverage endpoint returns correct domain mapping
|
||||
- Flow quality scoring formula produces expected results
|
||||
- PSA activity logging works on push
|
||||
- All endpoints team_admin gated
|
||||
|
||||
Frontend:
|
||||
- Tab switching loads correct data
|
||||
- Heatmap cells colored correctly
|
||||
- Flow quality table sorts correctly
|
||||
- PSA funnel displays conversion rates
|
||||
Reference in New Issue
Block a user