docs: add analytics & user feedback design document

Covers team analytics, personal analytics, flow analytics, step-level thumbs up/down feedback, and flow CSAT ratings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 23:43:36 -05:00
parent 293ceaa9e9
commit 448ce4b53f
1 changed files with 294 additions and 0 deletions
--- a/docs/plans/2026-02-15-analytics-feedback-design.md
+++ b/docs/plans/2026-02-15-analytics-feedback-design.md
@@ -0,0 +1,294 @@
+# Analytics & User Feedback — Design Document
+
+> **Date:** February 15, 2026
+> **Status:** Approved
+> **Audience:** Team admins + individual engineers (both equally)
+
+---
+
+## Goal
+
+Add analytics dashboards and user feedback systems to ResolutionFlow so MSP managers can measure team productivity and flow effectiveness, engineers can track their own performance, and flow authors get actionable feedback to improve their troubleshooting trees.
+
+## Architecture
+
+Live queries against existing PostgreSQL tables (sessions, trees, step_library) with one new table (session_ratings). No materialized views or external analytics services — direct aggregation queries with proper indexes. Time-series data returned as daily-bucketed arrays for Recharts visualization on the frontend.
+
+## Tech Stack Additions
+
+- **Backend:** New endpoint modules (`analytics.py`, `ratings.py`), one Alembic migration
+- **Frontend:** Recharts (charting), two new pages, one new modal, inline step feedback components
+- **No new infrastructure** — runs on existing PostgreSQL + Railway deployment
+
+---
+
+## 1. Feedback System Design
+
+### Step-Level Feedback: Thumbs Up / Thumbs Down
+
+**When:** Inline during session navigation, always visible on each step.
+
+**UX:**
+- Small thumb-up and thumb-down icons displayed on each step in TreeNavigationPage and ProceduralNavigationPage
+- Non-intrusive: muted icons that highlight on selection
+- First-time tooltip: "Rate this step to help improve flows" (dismissible, shown once)
+- After rating: selected thumb highlights (green for up, red for down), other thumb dims
+- Tapping the same thumb again un-rates (toggle behavior)
+
+**Data model:**
+- Uses existing `step_ratings` table with `was_helpful` boolean
+- One rating per user per step per session (unique constraint on step_id + user_id + session_id)
+- Updates `step_library.helpful_yes` / `helpful_no` aggregate counts
+
+**Endpoint:**
+- `POST /steps/{step_id}/feedback` — body: `{ session_id, was_helpful: true|false }`
+- `DELETE /steps/{step_id}/feedback/{session_id}` — un-rate
+
+### Flow-Level Feedback: CSAT 1-5 + Optional Comment
+
+**When:** After session completion, prompted after the SessionOutcomeModal.
+
+**UX:**
+- Modal with 1-5 star selector (or numbered buttons)
+- Optional comment textarea (500 char max)
+- "Skip" button to dismiss without rating
+- Shown once per completed session
+
+**Data model:**
+- New `session_ratings` table:
+  - `id` UUID PK
+  - `session_id` FK unique (one rating per session)
+  - `user_id` FK
+  - `tree_id` FK (denormalized for aggregation)
+  - `account_id` FK (denormalized for team scoping)
+  - `rating` Integer 1-5 (CHECK constraint)
+  - `comment` String(500), nullable
+  - `created_at` DateTime(timezone=True)
+
+**Endpoint:**
+- `POST /sessions/{session_id}/rate` — body: `{ rating: 1-5, comment?: string }`
+- `GET /trees/{tree_id}/ratings` — paginated list of ratings/comments for flow authors
+
+---
+
+## 2. Analytics Endpoints
+
+All analytics endpoints accept `?period=7d|30d|90d` (default 30d) and return data scoped to the user's account.
+
+### Team Analytics — `GET /analytics/team`
+
+**Access:** team_admin or super_admin only.
+
+**Response:**
+```json
+{
+  "summary": {
+    "total_sessions": 247,
+    "completed_sessions": 198,
+    "completion_rate": 0.801,
+    "avg_duration_minutes": 12.4,
+    "active_engineers": 8,
+    "outcome_breakdown": {
+      "resolved": 142,
+      "escalated": 31,
+      "workaround": 18,
+      "unresolved": 7
+    }
+  },
+  "time_series": [
+    { "date": "2026-02-01", "sessions": 12, "resolved": 8, "escalated": 2, "workaround": 1, "unresolved": 1 },
+    ...
+  ],
+  "top_flows": [
+    { "tree_id": "...", "name": "DNS Resolution", "sessions": 42, "completion_rate": 0.88, "avg_duration_minutes": 8.2, "avg_csat": 4.1 },
+    ...
+  ],
+  "top_engineers": [
+    { "user_id": "...", "name": "Jane Smith", "sessions": 34, "completion_rate": 0.91, "avg_duration_minutes": 10.1 },
+    ...
+  ]
+}
+```
+
+**Optional filter:** `?engineer_id=<uuid>` to scope to one engineer.
+
+### Personal Analytics — `GET /analytics/me`
+
+**Access:** Any authenticated user.
+
+**Response:** Same shape as team analytics but scoped to the requesting user only. No engineer leaderboard — replaced with "my top flows" list.
+
+### Flow Analytics — `GET /analytics/flows/{tree_id}`
+
+**Access:** Anyone who can view the flow.
+
+**Response:**
+```json
+{
+  "summary": {
+    "total_sessions": 42,
+    "completion_rate": 0.88,
+    "avg_duration_minutes": 8.2,
+    "avg_csat": 4.1,
+    "total_ratings": 28,
+    "outcome_breakdown": { ... }
+  },
+  "time_series": [
+    { "date": "2026-02-01", "sessions": 3, "avg_duration_minutes": 7.5 },
+    ...
+  ],
+  "step_feedback": [
+    { "node_id": "abc", "node_title": "Check DNS Settings", "helpful_yes": 18, "helpful_no": 2, "helpful_rate": 0.9 },
+    ...
+  ],
+  "recent_comments": [
+    { "rating": 5, "comment": "Very helpful flow", "user_name": "John", "created_at": "..." },
+    ...
+  ]
+}
+```
+
+---
+
+## 3. Frontend Pages
+
+### Team Analytics Page — `/analytics`
+
+**Access:** team_admin+ (hidden from engineers/viewers in nav).
+
+**Layout:**
+- Page header: "Team Analytics" + period dropdown (7d / 30d / 90d)
+- Row 1: Stat cards — Total Sessions, Completion Rate, Avg Duration, Active Engineers
+- Row 2: Time-series chart (sessions per day, stacked by outcome) — Recharts AreaChart
+- Row 3: Two columns:
+  - Left: Flow Leaderboard table (top flows by usage, with completion rate + avg duration + CSAT)
+  - Right: Engineer Leaderboard table (top engineers by session count, with success rate + avg duration)
+
+### My Analytics Page — `/analytics/me`
+
+**Access:** Any authenticated user.
+
+**Layout:**
+- Page header: "My Analytics" + period dropdown
+- Row 1: Stat cards — My Sessions, My Completion Rate, My Avg Duration, My Outcome Split
+- Row 2: Sessions-per-day line chart
+- Row 3: Two columns:
+  - Left: My Top Flows table (most-used flows with personal stats)
+  - Right: Outcome distribution donut chart (Recharts PieChart)
+
+### Flow Analytics Panel
+
+**Location:** New tab or expandable section on tree detail/editor views.
+
+**Layout:**
+- Summary stat cards: Usage, Completion Rate, Avg Duration, CSAT
+- Session trend mini-chart (sparkline or small area chart)
+- Step feedback table: each step with helpful rate bar + thumbs count
+- Recent CSAT comments list (latest 5-10)
+
+### Navigation
+
+- Sidebar: Add "Analytics" nav item with BarChart3 icon between "Sessions" and "Exports"
+- Team admins see `/analytics` (team view) as default
+- Engineers see `/analytics/me` as default
+- Both can navigate between views if they have permission
+
+---
+
+## 4. Database Migration
+
+**New table: `session_ratings`**
+
+```sql
+CREATE TABLE session_ratings (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    session_id UUID NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
+    user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+    tree_id UUID NOT NULL REFERENCES trees(id) ON DELETE CASCADE,
+    account_id UUID NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
+    rating INTEGER NOT NULL CHECK (rating >= 1 AND rating <= 5),
+    comment VARCHAR(500),
+    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+    UNIQUE (session_id)
+);
+```
+
+**New indexes for analytics queries:**
+
+```sql
+CREATE INDEX ix_session_ratings_tree_created ON session_ratings(tree_id, created_at);
+CREATE INDEX ix_session_ratings_account_created ON session_ratings(account_id, created_at);
+CREATE INDEX ix_sessions_account_completed ON sessions(account_id, completed_at);
+CREATE INDEX ix_sessions_account_tree_completed ON sessions(account_id, tree_id, completed_at);
+CREATE INDEX ix_step_ratings_step_helpful ON step_ratings(step_id, was_helpful);
+```
+
+**Modify `step_ratings` usage:**
+- Existing `rating` (1-5) and `review_text` columns remain in DB but are no longer populated
+- Only `was_helpful` boolean is used going forward
+- Add `session_id` to unique constraint if not already present: `UNIQUE(step_id, user_id, session_id)`
+
+---
+
+## 5. Charting Library
+
+**Recharts** (`recharts` npm package)
+- React-native, composable components
+- Supports: AreaChart (time-series), BarChart (comparisons), PieChart (outcome donut), LineChart (trends)
+- Lightweight (~45KB gzipped)
+- Dark theme compatible via custom colors matching our design tokens
+
+**Chart color palette** (matching design system):
+- Primary: `hsl(243, 75%, 59%)` (purple — matches `--primary`)
+- Resolved: `#34d399` (emerald-400)
+- Escalated: `#f87171` (red-400)
+- Workaround: `#fbbf24` (yellow-400)
+- Unresolved: `#94a3b8` (slate-400)
+
+---
+
+## 6. Step Feedback UX Detail
+
+**Inline thumbs placement:**
+- Positioned at the bottom of each step card, right-aligned
+- Two icons: ThumbsUp and ThumbsDown from Lucide
+- Default state: `text-muted-foreground` (subtle, not distracting)
+- Hover: icon scales slightly, tooltip appears
+- Selected up: `text-emerald-400` with subtle fill
+- Selected down: `text-red-400` with subtle fill
+- Toggle: clicking selected thumb un-selects it
+
+**First-time hint:**
+- On the first session where thumbs are available, show a subtle inline note below the first step: "New: Rate steps with thumbs to help improve flows"
+- Store dismissal in localStorage
+- Auto-dismisses after first thumb interaction
+
+**CSAT Modal (post-completion):**
+- Appears after SessionOutcomeModal closes
+- Five numbered buttons (1-5) or star icons in a row
+- Label: "How would you rate this flow?"
+- Sublabel: "Your feedback helps flow authors improve"
+- Optional textarea: "Any comments? (optional)"
+- Buttons: "Submit" (primary) + "Skip" (text link)
+- localStorage tracks which sessions have been rated to prevent re-prompting
+
+---
+
+## 7. Scope & Non-Goals
+
+**In scope:**
+- Team analytics dashboard with time-series charts
+- Personal analytics dashboard
+- Flow-level analytics panel
+- Step thumbs up/down inline feedback
+- Flow CSAT 1-5 + comment at session end
+- Period filtering (7d/30d/90d)
+
+**Not in scope (future):**
+- Real-time streaming analytics
+- Export analytics to CSV/PDF
+- Custom date range picker (just preset periods for v1)
+- Comparison mode (this period vs. last period)
+- Automation adoption metrics
+- Client/ticket correlation analytics
+- Email digest/reports