chore: clean up old plan docs and add analytics improvement notes
Remove 15 completed/superseded plan documents. Add analytics improvements reference and visual QA design migration notes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
chihlasm
364
docs/plans/plan-improvements-for-analytics.md
Normal file
364
docs/plans/plan-improvements-for-analytics.md
Normal file
@@ -0,0 +1,364 @@
|
||||
# Post-Implementation Improvements — Analytics & Feedback
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Context:** The analytics & feedback feature has just been implemented from `docs/plans/2026-02-15-analytics-feedback-implementation.md`. These are five targeted improvements identified from a comparative review. Apply them to the existing code on the current branch (`feat/analytics-feedback`).
|
||||
|
||||
**Reference files you'll need to modify:**
|
||||
- `backend/app/api/endpoints/analytics.py` — analytics query logic
|
||||
- `backend/app/schemas/analytics.py` — response schemas
|
||||
- `backend/app/api/endpoints/ratings.py` — rating endpoints
|
||||
- `backend/app/api/endpoints/steps.py` — existing step rating routes
|
||||
- `frontend/src/types/analytics.ts` — TypeScript types
|
||||
- `frontend/src/pages/TeamAnalyticsPage.tsx` — team dashboard
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Switch from Average to Median Duration
|
||||
|
||||
**Why:** One 3-hour debugging session shouldn't skew stats for a flow that normally takes 8 minutes. Median is the standard for time-based metrics in analytics.
|
||||
|
||||
**Backend changes:**
|
||||
|
||||
In `backend/app/schemas/analytics.py`:
|
||||
- Rename `avg_duration_minutes` → `median_duration_minutes` in `AnalyticsSummary`, `TopFlow`, and `TopEngineer`
|
||||
|
||||
In `backend/app/api/endpoints/analytics.py`:
|
||||
- Replace `func.avg(...)` duration calculations with a median approach
|
||||
- PostgreSQL supports `percentile_cont(0.5) WITHIN GROUP (ORDER BY ...)` for median
|
||||
- Example replacement for the `_build_summary` helper:
|
||||
|
||||
```python
|
||||
from sqlalchemy import text
|
||||
|
||||
# Replace the avg duration query with:
|
||||
duration_q = await db.execute(
|
||||
select(
|
||||
func.percentile_cont(0.5).within_group(
|
||||
func.extract('epoch', Session.completed_at - Session.started_at) / 60
|
||||
)
|
||||
).where(*base_filter, Session.completed_at.isnot(None))
|
||||
)
|
||||
median_duration = round(float(duration_q.scalar() or 0), 1)
|
||||
```
|
||||
|
||||
- Apply the same change to the top_flows and top_engineers subqueries (replace `func.avg` with `percentile_cont(0.5)` for duration)
|
||||
- Update all references from `avg_duration` to `median_duration` in the response construction
|
||||
|
||||
**Frontend changes:**
|
||||
|
||||
In `frontend/src/types/analytics.ts`:
|
||||
- Rename `avg_duration_minutes` → `median_duration_minutes` in `TopFlow`, `TopEngineer`, and `AnalyticsSummary`
|
||||
|
||||
In `frontend/src/pages/TeamAnalyticsPage.tsx` and `frontend/src/pages/MyAnalyticsPage.tsx`:
|
||||
- Update stat card label from "Avg Duration" → "Median Duration"
|
||||
- Update any references to `avg_duration_minutes` → `median_duration_minutes`
|
||||
|
||||
In `frontend/src/components/analytics/FlowAnalyticsPanel.tsx`:
|
||||
- Same rename for the flow summary stat card
|
||||
|
||||
**Verify:** `cd backend && python -m pytest --override-ini="addopts=" -v` and `cd frontend && npm run build`
|
||||
|
||||
**Commit:** `git commit -am "refactor: use median instead of average for duration metrics"`
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Add Step Dropoff Metrics to Flow Analytics
|
||||
|
||||
**Why:** Dropoff data is passive (requires no user action) and tells flow authors exactly where engineers get stuck and abandon sessions. This is the most actionable metric for flow improvement.
|
||||
|
||||
**Backend changes:**
|
||||
|
||||
In `backend/app/schemas/analytics.py`:
|
||||
- Add fields to `StepFeedbackSummary`:
|
||||
|
||||
```python
|
||||
class StepFeedbackSummary(BaseModel):
|
||||
node_id: str
|
||||
node_title: str
|
||||
helpful_yes: int
|
||||
helpful_no: int
|
||||
helpful_rate: float
|
||||
visit_count: int = 0 # NEW
|
||||
dropoff_count: int = 0 # NEW
|
||||
dropoff_rate: float = 0.0 # NEW
|
||||
```
|
||||
|
||||
In `backend/app/api/endpoints/analytics.py`, in the `get_flow_analytics` function:
|
||||
- After the existing step_feedback section, add dropoff calculation logic:
|
||||
|
||||
```python
|
||||
# Step dropoff analysis — build from session decisions JSONB
|
||||
# For each session in the period for this tree:
|
||||
# - Extract all node_ids from the decisions array
|
||||
# - The LAST node_id in an incomplete session = dropoff point
|
||||
# - Count visits per node and dropoffs per node
|
||||
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
|
||||
# Get all sessions for this tree in period
|
||||
sessions_q = await db.execute(
|
||||
select(Session.id, Session.decisions, Session.completed_at)
|
||||
.where(Session.tree_id == tree_id, Session.started_at >= period_start)
|
||||
)
|
||||
sessions_data = sessions_q.all()
|
||||
|
||||
# Build node visit and dropoff maps
|
||||
node_visits: dict[str, int] = {}
|
||||
node_dropoffs: dict[str, int] = {}
|
||||
|
||||
for sess in sessions_data:
|
||||
decisions = sess.decisions or []
|
||||
for decision in decisions:
|
||||
node_id = decision.get("node_id") or decision.get("nodeId", "")
|
||||
if node_id:
|
||||
node_visits[node_id] = node_visits.get(node_id, 0) + 1
|
||||
|
||||
# If session not completed, last decision node is a dropoff
|
||||
if not sess.completed_at and decisions:
|
||||
last_decision = decisions[-1]
|
||||
last_node = last_decision.get("node_id") or last_decision.get("nodeId", "")
|
||||
if last_node:
|
||||
node_dropoffs[last_node] = node_dropoffs.get(last_node, 0) + 1
|
||||
|
||||
# Merge dropoff data into step_feedback list
|
||||
# Get node titles from the tree's content JSONB
|
||||
tree_result = await db.execute(select(Tree.content).where(Tree.id == tree_id))
|
||||
tree_content = tree_result.scalar_one_or_none() or {}
|
||||
nodes = tree_content.get("nodes", [])
|
||||
node_title_map = {n.get("id", ""): n.get("title", n.get("label", "Unnamed")) for n in nodes}
|
||||
|
||||
# Build combined step feedback with visits + dropoffs + thumbs
|
||||
all_node_ids = set(list(node_visits.keys()) + [sf.node_id for sf in step_feedback])
|
||||
combined_feedback = []
|
||||
for nid in all_node_ids:
|
||||
visits = node_visits.get(nid, 0)
|
||||
dropoffs = node_dropoffs.get(nid, 0)
|
||||
# Find existing thumbs data if any
|
||||
existing = next((sf for sf in step_feedback if sf.node_id == nid), None)
|
||||
combined_feedback.append(StepFeedbackSummary(
|
||||
node_id=nid,
|
||||
node_title=node_title_map.get(nid, "Unknown Step"),
|
||||
helpful_yes=existing.helpful_yes if existing else 0,
|
||||
helpful_no=existing.helpful_no if existing else 0,
|
||||
helpful_rate=existing.helpful_rate if existing else 0.0,
|
||||
visit_count=visits,
|
||||
dropoff_count=dropoffs,
|
||||
dropoff_rate=round(dropoffs / visits, 3) if visits > 0 else 0.0,
|
||||
))
|
||||
|
||||
# Sort by dropoff_rate descending so worst steps are first
|
||||
combined_feedback.sort(key=lambda x: x.dropoff_rate, reverse=True)
|
||||
step_feedback = combined_feedback
|
||||
```
|
||||
|
||||
**Frontend changes:**
|
||||
|
||||
In `frontend/src/types/analytics.ts`:
|
||||
- Add to `StepFeedbackSummary`:
|
||||
|
||||
```typescript
|
||||
export interface StepFeedbackSummary {
|
||||
node_id: string
|
||||
node_title: string
|
||||
helpful_yes: number
|
||||
helpful_no: number
|
||||
helpful_rate: number
|
||||
visit_count: number // NEW
|
||||
dropoff_count: number // NEW
|
||||
dropoff_rate: number // NEW
|
||||
}
|
||||
```
|
||||
|
||||
In `frontend/src/components/analytics/FlowAnalyticsPanel.tsx`:
|
||||
- Add dropoff columns to the step feedback table: "Visits", "Dropoffs", "Dropoff Rate"
|
||||
- Highlight rows where `dropoff_rate > 0.2` with a subtle red/warning background
|
||||
|
||||
**Verify:** `cd backend && python -m pytest --override-ini="addopts=" -v` and `cd frontend && npm run build`
|
||||
|
||||
**Commit:** `git commit -am "feat: add step dropoff metrics to flow analytics"`
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Add Backward-Compatible /ratings Alias Routes
|
||||
|
||||
**Why:** The existing backend uses `/{step_id}/rate` but the frontend uses `/{step_id}/ratings`. This is a real inconsistency that needs fixing.
|
||||
|
||||
**Backend changes:**
|
||||
|
||||
In `backend/app/api/endpoints/steps.py`:
|
||||
- Find the existing routes: `POST /{step_id}/rate`, `PUT /{step_id}/rate`, `DELETE /{step_id}/rate`
|
||||
- Add alias routes that point to the same handler functions:
|
||||
|
||||
```python
|
||||
# After the existing rate_step function:
|
||||
@router.post("/{step_id}/ratings", response_model=StepRatingResponse, status_code=status.HTTP_201_CREATED,
|
||||
include_in_schema=False)
|
||||
async def rate_step_alias(
|
||||
step_id: UUID,
|
||||
rating_data: StepRatingCreate,
|
||||
db: AsyncSession = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user)
|
||||
):
|
||||
"""Alias for POST /{step_id}/rate — backward compatibility."""
|
||||
return await rate_step(step_id, rating_data, db, current_user)
|
||||
|
||||
|
||||
@router.put("/{step_id}/ratings", response_model=StepRatingResponse,
|
||||
include_in_schema=False)
|
||||
async def update_rating_alias(
|
||||
step_id: UUID,
|
||||
rating_data: StepRatingUpdate,
|
||||
db: AsyncSession = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user)
|
||||
):
|
||||
"""Alias for PUT /{step_id}/rate — backward compatibility."""
|
||||
return await update_rating(step_id, rating_data, db, current_user)
|
||||
|
||||
|
||||
@router.delete("/{step_id}/ratings", status_code=204,
|
||||
include_in_schema=False)
|
||||
async def delete_rating_alias(
|
||||
step_id: UUID,
|
||||
db: AsyncSession = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user)
|
||||
):
|
||||
"""Alias for DELETE /{step_id}/rate — backward compatibility."""
|
||||
return await delete_rating(step_id, db, current_user)
|
||||
```
|
||||
|
||||
Note: Use `include_in_schema=False` to keep the OpenAPI docs clean — only the canonical `/rate` routes appear in docs.
|
||||
|
||||
**Verify:** `cd backend && python -m pytest --override-ini="addopts=" -v`
|
||||
|
||||
**Commit:** `git commit -am "fix: add /ratings alias routes for backward compatibility"`
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Anonymize Feedback Comments in Analytics
|
||||
|
||||
**Why:** In a team tool where managers see feedback, engineers will give more honest feedback if their identity isn't attached. Anonymous feedback = better signal.
|
||||
|
||||
**Backend changes:**
|
||||
|
||||
In `backend/app/schemas/analytics.py`:
|
||||
- Remove `user_name` from `FlowRatingItem`:
|
||||
|
||||
```python
|
||||
class FlowRatingItem(BaseModel):
|
||||
rating: int
|
||||
comment: Optional[str]
|
||||
created_at: datetime
|
||||
# user_name removed — feedback is anonymous in analytics views
|
||||
```
|
||||
|
||||
In `backend/app/api/endpoints/analytics.py`, in `get_flow_analytics`:
|
||||
- Remove the `User.name` join from the recent_comments query:
|
||||
|
||||
```python
|
||||
# Recent comments — anonymous
|
||||
comments_q = await db.execute(
|
||||
select(SessionRating.rating, SessionRating.comment, SessionRating.created_at)
|
||||
.where(
|
||||
SessionRating.tree_id == tree_id,
|
||||
SessionRating.comment.isnot(None),
|
||||
SessionRating.comment != "",
|
||||
)
|
||||
.order_by(SessionRating.created_at.desc())
|
||||
.limit(10)
|
||||
)
|
||||
recent_comments = [
|
||||
FlowRatingItem(
|
||||
rating=row.rating,
|
||||
comment=row.comment,
|
||||
created_at=row.created_at,
|
||||
)
|
||||
for row in comments_q.all()
|
||||
]
|
||||
```
|
||||
|
||||
**Frontend changes:**
|
||||
|
||||
In `frontend/src/types/analytics.ts`:
|
||||
- Remove `user_name` from `FlowRatingItem`:
|
||||
|
||||
```typescript
|
||||
export interface FlowRatingItem {
|
||||
rating: number
|
||||
comment?: string
|
||||
created_at: string
|
||||
// user_name removed — anonymous feedback
|
||||
}
|
||||
```
|
||||
|
||||
In `frontend/src/components/analytics/FlowAnalyticsPanel.tsx`:
|
||||
- Remove any rendering of `user_name` in the recent comments list
|
||||
- Comments should show: star rating + comment text + relative timestamp only
|
||||
|
||||
**Verify:** `cd backend && python -m pytest --override-ini="addopts=" -v` and `cd frontend && npm run build`
|
||||
|
||||
**Commit:** `git commit -am "privacy: anonymize feedback comments in analytics views"`
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Add Explicit active_engineers Metric
|
||||
|
||||
**Why:** "How many of my techs actually used this tool this month" is a key adoption metric for MSP managers.
|
||||
|
||||
**Backend changes:**
|
||||
|
||||
In `backend/app/schemas/analytics.py`:
|
||||
- Verify `AnalyticsSummary` already has an `active_engineers` field. If not, add:
|
||||
|
||||
```python
|
||||
class AnalyticsSummary(BaseModel):
|
||||
total_sessions: int
|
||||
completed_sessions: int
|
||||
completion_rate: float
|
||||
median_duration_minutes: float
|
||||
active_engineers: int = 0 # ADD if missing
|
||||
outcome_breakdown: OutcomeBreakdown
|
||||
```
|
||||
|
||||
In `backend/app/api/endpoints/analytics.py`:
|
||||
- In the `_build_summary` helper (or the team analytics endpoint), add the active engineers count:
|
||||
|
||||
```python
|
||||
# Active engineers: distinct users with >= 1 session in window
|
||||
active_q = await db.execute(
|
||||
select(func.count(func.distinct(Session.user_id)))
|
||||
.where(*base_filter)
|
||||
)
|
||||
active_engineers = active_q.scalar() or 0
|
||||
```
|
||||
|
||||
- Include `active_engineers=active_engineers` in the `AnalyticsSummary(...)` construction
|
||||
- For personal analytics (`/me`), set `active_engineers=1` (it's always just the requesting user)
|
||||
|
||||
**Frontend changes:**
|
||||
|
||||
In `frontend/src/types/analytics.ts`:
|
||||
- Verify `AnalyticsSummary` has `active_engineers: number`. Add if missing.
|
||||
|
||||
In `frontend/src/pages/TeamAnalyticsPage.tsx`:
|
||||
- Verify there's a stat card for "Active Engineers". If missing, add one using the `summary.active_engineers` value.
|
||||
|
||||
**Verify:** `cd backend && python -m pytest --override-ini="addopts=" -v` and `cd frontend && npm run build`
|
||||
|
||||
**Commit:** `git commit -am "feat: add explicit active_engineers metric to team analytics"`
|
||||
|
||||
---
|
||||
|
||||
## Final: Full Test Suite + Build Verification
|
||||
|
||||
```bash
|
||||
cd backend && python -m pytest --override-ini="addopts=" -v
|
||||
cd frontend && npm run build
|
||||
```
|
||||
|
||||
Fix any failures, then:
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix: post-improvement test and build fixes"
|
||||
```
|
||||
Reference in New Issue
Block a user