resolutionflow/docs/plans/archive/plan-improvements-for-analytics.md

Post-Implementation Improvements — Analytics & Feedback

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Context: The analytics & feedback feature has just been implemented from docs/plans/2026-02-15-analytics-feedback-implementation.md. These are five targeted improvements identified from a comparative review. Apply them to the existing code on the current branch (feat/analytics-feedback).

Reference files you'll need to modify:

• backend/app/api/endpoints/analytics.py — analytics query logic
• backend/app/schemas/analytics.py — response schemas
• backend/app/api/endpoints/ratings.py — rating endpoints
• backend/app/api/endpoints/steps.py — existing step rating routes
• frontend/src/types/analytics.ts — TypeScript types
• frontend/src/pages/TeamAnalyticsPage.tsx — team dashboard

---

Task 1: Switch from Average to Median Duration

Why: One 3-hour debugging session shouldn't skew stats for a flow that normally takes 8 minutes. Median is the standard for time-based metrics in analytics.

Backend changes:

In backend/app/schemas/analytics.py:

• Rename avg_duration_minutes → median_duration_minutes in AnalyticsSummary, TopFlow, and TopEngineer

In backend/app/api/endpoints/analytics.py:

• Replace func.avg(...) duration calculations with a median approach
• PostgreSQL supports percentile_cont(0.5) WITHIN GROUP (ORDER BY ...) for median
• Example replacement for the _build_summary helper:

from sqlalchemy import text

# Replace the avg duration query with:
duration_q = await db.execute(
    select(
        func.percentile_cont(0.5).within_group(
            func.extract('epoch', Session.completed_at - Session.started_at) / 60
        )
    ).where(*base_filter, Session.completed_at.isnot(None))
)
median_duration = round(float(duration_q.scalar() or 0), 1)

• Apply the same change to the top_flows and top_engineers subqueries (replace func.avg with percentile_cont(0.5) for duration)
• Update all references from avg_duration to median_duration in the response construction

Frontend changes:

In frontend/src/types/analytics.ts:

• Rename avg_duration_minutes → median_duration_minutes in TopFlow, TopEngineer, and AnalyticsSummary

In frontend/src/pages/TeamAnalyticsPage.tsx and frontend/src/pages/MyAnalyticsPage.tsx:

• Update stat card label from "Avg Duration" → "Median Duration"
• Update any references to avg_duration_minutes → median_duration_minutes

In frontend/src/components/analytics/FlowAnalyticsPanel.tsx:

• Same rename for the flow summary stat card

Verify: cd backend && python -m pytest --override-ini="addopts=" -v and cd frontend && npm run build

Commit: git commit -am "refactor: use median instead of average for duration metrics"

---

Task 2: Add Step Dropoff Metrics to Flow Analytics

Why: Dropoff data is passive (requires no user action) and tells flow authors exactly where engineers get stuck and abandon sessions. This is the most actionable metric for flow improvement.

Backend changes:

In backend/app/schemas/analytics.py:

• Add fields to StepFeedbackSummary:

class StepFeedbackSummary(BaseModel):
    node_id: str
    node_title: str
    helpful_yes: int
    helpful_no: int
    helpful_rate: float
    visit_count: int = 0        # NEW
    dropoff_count: int = 0      # NEW
    dropoff_rate: float = 0.0   # NEW

In backend/app/api/endpoints/analytics.py, in the get_flow_analytics function:

• After the existing step_feedback section, add dropoff calculation logic:

# Step dropoff analysis — build from session decisions JSONB
# For each session in the period for this tree:
#   - Extract all node_ids from the decisions array
#   - The LAST node_id in an incomplete session = dropoff point
#   - Count visits per node and dropoffs per node

from sqlalchemy.dialects.postgresql import JSONB

# Get all sessions for this tree in period
sessions_q = await db.execute(
    select(Session.id, Session.decisions, Session.completed_at)
    .where(Session.tree_id == tree_id, Session.started_at >= period_start)
)
sessions_data = sessions_q.all()

# Build node visit and dropoff maps
node_visits: dict[str, int] = {}
node_dropoffs: dict[str, int] = {}

for sess in sessions_data:
    decisions = sess.decisions or []
    for decision in decisions:
        node_id = decision.get("node_id") or decision.get("nodeId", "")
        if node_id:
            node_visits[node_id] = node_visits.get(node_id, 0) + 1
    
    # If session not completed, last decision node is a dropoff
    if not sess.completed_at and decisions:
        last_decision = decisions[-1]
        last_node = last_decision.get("node_id") or last_decision.get("nodeId", "")
        if last_node:
            node_dropoffs[last_node] = node_dropoffs.get(last_node, 0) + 1

# Merge dropoff data into step_feedback list
# Get node titles from the tree's content JSONB
tree_result = await db.execute(select(Tree.content).where(Tree.id == tree_id))
tree_content = tree_result.scalar_one_or_none() or {}
nodes = tree_content.get("nodes", [])
node_title_map = {n.get("id", ""): n.get("title", n.get("label", "Unnamed")) for n in nodes}

# Build combined step feedback with visits + dropoffs + thumbs
all_node_ids = set(list(node_visits.keys()) + [sf.node_id for sf in step_feedback])
combined_feedback = []
for nid in all_node_ids:
    visits = node_visits.get(nid, 0)
    dropoffs = node_dropoffs.get(nid, 0)
    # Find existing thumbs data if any
    existing = next((sf for sf in step_feedback if sf.node_id == nid), None)
    combined_feedback.append(StepFeedbackSummary(
        node_id=nid,
        node_title=node_title_map.get(nid, "Unknown Step"),
        helpful_yes=existing.helpful_yes if existing else 0,
        helpful_no=existing.helpful_no if existing else 0,
        helpful_rate=existing.helpful_rate if existing else 0.0,
        visit_count=visits,
        dropoff_count=dropoffs,
        dropoff_rate=round(dropoffs / visits, 3) if visits > 0 else 0.0,
    ))

# Sort by dropoff_rate descending so worst steps are first
combined_feedback.sort(key=lambda x: x.dropoff_rate, reverse=True)
step_feedback = combined_feedback

Frontend changes:

In frontend/src/types/analytics.ts:

• Add to StepFeedbackSummary:

export interface StepFeedbackSummary {
  node_id: string
  node_title: string
  helpful_yes: number
  helpful_no: number
  helpful_rate: number
  visit_count: number      // NEW
  dropoff_count: number    // NEW
  dropoff_rate: number     // NEW
}

In frontend/src/components/analytics/FlowAnalyticsPanel.tsx:

• Add dropoff columns to the step feedback table: "Visits", "Dropoffs", "Dropoff Rate"
• Highlight rows where dropoff_rate > 0.2 with a subtle red/warning background

Verify: cd backend && python -m pytest --override-ini="addopts=" -v and cd frontend && npm run build

Commit: git commit -am "feat: add step dropoff metrics to flow analytics"

---

Task 3: Add Backward-Compatible /ratings Alias Routes

Why: The existing backend uses /{step_id}/rate but the frontend uses /{step_id}/ratings. This is a real inconsistency that needs fixing.

Backend changes:

In backend/app/api/endpoints/steps.py:

• Find the existing routes: POST /{step_id}/rate, PUT /{step_id}/rate, DELETE /{step_id}/rate
• Add alias routes that point to the same handler functions:

# After the existing rate_step function:
@router.post("/{step_id}/ratings", response_model=StepRatingResponse, status_code=status.HTTP_201_CREATED,
             include_in_schema=False)
async def rate_step_alias(
    step_id: UUID,
    rating_data: StepRatingCreate,
    db: AsyncSession = Depends(get_db),
    current_user: User = Depends(get_current_user)
):
    """Alias for POST /{step_id}/rate — backward compatibility."""
    return await rate_step(step_id, rating_data, db, current_user)


@router.put("/{step_id}/ratings", response_model=StepRatingResponse,
            include_in_schema=False)
async def update_rating_alias(
    step_id: UUID,
    rating_data: StepRatingUpdate,
    db: AsyncSession = Depends(get_db),
    current_user: User = Depends(get_current_user)
):
    """Alias for PUT /{step_id}/rate — backward compatibility."""
    return await update_rating(step_id, rating_data, db, current_user)


@router.delete("/{step_id}/ratings", status_code=204,
               include_in_schema=False)
async def delete_rating_alias(
    step_id: UUID,
    db: AsyncSession = Depends(get_db),
    current_user: User = Depends(get_current_user)
):
    """Alias for DELETE /{step_id}/rate — backward compatibility."""
    return await delete_rating(step_id, db, current_user)

Note: Use include_in_schema=False to keep the OpenAPI docs clean — only the canonical /rate routes appear in docs.

Verify: cd backend && python -m pytest --override-ini="addopts=" -v

Commit: git commit -am "fix: add /ratings alias routes for backward compatibility"

---

Task 4: Anonymize Feedback Comments in Analytics

Why: In a team tool where managers see feedback, engineers will give more honest feedback if their identity isn't attached. Anonymous feedback = better signal.

Backend changes:

In backend/app/schemas/analytics.py:

• Remove user_name from FlowRatingItem:

class FlowRatingItem(BaseModel):
    rating: int
    comment: Optional[str]
    created_at: datetime
    # user_name removed — feedback is anonymous in analytics views

In backend/app/api/endpoints/analytics.py, in get_flow_analytics:

• Remove the User.name join from the recent_comments query:

# Recent comments — anonymous
comments_q = await db.execute(
    select(SessionRating.rating, SessionRating.comment, SessionRating.created_at)
    .where(
        SessionRating.tree_id == tree_id,
        SessionRating.comment.isnot(None),
        SessionRating.comment != "",
    )
    .order_by(SessionRating.created_at.desc())
    .limit(10)
)
recent_comments = [
    FlowRatingItem(
        rating=row.rating,
        comment=row.comment,
        created_at=row.created_at,
    )
    for row in comments_q.all()
]

Frontend changes:

In frontend/src/types/analytics.ts:

• Remove user_name from FlowRatingItem:

export interface FlowRatingItem {
  rating: number
  comment?: string
  created_at: string
  // user_name removed — anonymous feedback
}

In frontend/src/components/analytics/FlowAnalyticsPanel.tsx:

• Remove any rendering of user_name in the recent comments list
• Comments should show: star rating + comment text + relative timestamp only

Verify: cd backend && python -m pytest --override-ini="addopts=" -v and cd frontend && npm run build

Commit: git commit -am "privacy: anonymize feedback comments in analytics views"

---

Task 5: Add Explicit active_engineers Metric

Why: "How many of my techs actually used this tool this month" is a key adoption metric for MSP managers.

Backend changes:

In backend/app/schemas/analytics.py:

• Verify AnalyticsSummary already has an active_engineers field. If not, add:

class AnalyticsSummary(BaseModel):
    total_sessions: int
    completed_sessions: int
    completion_rate: float
    median_duration_minutes: float
    active_engineers: int = 0  # ADD if missing
    outcome_breakdown: OutcomeBreakdown

In backend/app/api/endpoints/analytics.py:

• In the _build_summary helper (or the team analytics endpoint), add the active engineers count:

# Active engineers: distinct users with >= 1 session in window
active_q = await db.execute(
    select(func.count(func.distinct(Session.user_id)))
    .where(*base_filter)
)
active_engineers = active_q.scalar() or 0

• Include active_engineers=active_engineers in the AnalyticsSummary(...) construction
• For personal analytics (/me), set active_engineers=1 (it's always just the requesting user)

Frontend changes:

In frontend/src/types/analytics.ts:

• Verify AnalyticsSummary has active_engineers: number. Add if missing.

In frontend/src/pages/TeamAnalyticsPage.tsx:

• Verify there's a stat card for "Active Engineers". If missing, add one using the summary.active_engineers value.

Verify: cd backend && python -m pytest --override-ini="addopts=" -v and cd frontend && npm run build

Commit: git commit -am "feat: add explicit active_engineers metric to team analytics"

---

Final: Full Test Suite + Build Verification

cd backend && python -m pytest --override-ini="addopts=" -v
cd frontend && npm run build

Fix any failures, then:

git add -A
git commit -m "fix: post-improvement test and build fixes"