docs: add Phase 5 analytics enhancement implementation plan

10 tasks: flow tracking columns, coverage endpoint, flow quality endpoint, PSA activity logging, flow matching wiring, frontend tabs with heatmap, quality table, and PSA metrics panel. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 22:33:20 +00:00
parent 8b408f3418
commit 4d43ddcb90
1 changed files with 592 additions and 0 deletions
--- a/docs/plans/2026-03-19-phase5-analytics-impl.md
+++ b/docs/plans/2026-03-19-phase5-analytics-impl.md
@@ -0,0 +1,592 @@
+# Phase 5: Analytics Enhancement — Implementation Plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
+
+**Goal:** Extend the FlowPilot Analytics page with tabbed sections for coverage heatmap, flow quality scoring, and PSA time tracking metrics.
+
+**Architecture:** Add two new backend endpoints (`/coverage` and `/flow-quality`) alongside the existing `/analytics/flowpilot` endpoint. Add a `psa_activity_log` table for time entry tracking. Add flow usage columns to `trees`. Refactor the frontend analytics page into a tabbed layout with lazy-loaded sections.
+
+**Tech Stack:** FastAPI, SQLAlchemy 2.0 (async), React 19, TypeScript, Tailwind CSS v4, Recharts
+
+**Design doc:** `docs/plans/2026-03-19-phase5-analytics-enhancement-design.md`
+
+---
+
+## Task 1: Database — add flow tracking columns + PSA activity log table
+
+**Files:**
+- Modify: `backend/app/models/tree.py`
+- Create: `backend/app/models/psa_activity_log.py`
+- Create: migration file
+
+**Step 1: Add flow tracking columns to Tree model**
+
+In `backend/app/models/tree.py`, add after `gallery_sort_order`:
+
+```python
+# Flow quality tracking (Phase 5)
+usage_count: Mapped[int] = mapped_column(Integer, default=0)
+success_rate: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+last_matched_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
+```
+
+Import `Float` from sqlalchemy if not already imported.
+
+**Step 2: Create PSA activity log model**
+
+Create `backend/app/models/psa_activity_log.py`:
+
+```python
+"""PSA activity log — tracks time entries, note posts, and status updates pushed to PSA."""
+import uuid
+from datetime import datetime, timezone
+from typing import Optional
+
+from sqlalchemy import String, DateTime, ForeignKey, Float
+from sqlalchemy.orm import Mapped, mapped_column
+from sqlalchemy.dialects.postgresql import UUID
+
+from app.core.database import Base
+
+
+class PsaActivityLog(Base):
+    __tablename__ = "psa_activity_logs"
+
+    id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    account_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), ForeignKey("accounts.id", ondelete="CASCADE"), nullable=False, index=True
+    )
+    session_id: Mapped[Optional[uuid.UUID]] = mapped_column(
+        UUID(as_uuid=True), ForeignKey("ai_sessions.id", ondelete="SET NULL"), nullable=True
+    )
+    activity_type: Mapped[str] = mapped_column(String(50), nullable=False)  # "time_entry_posted", "note_posted", "status_updated"
+    hours_logged: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+    psa_ticket_id: Mapped[Optional[str]] = mapped_column(String(100), nullable=True)
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), default=lambda: datetime.now(timezone.utc)
+    )
+```
+
+**Step 3: Register model in imports**
+
+Ensure the model is importable. Check how other models are registered (e.g., in `backend/app/models/__init__.py` if it exists, or via alembic's `env.py`).
+
+**Step 4: Generate and run migration**
+
+```bash
+cd /projects/patherly/backend && alembic revision --autogenerate -m "add flow tracking columns and psa_activity_logs table"
+alembic upgrade head
+```
+
+**Step 5: Commit**
+
+```bash
+git commit -m "feat(analytics): add flow tracking columns and psa_activity_logs table"
+```
+
+---
+
+## Task 2: Backend — coverage endpoint
+
+**Files:**
+- Modify: `backend/app/api/endpoints/flowpilot_analytics.py`
+- Modify: `backend/app/schemas/flowpilot_analytics.py`
+- Create: `backend/tests/test_analytics_coverage.py`
+
+**Step 1: Add schemas**
+
+In `backend/app/schemas/flowpilot_analytics.py`, add:
+
+```python
+class CoverageDomainRow(BaseModel):
+    domain: str
+    flow_count: int
+    session_count: int
+    resolution_rate: float
+    escalation_rate: float
+    guided_rate: float
+    avg_resolution_minutes: float | None = None
+
+
+class CoverageResponse(BaseModel):
+    domains: list[CoverageDomainRow]
+    unmapped_session_count: int
+    total_domains: int
+```
+
+**Step 2: Add endpoint**
+
+In `backend/app/api/endpoints/flowpilot_analytics.py`, add:
+
+```python
+@router.get("/coverage", response_model=CoverageResponse)
+@limiter.limit("15/minute")
+async def get_coverage(
+    request: Request,
+    current_user: Annotated[User, Depends(get_current_active_user)],
+    db: Annotated[AsyncSession, Depends(get_db)],
+    _: None = Depends(require_team_admin),
+    period: str = Query("30d", pattern="^(7d|30d|90d)$"),
+):
+    """Coverage heatmap data — per-domain metrics for flow coverage analysis."""
+    if not current_user.account_id:
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="No account")
+
+    account_id = current_user.account_id
+    period_start = _get_period_start(period)
+
+    # Get all domains from sessions in period
+    domain_stats = await db.execute(
+        select(
+            AISession.problem_domain,
+            func.count(AISession.id).label("session_count"),
+            func.sum(case((AISession.status == "resolved", 1), else_=0)).label("resolved"),
+            func.sum(case((AISession.status == "escalated", 1), else_=0)).label("escalated"),
+            func.sum(case((AISession.confidence_tier == "guided", 1), else_=0)).label("guided"),
+        )
+        .where(
+            AISession.account_id == account_id,
+            AISession.created_at >= period_start,
+            AISession.problem_domain.isnot(None),
+        )
+        .group_by(AISession.problem_domain)
+    )
+    domain_rows = domain_rows_result = domain_stats.all()
+
+    # Count flows per domain via category name matching
+    # Trees have category_id → Category.name, sessions have problem_domain
+    # Match by comparing Category.name to problem_domain
+    from app.models.category import Category
+    flow_counts_result = await db.execute(
+        select(Category.name, func.count(Tree.id))
+        .join(Tree, Tree.category_id == Category.id)
+        .where(Tree.account_id == account_id, Tree.is_active == True)
+        .group_by(Category.name)
+    )
+    flow_counts = {row[0]: row[1] for row in flow_counts_result.all()}
+
+    # Count unmapped sessions (no problem_domain)
+    unmapped_result = await db.execute(
+        select(func.count(AISession.id))
+        .where(
+            AISession.account_id == account_id,
+            AISession.created_at >= period_start,
+            AISession.problem_domain.is_(None),
+        )
+    )
+    unmapped_count = unmapped_result.scalar() or 0
+
+    # Build response
+    domains = []
+    for row in domain_rows:
+        total = int(row.session_count)
+        resolved = int(row.resolved)
+        escalated = int(row.escalated)
+        guided = int(row.guided)
+        domain_name = row.problem_domain
+
+        domains.append(CoverageDomainRow(
+            domain=domain_name,
+            flow_count=flow_counts.get(domain_name, 0),
+            session_count=total,
+            resolution_rate=round(resolved / total, 3) if total else 0,
+            escalation_rate=round(escalated / total, 3) if total else 0,
+            guided_rate=round(guided / total, 3) if total else 0,
+        ))
+
+    # Sort by session count descending
+    domains.sort(key=lambda d: d.session_count, reverse=True)
+
+    return CoverageResponse(
+        domains=domains,
+        unmapped_session_count=unmapped_count,
+        total_domains=len(domains),
+    )
+```
+
+**Step 3: Write tests**
+
+Create `backend/tests/test_analytics_coverage.py` testing:
+- Endpoint requires team_admin auth
+- Returns domain breakdown with correct counts
+- Unmapped sessions counted
+- Empty state returns empty list
+
+**Step 4: Run tests and verify**
+
+```bash
+cd /projects/patherly/backend && python -m pytest tests/test_analytics_coverage.py -v --override-ini="addopts="
+```
+
+**Step 5: Commit**
+
+```bash
+git commit -m "feat(analytics): add coverage heatmap endpoint"
+```
+
+---
+
+## Task 3: Backend — flow quality endpoint
+
+**Files:**
+- Modify: `backend/app/api/endpoints/flowpilot_analytics.py`
+- Modify: `backend/app/schemas/flowpilot_analytics.py`
+- Create: `backend/tests/test_analytics_flow_quality.py`
+
+**Step 1: Add schemas**
+
+```python
+class FlowQualityRow(BaseModel):
+    flow_id: str
+    name: str
+    tree_type: str
+    usage_count: int
+    success_rate: float | None = None
+    last_matched_at: datetime | None = None
+    avg_confidence: float | None = None
+    quality_score: float
+
+
+class FlowQualityResponse(BaseModel):
+    flows: list[FlowQualityRow]
+    top_performers: list[FlowQualityRow]
+    needs_attention: list[FlowQualityRow]
+```
+
+**Step 2: Add endpoint**
+
+```python
+@router.get("/flow-quality", response_model=FlowQualityResponse)
+@limiter.limit("15/minute")
+async def get_flow_quality(
+    request: Request,
+    current_user: Annotated[User, Depends(get_current_active_user)],
+    db: Annotated[AsyncSession, Depends(get_db)],
+    _: None = Depends(require_team_admin),
+    period: str = Query("30d", pattern="^(7d|30d|90d)$"),
+    sort: str = Query("quality", pattern="^(quality|usage|success_rate)$"),
+):
+```
+
+Query logic:
+- Get all active flows for the account
+- For each flow, count sessions where `matched_flow_id == flow.id` in period
+- Calculate success_rate = resolved / total matched
+- Calculate quality_score = `(success_rate * 0.5) + (guided_rate * 0.3) + (recency_score * 0.2)`
+- Recency score: 1.0 if used today, decays linearly to 0.0 at 90 days
+- Top performers: top 5 by quality_score
+- Needs attention: flows with success_rate < 0.5 or not used in 30+ days
+
+**Step 3: Write tests, run, commit**
+
+```bash
+git commit -m "feat(analytics): add flow quality scoring endpoint"
+```
+
+---
+
+## Task 4: Backend — PSA activity logging + enhanced PSA metrics
+
+**Files:**
+- Modify: `backend/app/services/psa/connectwise/provider.py` (or wherever note/time entry posting happens)
+- Modify: `backend/app/api/endpoints/flowpilot_analytics.py`
+- Modify: `backend/app/schemas/flowpilot_analytics.py`
+
+**Step 1: Add PSA activity logging**
+
+Find where the ConnectWise provider posts notes and time entries. After a successful push, log to `psa_activity_logs`:
+
+```python
+from app.models.psa_activity_log import PsaActivityLog
+
+activity = PsaActivityLog(
+    account_id=account_id,
+    session_id=session_id,
+    activity_type="note_posted",  # or "time_entry_posted"
+    hours_logged=hours,  # for time entries
+    psa_ticket_id=ticket_id,
+)
+db.add(activity)
+await db.commit()
+```
+
+**Step 2: Add enhanced PSA schemas**
+
+```python
+class PsaFunnel(BaseModel):
+    total_sessions: int
+    linked_to_ticket: int
+    doc_pushed: int
+    time_entry_logged: int
+
+
+class PsaDailyTrend(BaseModel):
+    date: str
+    entries: int
+    hours: float
+
+
+class EnhancedPsaMetrics(BaseModel):
+    total_time_entries: int
+    total_hours_logged: float
+    avg_hours_per_session: float
+    push_funnel: PsaFunnel
+    daily_trend: list[PsaDailyTrend]
+```
+
+**Step 3: Add PSA metrics endpoint**
+
+```python
+@router.get("/psa-metrics", response_model=EnhancedPsaMetrics)
+```
+
+Query `psa_activity_logs` and `ai_sessions` to build the funnel and trend data.
+
+**Step 4: Write tests, run, commit**
+
+```bash
+git commit -m "feat(analytics): add PSA activity logging and enhanced PSA metrics endpoint"
+```
+
+---
+
+## Task 5: Backend — wire flow matching stats
+
+**Files:**
+- Modify: `backend/app/services/flowpilot_engine.py` (or wherever flow matching happens)
+
+**Step 1: Update flow stats on match**
+
+Find where `matched_flow_id` is set on an `AISession`. At that point, also update the matched flow:
+
+```python
+# When a flow is matched to a session:
+flow.usage_count = (flow.usage_count or 0) + 1
+flow.last_matched_at = datetime.now(timezone.utc)
+```
+
+**Step 2: Update success_rate on resolution**
+
+When a session resolves and has a `matched_flow_id`, recalculate that flow's success_rate:
+
+```python
+# After session resolves:
+if session.matched_flow_id:
+    total = await db.execute(
+        select(func.count(AISession.id))
+        .where(AISession.matched_flow_id == session.matched_flow_id)
+    )
+    resolved = await db.execute(
+        select(func.count(AISession.id))
+        .where(AISession.matched_flow_id == session.matched_flow_id, AISession.status == "resolved")
+    )
+    flow.success_rate = round(resolved.scalar() / total.scalar(), 3) if total.scalar() else None
+```
+
+**Step 3: Commit**
+
+```bash
+git commit -m "feat(analytics): wire flow usage tracking into session matching and resolution"
+```
+
+---
+
+## Task 6: Frontend — types and API client updates
+
+**Files:**
+- Modify: `frontend/src/types/flowpilot-analytics.ts`
+- Modify: `frontend/src/api/flowpilotAnalytics.ts`
+
+**Step 1: Add types**
+
+```typescript
+// Coverage
+export interface CoverageDomainRow {
+  domain: string
+  flow_count: number
+  session_count: number
+  resolution_rate: number
+  escalation_rate: number
+  guided_rate: number
+  avg_resolution_minutes: number | null
+}
+
+export interface CoverageResponse {
+  domains: CoverageDomainRow[]
+  unmapped_session_count: number
+  total_domains: number
+}
+
+// Flow Quality
+export interface FlowQualityRow {
+  flow_id: string
+  name: string
+  tree_type: string
+  usage_count: number
+  success_rate: number | null
+  last_matched_at: string | null
+  avg_confidence: number | null
+  quality_score: number
+}
+
+export interface FlowQualityResponse {
+  flows: FlowQualityRow[]
+  top_performers: FlowQualityRow[]
+  needs_attention: FlowQualityRow[]
+}
+
+// Enhanced PSA
+export interface PsaFunnel {
+  total_sessions: number
+  linked_to_ticket: number
+  doc_pushed: number
+  time_entry_logged: number
+}
+
+export interface PsaDailyTrend {
+  date: string
+  entries: number
+  hours: number
+}
+
+export interface EnhancedPsaMetrics {
+  total_time_entries: number
+  total_hours_logged: number
+  avg_hours_per_session: number
+  push_funnel: PsaFunnel
+  daily_trend: PsaDailyTrend[]
+}
+```
+
+**Step 2: Add API methods**
+
+```typescript
+async getCoverage(period: string = '30d'): Promise<CoverageResponse> {
+  const response = await apiClient.get<CoverageResponse>('/analytics/flowpilot/coverage', { params: { period } })
+  return response.data
+},
+
+async getFlowQuality(period: string = '30d', sort: string = 'quality'): Promise<FlowQualityResponse> {
+  const response = await apiClient.get<FlowQualityResponse>('/analytics/flowpilot/flow-quality', { params: { period, sort } })
+  return response.data
+},
+
+async getPsaMetrics(period: string = '30d'): Promise<EnhancedPsaMetrics> {
+  const response = await apiClient.get<EnhancedPsaMetrics>('/analytics/flowpilot/psa-metrics', { params: { period } })
+  return response.data
+},
+```
+
+**Step 3: Run build, commit**
+
+```bash
+cd /projects/patherly/frontend && npm run build
+git commit -m "feat(analytics): add coverage, flow quality, and PSA metrics types and API client"
+```
+
+---
+
+## Task 7: Frontend — tabbed layout + Coverage heatmap tab
+
+**Files:**
+- Modify: `frontend/src/pages/FlowPilotAnalyticsPage.tsx`
+- Create: `frontend/src/components/analytics/CoverageHeatmap.tsx`
+
+**Step 1: Refactor page into tabs**
+
+Add tab state and tab bar to `FlowPilotAnalyticsPage.tsx`:
+- Tabs: "Overview", "Coverage", "Flow Quality", "PSA"
+- Active tab: `bg-primary/10 text-foreground border-b-2 border-primary`
+- Inactive: `text-muted-foreground hover:text-foreground`
+- Move existing dashboard content into the Overview tab
+- Each non-overview tab fetches its data lazily on first selection
+
+**Step 2: Build CoverageHeatmap component**
+
+`frontend/src/components/analytics/CoverageHeatmap.tsx`:
+- `.glass-card-static` table container
+- Table headers: Domain, Flows, Sessions, Resolution %, Escalation %, Guided %
+- Cell coloring functions:
+  - Resolution: `bg-emerald-400/10 text-emerald-400` (>75%), `bg-amber-400/10 text-amber-400` (50-75%), `bg-rose-500/10 text-rose-500` (<50%)
+  - Escalation: green (<10%), amber (10-25%), red (>25%)
+  - Guided: green (>60%), amber (30-60%), red (<30%)
+  - Flows: green (5+), amber (1-4), red (0)
+- Domains with 0 flows show "Create Flow" link
+- Responsive: horizontal scroll on mobile (`overflow-x-auto`)
+
+**Step 3: Run build, commit**
+
+```bash
+git commit -m "feat(analytics): add tabbed layout and coverage heatmap"
+```
+
+---
+
+## Task 8: Frontend — Flow Quality tab
+
+**Files:**
+- Create: `frontend/src/components/analytics/FlowQualityTable.tsx`
+- Modify: `frontend/src/pages/FlowPilotAnalyticsPage.tsx`
+
+**Step 1: Build FlowQualityTable component**
+
+- `.glass-card-static` sortable table
+- Columns: Flow Name (link to editor), Usage, Success Rate, Last Used, Avg Confidence, Quality Score
+- Column headers clickable to sort
+- Top 5 rows: left border `border-l-2 border-emerald-400`
+- Bottom 5 rows: left border `border-l-2 border-rose-500`
+- "Needs attention" badge (`bg-amber-400/10 text-amber-400 font-label text-[0.625rem]`) on flows with success_rate < 50% or unused 30+ days
+- Quality score displayed as a colored bar (0-100% width, emerald/amber/rose)
+- Click flow name → navigate to `/trees/{id}/edit`
+
+**Step 2: Wire into tab, run build, commit**
+
+```bash
+git commit -m "feat(analytics): add flow quality scoring table"
+```
+
+---
+
+## Task 9: Frontend — PSA metrics tab
+
+**Files:**
+- Create: `frontend/src/components/analytics/PsaMetricsPanel.tsx`
+- Modify: `frontend/src/pages/FlowPilotAnalyticsPage.tsx`
+
+**Step 1: Build PsaMetricsPanel component**
+
+- **Metric cards row** (3 cards): Total Time Entries, Total Hours Logged, Avg Hours/Session
+- **Push success funnel**: horizontal bar visualization showing conversion at each step (sessions → linked → pushed → time entry). Show counts + percentage between steps.
+- **Trend chart**: Recharts `AreaChart` with dual Y-axes — entries (bars) and hours (area) over the period
+
+**Step 2: Wire into tab, run build, commit**
+
+```bash
+git commit -m "feat(analytics): add PSA metrics panel with funnel and trend chart"
+```
+
+---
+
+## Task 10: Final verification and docs
+
+**Step 1: Run full backend tests**
+
+```bash
+cd /projects/patherly/backend && python -m pytest --override-ini="addopts="
+```
+
+**Step 2: Run frontend build**
+
+```bash
+cd /projects/patherly/frontend && npm run build
+```
+
+**Step 3: Update CURRENT-STATE.md**
+
+Mark Phase 5 as complete. Update What's Next.
+
+**Step 4: Commit**
+
+```bash
+git commit -m "docs: update CURRENT-STATE.md — Phase 5 Analytics Enhancement complete"
+```