diff --git a/docs/plans/2026-03-19-phase5-analytics-impl.md b/docs/plans/2026-03-19-phase5-analytics-impl.md new file mode 100644 index 00000000..89253177 --- /dev/null +++ b/docs/plans/2026-03-19-phase5-analytics-impl.md @@ -0,0 +1,592 @@ +# Phase 5: Analytics Enhancement — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Extend the FlowPilot Analytics page with tabbed sections for coverage heatmap, flow quality scoring, and PSA time tracking metrics. + +**Architecture:** Add two new backend endpoints (`/coverage` and `/flow-quality`) alongside the existing `/analytics/flowpilot` endpoint. Add a `psa_activity_log` table for time entry tracking. Add flow usage columns to `trees`. Refactor the frontend analytics page into a tabbed layout with lazy-loaded sections. + +**Tech Stack:** FastAPI, SQLAlchemy 2.0 (async), React 19, TypeScript, Tailwind CSS v4, Recharts + +**Design doc:** `docs/plans/2026-03-19-phase5-analytics-enhancement-design.md` + +--- + +## Task 1: Database — add flow tracking columns + PSA activity log table + +**Files:** +- Modify: `backend/app/models/tree.py` +- Create: `backend/app/models/psa_activity_log.py` +- Create: migration file + +**Step 1: Add flow tracking columns to Tree model** + +In `backend/app/models/tree.py`, add after `gallery_sort_order`: + +```python +# Flow quality tracking (Phase 5) +usage_count: Mapped[int] = mapped_column(Integer, default=0) +success_rate: Mapped[Optional[float]] = mapped_column(Float, nullable=True) +last_matched_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) +``` + +Import `Float` from sqlalchemy if not already imported. + +**Step 2: Create PSA activity log model** + +Create `backend/app/models/psa_activity_log.py`: + +```python +"""PSA activity log — tracks time entries, note posts, and status updates pushed to PSA.""" +import uuid +from datetime import datetime, timezone +from typing import Optional + +from sqlalchemy import String, DateTime, ForeignKey, Float +from sqlalchemy.orm import Mapped, mapped_column +from sqlalchemy.dialects.postgresql import UUID + +from app.core.database import Base + + +class PsaActivityLog(Base): + __tablename__ = "psa_activity_logs" + + id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4) + account_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("accounts.id", ondelete="CASCADE"), nullable=False, index=True + ) + session_id: Mapped[Optional[uuid.UUID]] = mapped_column( + UUID(as_uuid=True), ForeignKey("ai_sessions.id", ondelete="SET NULL"), nullable=True + ) + activity_type: Mapped[str] = mapped_column(String(50), nullable=False) # "time_entry_posted", "note_posted", "status_updated" + hours_logged: Mapped[Optional[float]] = mapped_column(Float, nullable=True) + psa_ticket_id: Mapped[Optional[str]] = mapped_column(String(100), nullable=True) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=lambda: datetime.now(timezone.utc) + ) +``` + +**Step 3: Register model in imports** + +Ensure the model is importable. Check how other models are registered (e.g., in `backend/app/models/__init__.py` if it exists, or via alembic's `env.py`). + +**Step 4: Generate and run migration** + +```bash +cd /projects/patherly/backend && alembic revision --autogenerate -m "add flow tracking columns and psa_activity_logs table" +alembic upgrade head +``` + +**Step 5: Commit** + +```bash +git commit -m "feat(analytics): add flow tracking columns and psa_activity_logs table" +``` + +--- + +## Task 2: Backend — coverage endpoint + +**Files:** +- Modify: `backend/app/api/endpoints/flowpilot_analytics.py` +- Modify: `backend/app/schemas/flowpilot_analytics.py` +- Create: `backend/tests/test_analytics_coverage.py` + +**Step 1: Add schemas** + +In `backend/app/schemas/flowpilot_analytics.py`, add: + +```python +class CoverageDomainRow(BaseModel): + domain: str + flow_count: int + session_count: int + resolution_rate: float + escalation_rate: float + guided_rate: float + avg_resolution_minutes: float | None = None + + +class CoverageResponse(BaseModel): + domains: list[CoverageDomainRow] + unmapped_session_count: int + total_domains: int +``` + +**Step 2: Add endpoint** + +In `backend/app/api/endpoints/flowpilot_analytics.py`, add: + +```python +@router.get("/coverage", response_model=CoverageResponse) +@limiter.limit("15/minute") +async def get_coverage( + request: Request, + current_user: Annotated[User, Depends(get_current_active_user)], + db: Annotated[AsyncSession, Depends(get_db)], + _: None = Depends(require_team_admin), + period: str = Query("30d", pattern="^(7d|30d|90d)$"), +): + """Coverage heatmap data — per-domain metrics for flow coverage analysis.""" + if not current_user.account_id: + raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="No account") + + account_id = current_user.account_id + period_start = _get_period_start(period) + + # Get all domains from sessions in period + domain_stats = await db.execute( + select( + AISession.problem_domain, + func.count(AISession.id).label("session_count"), + func.sum(case((AISession.status == "resolved", 1), else_=0)).label("resolved"), + func.sum(case((AISession.status == "escalated", 1), else_=0)).label("escalated"), + func.sum(case((AISession.confidence_tier == "guided", 1), else_=0)).label("guided"), + ) + .where( + AISession.account_id == account_id, + AISession.created_at >= period_start, + AISession.problem_domain.isnot(None), + ) + .group_by(AISession.problem_domain) + ) + domain_rows = domain_rows_result = domain_stats.all() + + # Count flows per domain via category name matching + # Trees have category_id → Category.name, sessions have problem_domain + # Match by comparing Category.name to problem_domain + from app.models.category import Category + flow_counts_result = await db.execute( + select(Category.name, func.count(Tree.id)) + .join(Tree, Tree.category_id == Category.id) + .where(Tree.account_id == account_id, Tree.is_active == True) + .group_by(Category.name) + ) + flow_counts = {row[0]: row[1] for row in flow_counts_result.all()} + + # Count unmapped sessions (no problem_domain) + unmapped_result = await db.execute( + select(func.count(AISession.id)) + .where( + AISession.account_id == account_id, + AISession.created_at >= period_start, + AISession.problem_domain.is_(None), + ) + ) + unmapped_count = unmapped_result.scalar() or 0 + + # Build response + domains = [] + for row in domain_rows: + total = int(row.session_count) + resolved = int(row.resolved) + escalated = int(row.escalated) + guided = int(row.guided) + domain_name = row.problem_domain + + domains.append(CoverageDomainRow( + domain=domain_name, + flow_count=flow_counts.get(domain_name, 0), + session_count=total, + resolution_rate=round(resolved / total, 3) if total else 0, + escalation_rate=round(escalated / total, 3) if total else 0, + guided_rate=round(guided / total, 3) if total else 0, + )) + + # Sort by session count descending + domains.sort(key=lambda d: d.session_count, reverse=True) + + return CoverageResponse( + domains=domains, + unmapped_session_count=unmapped_count, + total_domains=len(domains), + ) +``` + +**Step 3: Write tests** + +Create `backend/tests/test_analytics_coverage.py` testing: +- Endpoint requires team_admin auth +- Returns domain breakdown with correct counts +- Unmapped sessions counted +- Empty state returns empty list + +**Step 4: Run tests and verify** + +```bash +cd /projects/patherly/backend && python -m pytest tests/test_analytics_coverage.py -v --override-ini="addopts=" +``` + +**Step 5: Commit** + +```bash +git commit -m "feat(analytics): add coverage heatmap endpoint" +``` + +--- + +## Task 3: Backend — flow quality endpoint + +**Files:** +- Modify: `backend/app/api/endpoints/flowpilot_analytics.py` +- Modify: `backend/app/schemas/flowpilot_analytics.py` +- Create: `backend/tests/test_analytics_flow_quality.py` + +**Step 1: Add schemas** + +```python +class FlowQualityRow(BaseModel): + flow_id: str + name: str + tree_type: str + usage_count: int + success_rate: float | None = None + last_matched_at: datetime | None = None + avg_confidence: float | None = None + quality_score: float + + +class FlowQualityResponse(BaseModel): + flows: list[FlowQualityRow] + top_performers: list[FlowQualityRow] + needs_attention: list[FlowQualityRow] +``` + +**Step 2: Add endpoint** + +```python +@router.get("/flow-quality", response_model=FlowQualityResponse) +@limiter.limit("15/minute") +async def get_flow_quality( + request: Request, + current_user: Annotated[User, Depends(get_current_active_user)], + db: Annotated[AsyncSession, Depends(get_db)], + _: None = Depends(require_team_admin), + period: str = Query("30d", pattern="^(7d|30d|90d)$"), + sort: str = Query("quality", pattern="^(quality|usage|success_rate)$"), +): +``` + +Query logic: +- Get all active flows for the account +- For each flow, count sessions where `matched_flow_id == flow.id` in period +- Calculate success_rate = resolved / total matched +- Calculate quality_score = `(success_rate * 0.5) + (guided_rate * 0.3) + (recency_score * 0.2)` +- Recency score: 1.0 if used today, decays linearly to 0.0 at 90 days +- Top performers: top 5 by quality_score +- Needs attention: flows with success_rate < 0.5 or not used in 30+ days + +**Step 3: Write tests, run, commit** + +```bash +git commit -m "feat(analytics): add flow quality scoring endpoint" +``` + +--- + +## Task 4: Backend — PSA activity logging + enhanced PSA metrics + +**Files:** +- Modify: `backend/app/services/psa/connectwise/provider.py` (or wherever note/time entry posting happens) +- Modify: `backend/app/api/endpoints/flowpilot_analytics.py` +- Modify: `backend/app/schemas/flowpilot_analytics.py` + +**Step 1: Add PSA activity logging** + +Find where the ConnectWise provider posts notes and time entries. After a successful push, log to `psa_activity_logs`: + +```python +from app.models.psa_activity_log import PsaActivityLog + +activity = PsaActivityLog( + account_id=account_id, + session_id=session_id, + activity_type="note_posted", # or "time_entry_posted" + hours_logged=hours, # for time entries + psa_ticket_id=ticket_id, +) +db.add(activity) +await db.commit() +``` + +**Step 2: Add enhanced PSA schemas** + +```python +class PsaFunnel(BaseModel): + total_sessions: int + linked_to_ticket: int + doc_pushed: int + time_entry_logged: int + + +class PsaDailyTrend(BaseModel): + date: str + entries: int + hours: float + + +class EnhancedPsaMetrics(BaseModel): + total_time_entries: int + total_hours_logged: float + avg_hours_per_session: float + push_funnel: PsaFunnel + daily_trend: list[PsaDailyTrend] +``` + +**Step 3: Add PSA metrics endpoint** + +```python +@router.get("/psa-metrics", response_model=EnhancedPsaMetrics) +``` + +Query `psa_activity_logs` and `ai_sessions` to build the funnel and trend data. + +**Step 4: Write tests, run, commit** + +```bash +git commit -m "feat(analytics): add PSA activity logging and enhanced PSA metrics endpoint" +``` + +--- + +## Task 5: Backend — wire flow matching stats + +**Files:** +- Modify: `backend/app/services/flowpilot_engine.py` (or wherever flow matching happens) + +**Step 1: Update flow stats on match** + +Find where `matched_flow_id` is set on an `AISession`. At that point, also update the matched flow: + +```python +# When a flow is matched to a session: +flow.usage_count = (flow.usage_count or 0) + 1 +flow.last_matched_at = datetime.now(timezone.utc) +``` + +**Step 2: Update success_rate on resolution** + +When a session resolves and has a `matched_flow_id`, recalculate that flow's success_rate: + +```python +# After session resolves: +if session.matched_flow_id: + total = await db.execute( + select(func.count(AISession.id)) + .where(AISession.matched_flow_id == session.matched_flow_id) + ) + resolved = await db.execute( + select(func.count(AISession.id)) + .where(AISession.matched_flow_id == session.matched_flow_id, AISession.status == "resolved") + ) + flow.success_rate = round(resolved.scalar() / total.scalar(), 3) if total.scalar() else None +``` + +**Step 3: Commit** + +```bash +git commit -m "feat(analytics): wire flow usage tracking into session matching and resolution" +``` + +--- + +## Task 6: Frontend — types and API client updates + +**Files:** +- Modify: `frontend/src/types/flowpilot-analytics.ts` +- Modify: `frontend/src/api/flowpilotAnalytics.ts` + +**Step 1: Add types** + +```typescript +// Coverage +export interface CoverageDomainRow { + domain: string + flow_count: number + session_count: number + resolution_rate: number + escalation_rate: number + guided_rate: number + avg_resolution_minutes: number | null +} + +export interface CoverageResponse { + domains: CoverageDomainRow[] + unmapped_session_count: number + total_domains: number +} + +// Flow Quality +export interface FlowQualityRow { + flow_id: string + name: string + tree_type: string + usage_count: number + success_rate: number | null + last_matched_at: string | null + avg_confidence: number | null + quality_score: number +} + +export interface FlowQualityResponse { + flows: FlowQualityRow[] + top_performers: FlowQualityRow[] + needs_attention: FlowQualityRow[] +} + +// Enhanced PSA +export interface PsaFunnel { + total_sessions: number + linked_to_ticket: number + doc_pushed: number + time_entry_logged: number +} + +export interface PsaDailyTrend { + date: string + entries: number + hours: number +} + +export interface EnhancedPsaMetrics { + total_time_entries: number + total_hours_logged: number + avg_hours_per_session: number + push_funnel: PsaFunnel + daily_trend: PsaDailyTrend[] +} +``` + +**Step 2: Add API methods** + +```typescript +async getCoverage(period: string = '30d'): Promise { + const response = await apiClient.get('/analytics/flowpilot/coverage', { params: { period } }) + return response.data +}, + +async getFlowQuality(period: string = '30d', sort: string = 'quality'): Promise { + const response = await apiClient.get('/analytics/flowpilot/flow-quality', { params: { period, sort } }) + return response.data +}, + +async getPsaMetrics(period: string = '30d'): Promise { + const response = await apiClient.get('/analytics/flowpilot/psa-metrics', { params: { period } }) + return response.data +}, +``` + +**Step 3: Run build, commit** + +```bash +cd /projects/patherly/frontend && npm run build +git commit -m "feat(analytics): add coverage, flow quality, and PSA metrics types and API client" +``` + +--- + +## Task 7: Frontend — tabbed layout + Coverage heatmap tab + +**Files:** +- Modify: `frontend/src/pages/FlowPilotAnalyticsPage.tsx` +- Create: `frontend/src/components/analytics/CoverageHeatmap.tsx` + +**Step 1: Refactor page into tabs** + +Add tab state and tab bar to `FlowPilotAnalyticsPage.tsx`: +- Tabs: "Overview", "Coverage", "Flow Quality", "PSA" +- Active tab: `bg-primary/10 text-foreground border-b-2 border-primary` +- Inactive: `text-muted-foreground hover:text-foreground` +- Move existing dashboard content into the Overview tab +- Each non-overview tab fetches its data lazily on first selection + +**Step 2: Build CoverageHeatmap component** + +`frontend/src/components/analytics/CoverageHeatmap.tsx`: +- `.glass-card-static` table container +- Table headers: Domain, Flows, Sessions, Resolution %, Escalation %, Guided % +- Cell coloring functions: + - Resolution: `bg-emerald-400/10 text-emerald-400` (>75%), `bg-amber-400/10 text-amber-400` (50-75%), `bg-rose-500/10 text-rose-500` (<50%) + - Escalation: green (<10%), amber (10-25%), red (>25%) + - Guided: green (>60%), amber (30-60%), red (<30%) + - Flows: green (5+), amber (1-4), red (0) +- Domains with 0 flows show "Create Flow" link +- Responsive: horizontal scroll on mobile (`overflow-x-auto`) + +**Step 3: Run build, commit** + +```bash +git commit -m "feat(analytics): add tabbed layout and coverage heatmap" +``` + +--- + +## Task 8: Frontend — Flow Quality tab + +**Files:** +- Create: `frontend/src/components/analytics/FlowQualityTable.tsx` +- Modify: `frontend/src/pages/FlowPilotAnalyticsPage.tsx` + +**Step 1: Build FlowQualityTable component** + +- `.glass-card-static` sortable table +- Columns: Flow Name (link to editor), Usage, Success Rate, Last Used, Avg Confidence, Quality Score +- Column headers clickable to sort +- Top 5 rows: left border `border-l-2 border-emerald-400` +- Bottom 5 rows: left border `border-l-2 border-rose-500` +- "Needs attention" badge (`bg-amber-400/10 text-amber-400 font-label text-[0.625rem]`) on flows with success_rate < 50% or unused 30+ days +- Quality score displayed as a colored bar (0-100% width, emerald/amber/rose) +- Click flow name → navigate to `/trees/{id}/edit` + +**Step 2: Wire into tab, run build, commit** + +```bash +git commit -m "feat(analytics): add flow quality scoring table" +``` + +--- + +## Task 9: Frontend — PSA metrics tab + +**Files:** +- Create: `frontend/src/components/analytics/PsaMetricsPanel.tsx` +- Modify: `frontend/src/pages/FlowPilotAnalyticsPage.tsx` + +**Step 1: Build PsaMetricsPanel component** + +- **Metric cards row** (3 cards): Total Time Entries, Total Hours Logged, Avg Hours/Session +- **Push success funnel**: horizontal bar visualization showing conversion at each step (sessions → linked → pushed → time entry). Show counts + percentage between steps. +- **Trend chart**: Recharts `AreaChart` with dual Y-axes — entries (bars) and hours (area) over the period + +**Step 2: Wire into tab, run build, commit** + +```bash +git commit -m "feat(analytics): add PSA metrics panel with funnel and trend chart" +``` + +--- + +## Task 10: Final verification and docs + +**Step 1: Run full backend tests** + +```bash +cd /projects/patherly/backend && python -m pytest --override-ini="addopts=" +``` + +**Step 2: Run frontend build** + +```bash +cd /projects/patherly/frontend && npm run build +``` + +**Step 3: Update CURRENT-STATE.md** + +Mark Phase 5 as complete. Update What's Next. + +**Step 4: Commit** + +```bash +git commit -m "docs: update CURRENT-STATE.md — Phase 5 Analytics Enhancement complete" +```