resolutionflow/docs/2026-03-18-flowpilot-first-pivot-phase3.md

# FlowPilot-First Pivot — Phase 3: Knowledge Flywheel, Script Generator & Analytics

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Close the learning loop. Every resolved AI session automatically proposes new flows or enhancements to existing ones. Team leads curate quality through a Review Queue. FlowPilot can invoke the Script Generator mid-session. Analytics track MTTR, resolution rates, and knowledge coverage to show the ROI.

**Architecture:** Builds on Phase 1 (AI sessions, FlowPilot Engine, Flow Matching) and Phase 2 (PSA integration, escalation handoff). Introduces the `flow_proposals` model, a post-session analysis pipeline, the Review Queue UI, in-session script generation, and an AI-enhanced analytics dashboard.

**Tech Stack:** FastAPI, SQLAlchemy 2.0 (async), Anthropic Claude (flow proposal generation), pgvector, React, TypeScript, Tailwind CSS v4 (`@tailwindcss/vite`), Recharts (analytics charts)

**Prerequisites:**
- Phase 1 complete (AI session core)
- Phase 2 complete (PSA integration, escalation handoff)
- Existing models: `ScriptCategory`, `ScriptTemplate`, `ScriptGeneration` (from Script Generator feature)
- Existing services: `script_template_engine.py`, `session_to_flow_service.py`, `embedding_service.py`
- Existing frontend: `ScriptLibraryPage.tsx`, `ScriptConfigurePane.tsx`, `ScriptParameterForm.tsx`, `ScriptPreview.tsx`, `PowerShellHighlighter.tsx`, `TeamAnalyticsPage.tsx`
- Existing schemas: `schemas/session_to_flow.py`, `schemas/script_template.py`

**Existing patterns to follow:**

- Session-to-flow: `app/services/session_to_flow_service.py` — converts legacy `Session` model to tree structures. **NOTE:** This service works with the legacy `Session` model (`session.decisions`, `session.outcome`, `session.scratchpad`), NOT `AISession`. The Knowledge Flywheel must build its own flow generation logic reading from `AISession.conversation_messages` and `AISession.steps`. Use this service as a reference for LLM prompt structure and tree format only.
- Script templates: `app/services/script_template_engine.py` — parameter substitution, validation, sanitization
- Embeddings: `app/services/embedding_service.py` — Voyage AI embeddings for vector search
- Analytics: `app/api/endpoints/analytics.py` — existing team analytics patterns
- Phase 1 engine: `app/services/flowpilot_engine.py` — structured JSON output contracts
- Frontend API pattern: `src/api/aiSessions.ts` uses `aiSessionsApi` object pattern

**Pivot architecture doc:** `docs/ResolutionFlow_Pivot_Architecture.docx`

---

## Context: What Phase 3 Adds

Phase 1 built the AI session core. Phase 2 connected it to PSA tickets. Phase 3 makes the system get smarter over time:

**Knowledge Flywheel:** Every resolved session is analyzed. The system proposes new flows from novel resolutions, suggests enhancements to existing flows when it discovers new branches, and reinforces proven flows when sessions follow known paths. Human-in-the-loop Review Queue ensures quality.

**In-Session Script Generator:** FlowPilot can invoke the Script Generator contextually during diagnosis. When it detects the engineer needs a PowerShell script (e.g., "reset this user's AD password"), it surfaces the script generator with parameters pre-filled from session context.

**AI-Enhanced Analytics:** MTTR trends, resolution rates by category, knowledge coverage heatmap, FlowPilot accuracy metrics, knowledge gap detection, and flow quality scoring.

---

## Slice 1: Flow Proposals Model & Post-Session Analysis

### Task 1: Create FlowProposal model

**Files:**
- Create: `backend/app/models/flow_proposal.py`

```python
"""Flow proposal model.

Generated by the Knowledge Flywheel after AI sessions resolve.
Represents a proposed new flow or enhancement awaiting human review.
"""
import uuid
from datetime import datetime, timezone
from typing import Optional, Any, TYPE_CHECKING

from sqlalchemy import String, Text, DateTime, ForeignKey, Integer, Float, CheckConstraint
from sqlalchemy.orm import Mapped, mapped_column, relationship
from sqlalchemy.dialects.postgresql import UUID, JSONB

from app.core.database import Base

if TYPE_CHECKING:
    from app.models.user import User
    from app.models.team import Team
    from app.models.account import Account
    from app.models.tree import Tree
    from app.models.ai_session import AISession


class FlowProposal(Base):
    """A proposed new flow or enhancement generated from an AI session.

    proposal_type:
    - new_flow: No similar flow exists. Full flow definition proposed.
    - enhancement: Similar flow exists but session discovered new branch/edge case.
    - branch_addition: A single new branch to add to an existing flow.

    status:
    - pending: Awaiting review
    - approved: Reviewed and published to knowledge base
    - modified: Reviewer edited before publishing
    - rejected: Reviewer decided not to publish (bad quality)
    - dismissed: Parked for later — not wrong, just not actionable now. Can resurface if supporting_session_count grows.
    - auto_reinforced: Session matched existing flow exactly (no review needed)
    """
    __tablename__ = "flow_proposals"
    __table_args__ = (
        CheckConstraint(
            "proposal_type IN ('new_flow', 'enhancement', 'branch_addition')",
            name="ck_flow_proposals_type",
        ),
        CheckConstraint(
            "status IN ('pending', 'approved', 'modified', 'rejected', 'dismissed', 'auto_reinforced')",
            name="ck_flow_proposals_status",
        ),
    )

    id: Mapped[uuid.UUID] = mapped_column(
        UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
    )
    account_id: Mapped[uuid.UUID] = mapped_column(
        UUID(as_uuid=True),
        ForeignKey("accounts.id", ondelete="CASCADE"),
        nullable=False,
        index=True,
    )
    team_id: Mapped[Optional[uuid.UUID]] = mapped_column(
        UUID(as_uuid=True),
        ForeignKey("teams.id", ondelete="SET NULL"),
        nullable=True,
        index=True,
    )
    source_session_id: Mapped[uuid.UUID] = mapped_column(
        UUID(as_uuid=True),
        ForeignKey("ai_sessions.id", ondelete="CASCADE"),
        nullable=False,
        index=True,
    )

    # ── Proposal details ──
    proposal_type: Mapped[str] = mapped_column(
        String(30), nullable=False,
    )
    target_flow_id: Mapped[Optional[uuid.UUID]] = mapped_column(
        UUID(as_uuid=True),
        ForeignKey("trees.id", ondelete="SET NULL"),
        nullable=True,
        comment="For enhancements: which existing flow to modify",
    )
    title: Mapped[str] = mapped_column(
        String(255), nullable=False,
        comment="Human-readable title for the proposed flow",
    )
    description: Mapped[Optional[str]] = mapped_column(
        Text, nullable=True,
        comment="AI-generated description of what this flow covers",
    )
    proposed_flow_data: Mapped[dict[str, Any]] = mapped_column(
        JSONB, nullable=False,
        comment="Complete flow/tree_structure definition (nodes, edges, conditions)",
    )
    proposed_diff: Mapped[Optional[dict[str, Any]]] = mapped_column(
        JSONB, nullable=True,
        comment="For enhancements: what changed vs existing flow",
    )

    # ── Scoring ──
    confidence_score: Mapped[float] = mapped_column(
        Float, nullable=False, default=0.0,
        comment="How confident the system is in this proposal (0.0-1.0)",
    )
    supporting_session_count: Mapped[int] = mapped_column(
        Integer, nullable=False, default=1,
        comment="Number of sessions with similar resolution paths",
    )
    supporting_session_ids: Mapped[list] = mapped_column(
        JSONB, nullable=False, default=list,
        comment="Array of session IDs that support this proposal",
    )
    problem_domain: Mapped[Optional[str]] = mapped_column(
        String(100), nullable=True,
    )

    # ── Review ──
    status: Mapped[str] = mapped_column(
        String(30), nullable=False, default="pending", index=True,
    )
    reviewed_by: Mapped[Optional[uuid.UUID]] = mapped_column(
        UUID(as_uuid=True),
        ForeignKey("users.id", ondelete="SET NULL"),
        nullable=True,
    )
    reviewer_notes: Mapped[Optional[str]] = mapped_column(
        Text, nullable=True,
    )
    published_flow_id: Mapped[Optional[uuid.UUID]] = mapped_column(
        UUID(as_uuid=True),
        ForeignKey("trees.id", ondelete="SET NULL"),
        nullable=True,
        comment="The flow that was created/updated when this proposal was approved",
    )

    # ── Timestamps ──
    created_at: Mapped[datetime] = mapped_column(
        DateTime(timezone=True), default=lambda: datetime.now(timezone.utc)
    )
    reviewed_at: Mapped[Optional[datetime]] = mapped_column(
        DateTime(timezone=True), nullable=True,
    )

    # ── Relationships ──
    account: Mapped["Account"] = relationship("Account")
    team: Mapped[Optional["Team"]] = relationship("Team")
    source_session: Mapped["AISession"] = relationship("AISession")
    target_flow: Mapped[Optional["Tree"]] = relationship("Tree", foreign_keys=[target_flow_id])
    published_flow: Mapped[Optional["Tree"]] = relationship("Tree", foreign_keys=[published_flow_id])
    reviewer: Mapped[Optional["User"]] = relationship("User")
```

Register in `app/models/__init__.py`:
```python
from .flow_proposal import FlowProposal
```

Add to `__all__`.

### Task 2: Create Alembic migration

Generate with:
```bash
cd /projects/patherly/backend
DATABASE_URL=postgresql://postgres:postgres@resolutionflow_postgres:5432/resolutionflow \
  venv/bin/alembic revision --autogenerate -m "add flow_proposals table"
```

Indexes: `account_id`, `team_id`, `source_session_id`, `status`, `target_flow_id`, `created_at`.

**Verification:** Run `alembic upgrade head`. Verify table exists.

```
git commit -m "feat(knowledge): add FlowProposal model + migration"
```

### Task 3: Build post-session analysis service (Knowledge Flywheel engine)

**Files:**
- Create: `backend/app/services/knowledge_flywheel.py`

**Architecture:**

This service runs after every successful session resolution. It analyzes the session and produces one of three outcomes:

**1. New Flow Proposal (`new_flow`):**
- Triggered when: Session resolved with `confidence_tier = "discovery"` or `"exploring"`, AND no flow was matched (or match_score < 0.5)
- Process: Make an LLM call (standard tier) with the full session conversation, asking it to:
  - Generate a flow title and description
  - Convert the diagnostic path into a tree_structure (nodes, edges, conditions)
  - Identify the key decision points and branching logic
  - Suggest match_keywords for future semantic matching
- Store as a `FlowProposal` with `proposal_type = "new_flow"` and `status = "pending"`

**2. Flow Enhancement Proposal (`enhancement`):**
- Triggered when: Session matched an existing flow (match_score > 0.5) but diverged at some point (engineer used free-text escape or chose a path not in the flow)
- Process: LLM call comparing the session path with the matched flow, identifying:
  - New branches that should be added
  - Options that should be added to existing questions
  - Steps that should be reordered based on what worked
- Store as `FlowProposal` with `proposal_type = "enhancement"`, `target_flow_id` set, and `proposed_diff` containing the changes

**3. Flow Reinforcement (`auto_reinforced`):**
- Triggered when: Session followed an existing flow closely (match_score > 0.8, no free-text escapes, resolution matched the flow's expected outcome)
- Process: No LLM call needed. Update the flow's `success_rate` and `last_matched_at`. Create a `FlowProposal` with `status = "auto_reinforced"` for tracking purposes only (no review needed).

**Key implementation details:**

- **Do NOT use `asyncio.create_task()`.** Use the APScheduler background job pattern established by `psa_retry_scheduler.py`. Add an `analysis_status` column to `AISession` (values: `null`, `pending`, `completed`, `failed`). Set it to `pending` in `resolve_session()`. A periodic scheduler job picks up pending sessions and runs analysis. This is resilient to server restarts and retryable on failure.
- The LLM call for flow generation should use the existing `ai_provider.generate_json()` with a specific system prompt for flow construction
- The generated `tree_structure` must match the existing tree format used by the Flow Editor (check `models/tree.py` → `tree_structure` JSONB schema). Troubleshooting flows use nested `children` nodes; procedural flows use linear `steps` arrays. The Knowledge Flywheel should generate **troubleshooting** tree format for diagnostic sessions.
- **Data source:** `AISession` uses `conversation_messages` (JSONB list of role/content dicts) and the `steps` relationship (`AISessionStep` with `step_type`, `content`, `selected_option`, `free_text_input`, `action_result`). Build session context from these — do NOT reference `session.decisions` or `session.scratchpad` (those are legacy `Session` model fields).
- For enhancement proposals, also generate a human-readable diff description (e.g., "Added new branch for 'Error code 0x80070005' at step 3")
- Track supporting sessions: if multiple sessions resolve a similar novel problem, increase `supporting_session_count` and `confidence_score` on the existing proposal rather than creating duplicates. Use `problem_domain` match + embedding similarity on `title + description` against existing pending proposals (threshold >0.85 cosine similarity → merge)
- **Rate limiting:** Add a daily budget cap for proposal generation LLM calls (configurable, default 50/day per account) to prevent runaway costs from high-volume teams

**System prompt for flow generation (excerpt):**

```python
FLOW_GENERATION_PROMPT = """You are a knowledge engineer converting a troubleshooting session into a reusable flow definition.

Given the session transcript below, generate a JSON flow definition that captures the diagnostic logic so other engineers can follow the same path.

## OUTPUT FORMAT
Respond with ONLY valid JSON:
{
  "title": "Short descriptive title",
  "description": "When to use this flow",
  "match_keywords": ["keyword1", "keyword2", ...],
  "problem_domain": "active_directory | networking | m365 | ...",
  "tree_structure": {
    "id": "root",
    "type": "question",
    "question": "First diagnostic question",
    "children": [
      {
        "id": "opt1",
        "label": "Option text",
        "type": "question | action | solution",
        ...
      }
    ]
  }
}

## RULES
- tree_structure must follow the ResolutionFlow tree format
- Every question node must have 2-5 children (options)
- Action nodes describe what the engineer should do
- Solution nodes describe the resolution
- Include the key diagnostic questions that narrowed down the problem
- Skip redundant or dead-end paths from the session
- match_keywords should be symptoms, error messages, and technology names
"""
```

**Verification:** Resolve an AI session (discovery mode, no matched flow). Wait 2-3 seconds. Check `flow_proposals` table — verify a `new_flow` proposal was created with a valid `tree_structure`. Resolve a session that matched an existing flow but diverged — verify an `enhancement` proposal was created. Resolve a session that followed a flow exactly — verify an `auto_reinforced` record was created and the flow's stats were updated.

```
git commit -m "feat(knowledge): add Knowledge Flywheel post-session analysis"
```

### Task 4: Wire Knowledge Flywheel into session resolution

**Files:**
- Edit: `backend/app/services/flowpilot_engine.py` — set `analysis_status = "pending"` in `resolve_session()`
- Create: `backend/app/services/knowledge_flywheel_scheduler.py` — APScheduler job
- Edit: `backend/app/main.py` — register the scheduler job
- Migration: add `analysis_status` column to `ai_sessions` table

**Migration:** Add `analysis_status` column:

```python
# String(20), nullable=True, default=None
# Values: null (not applicable), "pending", "completed", "failed"
op.add_column('ai_sessions', sa.Column('analysis_status', sa.String(20), nullable=True))
```

**In `resolve_session()`**, after documentation is generated and PSA push is queued:

```python
session.analysis_status = "pending"
```

**Scheduler (`knowledge_flywheel_scheduler.py`):**

Follow the same pattern as `psa_retry_scheduler.py`:

```python
async def process_pending_analyses() -> None:
    """Process resolved sessions awaiting Knowledge Flywheel analysis."""
    async with async_session_maker() as db:
        result = await db.execute(
            select(AISession)
            .options(selectinload(AISession.steps))
            .where(AISession.analysis_status == "pending")
            .limit(10)
        )
        sessions = result.scalars().all()
        for session in sessions:
            try:
                await analyze_session(session, db)
                session.analysis_status = "completed"
            except Exception as e:
                logger.warning("Knowledge Flywheel failed for %s: %s", session.id, e)
                session.analysis_status = "failed"
        await db.commit()
```

Register in `main.py` lifespan alongside the existing PSA retry scheduler (5-minute interval).

**Verification:** Resolve a session. Confirm the response returns immediately. Wait for the scheduler tick (~5 min or trigger manually). Check `flow_proposals` table — confirm the proposal was created and `analysis_status = "completed"`.

```
git commit -m "feat(knowledge): wire Knowledge Flywheel into session resolution via scheduler"
```

---

## Slice 2: Review Queue

### Task 5: Create Review Queue API endpoints

**Files:**
- Create: `backend/app/schemas/flow_proposal.py`
- Edit: `backend/app/api/endpoints/ai_sessions.py` (or create a new `flow_proposals.py` router)

**Schemas:**

```python
class FlowProposalSummary(BaseModel):
    id: UUID
    proposal_type: str
    title: str
    description: str | None
    problem_domain: str | None
    confidence_score: float
    supporting_session_count: int
    status: str
    target_flow_id: UUID | None
    target_flow_name: str | None  # Joined from trees table
    source_session_id: UUID
    created_at: datetime

    model_config = {"from_attributes": True}

class FlowProposalDetail(FlowProposalSummary):
    proposed_flow_data: dict[str, Any]
    proposed_diff: dict[str, Any] | None
    supporting_session_ids: list[str]
    reviewer_notes: str | None
    reviewed_by: UUID | None
    reviewed_at: datetime | None

class ReviewProposalRequest(BaseModel):
    action: str  # "approve" | "reject" | "modify"
    reviewer_notes: str | None = None
    modified_flow_data: dict[str, Any] | None = None  # Only for "modify"

class FlowProposalStats(BaseModel):
    pending_count: int
    approved_this_week: int
    rejected_this_week: int
    auto_reinforced_this_week: int
    top_domains: list[dict[str, Any]]  # [{domain, count}]
```

**Endpoints:**

```
GET    /api/v1/flow-proposals              — List proposals (filterable by status, type, domain)
GET    /api/v1/flow-proposals/stats        — Dashboard stats for the review queue
GET    /api/v1/flow-proposals/{id}         — Get proposal detail with full flow data
POST   /api/v1/flow-proposals/{id}/review  — Approve, reject, or modify a proposal
```

**Auth:** `require_engineer_or_admin` for listing/detail. Review actions (approve/reject/modify) require inline check: `if not (current_user.is_super_admin or current_user.is_team_admin): raise HTTPException(403, "Team admin required")`. No existing `require_team_admin` dep exists — add one to `api/deps.py` or use inline checks.

**Review flow:**

- **Approve:** Create a new `Tree` from `proposed_flow_data` (for `new_flow`) or update the existing tree (for `enhancement`). Set `tree.origin = "ai_generated"` or `"ai_enhanced"`. Set `tree.source_session_id`. Set proposal `status = "approved"`, `published_flow_id` = new tree ID.
- **Modify:** Same as approve, but use `modified_flow_data` instead of `proposed_flow_data`. Set proposal `status = "modified"`.
- **Reject:** Set proposal `status = "rejected"`. No flow changes.
- **Dismiss:** Set proposal `status = "dismissed"`. Unlike reject (bad quality), dismiss means "not now" — the proposal can resurface if `supporting_session_count` grows. Add `"dismissed"` to the `FlowProposal` status constraint.

**NOTE on `session_to_flow_service.py`:** This service works with the legacy `Session` model and CANNOT be called directly for `AISession`-based proposals. The Knowledge Flywheel generates `proposed_flow_data` in its own Task 3 — by the time we reach the Review Queue, the flow structure is already in the proposal. The review endpoint just needs to create a `Tree` from the pre-generated `proposed_flow_data` dict (set `tree_type`, `tree_structure`, `origin`, `source_session_id`, etc.). No LLM call needed at review time.

**Verification:** Create a few proposals via the Knowledge Flywheel. Hit the list endpoint. Review one (approve). Verify a new tree was created. Review another (reject). Verify no tree change.

```
git commit -m "feat(knowledge): add Review Queue API endpoints"
```

### Task 6: Build Review Queue frontend

**Files:**
- Create: `frontend/src/pages/ReviewQueuePage.tsx`
- Create: `frontend/src/components/flowpilot/ProposalCard.tsx`
- Create: `frontend/src/components/flowpilot/ProposalDetail.tsx`
- Create: `frontend/src/components/flowpilot/ProposalDiffView.tsx`
- Create: `frontend/src/components/flowpilot/ReviewActions.tsx`
- Create: `frontend/src/api/flowProposals.ts`
- Create: `frontend/src/types/flow-proposal.ts`
- Edit: `frontend/src/router.tsx`
- Edit sidebar navigation

**Page layout:** Two-panel design similar to the Script Library page.

**Left panel — Proposal list:**
- Filter tabs: "Pending" (default), "Approved", "Rejected", "Dismissed", "All"
- Filter by domain (dropdown)
- Sort by: newest, highest confidence, most supporting sessions
- Each card shows: title, proposal type badge (`new_flow` green, `enhancement` amber, `branch_addition` blue), domain badge, confidence score, supporting session count, created date

**Right panel — Proposal detail:**
- Full proposal info: title, description, source session link, confidence
- **For new_flow:** Flow preview — render the proposed `tree_structure` using a simplified read-only version of the flow editor or a tree visualization
- **For enhancement:** Diff view showing what would change on the target flow (added nodes highlighted green, modified nodes highlighted amber)
- Source session link — click to open the session that generated this proposal in read-only mode
- Supporting sessions list (if count > 1)

**Review actions (bottom bar):**

- "Approve & Publish" (green) — creates the flow immediately
- "Edit & Publish" — uses `navigate('/editor/new', { state: { preloadedStructure: proposedFlowData, proposalId } })` to open the Flow Editor with the proposed structure pre-loaded (same `location.state` pattern used by `CreateFlowDropdown` → `AIPromptDialog`, see Lesson 46)
- "Dismiss" (muted) — parks the proposal for later; can resurface if supporting sessions grow
- "Reject" (red) — with optional reason textarea
- Reviewer notes input

**Navigation:**
- Add "Review Queue" to the sidebar under "Knowledge Base" section
- Show a badge with pending count if > 0 (similar to notification badges)

**Verification:** Navigate to Review Queue. See pending proposals. Click one. See the flow preview. Approve it. Verify a new tree appears in My Trees. Click "Edit & Publish" on another — verify it opens in the Flow Editor with the proposed structure pre-loaded.

```
git commit -m "feat(knowledge): add Review Queue frontend"
```

---

## Slice 3: Knowledge Gap Detection

### Task 7: Build knowledge gap detection service

**Files:**
- Create: `backend/app/services/knowledge_gap_service.py`

**Architecture:**

This service aggregates signals from AI sessions to identify gaps in the knowledge base:

**Signal 1 — Frequent free-text escapes:**
- Query `ai_session_steps` where `was_free_text = true`
- Group by the `content` field (the question that was asked) and count
- High counts indicate FlowPilot's options don't cover a common scenario
- Return: list of questions with high free-text rates and the common free-text inputs

**Signal 2 — High escalation rate by domain:**
- Query `ai_sessions` where `status = "escalated"`, group by `problem_domain`
- Compare escalation rate vs resolution rate per domain
- Domains with >40% escalation rate are flagged as knowledge gaps

**Signal 3 — Discovery-mode resolutions:**
- Query `ai_sessions` where `status = "resolved"` AND `confidence_tier = "discovery"` at resolution
- These are novel problems that were solved — highest-value knowledge capture opportunities
- Group by `problem_domain` and rank by frequency

**Signal 4 — Repeated similar intake patterns (DESCOPED to Phase 4):**
~~Use embedding similarity on `intake_content.text` across recent sessions and cluster similar intakes.~~
**Reason:** The codebase has point-query embedding support (Voyage AI) but no batch embedding or clustering infrastructure. Implementing vector clustering (DBSCAN/k-means) over session embeddings is a significant undertaking. **Phase 3 alternative:** Use keyword frequency analysis on `problem_domain` + `problem_summary` text to find repeated unmatched patterns. Full embedding clustering deferred to Phase 4.

**Return type:**

```python
class KnowledgeGapReport(BaseModel):
    generated_at: datetime
    gaps: list[KnowledgeGap]

class KnowledgeGap(BaseModel):
    gap_type: str  # "weak_options" | "high_escalation" | "uncharted_territory" | "repeated_pattern"
    domain: str | None
    severity: str  # "high" | "medium" | "low"
    title: str
    description: str
    evidence: dict[str, Any]  # Supporting data (counts, examples, session IDs)
    suggested_action: str  # What to do about it
```

**Endpoint:**

```
GET /api/v1/analytics/knowledge-gaps
```

Returns the current knowledge gap report. Cache for 1 hour (expensive query).

**Verification:** Run several AI sessions across different domains. Some should escalate, some should use free-text. Hit the knowledge gaps endpoint. Verify it returns reasonable gap analysis.

```
git commit -m "feat(knowledge): add knowledge gap detection service"
```

---

## Slice 4: In-Session Script Generator

### Task 8: Enable FlowPilot to invoke Script Generator during sessions

**Files:**
- Edit: `backend/app/services/flowpilot_engine.py`
- Edit: `frontend/src/components/flowpilot/FlowPilotStepCard.tsx`
- Create: `frontend/src/components/flowpilot/InSessionScriptGenerator.tsx`

**Backend — System prompt enhancement:**

Add available script templates to FlowPilot's system prompt context. When building the system prompt in `_build_system_prompt()`, include:

```python
# Query available script templates
templates = await db.execute(
    select(ScriptTemplate)
    .where(ScriptTemplate.is_active == True)
    .where(or_(ScriptTemplate.team_id == None, ScriptTemplate.team_id == team_id))
    .order_by(ScriptTemplate.usage_count.desc())
    .limit(20)
)
template_list = templates.scalars().all()

# Add to system prompt
script_context = "\n--- AVAILABLE SCRIPTS ---\n"
for t in template_list:
    script_context += f"- {t.name} (ID: {t.id}): {t.description}\n"
    script_context += f"  Parameters: {', '.join(p['key'] for p in t.parameters_schema.get('parameters', []))}\n"
script_context += "\nWhen the engineer needs to run a script, suggest a script_generation action with the template_id and pre-fill parameters from the diagnostic context.\n"
```

**Backend — Structured output for script actions:**

FlowPilot already supports `action_type: "script_generation"` in its structured output contract (defined in Phase 1). When FlowPilot returns this type, the response includes:

```json
{
  "type": "action",
  "action_type": "script_generation",
  "template_id": "uuid-of-the-template",
  "pre_filled_params": {
    "sam_account_name": "jsmith",
    "ou_path": "OU=Users,DC=contoso,DC=com"
  },
  "instructions": "Generate a password reset script for this user",
  "confidence": 0.85
}
```

The backend should validate that `template_id` exists and is accessible to the user's team.

**IMPORTANT — Migration needed:** `ScriptGeneration.session_id` currently FKs to legacy `sessions.id`, NOT `ai_sessions.id`. Add a migration to add `ai_session_id` FK column to `script_generations`:

```python
op.add_column('script_generations', sa.Column(
    'ai_session_id', sa.UUID(), sa.ForeignKey('ai_sessions.id', ondelete='SET NULL'), nullable=True
))
op.create_index('ix_script_generations_ai_session_id', 'script_generations', ['ai_session_id'])
```

Update `ScriptGeneration` model to include `ai_session_id` mapped column. The existing `session_id` FK stays for legacy sessions.

**Frontend — `InSessionScriptGenerator` component:**

When `FlowPilotStepCard` receives a step with `action_type === "script_generation"`:

1. Render the step card with a script generation UI embedded inline
2. Reuse the existing `ScriptParameterForm` component from the Script Library
3. Pre-fill parameters from `pre_filled_params` in the step content
4. Engineer can edit parameters and generate the script
5. On generation, call the existing `POST /api/v1/scripts/generate` endpoint with `ai_session_id` set to the current AI session
6. Display the generated script with the existing `ScriptPreview` component (PowerShell syntax highlighting)
7. Copy/download buttons
8. "Script generated" event is captured in `ai_session_steps` with `step_type = "script_generation"` and `script_generation_id` FK populated
9. After generating, show "Continue" button → engineer reports result back to FlowPilot

**Key reuse:** Import and compose these existing components from `src/components/scripts/`:

- `ScriptParameterForm` — dynamic form from parameter schema
- `ScriptPreview` — PowerShell syntax highlighting
- `PowerShellHighlighter` — tokenizer

**Do NOT rebuild these components.** Wrap them in `InSessionScriptGenerator` which handles the session context (pre-filling, event capture, continue flow).

**Verification:** Start an AI session about an AD issue. Progress until FlowPilot suggests a script generation action. See the script generator appear inline. Parameters should be pre-filled from conversation context. Generate the script. Copy it. Click continue. Verify the step is captured in session docs with `script_generation_id` populated.

```
git commit -m "feat(ai-session): add in-session Script Generator integration"
```

---

## Slice 5: AI-Enhanced Analytics Dashboard

### Task 9: Build FlowPilot analytics API endpoints

**Files:**
- Create: `backend/app/schemas/flowpilot_analytics.py`
- Create: `backend/app/api/endpoints/flowpilot_analytics.py`
- Edit: `backend/app/api/router.py`

**Schemas:**

```python
class FlowPilotDashboard(BaseModel):
    """Top-level analytics dashboard data."""
    period: str  # "7d" | "30d" | "90d"
    total_sessions: int
    resolved_sessions: int
    escalated_sessions: int
    abandoned_sessions: int
    resolution_rate: float  # percentage
    avg_steps_to_resolution: float
    avg_session_duration_minutes: float
    avg_rating: float | None
    mttr_minutes: float | None  # Mean Time To Resolution
    mttr_trend: list[MTTRDataPoint]  # For chart
    sessions_by_domain: list[DomainBreakdown]
    confidence_breakdown: ConfidenceBreakdown
    knowledge_coverage: KnowledgeCoverage
    psa_metrics: PsaMetrics | None  # None if no PSA connection

class MTTRDataPoint(BaseModel):
    date: str  # ISO date
    mttr_minutes: float
    session_count: int

class DomainBreakdown(BaseModel):
    domain: str
    total: int
    resolved: int
    escalated: int
    resolution_rate: float

class ConfidenceBreakdown(BaseModel):
    guided_sessions: int
    guided_resolution_rate: float
    exploring_sessions: int
    exploring_resolution_rate: float
    discovery_sessions: int
    discovery_resolution_rate: float

class KnowledgeCoverage(BaseModel):
    total_flows: int
    ai_generated_flows: int
    total_proposals_pending: int
    proposals_approved_this_period: int
    proposals_rejected_this_period: int
    coverage_by_domain: list[DomainCoverage]

class DomainCoverage(BaseModel):
    domain: str
    flow_count: int
    session_count: int
    guided_rate: float  # % of sessions in this domain that hit "guided" confidence

class PsaMetrics(BaseModel):
    """PSA integration metrics (Phase 2 ROI data)."""
    ticket_link_rate: float  # % of sessions linked to a PSA ticket
    auto_push_success_rate: float  # % of pushes that succeeded on first try
    auto_push_retry_success_rate: float  # % that succeeded after retries
    total_time_entries_logged: int
    total_hours_logged: float
```

**Endpoints:**

```
GET /api/v1/analytics/flowpilot?period=30d           — Main dashboard data
GET /api/v1/analytics/flowpilot/mttr-trend?period=90d — MTTR trend chart data
GET /api/v1/analytics/flowpilot/knowledge-gaps        — Knowledge gap report (from Task 7)
```

**Auth:** Team admin or owner. Scope to account.

**Key queries:**

- MTTR: `AVG(resolved_at - created_at)` for sessions where `status = "resolved"`, grouped by date
- Confidence breakdown: `COUNT(*) GROUP BY confidence_tier` for resolved sessions
- Domain breakdown: `COUNT(*), SUM(CASE WHEN status='resolved')` grouped by `problem_domain`
- Knowledge coverage: Count flows per domain vs session count per domain. High session count + low flow count = poor coverage.

**Verification:** Run several AI sessions over different days (can seed test data). Hit the analytics endpoint. Verify all fields are populated with reasonable values.

```
git commit -m "feat(analytics): add FlowPilot analytics API"
```

### Task 10: Build FlowPilot analytics dashboard frontend

**Files:**
- Create: `frontend/src/pages/FlowPilotAnalyticsPage.tsx`
- Create: `frontend/src/components/flowpilot/analytics/MTTRChart.tsx`
- Create: `frontend/src/components/flowpilot/analytics/DomainBreakdownChart.tsx`
- Create: `frontend/src/components/flowpilot/analytics/ConfidenceBreakdown.tsx`
- Create: `frontend/src/components/flowpilot/analytics/KnowledgeCoverageMap.tsx`
- Create: `frontend/src/components/flowpilot/analytics/KnowledgeGapsPanel.tsx`
- Create: `frontend/src/api/flowpilotAnalytics.ts`
- Create: `frontend/src/types/flowpilot-analytics.ts`
- Edit: `frontend/src/router.tsx`

**Design:** Follow existing `TeamAnalyticsPage.tsx` patterns. Use Recharts for charts (already a project dependency).

**Layout:**

**Top row — Key metrics cards:**

- Total sessions (with trend arrow)
- Resolution rate (with trend)
- Average MTTR (with trend)
- Average rating (with star display)
- PSA ticket link rate (% of sessions linked to tickets)

**Second row — Charts:**

- MTTR trend area chart (Recharts `AreaChart` — matches existing `TeamAnalyticsPage.tsx` pattern, not `LineChart`)
- Domain breakdown bar chart (Recharts `BarChart`) — resolved vs escalated per domain

**Third row — Intelligence:**
- Confidence tier donut chart — guided vs exploring vs discovery, with resolution rate overlay
- Knowledge coverage heatmap — domains as rows, columns for flow count / session count / guided rate / gap severity. Color-coded: green (well covered), amber (needs work), red (major gap)

**Fourth row — Knowledge gaps:**
- `KnowledgeGapsPanel` — renders the knowledge gap report from Task 7
- Each gap as a card with severity badge, description, and suggested action
- "Create Flow" CTA on high-severity gaps → opens the Flow Editor with suggested structure

**Period selector:** Dropdown in the page header — 7 days, 30 days, 90 days.

**Navigation:** Add "FlowPilot Analytics" under the existing "Analytics" section in the sidebar.

**Verification:** Navigate to the analytics page. Select different periods. Verify charts render with real data. Check knowledge gaps section shows actionable insights.

```
git commit -m "feat(analytics): add FlowPilot analytics dashboard"
```

---

## Summary of All New/Modified Files

### Backend — New
```
app/models/flow_proposal.py                          # FlowProposal model
app/services/knowledge_flywheel.py                    # Post-session analysis engine
app/services/knowledge_flywheel_scheduler.py          # APScheduler job for async analysis
app/services/knowledge_gap_service.py                 # Knowledge gap detection
app/schemas/flow_proposal.py                          # Proposal schemas
app/schemas/flowpilot_analytics.py                    # Analytics schemas
app/api/endpoints/flow_proposals.py                   # Review Queue API
app/api/endpoints/flowpilot_analytics.py              # Analytics API
alembic/versions/xxx_add_flow_proposals.py            # Migration (flow_proposals table)
alembic/versions/xxx_add_analysis_status.py           # Migration (ai_sessions.analysis_status)
alembic/versions/xxx_add_ai_session_id_to_scripts.py  # Migration (script_generations.ai_session_id)
```

### Backend — Modified
```
app/models/__init__.py                                # Register FlowProposal
app/models/ai_session.py                              # Add analysis_status column
app/models/script_template.py                         # Add ai_session_id FK to ScriptGeneration
app/api/deps.py                                       # Add require_team_admin dependency
app/api/router.py                                     # Register new routers
app/main.py                                           # Register knowledge_flywheel_scheduler
app/services/flowpilot_engine.py                      # Set analysis_status, add script context to system prompt
```

### Frontend — New
```
src/pages/ReviewQueuePage.tsx                          # Review Queue page
src/pages/FlowPilotAnalyticsPage.tsx                   # Analytics dashboard
src/components/flowpilot/ProposalCard.tsx               # Proposal list card
src/components/flowpilot/ProposalDetail.tsx             # Proposal detail panel
src/components/flowpilot/ProposalDiffView.tsx           # Enhancement diff viewer
src/components/flowpilot/ReviewActions.tsx              # Approve/reject/modify bar
src/components/flowpilot/InSessionScriptGenerator.tsx   # Script gen embedded in session
src/components/flowpilot/analytics/MTTRChart.tsx        # MTTR trend chart
src/components/flowpilot/analytics/DomainBreakdownChart.tsx
src/components/flowpilot/analytics/ConfidenceBreakdown.tsx
src/components/flowpilot/analytics/KnowledgeCoverageMap.tsx
src/components/flowpilot/analytics/KnowledgeGapsPanel.tsx
src/api/flowProposals.ts                               # Proposals API client
src/api/flowpilotAnalytics.ts                          # Analytics API client
src/types/flow-proposal.ts                             # Proposal types
src/types/flowpilot-analytics.ts                       # Analytics types
```

### Frontend — Modified
```
src/components/flowpilot/FlowPilotStepCard.tsx         # Handle script_generation action type
src/router.tsx                                         # Review Queue + Analytics routes
src/components/sidebar/                                # New nav entries
```

---

## Database Changes

**Migration:** Create `flow_proposals` table with all columns, constraints, and indexes.

**Run migration:**
```bash
cd /projects/patherly/backend
DATABASE_URL=postgresql://postgres:postgres@resolutionflow_postgres:5432/resolutionflow \
  venv/bin/alembic upgrade head
```

---

## Testing Strategy

### Backend Unit Tests

**Files:** `backend/tests/test_knowledge_flywheel.py`

- Test new_flow proposal generation from a discovery-mode session
- Test enhancement proposal generation from a divergent session
- Test auto_reinforcement for matching sessions
- Test duplicate detection (similar proposals get merged)
- Test generated tree_structure matches the expected format

**Files:** `backend/tests/test_knowledge_gap_service.py`

- Test free-text escape detection
- Test escalation rate calculation by domain
- Test discovery-mode session grouping

**Files:** `backend/tests/test_flow_proposals_api.py`

- Test list/filter proposals
- Test approve → verify tree created
- Test modify → verify modified data used
- Test reject → verify no tree change
- Test RBAC (non-admin can't review)

### Frontend Manual Testing

1. Resolve several AI sessions (mix of discovery, exploring, guided)
2. Navigate to Review Queue — verify proposals appear
3. Approve a new_flow proposal — verify tree appears in library
4. Click "Edit & Publish" — verify Flow Editor opens with proposed structure
5. Start a session about an AD issue — progress until FlowPilot suggests a script — generate it inline
6. Navigate to FlowPilot Analytics — verify all charts render with data
7. Check Knowledge Gaps — verify actionable insights appear

---

## What Comes Next (Phase 4+ — NOT in scope here)

- **Template Marketplace:** Public templates gallery for SEO + lead gen
- **Multi-PSA support:** Autotask, Halo PSA, Datto PSA integrations
- **SSO/SAML:** Enterprise authentication
- **Custom AI training:** Per-account fine-tuning on company procedures
- **Mobile optimization:** Responsive design pass for tablet/phone sessions
- **Webhook integrations:** Slack notifications, Teams alerts on escalation
- **API for third-party tools:** Public API for RMM/PSA vendors to integrate