582 lines
26 KiB
Markdown
582 lines
26 KiB
Markdown
# AI-Guided Flow Creation — Design Document
|
|
|
|
> **Date:** 2026-02-12
|
|
> **Status:** Draft
|
|
> **Phase:** 3 (AI Intelligence)
|
|
> **Dependencies:** Global Categories, Flow Editor, Plan Limits, Session Tracking
|
|
> **Estimated Effort:** 3-4 weeks (backend + frontend + prompt engineering)
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
AI-Guided Flow Creation is an interactive wizard that helps engineers build troubleshooting and procedural flows through a structured conversation. Rather than generating a complete flow from a single description (expensive, low-quality, black box), the wizard asks targeted questions at each stage, using the engineer's domain expertise to shape the output while AI fills in the details.
|
|
|
|
**Core Principle:** The engineer knows the problem domain. The AI knows how to structure it. The wizard brings both together.
|
|
|
|
**Key Differentiator:** This is NOT a "describe your problem and AI builds everything" feature. It's a collaborative creation tool where AI assists at specific, bounded points — keeping costs predictable, output quality high, and the engineer in control.
|
|
|
|
---
|
|
|
|
## Why This Approach
|
|
|
|
| Approach | Cost/Flow | Quality | Engineer Control | Reusability |
|
|
|---|---|---|---|---|
|
|
| Full AI generation (one-shot) | ~$0.03 | Medium (needs heavy editing) | None until review | None |
|
|
| **Guided wizard (this design)** | **~$0.01-0.02** | **High (shaped by engineer)** | **At every step** | **High (cached prompts)** |
|
|
| Manual creation only | $0.00 | Varies | Full | None |
|
|
|
|
The guided approach wins on every axis because:
|
|
|
|
1. **Smaller, targeted API calls** replace one large generation call
|
|
2. **Fixed question patterns** enable aggressive prompt caching (90% input savings)
|
|
3. **Engineer input at each stage** means less post-generation editing
|
|
4. **Structured data collection** before any AI call means better prompts and better output
|
|
|
|
---
|
|
|
|
## User Flow
|
|
|
|
### Stage 1: Foundation (No AI — Pure UI)
|
|
|
|
The user provides structured metadata that will inform all subsequent AI calls.
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ CREATE A NEW FLOW │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Flow Type: │
|
|
│ ○ Troubleshooting (diagnostic branching — "what's wrong?")│
|
|
│ ○ Procedure (step-by-step — "how do I set this up?") │
|
|
│ │
|
|
│ Category: │
|
|
│ [▼ Networking ] │
|
|
│ │
|
|
│ Flow Name: │
|
|
│ [Printer Not Printing ] │
|
|
│ │
|
|
│ Brief Description: │
|
|
│ ┌─────────────────────────────────────────────────────┐ │
|
|
│ │ User reports print jobs stuck in queue, printer │ │
|
|
│ │ shows as offline or errors when printing │ │
|
|
│ └─────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Target Environment (optional): │
|
|
│ □ Windows □ macOS □ Linux │
|
|
│ □ Cloud/SaaS □ On-Premises □ Hybrid │
|
|
│ │
|
|
│ [Next →] │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**Data collected:** flow_type, category_id, name, description, environment tags.
|
|
**AI cost:** $0.00
|
|
|
|
---
|
|
|
|
### Stage 2: Structure Scaffolding (Light AI)
|
|
|
|
Based on Stage 1 input, AI suggests the top-level structure. The user confirms, removes, or adds items.
|
|
|
|
**For Troubleshooting flows — AI suggests initial symptom branches:**
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ WHAT SYMPTOMS WOULD A USER REPORT? │
|
|
│ │
|
|
│ Based on "Printer Not Printing", here are common starting │
|
|
│ points. Check the ones that apply, add your own: │
|
|
│ │
|
|
│ AI Suggestions: │
|
|
│ ☑ Print jobs stuck in queue / never print │
|
|
│ ☑ Printer shows offline │
|
|
│ ☑ Prints but output is garbled or wrong │
|
|
│ ☐ Printer not found / can't add printer │
|
|
│ ☑ Specific application can't print (others can) │
|
|
│ │
|
|
│ + Add your own symptom: [____________________________] │
|
|
│ │
|
|
│ [← Back] [Next →] │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**For Procedure flows — AI suggests major phases:**
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ WHAT ARE THE MAJOR PHASES? │
|
|
│ │
|
|
│ Based on "New Server Build", here are typical phases. │
|
|
│ Check the ones that apply, reorder, add your own: │
|
|
│ │
|
|
│ AI Suggestions: │
|
|
│ ☑ 1. Pre-requisites & Planning │
|
|
│ ☑ 2. OS Installation & Base Config │
|
|
│ ☑ 3. Network Configuration │
|
|
│ ☑ 4. Domain Join & Security │
|
|
│ ☑ 5. Role/Application Installation │
|
|
│ ☑ 6. Verification & Handoff │
|
|
│ │
|
|
│ + Add a phase: [____________________________] │
|
|
│ │
|
|
│ [← Back] [Next →] │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**API call:** 1 call to Haiku 4.5. Input: system prompt (cached) + stage 1 metadata. Output: 5-8 suggestions as JSON array.
|
|
**Estimated cost:** ~$0.003-0.005
|
|
|
|
---
|
|
|
|
### Stage 3: Branch/Step Detail (Light AI, per branch)
|
|
|
|
For each item the user selected in Stage 2, AI suggests the diagnostic steps or sub-steps. The user processes one branch/phase at a time.
|
|
|
|
**For Troubleshooting — diagnostic steps per symptom:**
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ BRANCH: "Print jobs stuck in queue" │
|
|
│ │
|
|
│ What steps should an engineer follow for this symptom? │
|
|
│ │
|
|
│ AI Suggestions: │
|
|
│ ☑ Check Print Spooler service status │
|
|
│ Action: Run Get-Service Spooler — restart if stopped │
|
|
│ │
|
|
│ ☑ Clear the print queue │
|
|
│ Action: Stop spooler, delete files in │
|
|
│ C:\Windows\System32\spool\PRINTERS, restart spooler │
|
|
│ │
|
|
│ ☑ Verify printer port and driver │
|
|
│ Decision: Is the printer networked or USB? │
|
|
│ → Networked: ping printer IP, check port config │
|
|
│ → USB: check cable, try different port │
|
|
│ │
|
|
│ ☑ Test with different application │
|
|
│ Decision: Does it print from Notepad? │
|
|
│ → Yes: Application-specific issue │
|
|
│ → No: System-level print problem │
|
|
│ │
|
|
│ + Add your own step: [____________________________] │
|
|
│ │
|
|
│ [✎ Edit any step] [← Back] [Next →] │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**For Procedure — detailed steps per phase:**
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ PHASE: "OS Installation & Base Config" │
|
|
│ │
|
|
│ What steps should be performed in this phase? │
|
|
│ │
|
|
│ AI Suggestions: │
|
|
│ ☑ Boot from installation media │
|
|
│ Action: Mount ISO / insert USB, boot to BIOS, set │
|
|
│ boot order │
|
|
│ │
|
|
│ ☑ Configure disk partitions │
|
|
│ Action: Create partitions per standard │
|
|
│ 📝 Record: "Partition layout used" │
|
|
│ │
|
|
│ ☑ Set hostname │
|
|
│ Action: Follow naming convention: SITE-ROLE-## │
|
|
│ 📝 Record: "Hostname assigned" │
|
|
│ │
|
|
│ ☑ Configure local admin account │
|
|
│ Action: Set password per policy, disable built-in │
|
|
│ Administrator │
|
|
│ │
|
|
│ ☑ Install Windows Updates │
|
|
│ Action: Run full update cycle, reboot, repeat until │
|
|
│ clean │
|
|
│ │
|
|
│ + Add your own step: [____________________________] │
|
|
│ │
|
|
│ [← Back] [Next →] │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**Note the 📝 Record fields** — for procedure flows, AI can suggest which steps should capture data (hostname, IP, etc.). This feeds directly into the session documentation.
|
|
|
|
**API call:** 1 call per branch/phase. Input: system prompt (cached) + stage 1 metadata + branch name + context from other branches. Output: 3-6 detailed steps as JSON.
|
|
**Estimated cost:** ~$0.003-0.005 per branch. 5 branches = ~$0.015-0.025
|
|
|
|
---
|
|
|
|
### Stage 4: Review & Refine (No AI)
|
|
|
|
The complete flow is assembled and shown in the existing tree/flow editor. User can:
|
|
|
|
- Rearrange nodes
|
|
- Edit text on any step
|
|
- Add/remove branches
|
|
- Preview the flow as an end-user would see it
|
|
- Save as draft or publish
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ REVIEW YOUR FLOW │
|
|
│ │
|
|
│ "Printer Not Printing" — Troubleshooting Flow │
|
|
│ Category: Networking | 5 branches | 23 steps │
|
|
│ │
|
|
│ [Visual tree/flow editor — existing component] │
|
|
│ │
|
|
│ ┌─ Root: What is the printing issue? │
|
|
│ ├── Jobs stuck in queue (5 steps) │
|
|
│ ├── Printer shows offline (4 steps) │
|
|
│ ├── Garbled output (3 steps) │
|
|
│ ├── App-specific failure (4 steps) │
|
|
│ └── Can't add printer (4 steps) │
|
|
│ │
|
|
│ [Save as Draft] [Publish] │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**AI cost:** $0.00
|
|
|
|
---
|
|
|
|
## Total Cost Per Flow
|
|
|
|
| Stage | AI Calls | Estimated Cost |
|
|
|---|---|---|
|
|
| 1. Foundation | 0 | $0.000 |
|
|
| 2. Structure Scaffolding | 1 | $0.003-0.005 |
|
|
| 3. Branch/Step Detail | 1 per branch (avg 5) | $0.015-0.025 |
|
|
| 4. Review & Refine | 0 | $0.000 |
|
|
| **Total** | **~6 calls** | **$0.018-0.030** |
|
|
|
|
**With prompt caching active** (system prompt cached across all users):
|
|
- First user of the day: ~$0.025 (full system prompt cost)
|
|
- Subsequent users: ~$0.012-0.018 (90% savings on system prompt input)
|
|
|
|
---
|
|
|
|
## Cost Projections By Plan
|
|
|
|
### Per-User Monthly Cost to ResolutionFlow
|
|
|
|
| Plan | AI Flows/Month | Est. Cost/Month | Subscription Price | Margin |
|
|
|---|---|---|---|---|
|
|
| Free | 2 | $0.04-0.06 | $0 | -$0.06 (acquisition cost) |
|
|
| Pro | 30 (1/day) | $0.54-0.90 | $29 | 97%+ |
|
|
| Pro (heavy) | 300 (10/day) | $5.40-9.00 | $29 | 69-81% |
|
|
| Team (per user) | 300 (10/day) | $5.40-9.00 | ~$20/seat | 55-73% |
|
|
|
|
### At Scale (Monthly Platform Cost)
|
|
|
|
| Scenario | Users | Flows/Month | Monthly AI Cost |
|
|
|---|---|---|---|
|
|
| Beta (15 users) | 15 | ~150 | $2.70-4.50 |
|
|
| Early growth | 100 | ~1,500 | $27-45 |
|
|
| Scaling | 500 | ~7,500 | $135-225 |
|
|
| Target ($10K MRR) | ~400 | ~6,000 | $108-180 |
|
|
|
|
**Bottom line:** Even at 10/day per user, AI costs stay under 30% of subscription revenue. At typical usage (1-3/day), AI costs are negligible — under 5% of revenue.
|
|
|
|
---
|
|
|
|
## Plan Limits Integration
|
|
|
|
### New Fields for plan_limits Table
|
|
|
|
```sql
|
|
ALTER TABLE plan_limits
|
|
ADD COLUMN ai_flows_per_month INTEGER DEFAULT NULL,
|
|
ADD COLUMN ai_calls_per_flow INTEGER DEFAULT NULL;
|
|
```
|
|
|
|
| Plan | ai_flows_per_month | ai_calls_per_flow |
|
|
|---|---|---|
|
|
| Free | 2 | 6 |
|
|
| Pro | 50 | 10 |
|
|
| Team | 200 (per account) | 10 |
|
|
|
|
`ai_calls_per_flow` caps how many branches can get AI suggestions per flow. This prevents a user from creating a 50-branch monstrosity that costs $0.25 per generation.
|
|
|
|
### Tracking Table
|
|
|
|
```sql
|
|
CREATE TABLE ai_usage_log (
|
|
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
account_id UUID NOT NULL REFERENCES accounts(id),
|
|
user_id UUID NOT NULL REFERENCES users(id),
|
|
usage_type VARCHAR(50) NOT NULL, -- 'flow_scaffold', 'branch_detail', 'branch_suggest'
|
|
model VARCHAR(100) NOT NULL, -- 'claude-haiku-4-5'
|
|
input_tokens INTEGER NOT NULL,
|
|
output_tokens INTEGER NOT NULL,
|
|
estimated_cost_usd NUMERIC(10, 6) NOT NULL,
|
|
flow_id UUID REFERENCES trees(id),
|
|
metadata JSONB DEFAULT '{}', -- stage, branch name, etc.
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
|
);
|
|
|
|
CREATE INDEX idx_ai_usage_account_month
|
|
ON ai_usage_log (account_id, created_at);
|
|
```
|
|
|
|
This gives you:
|
|
- Per-account usage tracking for limit enforcement
|
|
- Cost visibility in admin dashboard
|
|
- Data to optimize prompts over time (which calls use the most tokens?)
|
|
|
|
---
|
|
|
|
## Backend Implementation
|
|
|
|
### New Files
|
|
|
|
```
|
|
app/
|
|
services/
|
|
ai_flow_generator.py # Core AI service — prompt construction, API calls, parsing
|
|
api/
|
|
endpoints/
|
|
ai_generation.py # API endpoints for wizard stages
|
|
core/
|
|
ai_config.py # Model selection, pricing constants, feature flag checks
|
|
```
|
|
|
|
### API Endpoints
|
|
|
|
```
|
|
POST /api/v1/ai/flow/scaffold
|
|
Body: { flow_type, category_id, name, description, environment_tags }
|
|
Returns: { suggestions: ["symptom1", "symptom2", ...] }
|
|
Auth: Required, plan limit checked
|
|
AI: 1 Haiku call
|
|
|
|
POST /api/v1/ai/flow/branch-detail
|
|
Body: { flow_type, category_id, name, description, branch_name, existing_branches }
|
|
Returns: { steps: [ { type, title, action/question, help_text, sub_branches?, record_fields? } ] }
|
|
Auth: Required, plan limit checked, per-flow call limit checked
|
|
AI: 1 Haiku call
|
|
|
|
POST /api/v1/ai/flow/assemble
|
|
Body: { flow_type, category_id, name, description, branches: [ { name, steps: [...] } ] }
|
|
Returns: { tree_structure: { ... } } // Valid tree structure ready for create_tree
|
|
Auth: Required
|
|
AI: 0 calls — pure assembly logic, runs through normalize_node + validation
|
|
```
|
|
|
|
### Prompt Architecture
|
|
|
|
The system prompt is the same across all users and all calls within a stage. This is critical for prompt caching.
|
|
|
|
**System Prompt (Stage 2 — Troubleshooting):**
|
|
```
|
|
You are a senior MSP engineer helping build troubleshooting decision trees.
|
|
Given a problem description and category, suggest 4-7 initial symptom branches
|
|
that a support engineer would encounter.
|
|
|
|
Rules:
|
|
- Each suggestion should be a distinct, common symptom (not overlapping)
|
|
- Order from most common to least common
|
|
- Use plain language that a Tier 1 engineer would understand
|
|
- Focus on observable symptoms, not root causes
|
|
- Respond in JSON format only: { "suggestions": ["symptom1", "symptom2"] }
|
|
```
|
|
|
|
**System Prompt (Stage 2 — Procedure):**
|
|
```
|
|
You are a senior MSP engineer helping build procedural checklists.
|
|
Given a project description and category, suggest 4-8 major phases
|
|
that the project should follow.
|
|
|
|
Rules:
|
|
- Phases should be in logical execution order
|
|
- Each phase should be a distinct stage of work
|
|
- Use standard MSP/IT terminology
|
|
- Respond in JSON format only: { "suggestions": ["phase1", "phase2"] }
|
|
```
|
|
|
|
**System Prompt (Stage 3 — Branch Detail):**
|
|
```
|
|
You are a senior MSP engineer helping build troubleshooting steps.
|
|
Given a symptom/branch name and context, suggest 3-6 diagnostic steps.
|
|
|
|
For each step, provide:
|
|
- type: "action" (do something), "decision" (yes/no check), or "solution" (resolution)
|
|
- title: short step name
|
|
- content: detailed instructions including commands where relevant
|
|
- sub_branches: (for decisions only) what the yes/no paths look like
|
|
- record_fields: (for procedures only) data the engineer should document
|
|
|
|
Rules:
|
|
- Include specific commands (PowerShell preferred for Windows environments)
|
|
- Action steps should be concrete and actionable
|
|
- Decision steps should have clear yes/no outcomes
|
|
- End branches with solution nodes where possible
|
|
- Respond in JSON format only
|
|
```
|
|
|
|
**Caching strategy:** System prompts are sent as cacheable prefixes. With a 5-minute TTL, any user hitting the wizard within 5 minutes of another user gets 90% savings on input tokens. During active hours, this cache will almost always be warm.
|
|
|
|
---
|
|
|
|
## Frontend Implementation
|
|
|
|
### New Components
|
|
|
|
```
|
|
frontend/src/
|
|
pages/
|
|
FlowWizardPage.tsx # Main wizard container with stage navigation
|
|
components/
|
|
wizard/
|
|
WizardStageFoundation.tsx # Stage 1: type, category, name, description
|
|
WizardStageScaffold.tsx # Stage 2: AI suggestions with checkboxes
|
|
WizardStageBranchDetail.tsx # Stage 3: per-branch step suggestions
|
|
WizardStageReview.tsx # Stage 4: assembled flow preview
|
|
WizardProgress.tsx # Stage indicator bar
|
|
SuggestionCheckbox.tsx # Reusable AI suggestion with check/uncheck
|
|
CustomItemInput.tsx # "Add your own" input field
|
|
api/
|
|
aiGeneration.ts # API client for AI endpoints
|
|
hooks/
|
|
useAiWizard.ts # State management for wizard flow
|
|
```
|
|
|
|
### Wizard State Management
|
|
|
|
```typescript
|
|
interface WizardState {
|
|
// Stage 1
|
|
flowType: 'troubleshooting' | 'procedure';
|
|
categoryId: string;
|
|
name: string;
|
|
description: string;
|
|
environmentTags: string[];
|
|
|
|
// Stage 2
|
|
scaffoldSuggestions: string[]; // AI-generated
|
|
selectedBranches: string[]; // User-confirmed
|
|
customBranches: string[]; // User-added
|
|
|
|
// Stage 3
|
|
branchDetails: Record<string, BranchStep[]>; // Per-branch steps
|
|
currentBranchIndex: number;
|
|
|
|
// Stage 4
|
|
assembledStructure: TreeStructure | null;
|
|
|
|
// Meta
|
|
currentStage: 1 | 2 | 3 | 4;
|
|
aiCallsUsed: number;
|
|
isGenerating: boolean;
|
|
}
|
|
```
|
|
|
|
### UX Considerations
|
|
|
|
1. **Loading states:** AI calls take 2-5 seconds. Show a subtle loading animation with context ("Analyzing common symptoms..."), not a spinner.
|
|
|
|
2. **Graceful degradation:** If an AI call fails, show an empty state with "Add your own" prominent. The wizard should never be blocked by AI failure.
|
|
|
|
3. **Edit-in-place:** Users should be able to click any AI suggestion to edit the text before accepting it.
|
|
|
|
4. **Skip AI:** Every stage should have a "Skip suggestions, I'll add my own" option. Power users may want to build manually but still benefit from the wizard structure.
|
|
|
|
5. **Mobile consideration:** The wizard should work on tablet at minimum. MSP engineers may use it on-site with an iPad.
|
|
|
|
---
|
|
|
|
## Environment & Configuration
|
|
|
|
### Environment Variables
|
|
|
|
```
|
|
# .env / Railway
|
|
ANTHROPIC_API_KEY=sk-ant-...
|
|
AI_GENERATION_MODEL=claude-haiku-4-5-20251001
|
|
AI_GENERATION_ENABLED=true # Kill switch
|
|
AI_MAX_RETRIES=2
|
|
AI_REQUEST_TIMEOUT=30 # seconds
|
|
```
|
|
|
|
### Feature Flag Integration
|
|
|
|
Use existing feature_flags system:
|
|
|
|
```json
|
|
{
|
|
"ai_flow_generation": {
|
|
"enabled": true,
|
|
"plans": ["pro", "team"],
|
|
"beta_override": true // Allow specific free users during beta
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Rollout Strategy
|
|
|
|
### Phase 3a: Internal Testing (Week 1)
|
|
|
|
- Backend endpoints + prompt engineering
|
|
- Test with admin account only
|
|
- Iterate on prompt quality — this is where most time goes
|
|
- Validate cost estimates against real API usage
|
|
|
|
### Phase 3b: Beta Testing (Week 2-3)
|
|
|
|
- Frontend wizard UI
|
|
- Enable for beta testers (feature flag)
|
|
- Collect feedback on suggestion quality
|
|
- Monitor ai_usage_log for actual costs
|
|
|
|
### Phase 3c: General Availability (Week 4)
|
|
|
|
- Enable for all Pro/Team plans
|
|
- Free tier gets limited access (2/month)
|
|
- Admin dashboard shows AI cost metrics
|
|
- Plan limit enforcement active
|
|
|
|
---
|
|
|
|
## Future Enhancements (Not in Initial Scope)
|
|
|
|
These are explicitly deferred to keep initial scope manageable:
|
|
|
|
1. **AI-suggested branch improvements** — "Based on 15 unresolved sessions on this branch, AI suggests adding these steps." Requires session outcome tracking (separate feature).
|
|
|
|
2. **AI step refinement** — User selects an existing step and says "make this more detailed" or "add the PowerShell commands." Single-step AI call, very cheap.
|
|
|
|
3. **Template flows from wizard patterns** — If many users create similar flows (e.g., "printer troubleshooting"), cache the assembled structure as a template. Future users get instant results with zero AI cost.
|
|
|
|
4. **Multi-language generation** — Same wizard, but AI generates steps in the user's language. Prompt change only, no architecture change.
|
|
|
|
5. **AI from session data** — After enough sessions are tracked, use anonymized session paths to improve AI suggestions. "80% of engineers check the spooler service first" → AI learns to suggest that first.
|
|
|
|
---
|
|
|
|
## Key Risks & Mitigations
|
|
|
|
| Risk | Impact | Mitigation |
|
|
|---|---|---|
|
|
| Anthropic API outage | Wizard Stage 2-3 blocked | Graceful fallback to manual entry; AI is enhancement, not requirement |
|
|
| Poor suggestion quality | Users lose trust in feature | Extensive prompt testing before launch; user can always edit/override |
|
|
| Cost overrun from heavy usage | Margin erosion | Per-account monthly limits in plan_limits; ai_usage_log for monitoring |
|
|
| Prompt injection via user input | Unexpected AI output | Sanitize user input; structured JSON output parsing; never execute AI output as code |
|
|
| Haiku model deprecated | Service interruption | Model name in env variable, not hardcoded; swap to successor model |
|
|
| Users treat AI as authoritative | Bad troubleshooting steps followed blindly | "Draft" status by default; review stage required; clear "AI-suggested" labeling |
|
|
|
|
---
|
|
|
|
## Success Metrics
|
|
|
|
| Metric | Target | How to Measure |
|
|
|---|---|---|
|
|
| Wizard completion rate | >70% of starts finish Stage 4 | Track stage progression in analytics |
|
|
| AI suggestion acceptance rate | >50% of suggestions kept | Compare suggestions shown vs. selected |
|
|
| Time to create flow (wizard vs manual) | 60%+ faster with wizard | Compare creation timestamps |
|
|
| Post-wizard editing rate | <30% of nodes edited after assembly | Compare Stage 4 structure vs. published version |
|
|
| Cost per flow | <$0.03 average | ai_usage_log aggregation |
|
|
| User satisfaction | Positive feedback from beta | Direct feedback + usage retention |
|