resolutionflow/docs/plans/archive/2026-03-23-solutions-library-design.md

Solutions Library + Smart RAG + Community Knowledge — Design Spec

Status: SPEC ONLY — not implementing yet. Build after colleague pilot (Week 3-4). Date: 2026-03-23 (updated 2026-03-23 — added Community tier) Context: GTM validation plan — copilot-first, team knowledge flywheel

---

Problem

Engineers solve the same problems repeatedly across an MSP. Today that knowledge lives in engineers' heads, scattered PSA ticket notes, or nowhere. When an engineer resolves a tricky issue through the FlowPilot copilot, that knowledge dies with the session. The next engineer who hits the same issue starts from scratch.

Solution

Solutions Library — a team knowledge base that builds itself from resolved copilot sessions and feeds back into future sessions via RAG.

Two halves:

1. Capture & Dedup — save resolutions from copilot sessions, prevent duplicates
2. Smart RAG — FlowPilot pulls from the Solutions Library during live sessions and surfaces relevant prior resolutions

How It Works

1. Resolution Capture (post-session)

When an engineer resolves a copilot session, FlowPilot auto-generates a structured resolution:

{
  "title": "Exchange Online mailbox not receiving email",
  "problem": "User reports not receiving emails in Outlook. OWA also shows no new mail.",
  "root_cause": "Mail flow rule blocking external senders due to tenant-wide transport rule misconfiguration",
  "resolution_steps": [
    "Checked MX records — correct",
    "Ran message trace in Exchange Admin Center — messages queued",
    "Found transport rule 'Block External' was enabled tenant-wide instead of per-group",
    "Disabled rule, emails delivered within 5 minutes"
  ],
  "environment_tags": ["exchange-online", "mail-flow", "transport-rules"],
  "auto_detected_category": "Microsoft 365"
}

Engineer gets prompted: "Save this as a reusable solution?"

• One-click save with auto-generated content
• Can edit title, tags, steps before saving
• Can skip (not every session produces reusable knowledge)

2. Dedup Check (on save)

Before saving, system does a similarity search (embedding cosine similarity) against existing solutions.

If similarity > 0.85 (strong match):

• Show existing solution side-by-side with new one
• Three options:
  • Merge — update existing solution with new context/steps (keeps the better version, increments usage count)
  • Keep Both — they look similar but are actually different problems
  • Discard — it's the same thing, don't save

If similarity 0.6-0.85 (partial match):

• Show as "Related solutions" but save as new by default
• Engineer can choose to merge if they recognize it's the same

If similarity < 0.6:

• Save directly, no prompt

3. RAG During Live Sessions

When an engineer starts or progresses through a copilot conversation:

1. After the first 2-3 message exchanges (enough context to understand the problem), FlowPilot searches the Solutions Library
2. Uses the conversation context as the query (not just the initial message)
3. If a solution scores above threshold, FlowPilot surfaces it naturally:

"I found a similar issue. Sarah resolved an Exchange mail flow problem 3 days ago — she found a transport rule was blocking external senders. Want me to walk you through her resolution?"

If engineer says yes:

• FlowPilot presents the resolution steps one at a time
• Engineer confirms each step worked or skips
• At the end, the solution's usage count increments

If engineer says no:

• FlowPilot continues open-ended troubleshooting
• The suggestion is noted (helps tune future relevance)

Retrieval rules:

• Only surface solutions from the same team
• Max 1 suggestion per session (don't nag)
• Don't suggest solutions the same engineer saved (they already know)
• Prefer recent solutions over old ones (tie-breaker)

4. Confidence Scoring

Each solution gets a confidence score (0-100):

Event	Score change
Saved from resolved session	+50 (base)
Another engineer uses it successfully	+15
Engineer accepts RAG suggestion	+10
Engineer rejects RAG suggestion	-5
Multiple engineers save similar (merged)	+20
Not suggested in 90 days	-10 (decay)

High-confidence solutions are suggested more aggressively. Low-confidence solutions still appear in search but aren't proactively surfaced.

5. Solutions Library UI

Replaces the current Step Library page. Card-based grid with:

Each solution card shows:

• Title (e.g., "Exchange Online mailbox not receiving email")
• Problem summary (2 lines, truncated)
• Root cause (1 line)
• Tags (environment, category)
• Saved by [engineer name] · [date]
• Used [N] times · Confidence [high/medium/low]

Page features:

• Search (full-text + semantic)
• Filter by tag, engineer, confidence, recency
• Sort by most used, most recent, highest confidence
• Manual "Add Solution" button (not just from sessions)
• Edit/delete for solutions you created (team admins can edit any)

---

Data Model

solutions table

Column	Type	Notes
id	UUID	PK
team_id	UUID	FK to teams
created_by	UUID	FK to users
title	VARCHAR(255)
problem_description	TEXT	What the user reported
root_cause	TEXT	What was actually wrong
resolution_steps	JSONB	Array of step strings
environment_tags	JSONB	Array of tag strings
category	VARCHAR(100)	Auto-detected or manual
source_session_id	UUID	FK to ai_sessions (nullable — manual entries have no source)
embedding	VECTOR(1536)	For similarity search (pgvector)
confidence_score	INTEGER	0-100, default 50
use_count	INTEGER	Times used via RAG suggestion
last_used_at	TIMESTAMPTZ
created_at	TIMESTAMPTZ
updated_at	TIMESTAMPTZ

solution_events table (for confidence scoring)

Column	Type	Notes
id	UUID	PK
solution_id	UUID	FK to solutions
event_type	VARCHAR(30)	'used', 'accepted', 'rejected', 'merged', 'decayed'
user_id	UUID	FK to users (nullable for decay events)
session_id	UUID	FK to ai_sessions (nullable)
created_at	TIMESTAMPTZ

---

Existing Infrastructure to Reuse

What exists	Where	How it maps
Knowledge Flywheel	services/knowledge_flywheel.py	Session analysis → can generate solution drafts
Knowledge Gap Service	services/knowledge_gap_service.py	Detects weak options → can flag sessions worth saving
RAG in assistant chat	services/ai_chat_service.py	Already does retrieval — extend to Solutions Library
Step Library UI	components/step-library/	Restyle as Solutions Library
pgvector	Already in Docker image (pgvector/pgvector:pg16)	Embedding storage + similarity search
FlowPilot session conclusion	components/flowpilot/	Add "Save as Solution" prompt

---

Implementation Phases (future)

Phase 1: Capture & Library

• Solutions table + migrations
• Post-session "Save as Solution" prompt in FlowPilot
• Auto-generate resolution summary from session transcript
• Solutions Library page (replaces Step Library)
• Manual add/edit/delete

Phase 2: Dedup

• Embedding generation on save (Anthropic or OpenAI embeddings)
• Similarity search on save
• Merge/Keep Both/Discard UI

Phase 3: Smart RAG

• Mid-session similarity search
• Natural language suggestion in FlowPilot conversation
• Accept/reject tracking
• Confidence scoring + decay job

Phase 4: Team Intelligence

• "Trending solutions" on dashboard
• "Your team resolved 12 Exchange issues this week" insights
• Solution suggestions in the copilot intake ("Common issues today: VPN, Exchange, AD lockouts")

---

Open Questions (to answer during pilot)

1. Do engineers actually want to save resolutions, or is it friction?
2. How similar do problems need to be before a suggestion is helpful vs. annoying?
3. Should solutions be editable by the whole team, or only the creator + admins?
4. What's the right moment to prompt "Save as Solution" — right after resolution, or in a follow-up?
5. Do engineers trust AI-generated resolution summaries, or do they want to write their own?

These questions should be answerable after 2-4 weeks of pilot usage.

---

Community Solutions Library

Overview

The Solutions Library has three tiers of knowledge:

Tier	Source	Who sees it
Private	Your team's resolutions	Your team only
Community	Anonymized resolutions from all paid users	All paid users
None	—	Free users (upgrade CTA)

This creates a network effect moat — every paid user who resolves a ticket makes the product better for all paid users. Solo MSP engineers (who don't have a team to build a knowledge base) get collective wisdom from day one.

Sharing Permissions

• Any engineer can share their own resolutions to Community (opt-in per resolution)
• Team admins can retract any community solution shared by their team
• Team admins can edit any community solution from their team (in case something slips through AI sanitization)
• Team admins can toggle community sharing on/off at the account level (enabled by default)
• When sharing is disabled, the "Share to Community" button disappears for all team members; existing community solutions from that team remain live but no new ones can be added

AI Sanitization Pipeline

When an engineer clicks "Share to Community," the resolution goes through an AI sanitization pass before publishing.

What gets stripped/replaced:

Data type	Example	Becomes
Company names	"Contoso Corp"	[Company]
Server/host names	"DC01.contoso.local"	[Domain Controller]
IP addresses	"192.168.1.50"	[Internal IP]
Usernames	"jsmith@contoso.com"	[User]
Passwords/secrets	"P@ssw0rd123"	[REDACTED]
Ticket IDs	"TKT-2847"	[Ticket]
Client names	"Acme Medical Group"	[Client]
File paths with org info	\\contoso-fs01\shares	[File Server]\shares

Sanitization pipeline:

1. LLM pass — sanitization prompt: "Replace all identifying information with descriptive placeholders while preserving technical accuracy"
2. Regex pass — safety net to catch IP patterns, email formats, UNC paths the LLM might miss
3. Preview — show the engineer the sanitized version before publishing; they can edit or cancel
4. Low-confidence highlights — if the LLM can't tell whether something is a product name or company name (e.g., "Apollo"), it highlights those sections in yellow for the engineer to confirm

The engineer always approves the sanitized preview before anything goes live.

AI Confidence Assessment

When a sanitized solution is submitted, AI runs a quality assessment before publishing.

AI Confidence Score (0-100):

Signal	Score impact
Session ended in "resolved" status	Required (gate — only resolved sessions can be shared)
Clear root cause identified	+25
Specific, actionable resolution steps	+20
Problem is generalizable (not hyper-specific to one environment)	+15
Resolution steps are reproducible by another engineer	+15
Session had multiple troubleshooting steps (not a one-liner)	+10
Similar solution already exists in community	-15
Resolution is vague ("restarted and it worked")	-20

Publishing thresholds (admin-configurable):

• Score 70+ — Auto-published, visible immediately
• Score 40-69 — Published with "Unverified" badge, needs community votes to graduate
• Score < 40 — Rejected with feedback ("This resolution is too vague to be useful. Try adding the specific steps you took.")

Community Voting

• Any paid user can upvote or downvote a community solution
• Upvote = "This helped me" / Downvote = "This didn't work for me"
• 5 net upvotes → "Unverified" badge removed, becomes "Community Verified"
• 3 net downvotes → Solution hidden, flagged for staff review
• Users can leave a comment with their vote — comments go to a ResolutionFlow staff moderation queue

Ongoing confidence adjustment:

Event	Score change
Community upvote	+5
Community downvote	-8
Used successfully via RAG (community)	+10
Rejected via RAG (community)	-3
Staff review confirms quality	+20
Not used in 120 days	-5 (decay)

Admin Controls (Super Admin Panel)

A new "Community" tab in the /admin panel with configurable thresholds:

Setting	Default	Description
Auto-publish threshold	70	Min AI confidence score to publish immediately
Review threshold	40	Min score to publish with "Unverified" badge
Upvotes to verify	5	Net upvotes needed to remove "Unverified"
Downvotes to hide	3	Net downvotes to auto-hide for review
Confidence decay days	120	Days of inactivity before score decays
Confidence decay amount	5	Points lost per decay cycle
Community sharing enabled	true	Global kill switch for all community features

These live in the existing platform_settings table (JSONB). The admin panel also shows the moderation queue — flagged solutions and user comments requiring staff review.

RAG Priority (Community Extension)

During live FlowPilot sessions, retrieval works in two passes:

1. Team pass — search team's private Solutions Library (existing behavior)
2. Community pass — search community solutions (only if team has community_sharing_enabled and user is on a paid plan)

How FlowPilot presents community solutions differently:

• Team solutions: "Your colleague Sarah resolved a similar Exchange issue 3 days ago..."
• Community solutions: "A community solution with 12 upvotes matches your issue — an engineer found that a transport rule was blocking external senders. Want me to walk you through it?"

Community RAG rules:

• Only surface community solutions with community_status = 'published' (not 'unverified' or 'hidden')
• Require AI confidence score 60+ for RAG surfacing
• Team solutions always rank above community solutions at equal similarity
• Still max 1 suggestion per session (team OR community, whichever scores higher)
• If team solution exists at similarity > 0.7, don't bother checking community (team context is better)

Solutions Library UI (Community Tab)

Add a tab switcher at the top of the existing Solutions Library page:

Tab	What it shows
My Team	Private team solutions (current behavior)
Community	All published community solutions, searchable/filterable

Community tab adds:

• Upvote/downvote buttons on each card
• "Community Verified" or "Unverified" badge
• "Used by X engineers" count instead of team member name
• Vote comment modal (optional, sends to staff queue)
• Filter by: confidence, votes, category, recency

Pricing Gate

Feature	Free	Paid
Private Solutions Library (team)	Limited (10 solutions)	Unlimited
Save from FlowPilot sessions	Yes	Yes
Community browsing	Upgrade CTA	Full access
Community RAG in sessions	No	Yes
Share to Community	No	Yes
Vote on community solutions	No	Yes
Reputation badges	No	Yes

Free users see the "Community" tab but with a blurred preview + upgrade CTA: "Join 500+ MSP engineers sharing solutions. Upgrade to access the community knowledge base."

Reputation & Incentives

Visible reputation:

• Profile badge showing contribution tier: Contributor (1+ shared) → Expert (10+) → Authority (50+)
• "Your solutions helped X engineers this month" notification (monthly digest)
• Contributor count visible on user avatar tooltip in the app

Psychological nudges (v1):

• Social proof on share prompt: After resolving a session, the save prompt says: "47 engineers solved similar Exchange issues this week using community solutions. Share yours?"
• First-share celebration: Confetti moment + "Welcome to the community!" toast
• Streak tracking: "You've shared 3 solutions this week" (consistency bias)
• Loss aversion (gentle): "You resolved 8 tickets this month but only shared 2 — 6 solutions that could help others"

Future incentives (post-v1):

• Leaderboard on the Community page: top contributors this month (opt-in visibility)
• Early access to new features for top contributors
• "Featured Contributor" badge on profile
• Potential: discount on subscription for consistent contributors

Community Data Model Extensions

New columns on solutions table:

Column	Type	Notes
visibility	VARCHAR(20)	'private', 'community' — default 'private'
community_status	VARCHAR(20)	'published', 'unverified', 'hidden', 'rejected' — null for private
ai_confidence_score	INTEGER	0-100, from sanitization/quality assessment
sanitized_content	JSONB	Anonymized version (title, problem, root_cause, steps)
source_solution_id	UUID	FK to solutions — links community copy to private original
upvote_count	INTEGER	default 0
downvote_count	INTEGER	default 0
shared_by_team_id	UUID	FK to accounts — which team shared it (for retraction)
shared_at	TIMESTAMPTZ	When it was published to community

New table: solution_votes

Column	Type	Notes
id	UUID	PK
solution_id	UUID	FK to solutions (community ones)
user_id	UUID	FK to users
vote	SMALLINT	+1 or -1
comment	TEXT	Optional — goes to staff review queue
created_at	TIMESTAMPTZ
UNIQUE	(solution_id, user_id)	One vote per user per solution

New table: community_reports

Column	Type	Notes
id	UUID	PK
solution_id	UUID	FK to solutions
reporter_id	UUID	FK to users
comment	TEXT	From the vote comment
status	VARCHAR(20)	'pending', 'reviewed', 'actioned'
reviewed_by	UUID	FK to users (staff) — nullable
created_at	TIMESTAMPTZ
reviewed_at	TIMESTAMPTZ

New columns on accounts (team settings):

Column	Type	Notes
community_sharing_enabled	BOOLEAN	default true — team admin toggle

New columns on users (reputation):

Column	Type	Notes
community_shares_count	INTEGER	default 0 — denormalized for fast badge lookup
community_helped_count	INTEGER	default 0 — how many times their solutions were used

Platform settings (admin-tunable, in existing JSONB):

{
  "community_auto_publish_threshold": 70,
  "community_review_threshold": 40,
  "community_upvotes_to_verify": 5,
  "community_downvotes_to_hide": 3,
  "community_decay_days": 120,
  "community_decay_amount": 5,
  "community_enabled": true
}

Community Implementation Phases

Extends the existing implementation phases:

Phase 5: Community Sharing

• visibility and community_status columns on solutions
• AI sanitization pipeline (LLM + regex + preview)
• "Share to Community" button on solution cards
• Team admin toggle for community sharing
• Team admin retract/edit for community solutions

Phase 6: Community Quality & Voting

• AI confidence assessment on community submissions
• solution_votes table + upvote/downvote UI
• community_reports table + staff moderation queue
• Admin-configurable thresholds in platform settings
• "Community" tab in admin panel

Phase 7: Community RAG

• Two-pass RAG (team-first, community-second)
• Community solution presentation in FlowPilot
• Paid-plan gating for community features
• Blurred preview + upgrade CTA for free users

Phase 8: Reputation & Growth

• Contribution tier badges (Contributor → Expert → Authority)
• Social proof nudges on share prompt
• First-share celebration
• "Helped X engineers" monthly digest
• Streak tracking

---

Related: FlowPilot Engagement Nudges (Future Design)

Separate from community sharing — how do we nudge engineers to resolve issues through FlowPilot in the first place? This is the top of the funnel. Community sharing nudges are pointless if engineers aren't using FlowPilot to resolve.

To design:

• What triggers bring engineers back to FlowPilot?
• How do we make the resolve action feel rewarding (dopamine loop)?
• Post-resolution moments: celebration, streak tracking, team leaderboard
• Push notifications / email digests for stale sessions
• "Your team resolved 12 tickets today" social proof

Design this as a separate spec — it deserves focused thinking.