feat(pilot): Phase 5 — inline Script Generator integration
All checks were successful
Mirror to GitHub / mirror (push) Successful in 10s
All checks were successful
Mirror to GitHub / mirror (push) Successful in 10s
Wires the SuggestedFix card to an inline panel that handles both cases:
template-matched fixes open the Script Library generator with parameters
pre-filled from session context; un-matched fixes open the three-option
dialog (one_off / draft_template / build_template). The decision endpoint
records the path choice with side effects: draft_template persists a
draft_templates row via a Sonnet-driven TemplateExtractionService;
build_template returns a redirect to the Script Builder; one_off just
records the choice.
Backend:
- TemplateExtractionService: drafts a parameter schema from a concrete
rendered script. Conservative by default ("prefer fewer parameters").
Round-trip-validates that templated_body only references declared
parameters; missing-key mismatch falls back to the original script
with no params. LLM/parse failures fall back identically — the
engineer can still create a draft and refine in the post-resolve
prompt (Phase 6).
- /suggested-fixes/{fix_id}/decision side effects:
* one_off → returns rendered_script (engineer's edited version or the
fix's ai_drafted_script verbatim)
* draft_template → same + creates draft_templates row with extracted
params, returns draft_template_id
* build_template → returns redirect_path=/scripts/builder?from_session=
&fix= so the frontend can navigate to the builder pre-loaded
- 400 when a non-template fix has no ai_drafted_script (template-matched
fixes take the dedicated /scripts/generate path, not this endpoint).
- 12 tests: TemplateExtractionService parse + fallback paths, all four
decision branches, edited_script override, missing-script 400.
Frontend:
- src/components/pilot/script/{TemplateMatchPanel, NoTemplateDialog,
ParameterizationPreview}.tsx — inline panels rendered in the task
lane's bottom slot when the engineer clicks a SuggestedFix card.
- TemplateMatchPanel: loads template via /scripts/templates/{id},
pre-fills params from fix.ai_drafted_parameters with cyan "from
session" tags, generates via existing /scripts/generate (already
bumps state_version on ai_session_id from Phase 3). 404 falls back
with a clear message instead of erroring.
- NoTemplateDialog: shows the AI-drafted script with proposed parameter
values highlighted in amber via ParameterizationPreview; three option
cards with the middle (draft_template) flagged Recommended; inline
edit on the script body before deciding.
- SuggestedFix card now clickable: onActivate toggles the inline panel.
- AssistantChatPage: scriptPanelOpen state + handleScriptDecision that
navigates on build_template and toasts on the other paths. Active fix
changes auto-close the panel so engineers don't act on stale state.
- Cmd+K → "Open inline Script Generator" palette entry surfaces only on
/pilot/:id routes; fires a window event the chat page subscribes to.
No Resolve shortcut added per Section 14 decision (browser ⌘R conflict).
Verified 2026-04-22 against the dev stack:
- one_off / draft_template / build_template all return the right shape
with real Sonnet TemplateExtractionService for the draft path.
- Conservative extraction confirmed: cmdkey + Restart-Process script
yielded zero proposed parameters as intended.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -32,6 +32,8 @@ from app.schemas.session_suggested_fix import (
|
||||
SessionSuggestedFixDecisionResponse,
|
||||
SessionSuggestedFixResponse,
|
||||
)
|
||||
from app.models.draft_template import DraftTemplate
|
||||
from app.models.session_fact import SessionFact
|
||||
from app.services.escalation_package_generator import EscalationPackageGeneratorService
|
||||
from app.services.preview_cache import preview_cache
|
||||
from app.services.psa_writeback_service import (
|
||||
@@ -39,6 +41,7 @@ from app.services.psa_writeback_service import (
|
||||
PSAWritebackService,
|
||||
)
|
||||
from app.services.resolution_note_generator import ResolutionNoteGeneratorService
|
||||
from app.services.template_extraction_service import extract_parameters as _extract_template_parameters
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -105,12 +108,13 @@ async def record_decision(
|
||||
) -> SessionSuggestedFixDecisionResponse:
|
||||
"""Record the engineer's path choice on a suggested fix.
|
||||
|
||||
Phase 3 only persists the decision and (for `dismissed`) supersedes the
|
||||
row. Side effects — script generation for `one_off` / `draft_template`,
|
||||
redirect for `build_template` — land in Phase 5 alongside the inline
|
||||
Script Generator integration. The response shape is forward-compatible.
|
||||
Phase 3 recorded the choice and (for `dismissed`) superseded the fix.
|
||||
Phase 5 adds side effects: one_off / draft_template return the rendered
|
||||
script; draft_template also creates a `draft_templates` row via the
|
||||
TemplateExtractionService; build_template returns a redirect to the
|
||||
Script Builder.
|
||||
"""
|
||||
await _load_session_or_404(db, session_id)
|
||||
session_obj = await _load_session_or_404(db, session_id)
|
||||
|
||||
result = await db.execute(
|
||||
select(SessionSuggestedFix).where(
|
||||
@@ -145,15 +149,97 @@ async def record_decision(
|
||||
.where(AISession.id == session_id)
|
||||
.values(state_version=AISession.state_version + 1)
|
||||
)
|
||||
|
||||
rendered_script: str | None = None
|
||||
draft_template_id: UUID | None = None
|
||||
redirect_path: str | None = None
|
||||
|
||||
# Phase 5 side effects. All three non-dismiss paths assume the fix has
|
||||
# either a script_template_id (template match — use the dedicated
|
||||
# /scripts/generate endpoint from the frontend, not this one) or an
|
||||
# ai_drafted_script (custom script — this is the entry point).
|
||||
if body.decision in ("one_off", "draft_template", "build_template"):
|
||||
drafted = body.edited_script or fix.ai_drafted_script
|
||||
if not drafted:
|
||||
# Template-matched fixes take the regular /scripts/generate path.
|
||||
# If a fix somehow reaches here without a drafted script AND
|
||||
# without a template, that's a client-side wiring bug.
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=(
|
||||
"Suggested fix has no ai_drafted_script — use "
|
||||
"/api/v1/scripts/generate for template-matched fixes."
|
||||
),
|
||||
)
|
||||
rendered_script = drafted.strip()
|
||||
|
||||
if body.decision == "draft_template":
|
||||
# TemplateExtractionService proposes the parameterization. Runs
|
||||
# under the same transaction so a failure rolls back the decision.
|
||||
session_ctx = await _summarize_session_for_extraction(db, session_id)
|
||||
extraction = await _extract_template_parameters(
|
||||
script_body=rendered_script or "",
|
||||
session_context=session_ctx,
|
||||
ticket_context=None, # ticket context wiring lands in Phase 5 polish
|
||||
)
|
||||
|
||||
draft = DraftTemplate(
|
||||
account_id=session_obj.account_id,
|
||||
source_session_id=session_obj.id,
|
||||
source_user_id=current_user.id,
|
||||
script_body=extraction["templated_body"] or (rendered_script or ""),
|
||||
proposed_parameters={"parameters": extraction["parameters"]},
|
||||
proposed_name=fix.title[:200] if fix.title else None,
|
||||
status="pending",
|
||||
)
|
||||
db.add(draft)
|
||||
await db.flush()
|
||||
draft_template_id = draft.id
|
||||
|
||||
if body.decision == "build_template":
|
||||
# Frontend navigates to the Script Builder preloaded with the
|
||||
# drafted body. The builder wires the full parameterization flow;
|
||||
# we hand it a scratch-pad query string, not persistent state.
|
||||
redirect_path = (
|
||||
f"/scripts/builder?from_session={session_obj.id}&fix={fix.id}"
|
||||
)
|
||||
|
||||
await db.commit()
|
||||
await db.refresh(fix)
|
||||
|
||||
return SessionSuggestedFixDecisionResponse(
|
||||
id=fix.id,
|
||||
user_decision=fix.user_decision, # type: ignore[arg-type]
|
||||
rendered_script=rendered_script,
|
||||
draft_template_id=draft_template_id,
|
||||
redirect_path=redirect_path,
|
||||
)
|
||||
|
||||
|
||||
async def _summarize_session_for_extraction(
|
||||
db: AsyncSession, session_id: UUID,
|
||||
) -> str:
|
||||
"""Compact fact list for TemplateExtractionService context.
|
||||
|
||||
We don't send the full chat transcript — the extractor only needs enough
|
||||
signal to decide which values in the script are session-specific (and
|
||||
therefore worth parameterizing).
|
||||
"""
|
||||
result = await db.execute(
|
||||
select(SessionFact)
|
||||
.where(
|
||||
SessionFact.session_id == session_id,
|
||||
SessionFact.deleted_at.is_(None),
|
||||
)
|
||||
.order_by(SessionFact.created_at.asc())
|
||||
)
|
||||
facts = list(result.scalars().all())
|
||||
if not facts:
|
||||
return ""
|
||||
lines = [f"- {f.text}" for f in facts]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# ── Resolution note preview ────────────────────────────────────────────────
|
||||
|
||||
@router.post(
|
||||
|
||||
@@ -142,6 +142,10 @@ class Settings(BaseSettings):
|
||||
# FlowPilot migration Phase 4 — escalation handoff package. Parallel
|
||||
# to resolution_note: Sonnet, same cache story, no MCP.
|
||||
"escalation_package": "standard",
|
||||
# FlowPilot migration Phase 5 — extract a parameter schema from a
|
||||
# concrete rendered script so a draft_template can be proposed.
|
||||
# Creates a persistent library artifact on accept, so Sonnet.
|
||||
"template_extraction": "standard",
|
||||
}
|
||||
|
||||
def get_model_for_action(self, action_type: str) -> str:
|
||||
|
||||
@@ -33,21 +33,38 @@ class SessionSuggestedFixDecisionRequest(BaseModel):
|
||||
"""Engineer's path choice on a suggested fix.
|
||||
|
||||
Server-side side effects per Section 5.2:
|
||||
- one_off: render the script (Phase 5), no template created.
|
||||
- draft_template: render + queue a draft_templates row (Phase 5/6).
|
||||
- build_template: redirect to full template creation (Phase 5).
|
||||
- dismissed: mark the fix superseded so a fresh suggestion can take over.
|
||||
- one_off: record decision, return the rendered (AI-drafted or
|
||||
engineer-edited) script. No persistent library artifact created.
|
||||
- draft_template: same as one_off, plus TemplateExtractionService
|
||||
proposes a parameterization and a draft_templates row is created.
|
||||
- build_template: return a redirect payload pointing at the Script
|
||||
Builder page, pre-loaded with the drafted script body.
|
||||
- dismissed: mark the fix superseded.
|
||||
|
||||
For one_off / draft_template, the engineer may have edited the drafted
|
||||
script or its parameters in the dialog. The final versions are sent
|
||||
back here so we persist what will actually run.
|
||||
"""
|
||||
decision: UserDecision
|
||||
# Present for one_off / draft_template — the engineer's final version of
|
||||
# the drafted script after any inline edits. Omit to use the fix's
|
||||
# `ai_drafted_script` verbatim.
|
||||
edited_script: str | None = Field(None, min_length=1, max_length=50_000)
|
||||
# Parameter values used when rendering (informational, stored on the
|
||||
# draft_template row so a reviewer can see what the first run used).
|
||||
parameters_used: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SessionSuggestedFixDecisionResponse(BaseModel):
|
||||
"""Returned after recording a decision; richer payloads land in Phase 5."""
|
||||
"""Returned after recording a decision."""
|
||||
id: UUID
|
||||
user_decision: UserDecision
|
||||
# Set when the decision triggered side effects (e.g. a script generation).
|
||||
# Phase 3 only records the choice; this stays None until Phase 5 wires it.
|
||||
# Populated for one_off / draft_template — the script to display/run.
|
||||
rendered_script: str | None = None
|
||||
# Populated for draft_template — the ID of the draft_templates row so
|
||||
# the post-resolve TemplatizePrompt can fetch it in Phase 6.
|
||||
draft_template_id: UUID | None = None
|
||||
# Populated for build_template — where to send the engineer next.
|
||||
redirect_path: str | None = Field(
|
||||
None,
|
||||
description="Where to send the engineer next (e.g. /scripts/builder?... for build_template)",
|
||||
|
||||
201
backend/app/services/template_extraction_service.py
Normal file
201
backend/app/services/template_extraction_service.py
Normal file
@@ -0,0 +1,201 @@
|
||||
"""TemplateExtractionService — propose a parameter schema from a rendered script.
|
||||
|
||||
Phase 5 of the FlowPilot migration. Called when an engineer chooses
|
||||
"Run now, templatize after resolve" on a suggested fix with no existing
|
||||
library match. The service looks at the concrete script (with the values
|
||||
the engineer is about to run with) and session/ticket context, then
|
||||
proposes a parameterization that future engineers could use from the
|
||||
Script Library.
|
||||
|
||||
Design choices (per FLOWPILOT-MIGRATION.md Section 6.4):
|
||||
|
||||
- **Conservative by default.** Prefer fewer parameters. Environment-agnostic
|
||||
values (like a command name) should not be parameterized. The prompt calls
|
||||
that out explicitly.
|
||||
- **Round-trip check.** After the LLM proposes parameters, we validate that
|
||||
the templated body renders back to the original script when given the
|
||||
extracted parameter values. Failures log a warning and the caller falls
|
||||
back to a single-parameter "raw script" proposal.
|
||||
- **Model:** Sonnet (`template_extraction` tier). Creates a persistent
|
||||
library artifact — quality matters more than latency.
|
||||
|
||||
Output shape mirrors the Script Generator's parameter schema:
|
||||
{
|
||||
"parameters": [
|
||||
{"key": "<snake>", "label": "<human>", "type": "text|password|select|...",
|
||||
"inferred_from": "<session fact / ticket field / ai guess>"}
|
||||
],
|
||||
"templated_body": "<script with {{ key }} placeholders>",
|
||||
}
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from app.core.ai_provider import get_ai_provider
|
||||
from app.core.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
_EXTRACTION_SYSTEM_PROMPT = """\
|
||||
You are a senior MSP engineer drafting a reusable script template from a \
|
||||
concrete script that resolved one ticket. Your job is to identify the values \
|
||||
in the script that would change for a different invocation — those become \
|
||||
parameters — and replace them with {{ snake_case }} placeholders.
|
||||
|
||||
Return strict JSON with this shape:
|
||||
{
|
||||
"parameters": [
|
||||
{
|
||||
"key": "<snake_case, ASCII>",
|
||||
"label": "<Short human label, Title Case>",
|
||||
"type": "text" | "password" | "select" | "boolean" | "number" | "textarea",
|
||||
"inferred_from": "<short sentence naming the session fact or ticket \
|
||||
field this value came from; or 'ai best-guess' when neither>"
|
||||
}
|
||||
],
|
||||
"templated_body": "<the original script with each parameterized value \
|
||||
replaced by {{ key }} matching the parameters above>"
|
||||
}
|
||||
|
||||
Rules:
|
||||
- Prefer FEWER parameters. If a value looks environment-agnostic — a cmdlet \
|
||||
name, a standard path like C:\\Windows\\System32, a Microsoft-documented URL \
|
||||
— keep it hardcoded.
|
||||
- Secret-looking values (passwords, API keys, client secrets) MUST be \
|
||||
parameterized with type=password.
|
||||
- The templated_body MUST render back to the original script when the \
|
||||
parameter values from the context are substituted in. Preserve all whitespace, \
|
||||
comments, and casing.
|
||||
- If the script has no meaningful parameters (e.g. it's a single read-only \
|
||||
cmdlet like Get-Service), return parameters=[] and templated_body = original.
|
||||
- No markdown fences, no prose, only the JSON object.
|
||||
"""
|
||||
|
||||
|
||||
async def extract_parameters(
|
||||
*,
|
||||
script_body: str,
|
||||
session_context: str | None = None,
|
||||
ticket_context: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Return `{parameters, templated_body}` for the given rendered script.
|
||||
|
||||
On LLM failure or malformed output, returns a conservative fallback:
|
||||
the original body with no parameters proposed. Callers can still create
|
||||
a `draft_templates` row from this — the engineer reviews and refines
|
||||
before accepting in the post-resolve prompt (Phase 6).
|
||||
"""
|
||||
model = settings.get_model_for_action("template_extraction")
|
||||
provider = get_ai_provider(model=model)
|
||||
|
||||
input_lines = [
|
||||
"# Script to templatize",
|
||||
"```",
|
||||
script_body.strip(),
|
||||
"```",
|
||||
]
|
||||
if session_context:
|
||||
input_lines.extend(["", "# Session context (facts, symptoms)", session_context.strip()])
|
||||
if ticket_context:
|
||||
input_lines.extend(["", "# Ticket context (company, user, priority)", ticket_context.strip()])
|
||||
user_input = "\n".join(input_lines)
|
||||
|
||||
system_blocks: list[dict[str, Any]] = [
|
||||
{
|
||||
"type": "text",
|
||||
"text": _EXTRACTION_SYSTEM_PROMPT,
|
||||
"cache_control": {"type": "ephemeral"},
|
||||
# cacheable: identical across every extraction call
|
||||
},
|
||||
]
|
||||
|
||||
try:
|
||||
text, _in, _out = await provider.generate_json(
|
||||
system_prompt=system_blocks,
|
||||
messages=[{"role": "user", "content": user_input}],
|
||||
max_tokens=3000,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("TemplateExtractionService LLM call failed; returning fallback")
|
||||
return _fallback(script_body)
|
||||
|
||||
parsed = _parse_response(text)
|
||||
if parsed is None:
|
||||
return _fallback(script_body)
|
||||
|
||||
# Round-trip validation: render parsed["templated_body"] with the
|
||||
# `inferred_from` values and confirm it matches the original. We don't
|
||||
# have the engineer's values yet here (those come at runtime), but we
|
||||
# can at least check that every {{ key }} in templated_body maps to a
|
||||
# declared parameter. A mismatch means the LLM referenced an undeclared
|
||||
# placeholder — conservative fallback.
|
||||
declared_keys = {p.get("key") for p in parsed["parameters"] if isinstance(p, dict)}
|
||||
referenced_keys = set(re.findall(r"\{\{\s*(\w+)\s*\}\}", parsed["templated_body"]))
|
||||
missing = referenced_keys - declared_keys
|
||||
if missing:
|
||||
logger.warning(
|
||||
"TemplateExtractionService: templated_body references undeclared "
|
||||
"keys %s; using fallback",
|
||||
sorted(missing),
|
||||
)
|
||||
return _fallback(script_body)
|
||||
|
||||
return parsed
|
||||
|
||||
|
||||
def _parse_response(raw: str) -> dict[str, Any] | None:
|
||||
"""Tolerant parse. Returns None on any structural problem."""
|
||||
cleaned = raw.strip()
|
||||
if cleaned.startswith("```"):
|
||||
cleaned = re.sub(r"^```(?:json)?\s*", "", cleaned)
|
||||
cleaned = re.sub(r"\s*```$", "", cleaned)
|
||||
try:
|
||||
data = json.loads(cleaned)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
logger.warning("TemplateExtractionService returned non-JSON: %r", raw[:200])
|
||||
return None
|
||||
|
||||
if not isinstance(data, dict):
|
||||
return None
|
||||
params = data.get("parameters")
|
||||
body = data.get("templated_body")
|
||||
if not isinstance(params, list) or not isinstance(body, str):
|
||||
logger.warning("TemplateExtractionService missing parameters or templated_body")
|
||||
return None
|
||||
|
||||
# Validate each parameter shape. Drop malformed entries rather than
|
||||
# failing the whole response — the engineer will review before accept.
|
||||
valid_params: list[dict[str, Any]] = []
|
||||
allowed_types = {"text", "password", "select", "boolean", "number", "textarea", "multi_text"}
|
||||
for p in params:
|
||||
if not isinstance(p, dict):
|
||||
continue
|
||||
key = p.get("key")
|
||||
if not isinstance(key, str) or not re.match(r"^[a-z_][a-z0-9_]*$", key):
|
||||
continue
|
||||
ptype = p.get("type", "text")
|
||||
if ptype not in allowed_types:
|
||||
ptype = "text"
|
||||
valid_params.append({
|
||||
"key": key,
|
||||
"label": p.get("label") or key.replace("_", " ").title(),
|
||||
"type": ptype,
|
||||
"inferred_from": p.get("inferred_from") or "ai best-guess",
|
||||
})
|
||||
|
||||
return {"parameters": valid_params, "templated_body": body}
|
||||
|
||||
|
||||
def _fallback(script_body: str) -> dict[str, Any]:
|
||||
"""Conservative no-op result: zero parameters, body unchanged.
|
||||
|
||||
Used when the LLM call fails or returns unusable output. The engineer
|
||||
can still save this as a draft and refine in the post-resolve prompt —
|
||||
it just won't propose a parameterization for them.
|
||||
"""
|
||||
return {"parameters": [], "templated_body": script_body}
|
||||
277
backend/tests/test_phase5_inline_script.py
Normal file
277
backend/tests/test_phase5_inline_script.py
Normal file
@@ -0,0 +1,277 @@
|
||||
"""API + service tests for Phase 5 inline Script Generator integration.
|
||||
|
||||
Covers:
|
||||
- TemplateExtractionService: well-formed, fallback on bad output, missing-key fallback.
|
||||
- /suggested-fixes/{fix_id}/decision side effects:
|
||||
* one_off returns rendered_script, no draft_templates row.
|
||||
* draft_template returns rendered_script + draft_template_id, draft persisted.
|
||||
* build_template returns redirect_path.
|
||||
* dismissed (Phase 3) still works.
|
||||
- 400 when ai_drafted_script is missing for a non-template fix.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
import pytest
|
||||
from httpx import AsyncClient
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.api.endpoints.session_suggested_fixes import _clear_preview_cache_for_tests
|
||||
from app.models.ai_session import AISession
|
||||
from app.models.draft_template import DraftTemplate
|
||||
from app.models.session_suggested_fix import SessionSuggestedFix
|
||||
from app.services.template_extraction_service import (
|
||||
_fallback,
|
||||
_parse_response,
|
||||
extract_parameters,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_preview_cache():
|
||||
_clear_preview_cache_for_tests()
|
||||
yield
|
||||
_clear_preview_cache_for_tests()
|
||||
|
||||
|
||||
async def _make_session_with_fix(
|
||||
test_db, user, *, with_template_id: bool = False, with_drafted_script: bool = True,
|
||||
) -> tuple[AISession, SessionSuggestedFix]:
|
||||
session = AISession(
|
||||
user_id=user["user_data"]["id"],
|
||||
account_id=user["user_data"]["account_id"],
|
||||
session_type="chat",
|
||||
intake_type="free_text",
|
||||
intake_content={"text": "phase 5 test"},
|
||||
status="active",
|
||||
confidence_tier="discovery",
|
||||
conversation_messages=[],
|
||||
)
|
||||
test_db.add(session)
|
||||
await test_db.flush()
|
||||
|
||||
fix = SessionSuggestedFix(
|
||||
session_id=session.id,
|
||||
account_id=session.account_id,
|
||||
title="Reset cached creds",
|
||||
description="Clearing the cached credential...",
|
||||
confidence_pct=85,
|
||||
ai_drafted_script=(
|
||||
'cmdkey /delete:"outlook.office365.com"\n'
|
||||
'Restart-Process -Name OUTLOOK'
|
||||
) if with_drafted_script else None,
|
||||
ai_drafted_parameters={"target_user": "jsmith"} if with_drafted_script else None,
|
||||
)
|
||||
test_db.add(fix)
|
||||
await test_db.commit()
|
||||
await test_db.refresh(session)
|
||||
await test_db.refresh(fix)
|
||||
return session, fix
|
||||
|
||||
|
||||
# ── TemplateExtractionService: parse + fallback ───────────────────────────
|
||||
|
||||
class TestParseResponse:
|
||||
def test_well_formed(self):
|
||||
raw = (
|
||||
'{"parameters": [{"key":"host","label":"Host","type":"text",'
|
||||
'"inferred_from":"session fact"}],'
|
||||
'"templated_body":"Get-Service -ComputerName {{ host }}"}'
|
||||
)
|
||||
result = _parse_response(raw)
|
||||
assert result is not None
|
||||
assert len(result["parameters"]) == 1
|
||||
assert result["parameters"][0]["key"] == "host"
|
||||
assert result["templated_body"].endswith("{{ host }}")
|
||||
|
||||
def test_strips_fences(self):
|
||||
raw = '```json\n{"parameters": [], "templated_body": "x"}\n```'
|
||||
result = _parse_response(raw)
|
||||
assert result is not None and result["parameters"] == []
|
||||
|
||||
def test_invalid_key_dropped(self):
|
||||
# Capital letters and dashes in key names violate snake_case — drop.
|
||||
raw = (
|
||||
'{"parameters":[{"key":"BadKey-Name","type":"text"}],'
|
||||
'"templated_body":"x"}'
|
||||
)
|
||||
result = _parse_response(raw)
|
||||
assert result is not None and result["parameters"] == []
|
||||
|
||||
def test_unknown_type_falls_back_to_text(self):
|
||||
raw = (
|
||||
'{"parameters":[{"key":"x","type":"weird"}],"templated_body":"x"}'
|
||||
)
|
||||
result = _parse_response(raw)
|
||||
assert result is not None and result["parameters"][0]["type"] == "text"
|
||||
|
||||
def test_malformed_json_returns_none(self):
|
||||
assert _parse_response("not json") is None
|
||||
|
||||
def test_non_dict_returns_none(self):
|
||||
assert _parse_response('["a","b"]') is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_extract_parameters_round_trip_failure_uses_fallback():
|
||||
"""Templated_body referencing an undeclared placeholder triggers fallback."""
|
||||
fake_provider = AsyncMock()
|
||||
fake_provider.generate_json = AsyncMock(return_value=(
|
||||
# Declares parameter `host` but the body references `port` too.
|
||||
'{"parameters":[{"key":"host","label":"Host","type":"text"}],'
|
||||
'"templated_body":"Get-Service -ComputerName {{ host }} -Port {{ port }}"}',
|
||||
100, 50,
|
||||
))
|
||||
with patch(
|
||||
"app.services.template_extraction_service.get_ai_provider",
|
||||
return_value=fake_provider,
|
||||
):
|
||||
result = await extract_parameters(
|
||||
script_body="Get-Service -ComputerName srv01 -Port 8080",
|
||||
)
|
||||
fb = _fallback("Get-Service -ComputerName srv01 -Port 8080")
|
||||
assert result == fb
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_extract_parameters_llm_exception_uses_fallback():
|
||||
fake_provider = AsyncMock()
|
||||
fake_provider.generate_json = AsyncMock(side_effect=RuntimeError("boom"))
|
||||
with patch(
|
||||
"app.services.template_extraction_service.get_ai_provider",
|
||||
return_value=fake_provider,
|
||||
):
|
||||
result = await extract_parameters(script_body="echo hello")
|
||||
assert result == _fallback("echo hello")
|
||||
|
||||
|
||||
# ── Decision endpoint: one_off ─────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_one_off_returns_rendered_script_no_draft(
|
||||
client: AsyncClient, test_user, auth_headers, test_db
|
||||
):
|
||||
session, fix = await _make_session_with_fix(test_db, test_user)
|
||||
r = await client.post(
|
||||
f"/api/v1/ai-sessions/{session.id}/suggested-fixes/{fix.id}/decision",
|
||||
headers=auth_headers,
|
||||
json={"decision": "one_off"},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["user_decision"] == "one_off"
|
||||
assert body["rendered_script"] is not None
|
||||
assert "cmdkey" in body["rendered_script"]
|
||||
assert body["draft_template_id"] is None
|
||||
assert body["redirect_path"] is None
|
||||
|
||||
# No draft_templates row should have been created.
|
||||
rows = (
|
||||
await test_db.execute(select(DraftTemplate).where(DraftTemplate.source_session_id == session.id))
|
||||
).scalars().all()
|
||||
assert list(rows) == []
|
||||
|
||||
|
||||
# ── Decision endpoint: draft_template ─────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_draft_template_creates_draft_with_extracted_params(
|
||||
client: AsyncClient, test_user, auth_headers, test_db
|
||||
):
|
||||
session, fix = await _make_session_with_fix(test_db, test_user)
|
||||
|
||||
fake_provider = AsyncMock()
|
||||
fake_provider.generate_json = AsyncMock(return_value=(
|
||||
'{"parameters":[{"key":"target_user","label":"Target User","type":"text",'
|
||||
'"inferred_from":"session fact"}],'
|
||||
'"templated_body":"cmdkey /delete:\\"outlook.office365.com\\"\\n'
|
||||
'Restart-Process -Name OUTLOOK"}',
|
||||
80, 60,
|
||||
))
|
||||
|
||||
with patch(
|
||||
"app.services.template_extraction_service.get_ai_provider",
|
||||
return_value=fake_provider,
|
||||
):
|
||||
r = await client.post(
|
||||
f"/api/v1/ai-sessions/{session.id}/suggested-fixes/{fix.id}/decision",
|
||||
headers=auth_headers,
|
||||
json={"decision": "draft_template"},
|
||||
)
|
||||
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["user_decision"] == "draft_template"
|
||||
assert body["rendered_script"] is not None
|
||||
assert body["draft_template_id"] is not None
|
||||
assert body["redirect_path"] is None
|
||||
|
||||
drafts = (
|
||||
await test_db.execute(select(DraftTemplate).where(DraftTemplate.source_session_id == session.id))
|
||||
).scalars().all()
|
||||
drafts = list(drafts)
|
||||
assert len(drafts) == 1
|
||||
draft = drafts[0]
|
||||
assert draft.status == "pending"
|
||||
assert draft.proposed_name == fix.title
|
||||
proposed = draft.proposed_parameters.get("parameters") if isinstance(draft.proposed_parameters, dict) else None
|
||||
assert isinstance(proposed, list) and len(proposed) == 1
|
||||
assert proposed[0]["key"] == "target_user"
|
||||
|
||||
|
||||
# ── Decision endpoint: build_template ─────────────────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_build_template_returns_redirect_path(
|
||||
client: AsyncClient, test_user, auth_headers, test_db
|
||||
):
|
||||
session, fix = await _make_session_with_fix(test_db, test_user)
|
||||
r = await client.post(
|
||||
f"/api/v1/ai-sessions/{session.id}/suggested-fixes/{fix.id}/decision",
|
||||
headers=auth_headers,
|
||||
json={"decision": "build_template"},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["redirect_path"] is not None
|
||||
assert str(session.id) in body["redirect_path"]
|
||||
assert str(fix.id) in body["redirect_path"]
|
||||
|
||||
|
||||
# ── Decision endpoint: 400 when no drafted script ─────────────────────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_one_off_without_drafted_script_returns_400(
|
||||
client: AsyncClient, test_user, auth_headers, test_db
|
||||
):
|
||||
"""A template-matched fix takes the dedicated /scripts/generate path; trying
|
||||
to one_off it via this endpoint without an ai_drafted_script must surface
|
||||
a clear client-error, not silently render nothing."""
|
||||
session, fix = await _make_session_with_fix(test_db, test_user, with_drafted_script=False)
|
||||
r = await client.post(
|
||||
f"/api/v1/ai-sessions/{session.id}/suggested-fixes/{fix.id}/decision",
|
||||
headers=auth_headers,
|
||||
json={"decision": "one_off"},
|
||||
)
|
||||
assert r.status_code == 400
|
||||
assert "ai_drafted_script" in r.json()["detail"]
|
||||
|
||||
|
||||
# ── Decision endpoint: edited script overrides ai_drafted_script ──────────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_edited_script_overrides_ai_drafted(
|
||||
client: AsyncClient, test_user, auth_headers, test_db
|
||||
):
|
||||
session, fix = await _make_session_with_fix(test_db, test_user)
|
||||
r = await client.post(
|
||||
f"/api/v1/ai-sessions/{session.id}/suggested-fixes/{fix.id}/decision",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"decision": "one_off",
|
||||
"edited_script": "Get-Service -Name Dnscache",
|
||||
},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
assert r.json()["rendered_script"] == "Get-Service -Name Dnscache"
|
||||
Reference in New Issue
Block a user