fix: strengthen prompt to prevent next_node_id mismatches, keep strict validation

Rather than lowering the validation bar, improve the system prompt:
- Rule 6 now explicitly states next_node_id must match a direct child's id
- Added rule 10: build tree bottom-up to avoid forward-reference errors
- Corrective prompt now calls out the ID mismatch constraint specifically

Reverts the strict=False fallback — flows must be correct before saving.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
chihlasm
2026-02-21 13:48:50 -05:00
parent 21f542694e
commit c562a82f5d
2 changed files with 7 additions and 10 deletions

View File

@@ -77,10 +77,11 @@ Rules:
3. Every branch path MUST end in a solution node — no dead ends 3. Every branch path MUST end in a solution node — no dead ends
4. Include realistic MSP commands (PowerShell preferred for Windows) 4. Include realistic MSP commands (PowerShell preferred for Windows)
5. Use unique node IDs prefixed with the branch context (e.g., "dns-check-service") 5. Use unique node IDs prefixed with the branch context (e.g., "dns-check-service")
6. Every option's next_node_id must match an existing child node's id 6. CRITICAL — next_node_id must exactly match the "id" of a direct child in the "children" array of that same node. Never reference an ID that does not appear as a child of the current node.
7. All option labels must be meaningful and specific 7. All option labels must be meaningful and specific
8. Decision nodes must have at least 2 options 8. Decision nodes must have at least 2 options
9. Return a single root node with its children nested inside 9. Return a single root node with its children nested inside
10. Build the tree bottom-up in your head: create leaf nodes first, then reference their IDs in parent options
Few-shot example (abbreviated): Few-shot example (abbreviated):
{"id": "dns-root", "type": "decision", "question": "Can the client resolve any DNS names?", "help_text": "Run: nslookup google.com", "options": [{"id": "dns-opt-none", "label": "No DNS resolution at all", "next_node_id": "dns-check-service"}, {"id": "dns-opt-partial", "label": "Some names resolve, others don't", "next_node_id": "dns-check-specific"}], "children": [{"id": "dns-check-service", "type": "action", "title": "Check DNS Service", "description": "Verify the DNS Client service is running", "commands": ["Get-Service -Name Dnscache"], "expected_outcome": "Service should be Running", "children": [{"id": "dns-resolved", "type": "solution", "title": "DNS Service Restored", "description": "DNS client service was stopped", "resolution_steps": ["Restart DNS Client service", "Flush DNS cache: ipconfig /flushdns", "Test resolution"]}]}, {"id": "dns-check-specific", "type": "solution", "title": "Selective DNS Failure", "description": "Specific records missing or stale", "resolution_steps": ["Check DNS server configuration", "Verify zone records", "Clear DNS cache"]}]}""" {"id": "dns-root", "type": "decision", "question": "Can the client resolve any DNS names?", "help_text": "Run: nslookup google.com", "options": [{"id": "dns-opt-none", "label": "No DNS resolution at all", "next_node_id": "dns-check-service"}, {"id": "dns-opt-partial", "label": "Some names resolve, others don't", "next_node_id": "dns-check-specific"}], "children": [{"id": "dns-check-service", "type": "action", "title": "Check DNS Service", "description": "Verify the DNS Client service is running", "commands": ["Get-Service -Name Dnscache"], "expected_outcome": "Service should be Running", "children": [{"id": "dns-resolved", "type": "solution", "title": "DNS Service Restored", "description": "DNS client service was stopped", "resolution_steps": ["Restart DNS Client service", "Flush DNS cache: ipconfig /flushdns", "Test resolution"]}]}, {"id": "dns-check-specific", "type": "solution", "title": "Selective DNS Failure", "description": "Specific records missing or stale", "resolution_steps": ["Check DNS server configuration", "Verify zone records", "Clear DNS cache"]}]}"""
@@ -91,6 +92,8 @@ CORRECTIVE_PROMPT_TEMPLATE = """Your previous JSON was invalid for ResolutionFlo
Validation errors: Validation errors:
{error_list} {error_list}
IMPORTANT: If any error mentions a next_node_id referencing a non-existent child, you must ensure every option's next_node_id exactly matches the "id" field of one of the node's direct children. The child node must exist in the "children" array of the same parent node.
Return a corrected full JSON object only. No markdown, no prose, no code fences. Return a corrected full JSON object only. No markdown, no prose, no code fences.
Fix ALL listed errors while maintaining the same troubleshooting/procedural logic.""" Fix ALL listed errors while maintaining the same troubleshooting/procedural logic."""
@@ -234,14 +237,8 @@ async def generate_branch_detail(
continue continue
raise ValueError(f"AI returned invalid JSON after retry: {e}") raise ValueError(f"AI returned invalid JSON after retry: {e}")
# On the final attempt, use strict=False to accept cross-reference errors = validate_generated_tree(branch_tree)
# mismatches (next_node_id pointing to wrong child) — these are
# minor structural issues the user can fix in the editor.
is_final_attempt = attempt == 2
errors = validate_generated_tree(branch_tree, strict=not is_final_attempt)
if not errors: if not errors:
if is_final_attempt:
logger.warning("branch_detail accepted on final attempt (lenient validation): branch=%s", branch_name)
cost = _estimate_cost(total_input, total_output) cost = _estimate_cost(total_input, total_output)
return branch_tree, total_input, total_output, cost return branch_tree, total_input, total_output, cost

View File

@@ -24,7 +24,7 @@ class TreeValidationError(Exception):
super().__init__(f"Tree validation failed: {'; '.join(errors)}") super().__init__(f"Tree validation failed: {'; '.join(errors)}")
def validate_generated_tree(tree: dict[str, Any], strict: bool = True) -> list[str]: def validate_generated_tree(tree: dict[str, Any]) -> list[str]:
"""Validate an AI-generated tree structure. """Validate an AI-generated tree structure.
Returns a list of error strings. Empty list means valid. Returns a list of error strings. Empty list means valid.
@@ -106,7 +106,7 @@ def validate_generated_tree(tree: dict[str, Any], strict: bool = True) -> list[s
next_id = opt.get("next_node_id") next_id = opt.get("next_node_id")
if next_id: if next_id:
all_referenced_ids.add(next_id) all_referenced_ids.add(next_id)
if strict and child_ids and next_id not in child_ids: if child_ids and next_id not in child_ids:
errors.append( errors.append(
f"Option '{opt.get('label', '?')}' in node '{node_id}' " f"Option '{opt.get('label', '?')}' in node '{node_id}' "
f"references non-existent child '{next_id}'" f"references non-existent child '{next_id}'"