feat(ai): robust response extraction + structured-output foundation

Harden the Anthropic provider and lay the groundwork for schema-constrained
JSON, optimizing the existing claude-sonnet-4-6 / claude-haiku-4-5 usage
(no model changes).

ai_provider.py:
- _extract_text_from_response replaces fragile response.content[0].text:
  skips non-text leading blocks (e.g. thinking), returns the first text
  block, logs an anthropic.stop_reason warning on max_tokens/refusal
  (truncation now observable), and raises ValueError on a no-text response.
- generate_json gains an optional `schema` param. Anthropic wires it to
  output_config.format (structured outputs); schema=None preserves the exact
  prior call for every existing caller. Gemini accepts-and-ignores it.

kb_conversion_service.py:
- TROUBLESHOOTING_SCHEMA / PROCEDURAL_SCHEMA + _schema_for_target_type(),
  modelled as a strict superset of every field the prompts emit.
- convert_document passes the schema only when the new
  AI_KB_CONVERT_STRUCTURED_OUTPUT setting is True (default False). The
  _try_repair_json fallback stays as belt-and-suspenders.

Tests: 14 provider + 7 schema, TDD (red-green). Live constrained-decoding
smoke-test still required before enabling the flag in production.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

backend/app/core/config.py

@@ -155,6 +155,12 @@ class Settings(BaseSettings):
     AI_CONVERSATION_TTL_HOURS: int = 24
     AI_MAX_CALLS_PER_FLOW: int = 10
     AI_REQUEST_TIMEOUT_SECONDS: int = 120
+    # When True, KB conversion constrains the Anthropic response with a JSON
+    # schema (structured outputs) instead of relying on prompt-only JSON +
+    # downstream fence-stripping / brace-repair. Default OFF: enable in staging
+    # and smoke-test constrained decoding against the live model before turning
+    # it on in production. Only affects the Anthropic backend.
+    AI_KB_CONVERT_STRUCTURED_OUTPUT: bool = False
     # AI Provider selection
     AI_PROVIDER: str = "anthropic"  # "gemini" or "anthropic"
     GOOGLE_AI_API_KEY: Optional[str] = None