chihlasm/resolutionflow
1
0
Fork 0
Code Issues 23 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(ai): robust response extraction + structured-output foundation (flag-gated) #188

Merged
chihlasm merged 3 commits from feat/ai-structured-outputs into main 2026-05-29 04:23:28 +00:00
Conversation 0 Commits 3 Files Changed 7 +533 -8
chihlasm commented 2026-05-29 02:45:34 +00:00
Owner
Copy Link

Summary

Hardens the Anthropic provider and lays the groundwork for schema-constrained JSON in kb_conversion. No model changes (still claude-sonnet-4-6 / claude-haiku-4-5). New behavior is gated behind AI_KB_CONVERT_STRUCTURED_OUTPUT (default False) so prod behavior is unchanged until staging-validated.

Decision recorded in .ai/DECISIONS.md: structured outputs scope to flat-array generate_json (kb_conversion) only — ai_fix and knowledge_flywheel emit recursive/nested decision trees that Anthropic's "no recursive schemas" limit excludes, so their fence-strippers stay.

What changes

backend/app/core/ai_provider.py

  • _extract_text_from_response replaces fragile response.content[0].text: skips non-text leading blocks (e.g. thinking blocks), returns first text block, logs an anthropic.stop_reason warning on max_tokens/refusal (truncation now observable), raises ValueError on no-text response.
  • generate_json gains optional schema param. Anthropic wires it to output_config.format (structured outputs); schema=None preserves exact prior call for every existing caller. Gemini accepts-and-ignores.

backend/app/core/kb_conversion_service.py

  • TROUBLESHOOTING_SCHEMA / PROCEDURAL_SCHEMA + _schema_for_target_type(), modelled as a strict superset of every field the prompts emit.
  • convert_document passes the schema only when AI_KB_CONVERT_STRUCTURED_OUTPUT is True (default False). The _try_repair_json fallback stays as belt-and-suspenders.

Cleanup

  • .gitignore — stop core.<pid> crash dumps from showing up as untracked.

Test plan

  • 14 provider tests + 7 schema tests (TDD red-green) passing locally
  • CI green
  • Staging: enable AI_KB_CONVERT_STRUCTURED_OUTPUT=true, run a live constrained-decoding smoke test, confirm token count + JSON validity
  • If staging looks good, flip flag in prod
## Summary Hardens the Anthropic provider and lays the groundwork for schema-constrained JSON in `kb_conversion`. No model changes (still claude-sonnet-4-6 / claude-haiku-4-5). New behavior is gated behind `AI_KB_CONVERT_STRUCTURED_OUTPUT` (default `False`) so prod behavior is unchanged until staging-validated. Decision recorded in [`.ai/DECISIONS.md`](../src/branch/feat/ai-structured-outputs/.ai/DECISIONS.md): structured outputs scope to flat-array `generate_json` (`kb_conversion`) only — `ai_fix` and `knowledge_flywheel` emit recursive/nested decision trees that Anthropic's "no recursive schemas" limit excludes, so their fence-strippers stay. ## What changes **`backend/app/core/ai_provider.py`** - `_extract_text_from_response` replaces fragile `response.content[0].text`: skips non-text leading blocks (e.g. thinking blocks), returns first text block, logs an `anthropic.stop_reason` warning on `max_tokens`/refusal (truncation now observable), raises `ValueError` on no-text response. - `generate_json` gains optional `schema` param. Anthropic wires it to `output_config.format` (structured outputs); `schema=None` preserves exact prior call for every existing caller. Gemini accepts-and-ignores. **`backend/app/core/kb_conversion_service.py`** - `TROUBLESHOOTING_SCHEMA` / `PROCEDURAL_SCHEMA` + `_schema_for_target_type()`, modelled as a strict superset of every field the prompts emit. - `convert_document` passes the schema only when `AI_KB_CONVERT_STRUCTURED_OUTPUT` is `True` (default `False`). The `_try_repair_json` fallback stays as belt-and-suspenders. **Cleanup** - `.gitignore` — stop `core.<pid>` crash dumps from showing up as untracked. ## Test plan - [x] 14 provider tests + 7 schema tests (TDD red-green) passing locally - [ ] CI green - [ ] Staging: enable `AI_KB_CONVERT_STRUCTURED_OUTPUT=true`, run a live constrained-decoding smoke test, confirm token count + JSON validity - [ ] If staging looks good, flip flag in prod
chihlasm added 3 commits 2026-05-29 02:45:34 +00:00
feat(ai): robust response extraction + structured-output foundation 067574ad6a 
Harden the Anthropic provider and lay the groundwork for schema-constrained
JSON, optimizing the existing claude-sonnet-4-6 / claude-haiku-4-5 usage
(no model changes).

ai_provider.py:
- _extract_text_from_response replaces fragile response.content[0].text:
  skips non-text leading blocks (e.g. thinking), returns the first text
  block, logs an anthropic.stop_reason warning on max_tokens/refusal
  (truncation now observable), and raises ValueError on a no-text response.
- generate_json gains an optional `schema` param. Anthropic wires it to
  output_config.format (structured outputs); schema=None preserves the exact
  prior call for every existing caller. Gemini accepts-and-ignores it.

kb_conversion_service.py:
- TROUBLESHOOTING_SCHEMA / PROCEDURAL_SCHEMA + _schema_for_target_type(),
  modelled as a strict superset of every field the prompts emit.
- convert_document passes the schema only when the new
  AI_KB_CONVERT_STRUCTURED_OUTPUT setting is True (default False). The
  _try_repair_json fallback stays as belt-and-suspenders.

Tests: 14 provider + 7 schema, TDD (red-green). Live constrained-decoding
smoke-test still required before enabling the flag in production.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
chore: gitignore core dumps (core.<pid>) 3fde3369c8 
Stop crashed-process core dumps (core.144926, etc.) from showing up as
untracked noise / being committed by accident.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
docs(decisions): scope structured outputs to flat-array JSON (close 3c)
Some checks failed
Mirror to GitHub / mirror (push) Failing after 6s
Details
CI / frontend (pull_request) Successful in 7m12s
Details
CI / backend (pull_request) Successful in 11m51s
Details
CI / e2e (pull_request) Successful in 10m7s
Details
02db15f118 
Record the 3c finding: Anthropic structured outputs apply only to flat-array
generate_json outputs (kb_conversion). ai_fix and knowledge_flywheel flow-gen
emit recursive/nested decision trees that the "no recursive schemas" limit
excludes; their fence-strippers stay. Documents the deferred kb-only
_try_repair_json removal pending staging validation of the
AI_KB_CONVERT_STRUCTURED_OUTPUT flag.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
chihlasm merged commit 1d92893573 into main 2026-05-29 04:23:28 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
10x-analysis
From 10x product analysis
 backend
FastAPI/Python work
 bug
Something isn't working
 collaboration
Team sharing, escalation, notifications
 database
PostgreSQL/schema changes
 documentation
Documentation updates
 duplicate
This issue or pull request already exists
 enhancement
Improvement to existing feature
 feature
New functionality
 frontend
React work
 good first issue
Good for newcomers
 help wanted
Extra attention is needed
 integration
PSA, external service integrations
 intelligence
Analytics, AI, data-driven features
 invalid
This doesn't seem right
 priority: high
Do this soon
 priority: low
Nice to have
 priority: medium
Important but not urgent
 question
Further information is requested
 quick-win
Low effort, high value
 strategic
Long-term strategic bet
 ux
User experience improvements
 wontfix
This will not be worked on
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
chihlasm
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: chihlasm/resolutionflow#188
Delete Branch "feat/ai-structured-outputs"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?