* docs: add tenant data isolation design spec Complete architecture plan for multi-tenant data isolation across all layers (PostgreSQL RLS, application-layer filtering, schema migration, testing strategy, and phased rollout checklist). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs: add background job isolation policy to tenant isolation spec Documents policy for all 5 existing background jobs: - Knowledge Flywheel and PSA Retry flagged for account_id threading - Chat Retention already follows correct pattern (model for others) - Maintenance Schedule Firing needs account_id in queries + Session creation - AI Conversation Expiry approved as cross-tenant with justification Adds approved cross-tenant query registry and Phase 2 checklist items. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs: add tenant isolation Phase 0 implementation plan 8 tasks covering: CRITICAL copilot hotfix, tenant_filter() helper, get_tenant_context dependency, analytics/category/AI session gap fixes, full UUID endpoint audit, TargetList dead code audit, teams orphan check, and CI grep check for missing tenant filters. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add tenant_filter() helper and get_tenant_context dependency tenant_filter(model, account_id) is the canonical app-layer tenant scoping expression. Every query on a tenant table must use it. build_tree_access_filter and build_step_visibility_filter updated to call tenant_filter() internally for the account_id match. get_tenant_context is a FastAPI dependency that returns account_id or raises 403 if the user has no account — prevents raw access to current_user.account_id and centralises the null check. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: scope analytics/flows/{tree_id} to requesting account Any authenticated user could read flow analytics (session counts, completion rates, CSAT) for any tree UUID. Now returns 404 if the tree doesn't belong to the requesting account. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: scope category tree_count to requesting account tree_count on GET /categories/{id} was including trees from all accounts, leaking cross-tenant row counts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: restrict AI session search to current user only Search endpoint used OR(user_id, account_id), exposing other users' problem_summary and problem_domain within the same account. Sessions are user-scoped only — cross-user access requires explicit escalation or sharing. List and search endpoints now behave consistently. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: add ownership check and 404 responses to ai-sessions endpoints Cross-tenant isolation audit found: - retry-psa-push had NO ownership check (CRITICAL) — any user could retry any session's PSA push - save_task_lane used db.get() without ownership filter, returned 403 revealing existence - get_session returned 403 instead of 404 for unauthorized access - stream_documentation returned 403 instead of 404 All now use query-level user_id filtering and return 404 to avoid revealing existence. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: return 404 instead of 403 for cross-tenant session access All session endpoints (get, update, complete, scratchpad, variables, export, ticket-link) now return 404 instead of 403 when a user tries to access another user's session. This prevents confirming existence of resources across tenant boundaries. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: return 404 instead of 403 for cross-tenant tree access get_tree and update_tree now return 404 when a user cannot access a tree (private tree from another account). Prevents confirming resource existence across tenant boundaries. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: return 404 instead of 403 for cross-tenant step access get_step_or_404 now returns 404 when can_view_step or can_edit_step fails, preventing confirmation of step existence across tenant boundaries. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: return 404 instead of 403 for cross-tenant upload access get_upload_url and delete_upload now return 404 when the upload belongs to a different account/user, preventing resource existence confirmation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: return 404 instead of 403 for cross-tenant share access revoke_share and create_share now return 404 when the caller is not the owner, preventing resource existence confirmation across users. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: return 404 instead of 403 for cross-team tree access in maintenance schedules _get_tree_or_403 now returns 404 when the user's team does not match, preventing confirmation of tree existence across teams. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: return 404 instead of 403 for cross-account tag access get_tag now returns 404 for account-specific tags that belong to another account, preventing resource existence confirmation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: return 404 instead of 403 for cross-account step category access get_step_category now returns 404 for account-specific categories that belong to another account, preventing resource existence confirmation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: add cross-tenant isolation tests for Task 6 UUID audit Tests cover: - Tree GET/PUT returns 404 for cross-account access - Session GET returns 404 for cross-user access - AI session GET returns 404 for cross-user access - AI session retry-psa-push requires ownership - Upload URL returns 404 for cross-account access - Share revoke returns 404 for cross-user access Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: return 404 (not 403) for get_documentation cross-user access; add missing Task 6 tests get_documentation was revealing session existence via 403. Added pre-check query filtering by session_id AND user_id before calling the engine. Also add cross-tenant isolation tests for steps, tags, step_categories, and maintenance_schedules endpoints fixed in Task 6 (TDD was skipped). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address Task 6 quality review — rename helper, restore 403 for intra-account, add docs test - Rename _get_tree_or_403 → _get_tree_or_404 in maintenance_schedules.py (function now raises 404, old name was misleading) - Restore HTTP 403 for intra-account permission failures in update_tree: same-account users who can see a tree but can't edit it got 404 (wrong); only cross-account lookups should return 404 to avoid confirming existence - Apply same 403/404 distinction to update_tree_visibility - Add test: get_documentation must return 404 for cross-user session access - Add comment documenting owner-only design for documentation endpoints Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: Task 7+8 — TargetList audit, CI tenant-filter grep check Task 7: TargetList dead code audit - Found active code references in 12+ files across backend and frontend (full CRUD API + frontend page + MaintenanceScheduleSection + BatchLaunchModal) - Decision: migrate to account_id in Phase 1 (cannot drop) - DB row count not available from code-server — must verify from VPS SSH before Phase 1 migration - Teams orphan check query documented; must run from VPS SSH before Phase 1 - Results documented in spec Section 9 Task 8: CI tenant-filter enforcement check (warn mode) - Create backend/scripts/check_tenant_filters.py Scans endpoint and service files for select() on tenant tables without tenant_filter/account_id/user_id in surrounding context. Currently reports 109 warnings (Phase 1 backlog). Exits 0 (warn mode). - Add Check tenant filter enforcement step to backend CI job Add --fail flag after Phase 1 backlog clears to make it blocking. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs: record Phase 0 audit results — 0 orphaned teams, 0 target_list rows Both checks confirmed 2026-04-09 from production DB. Phase 1 migration is safe to proceed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
324 lines
12 KiB
Python
324 lines
12 KiB
Python
"""File upload endpoints — S3-compatible object storage.
|
|
|
|
POST /uploads — Upload a file (multipart)
|
|
GET /uploads/{id}/url — Get presigned download URL
|
|
GET /uploads — List uploads for a session
|
|
DELETE /uploads/{id} — Delete an upload
|
|
"""
|
|
import logging
|
|
from typing import Annotated, Optional
|
|
from uuid import UUID
|
|
|
|
from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, UploadFile, status
|
|
from sqlalchemy import select, func
|
|
from sqlalchemy.ext.asyncio import AsyncSession
|
|
|
|
from app.api.deps import get_current_active_user, get_db
|
|
from app.core.config import settings
|
|
from app.core.rate_limit import limiter
|
|
from app.models.file_upload import FileUpload
|
|
from app.models.user import User
|
|
from app.schemas.upload import FileUploadResponse
|
|
from app.services import storage_service
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
router = APIRouter(prefix="/uploads", tags=["uploads"])
|
|
|
|
|
|
def _check_storage_configured() -> None:
|
|
"""Raise 503 if object storage is not configured."""
|
|
if not settings.STORAGE_ENDPOINT:
|
|
raise HTTPException(
|
|
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
|
detail="File storage is not configured",
|
|
)
|
|
|
|
|
|
async def _store_document_content(upload, text_content: str, doc_type: str) -> None:
|
|
"""Store extracted document text and optionally generate an AI summary."""
|
|
from app.services.assistant_chat_service import _call_ai
|
|
|
|
if text_content:
|
|
upload.extracted_content = text_content[:10000]
|
|
|
|
if len(text_content) > 2000:
|
|
summary, _, _ = await _call_ai(
|
|
system_base="You are a technical document analyst for IT troubleshooting.",
|
|
rag_context="",
|
|
history=[],
|
|
new_message=f"Summarize this {doc_type} content in 2-3 sentences:\n\n{text_content[:5000]}",
|
|
max_tokens=200,
|
|
)
|
|
upload.content_summary = summary
|
|
upload.ai_description = summary
|
|
else:
|
|
upload.ai_description = f"{doc_type}: {upload.filename}"
|
|
else:
|
|
upload.ai_description = f"{doc_type} (no extractable text): {upload.filename}"
|
|
|
|
|
|
async def _generate_ai_description(upload_id: UUID, file_data: bytes, content_type: str) -> None:
|
|
"""Background task: generate AI description for uploaded file."""
|
|
try:
|
|
from app.core.database import async_session_maker
|
|
from app.services.assistant_chat_service import _call_ai
|
|
import base64
|
|
|
|
async with async_session_maker() as db:
|
|
result = await db.execute(
|
|
select(FileUpload).where(FileUpload.id == upload_id)
|
|
)
|
|
upload = result.scalar_one_or_none()
|
|
if not upload:
|
|
return
|
|
|
|
if content_type.startswith("image/"):
|
|
b64_data = base64.b64encode(file_data).decode("utf-8")
|
|
description, _, _ = await _call_ai(
|
|
system_base="You are a technical image analyst for IT troubleshooting.",
|
|
rag_context="",
|
|
history=[],
|
|
new_message="Describe this image in one sentence for a troubleshooting context log.",
|
|
images=[{"media_type": content_type, "data": b64_data}],
|
|
max_tokens=100,
|
|
)
|
|
upload.ai_description = description
|
|
elif content_type == "application/pdf":
|
|
try:
|
|
from pypdf import PdfReader
|
|
import io as _io
|
|
|
|
reader = PdfReader(_io.BytesIO(file_data))
|
|
pages_text = []
|
|
for page in reader.pages:
|
|
page_text = page.extract_text()
|
|
if page_text:
|
|
pages_text.append(page_text)
|
|
text_content = "\n\n".join(pages_text)
|
|
except Exception:
|
|
logger.warning("PDF text extraction failed for upload %s", upload_id)
|
|
text_content = ""
|
|
|
|
await _store_document_content(upload, text_content, "PDF")
|
|
|
|
elif content_type == "application/vnd.openxmlformats-officedocument.wordprocessingml.document":
|
|
try:
|
|
from docx import Document as DocxDocument
|
|
import io as _io
|
|
|
|
doc = DocxDocument(_io.BytesIO(file_data))
|
|
text_content = "\n\n".join(
|
|
p.text for p in doc.paragraphs if p.text.strip()
|
|
)
|
|
except Exception:
|
|
logger.warning("DOCX text extraction failed for upload %s", upload_id)
|
|
text_content = ""
|
|
|
|
await _store_document_content(upload, text_content, "Word document")
|
|
|
|
elif content_type.startswith("text/") or content_type in (
|
|
"application/json", "application/xml", "application/yaml",
|
|
):
|
|
try:
|
|
text_content = file_data.decode("utf-8")
|
|
except UnicodeDecodeError:
|
|
text_content = file_data.decode("latin-1")
|
|
|
|
upload.extracted_content = text_content[:10000]
|
|
|
|
if len(text_content) > 2000:
|
|
summary, _, _ = await _call_ai(
|
|
system_base="You are a technical log/config analyst.",
|
|
rag_context="",
|
|
history=[],
|
|
new_message=f"Summarize this file content in 2-3 sentences:\n\n{text_content[:5000]}",
|
|
max_tokens=200,
|
|
)
|
|
upload.content_summary = summary
|
|
upload.ai_description = summary
|
|
else:
|
|
upload.ai_description = f"Text file: {upload.filename}"
|
|
|
|
await db.commit()
|
|
except Exception:
|
|
logger.exception(f"Failed to generate AI description for upload {upload_id}")
|
|
|
|
|
|
@router.post("", response_model=FileUploadResponse, status_code=status.HTTP_201_CREATED)
|
|
@limiter.limit("10/minute")
|
|
async def upload_file(
|
|
request: Request,
|
|
file: UploadFile = File(...),
|
|
session_id: Optional[str] = Form(None),
|
|
current_user: Annotated[User, Depends(get_current_active_user)] = None,
|
|
db: Annotated[AsyncSession, Depends(get_db)] = None,
|
|
) -> FileUploadResponse:
|
|
"""Upload a file and store it in S3-compatible object storage."""
|
|
_check_storage_configured()
|
|
|
|
file_data = await file.read()
|
|
content_type = file.content_type or "application/octet-stream"
|
|
size_bytes = len(file_data)
|
|
|
|
# Validate content type and size
|
|
error = storage_service.validate_upload(content_type, size_bytes)
|
|
if error:
|
|
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=error)
|
|
|
|
# Parse and validate session_id if provided
|
|
parsed_session_id: Optional[UUID] = None
|
|
if session_id:
|
|
try:
|
|
parsed_session_id = UUID(session_id)
|
|
except ValueError:
|
|
raise HTTPException(
|
|
status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid session_id"
|
|
)
|
|
|
|
# Check per-session limits
|
|
count_result = await db.execute(
|
|
select(func.count()).select_from(FileUpload).where(
|
|
FileUpload.session_id == parsed_session_id
|
|
)
|
|
)
|
|
session_count = count_result.scalar() or 0
|
|
if session_count >= storage_service.MAX_FILES_PER_SESSION:
|
|
raise HTTPException(
|
|
status_code=status.HTTP_400_BAD_REQUEST,
|
|
detail=f"Session has reached the maximum of {storage_service.MAX_FILES_PER_SESSION} files",
|
|
)
|
|
|
|
size_result = await db.execute(
|
|
select(func.sum(FileUpload.size_bytes)).where(
|
|
FileUpload.session_id == parsed_session_id
|
|
)
|
|
)
|
|
session_bytes = size_result.scalar() or 0
|
|
if session_bytes + size_bytes > storage_service.MAX_BYTES_PER_SESSION:
|
|
raise HTTPException(
|
|
status_code=status.HTTP_400_BAD_REQUEST,
|
|
detail="Session has exceeded the maximum total upload size (50 MB)",
|
|
)
|
|
|
|
# Upload to S3
|
|
storage_key = await storage_service.upload_file(
|
|
file_data=file_data,
|
|
filename=file.filename or "upload",
|
|
content_type=content_type,
|
|
account_id=str(current_user.account_id),
|
|
)
|
|
|
|
# Persist metadata
|
|
upload = FileUpload(
|
|
account_id=current_user.account_id,
|
|
uploaded_by=current_user.id,
|
|
session_id=parsed_session_id,
|
|
filename=file.filename or "upload",
|
|
content_type=content_type,
|
|
size_bytes=size_bytes,
|
|
storage_key=storage_key,
|
|
)
|
|
db.add(upload)
|
|
await db.commit()
|
|
await db.refresh(upload)
|
|
|
|
import asyncio
|
|
asyncio.create_task(
|
|
_generate_ai_description(upload.id, file_data, content_type)
|
|
)
|
|
|
|
presigned_url = storage_service.get_presigned_url(upload.storage_key)
|
|
|
|
return FileUploadResponse(
|
|
id=upload.id,
|
|
filename=upload.filename,
|
|
content_type=upload.content_type,
|
|
size_bytes=upload.size_bytes,
|
|
url=presigned_url,
|
|
created_at=upload.created_at,
|
|
)
|
|
|
|
|
|
@router.get("/{upload_id}/url")
|
|
async def get_upload_url(
|
|
upload_id: UUID,
|
|
current_user: Annotated[User, Depends(get_current_active_user)],
|
|
db: Annotated[AsyncSession, Depends(get_db)],
|
|
) -> dict:
|
|
"""Get a presigned download URL for an uploaded file."""
|
|
_check_storage_configured()
|
|
|
|
result = await db.execute(select(FileUpload).where(FileUpload.id == upload_id))
|
|
upload = result.scalar_one_or_none()
|
|
|
|
if upload is None:
|
|
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Upload not found")
|
|
|
|
# Verify the upload belongs to the user's account — 404 to avoid revealing existence
|
|
if upload.account_id != current_user.account_id and not current_user.is_super_admin:
|
|
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Upload not found")
|
|
|
|
url = storage_service.get_presigned_url(upload.storage_key)
|
|
return {"url": url}
|
|
|
|
|
|
@router.get("", response_model=list[FileUploadResponse])
|
|
async def list_uploads(
|
|
session_id: UUID,
|
|
current_user: Annotated[User, Depends(get_current_active_user)],
|
|
db: Annotated[AsyncSession, Depends(get_db)],
|
|
) -> list[FileUploadResponse]:
|
|
"""List uploads for a session."""
|
|
_check_storage_configured()
|
|
|
|
result = await db.execute(
|
|
select(FileUpload).where(
|
|
FileUpload.session_id == session_id,
|
|
FileUpload.account_id == current_user.account_id,
|
|
)
|
|
)
|
|
uploads = result.scalars().all()
|
|
|
|
responses = []
|
|
for upload in uploads:
|
|
presigned_url = storage_service.get_presigned_url(upload.storage_key)
|
|
responses.append(
|
|
FileUploadResponse(
|
|
id=upload.id,
|
|
filename=upload.filename,
|
|
content_type=upload.content_type,
|
|
size_bytes=upload.size_bytes,
|
|
url=presigned_url,
|
|
created_at=upload.created_at,
|
|
)
|
|
)
|
|
return responses
|
|
|
|
|
|
@router.delete("/{upload_id}", status_code=status.HTTP_204_NO_CONTENT)
|
|
async def delete_upload(
|
|
upload_id: UUID,
|
|
current_user: Annotated[User, Depends(get_current_active_user)],
|
|
db: Annotated[AsyncSession, Depends(get_db)],
|
|
) -> None:
|
|
"""Delete an uploaded file."""
|
|
_check_storage_configured()
|
|
|
|
result = await db.execute(select(FileUpload).where(FileUpload.id == upload_id))
|
|
upload = result.scalar_one_or_none()
|
|
|
|
if upload is None:
|
|
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Upload not found")
|
|
|
|
# Verify ownership — 404 to avoid revealing existence
|
|
if upload.uploaded_by != current_user.id and not current_user.is_super_admin:
|
|
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Upload not found")
|
|
|
|
# Delete from S3
|
|
await storage_service.delete_file(upload.storage_key)
|
|
|
|
# Delete DB record
|
|
await db.delete(upload)
|
|
await db.commit()
|