T3: Production Architecture Patterns

Table of Contents

• T3.1 Production AI is Different
• T3.2 The AI Application Architecture Stack
• T3.3 Hosting Patterns: Where Agents Live
• T3.4 API Gateway for AI
• T3.5 Latency Optimization Patterns
• T3.6 Cost Control Architecture
• T3.7 Multi-Agent Production Patterns
• T3.8 Monitoring & Observability for AI
• T3.9 Resilience Patterns
• T3.10 The Production Readiness Checklist
• Learning Outcomes
• Prerequisites
• Applied Scenario: Operate a Cost-Aware Model Router
• Knowledge Check
• Key Takeaways

Learning Outcomes

After completing this module, you can:

• Choose production hosting, gateway, routing, caching, and resilience patterns from explicit SLOs.
• Instrument model, retrieval, tool, safety, cost, and business outcome telemetry.
• Design graceful degradation and rollback for provider, region, quota, and model failures.
• Complete a production readiness review with accountable operational ownership.

Prerequisites

Prerequisites: Complete O2, O4, and O5.

T3.1 Production AI is Different

Your POC worked beautifully in the demo. Now you need to serve 10,000 users concurrently, handle API rate limits, manage costs, respond in under 2 seconds, never hallucinate on financial data, and stay available 99.9% of the time.

Welcome to production AI.

T3.2 The AI Application Architecture Stack

Every production AI system has these layers. Missing any one of them is a production incident waiting to happen.

T3.3 Hosting Patterns: Where Agents Live

Pattern Comparison

Decision Matrix

Recheck the Container Apps serverless GPU regions and constraints  and Functions hosting-plan timeouts  during design. Deploying Foundry catalog models directly to Container Apps serverless GPUs is a separate preview capability from running your own GPU container.

Container Apps — The Sweet Spot for Most AI Workloads

T3.4 API Gateway for AI

Azure API Management (APIM) becomes critical for production AI — it’s the control plane for all AI traffic.

AI Gateway Capabilities

Multi-Region AI Gateway

T3.5 Latency Optimization Patterns

Where Latency Hides

Streaming for Perceived Performance

Instead of waiting for the full response, stream tokens to the user as they’re generated:

Without streaming:  [---- 3 seconds of nothing ----] Full response appears
With streaming:     H-e-l-l-o-,- -h-e-r-e-'-s- -y-o-u-r- -a-n-s-w-e-r-... (progressive)

TTFT (Time To First Token) drops from 3s to ~200ms. The user sees progress immediately.

Caching Strategies

T3.6 Cost Control Architecture

Token Economics

Cost per request = (input_tokens × input_rate) + (output_tokens × output_rate)

Provider-neutral example:
    Input tokens:  system + user + retrieved context
    Output tokens: generated response + billed reasoning tokens, when applicable
    Request cost:  (input tokens × current input rate)
                             + (output tokens × current output rate)
                             + tool, retrieval, cache, and hosting charges

    Monthly cost: request cost × requests/day × active days

Use current target-region pricing and measured token distributions from representative traffic. Model routing, caching, batching, and prompt reduction should be justified by evaluation data, not generic savings percentages.

Cost Optimization Decision Tree

T3.7 Multi-Agent Production Patterns

Pattern 1: Supervisor Agent

When: Clear domain boundaries, need routing intelligence, want centralized control.

Pattern 2: Pipeline (Sequential Handoff)

When: Document processing, data pipelines, workflows with clear sequential steps.

Pattern 3: Swarm (Peer-to-Peer)

When: Creative tasks, complex reasoning, research — agents negotiate and collaborate without a central controller.

Hosting Multi-Agent: The Microservices Approach

Agent 1 (Supervisor)   → Container App (scale 2-10)
Agent 2 (Billing)      → Container App (scale 0-5)
Agent 3 (Tech Support) → Container App (scale 0-5)
Agent 4 (Product)      → Container App (scale 0-5)

Communication: Dapr pub/sub (async) or HTTP (sync)
State: Cosmos DB (conversation memory)
Observability: Application Insights (distributed tracing)

T3.8 Monitoring & Observability for AI

The AI Observability Stack

Key Metrics Dashboard

T3.9 Resilience Patterns

Circuit Breaker for AI Endpoints

Fallback Chain

Primary:    Azure OpenAI (East US) GPT-4o
  ↓ (429 or 500)
Fallback 1: Azure OpenAI (West US) GPT-4o
  ↓ (429 or 500)
Fallback 2: Azure OpenAI (East US) GPT-4o-mini (degraded quality, lower cost)
  ↓ (both down)
Fallback 3: Cached response for similar queries
  ↓ (no cache hit)
Fallback 4: "I'm experiencing high demand. Please try again shortly."

T3.10 The Production Readiness Checklist

Before going live, verify every item:

Architecture & Infrastructure

• Hosting platform selected (Container Apps / AKS / App Service)
• Private endpoints for all AI services
• Managed Identity (no API keys in code)
• Multi-region deployment (if SLA requires >99.9%)
• Auto-scaling configured with appropriate min/max
• Load testing completed at 2x expected peak

Security

• Prompt injection detection enabled
• Content Safety filters configured
• Input validation on all user inputs
• PII detection and redaction
• RBAC configured with least privilege
• Audit logging for all AI interactions

Quality & Reliability

• Evaluation pipeline running (offline + online)
• Hallucination rate measured and <5%
• Groundedness score >0.85
• Circuit breaker and fallback chain configured
• Rate limiting per user/application
• Retry with exponential backoff for transient errors

Cost Management

• Token budget per user/team configured
• Cost alerts at 50%, 80%, 100% of budget
• Semantic caching for common queries
• Model tier optimization (mini for simple, full for complex)
• Cost dashboard accessible to stakeholders

Monitoring & Operations

• Application Insights with custom AI metrics
• Distributed tracing across agent interactions
• Alerts for latency, errors, cost, quality degradation
• Runbook for common incidents (rate limits, model outage)
• On-call rotation for AI-specific issues

Applied Scenario: Operate a Cost-Aware Model Router

Situation: A shared API serves extraction, chat, and complex reasoning with different latency and quality requirements.

Routing policy: Use deterministic task metadata and measured evaluation bands. Never route solely from a model-generated self-assessment.

def route(request):
    if request.contract == "strict-json" and request.complexity == "low":
        return "economy-deployment"
    if request.deadline_ms < 1500:
        return "low-latency-deployment"
    if request.requires_deep_reasoning:
        return "reasoning-deployment"
    return "standard-deployment"

Validation: Shadow traffic before promotion, compare quality by task class, enforce spend and latency budgets, test every fallback, and record the deployment IDs used for each decision.

Key Takeaways

FrootAI T3 — Production AI is where engineering meets intelligence. Build it like infrastructure. Monitor it like a service. Budget it like a business.

Knowledge Check

1. Why should routing use task classes rather than a universal best model?

2. What must happen when the preferred deployment is unavailable?

3. Which telemetry is required to tune a router?