Module O4: Microsoft Foundry — Platform Deep Dive

Learning Outcomes

After completing this module, you can:

• Model Microsoft Foundry resources and projects as governance and application boundaries.
• Choose deployment, networking, identity, quota, and evaluation patterns from workload requirements.
• Separate current Foundry guidance from classic migration references.
• Create an evidence-backed production readiness decision for a Foundry workload.

Prerequisites

Prerequisites: Complete F1, O2, and have basic Azure CLI and RBAC experience.

3.1 What is Microsoft Foundry?

Microsoft Foundry is Microsoft’s unified platform for building, evaluating, governing, and operating AI applications and agents on Azure. The current platform brings models, agents, evaluations, and Foundry Tools under a Foundry resource with child projects.

The Evolution

The naming journey matters because customers still encounter every generation in repositories and deployed resources.

The current Foundry resource is an Azure resource: Microsoft.CognitiveServices/accounts with kind: AIServices. A Foundry project is its Microsoft.CognitiveServices/accounts/projects child resource. This differs from the classic Hub/Project architecture based on Microsoft.MachineLearningServices/workspaces.

Three Interfaces, One Platform

3.2 Architecture & Resource Model

The current architecture uses a Foundry resource as the governance and capability boundary and child projects as application isolation boundaries.

Foundry Resource and Project Model

Foundry Resource (Parent)

The Foundry resource is the shared administrative boundary. It provides:

• Model deployments — Microsoft and partner models exposed through supported endpoints
• Agent and evaluation capabilities — shared platform services consumed by projects
• Networking configuration — public access, private endpoints, managed VNet
• Security policies — RBAC role assignments, managed identity, customer-managed keys
• Foundry Tools — Speech, Vision, Language, Content Safety, and related prebuilt capabilities

A Foundry resource maps to Microsoft.CognitiveServices/accounts with kind: AIServices.

Project (Child Resource)

A Project is an isolated development and operations boundary scoped to an AI application or workload. It maintains its own:

• Model deployments and endpoints
• Agent versions and application assets
• Evaluation runs and datasets
• Fine-tuning jobs
• Artifacts and logs

A Project maps to Microsoft.CognitiveServices/accounts/projects beneath its parent Foundry resource.

Resource Relationship Summary

RBAC Model

Microsoft Foundry uses Azure RBAC with purpose-built roles.

Networking Options

3.3 Model Catalog

The model catalog is the front door of Microsoft Foundry. Its contents, deployment types, licenses, regions, and lifecycle states change continuously; query the current Foundry model catalog  before selecting a model. Do not use a fixed catalog count for architecture or procurement decisions.

Model Providers

Two Deployment Paradigms

The catalog offers two fundamentally different ways to run a model. Understanding this distinction is critical for cost planning and architecture.

Serverless API Endpoints (MaaS)

Serverless API endpoints are the simplest path from model selection to production. You select a model from the catalog, accept the terms, and receive an API endpoint and key. No compute provisioning, no VM sizing, no scaling configuration.

Key characteristics:

• Pay-per-token pricing — you pay only for the tokens you consume (input + output)
• No infrastructure to manage — Microsoft handles scaling, availability, and hardware
• Azure OpenAI-compatible API — same SDK and API contract as Azure OpenAI deployments
• Immediate availability — endpoint is live within seconds of deployment
• Regional availability matters — not all models are available in all regions

Managed Online Endpoints

Managed Online Endpoints give you dedicated compute for model serving. You deploy a model (from the catalog or your own custom model) to a specific VM SKU and control scaling.

Key characteristics:

• Dedicated VMs — choose from CPU or GPU SKUs (e.g., Standard_NC24ads_A100_v4)
• Autoscaling — scale based on request count, CPU, or custom metrics
• Blue-green deployments — traffic splitting across multiple deployments for safe rollouts
• Custom containers — bring your own inference server (e.g., vLLM, TGI, Triton)
• VNet integration — deploy endpoints inside a managed VNet with private endpoint access

Comparing Models in the Catalog

The portal provides built-in tools for model comparison:

1. Benchmark scores — view standardized benchmarks (MMLU, HumanEval, MT-Bench) side by side
2. Model cards — detailed descriptions of capabilities, limitations, and intended use cases
3. Try it out — interactive playground to test models with your own prompts before deploying
4. Pricing calculator — estimate costs based on expected token volume
5. Region availability — check which regions support which models

3.4 Model Deployments

Deployment is where architecture decisions meet operational reality. Azure AI Foundry supports three deployment types, each with different performance, cost, and management characteristics.

Deployment Type 1: Azure OpenAI Deployments

These are deployments of Microsoft’s proprietary models (GPT-4o, GPT-4.1, o3, etc.) through the Azure OpenAI Service resource connected to your Hub.

Deployment Type 2: Serverless API Deployments

These are pay-per-token deployments for partner models (Llama, Mistral, Cohere) and some Microsoft models that use the Models as a Service (MaaS) infrastructure.

• No compute to manage
• You accept the model provider’s terms of use
• Charged per million input/output tokens
• Endpoint is Azure OpenAI-compatible (same SDK, same API shape)

Deployment Type 3: Managed Online Endpoints

These are dedicated-compute deployments for any model — from the catalog or custom.

• You select the VM SKU and instance count
• Support for autoscaling (min/max replicas, scaling metric)
• Blue-green deployment with traffic splitting
• Custom inference containers supported
• Full VNet integration and private endpoint access

Deployment Types Comparison

Quotas and Rate Limits

Every deployment type has quotas. Platform architects must plan for quota as a first-class infrastructure concern.

3.5 Prompt Flow

Prompt Flow is Azure AI Foundry’s visual orchestration tool for building LLM applications. If you are familiar with Azure Logic Apps or Power Automate, the mental model is similar — but purpose-built for AI workflows.

What is Prompt Flow?

Prompt Flow lets you build DAG-based flows (Directed Acyclic Graphs) where each node performs a specific operation: call an LLM, execute Python code, process a prompt template, or invoke a tool. The output of one node feeds into the next, creating a composable pipeline.

Node Types

Use Case: Building a RAG Pipeline in Prompt Flow

Here is how you would build a Retrieval-Augmented Generation pipeline visually in Prompt Flow:

Step 1: Input Node — Accept the user’s query as a string input.

Step 2: Embedding Node (Python) — Call the Azure OpenAI embedding model to convert the query into a vector.

Step 3: Search Node (Tool) — Query Azure AI Search with the vector to retrieve the top-k most relevant document chunks.

Step 4: Prompt Node — Construct an augmented prompt that injects the retrieved chunks as context, along with the user query.

Step 5: LLM Node — Send the augmented prompt to GPT-4o for answer generation.

Step 6: Output Node — Return the generated answer along with source citations.

The entire flow is defined as a YAML file (flow.dag.yaml) that can be version-controlled in Git, making it CI/CD-friendly.

Evaluation Flows

Prompt Flow supports a special type of flow called an evaluation flow. Instead of processing user queries, evaluation flows score the quality of outputs produced by your main flow.

An evaluation flow typically:

1. Takes the main flow’s output (answer), the ground-truth answer, and the original question as inputs
2. Calls an LLM (or runs custom Python logic) to score the output on metrics like groundedness, relevance, and coherence
3. Outputs numerical scores that can be aggregated across a test dataset

This enables automated quality gates in your CI/CD pipeline — if the evaluation scores drop below a threshold, the deployment is blocked.

Deploying a Flow as an Endpoint

Once you have built and tested a Prompt Flow:

1. Build the flow into a Docker container (Foundry handles this automatically)
2. Deploy the container to a Managed Online Endpoint
3. Configure autoscaling, traffic splitting, and authentication
4. Monitor with built-in metrics (latency, throughput, error rate, token consumption)

The deployed flow exposes a REST API endpoint that your application calls — just like any other microservice.

3.6 Model Evaluation

Evaluation is the most underinvested area in most AI projects and the most important area for production readiness. Azure AI Foundry provides built-in evaluation capabilities that let you systematically measure model quality before deployment.

Why Evaluation Matters

Built-in Evaluation Metrics

Azure AI Foundry provides LLM-as-a-judge evaluation metrics that use a grader model (typically GPT-4o) to score your application’s outputs.

Custom Evaluation Metrics

When built-in metrics are not sufficient, you can define custom evaluation metrics using:

• Python functions — Write a Python function that takes the model output and returns a score
• LLM-as-a-judge prompts — Write a custom prompt that instructs GPT-4o to score the output on your domain-specific criteria (e.g., “Does this medical summary include all required ICD-10 codes?”)
• Composite metrics — Combine multiple metrics into a single quality score with weighted averages

Red-Teaming Evaluations

Red-teaming tests whether your AI application can be manipulated into producing harmful, biased, or policy-violating outputs.

Azure AI Foundry supports red-teaming through:

1. Automated adversarial testing — Built-in adversarial datasets that probe for jailbreaks, prompt injections, and content policy violations
2. Custom red-team datasets — Define your own adversarial prompts tailored to your application’s domain
3. Azure AI Content Safety integration — Automatically score outputs for hate speech, violence, self-harm, and sexual content severity levels
4. Human-in-the-loop review — Export flagged outputs for manual review by your safety team

Evaluation Datasets and Test Suites

A robust evaluation requires a well-curated test dataset. Best practices:

3.7 Fine-Tuning

Fine-tuning is the process of further training a pre-trained model on your domain-specific data to improve its performance on your specific use case. Azure AI Foundry supports fine-tuning for select models directly within the platform.

When to Fine-Tune (And When NOT To)

Fine-Tuning Decision Matrix

Supported Models for Fine-Tuning

Fine-Tuning Workflow

Data preparation format (JSONL):

{"messages": [{"role": "system", "content": "You are a medical coding specialist."}, {"role": "user", "content": "Patient presents with acute bronchitis and mild dehydration."}, {"role": "assistant", "content": "ICD-10 Codes:\n- J20.9 (Acute bronchitis, unspecified)\n- E86.0 (Dehydration)"}]}
{"messages": [{"role": "system", "content": "You are a medical coding specialist."}, {"role": "user", "content": "Follow-up for type 2 diabetes, well controlled."}, {"role": "assistant", "content": "ICD-10 Code:\n- E11.65 (Type 2 diabetes mellitus with hyperglycemia)"}]}

LoRA and Parameter-Efficient Fine-Tuning

LoRA (Low-Rank Adaptation) is a technique that fine-tunes only a small number of additional parameters (adapters) rather than updating all model weights. This has major implications for architects:

Cost and Compute Requirements

Fine-tuning costs are driven by three factors:

1. Training compute — GPU hours consumed during training (typically Standard_NC24ads_A100_v4 or similar)
2. Hosting cost — Fine-tuned Azure OpenAI models incur higher per-token costs than base models; custom models on Managed Online Endpoints cost per VM-hour
3. Data preparation — Human time to curate, clean, and validate training data (often the most expensive part)

3.8 Azure AI Services (Integrated)

Azure AI Foundry integrates with the broader Azure AI Services family — pre-built, task-specific AI capabilities that were previously known as Azure Cognitive Services. These services complement generative AI models by handling specialized tasks like speech recognition, document parsing, and content moderation.

Service Overview

How These Integrate with Foundry Projects

Key integration patterns:

1. Document Intelligence as RAG Ingestion — Use Document Intelligence to extract text, tables, and structure from PDFs and images, then chunk and embed the output for vector search
2. Content Safety as a Guardrail — Content Safety filters run automatically on Azure OpenAI deployments; you can also invoke them explicitly in Prompt Flow for custom models
3. Speech as an I/O Layer — Add voice input/output to any Prompt Flow by using Speech STT (input) and TTS (output) nodes
4. Language for Pre-Processing — Use PII detection to redact sensitive data before sending to an LLM; use NER to extract entities for structured queries

3.9 Infrastructure Considerations

This section addresses the infrastructure decisions that platform architects must make when deploying Azure AI Foundry in production.

Compute Options

Networking Deep Dive

Production deployments of Azure AI Foundry require careful networking design. The following table summarizes the network endpoints you need to plan for:

Data Residency and Compliance

RBAC and Security Model (Expanded)

Security best practices:

1. Use Managed Identity for all service-to-service communication. Avoid storing API keys in Key Vault when managed identity is supported.
2. Enable Managed VNet with data exfiltration protection for regulated workloads.
3. Apply RBAC at the Project level — not the Hub level — to enforce least privilege.
4. Use Conditional Access policies in Entra ID to enforce MFA and compliant device requirements for Foundry portal access.
5. Enable diagnostic logging to send Hub and Project audit logs to a Log Analytics workspace or SIEM.

Cost Management

3.10 Microsoft Foundry vs AWS Bedrock vs Google Vertex AI

Platform architects in multi-cloud environments should compare ownership boundaries and live workload evidence, not frozen model counts or benchmark headlines. Provider catalogs, regions, quotas, and feature support change independently.

Feature Comparison

Durable Decision Signals

1. Existing platform ownership — Prefer the cloud where identity, networking, data governance, incident response, and platform expertise are already mature unless workload evidence justifies another boundary.
2. Required live capability — Verify the exact model, region, deployment type, quota, fine-tuning support, safety controls, and endpoint behavior in the target account or subscription.
3. Data and regulatory boundary — Compare where prompts, outputs, logs, embeddings, and training data are processed and retained.
4. Operational evidence — Benchmark the real workload for quality, p95 latency, throughput, failure behavior, and total cost instead of comparing catalog counts.
5. Exit strategy — Keep business contracts, evaluation datasets, and tool interfaces separable from provider-specific deployment code where portability has measurable value.

Multi-Cloud Considerations

For organizations running multi-cloud AI strategies:

• Own an application-level contract — Define the messages, tool schemas, structured outputs, streaming events, errors, and telemetry your application requires. Provider adapters must pass contract tests; a similar API shape alone does not guarantee portability.
• Abstract only proven differences — Keep orchestration and business policy independent of provider SDKs where the workload exercises multiple providers. Do not add a framework solely to claim theoretical portability.
• Separate model selection from infrastructure — Resolve deployment identifiers and routing policy from configuration while retaining explicit adapters for provider-specific identity, safety, quota, and lifecycle behavior.
• Evaluate per workload — Compare qualified deployments with the same versioned dataset, SLOs, failure cases, residency requirements, and cost model. Record the verification date and exact deployment identifiers.

Applied Scenario: Approve a Foundry Project for Production

Situation: A platform team must onboard a RAG agent with private data, separate application ownership, managed identity, and regional recovery requirements.

Discovery commands: Inspect the target subscription instead of copying resource IDs, regions, or quotas from documentation.

az account show --query "{subscription:id, tenant:tenantId}" -o json
az cognitiveservices account list --query "[].{name:name, kind:kind, location:location, id:id}" -o table
az role assignment list --assignee "<managed-identity-object-id>" --all -o table

Decision record: Capture the parent Foundry resource, child project, identity assignments, private connectivity, model deployment name, quota evidence, evaluation baseline, telemetry destination, and rollback model.

{
  "projectBoundary": "one workload and owning team",
  "identity": "managed identity; no embedded API keys",
  "network": "private ingress and approved egress",
  "releaseGates": ["evaluation baseline", "quota evidence", "rollback test"],
  "liveChecks": ["model catalog", "target-region availability", "RBAC assignments"]
}

Validation: Deny public access where required, prove least-privilege access, exercise quota exhaustion and regional fallback, and retain dated catalog evidence.

Key Takeaways

What is Next?

You now understand the platform where AI applications are built and deployed. In the next module, we explore the consumer-facing side of Microsoft’s AI strategy — the Copilot ecosystem that sits on top of this platform.

Next: Module 4: Microsoft Copilot Ecosystem — M365 Copilot, Copilot Studio, Copilot Actions, GitHub Copilot, Copilot for Azure, and the extensibility model that connects them all.

Previous: Module 2: LLM Landscape

Knowledge Check

1. What boundary does a Foundry project represent?

2. Why can documentation not guarantee model availability?

3. What makes a Foundry deployment production-ready?