Microsoft announced on July 27, 2026 that the AI Gateway tier of Azure API Management is available in public preview.

The new tier provides a purpose-built management and runtime experience for publishing, securing, governing, and observing AI models and Model Context Protocol tools.

It is built on Azure API Management, but its portal, asset model, policies, and developer experience are organized around models, MCP servers, and tools instead of traditional APIs.

What Changed?

Azure API Management already provides AI gateway capabilities across existing tiers.

The new announcement introduces a dedicated AI Gateway tier for organizations that want a focused AI workload experience.

Platform teams can use it to:

• Publish models from Microsoft Foundry and other supported providers

• Place multiple model providers behind governed gateway endpoints

• Publish remote, OpenAPI-generated, and connector-backed MCP tools

• Federate several MCP servers behind one endpoint

• Apply request and token controls through policy cards

• Add content-safety and fallback behavior

• Issue runtime access keys to application teams

• Send AI telemetry to OpenTelemetry-compatible destinations

• Manage gateway resources through Azure and Microsoft Entra controls

The gateway resource runs in the customer's Azure subscription, uses the customer's Entra tenant, and sends telemetry to configured destinations.

A Tier Built Around AI Assets

Traditional API Management begins with APIs, operations, products, subscriptions, and XML-based policies.

The AI Gateway tier begins with models, MCP servers, tools, runtime keys, and AI-specific controls.

That changes the management experience without discarding the gateway operating model. Applications call a stable gateway endpoint, the gateway authenticates the request, applies the policies attached to the target asset, routes to the configured backend, and emits telemetry.

The focused experience matters for platform teams supporting many AI application teams. Instead of requiring each application to manage provider credentials, rate limits, model endpoints, safety controls, and observability independently, the organization can provide a governed access layer.

Multi-Model Support

Most enterprises will not standardize on one model or provider.

Model selection can depend on quality, latency, price, geography, modality, contractual requirements, or a specialized capability.

The preview can publish models hosted through Microsoft Foundry, including supported OpenAI, Anthropic, Mistral, and other Foundry models. Microsoft also describes support for compatible models hosted through services such as AWS Bedrock, Google Vertex AI, OpenAI, and Anthropic.

Applications continue to use supported provider formats, including OpenAI-compatible APIs and Anthropic's Messages API, while the gateway holds the backend connection and credentials.

This creates a stable policy and access boundary, but it does not make every provider identical. Model behavior, API features, quotas, data handling, content filters, regions, latency, and pricing still vary.

MCP Servers and Enterprise Tools

The AI Gateway tier also treats MCP servers as managed assets.

Teams can publish:

• An existing remote MCP server over supported transports

• Selected operations from a REST API by providing an OpenAPI specification

• Connector-backed tools from the Power Platform and Logic Apps ecosystem

• A federated MCP server that combines tools from several backends

Backend authentication can use supported API keys, OAuth client credentials, managed identity, or mutual TLS depending on the connection.

This can reduce the number of direct integrations an agent must manage. It also creates a high-value security boundary because a single gateway may expose tools that read data or take business actions.

Tool publication should therefore be treated as privileged platform work. Every tool needs an owner, approved identity, narrow permissions, input validation, audit telemetry, versioning strategy, and a clear decision about which actions require human approval.

Governance Through Policy Cards

The preview presents AI governance controls as portal policy cards instead of requiring teams to author API Management XML expressions.

Microsoft's announcement highlights controls for:

• Request rate limits

• Token rate limits and quotas

• Azure AI Content Safety

• Fallback to a secondary model

The same settings are represented as JSON properties, allowing teams to manage them through infrastructure as code and evaluate them with Azure Policy.

The card experience lowers the barrier to configuration, but the engineering decisions remain important.

A token quota needs a consumer identity and an escalation process. A fallback needs tested behavior when the primary and secondary models produce different output. Content safety needs workload-specific thresholds and a response path. Rate limits need to reflect concurrency, retry behavior, and downstream capacity.

OpenTelemetry-Based Observability

The AI Gateway tier emits token and request telemetry using OpenTelemetry and GenAI semantic conventions.

Microsoft lists destinations including Application Insights and other OTLP-compatible platforms such as Datadog, Splunk, and Grafana Cloud.

This creates an opportunity to standardize:

• Request volume

• Token consumption

• Latency

• Failure rates

• Model and tool usage

• Consumer and environment attribution

• Budget and capacity alerts

Telemetry design should be reviewed for privacy and data handling. Prompt or response content, tool parameters, user identifiers, and business metadata can be sensitive even when the operational goal is only cost or reliability monitoring.

Identity and Access Need Careful Review

Administrators use Microsoft Entra ID to manage the gateway.

Applications call published assets with runtime access keys, while the gateway authenticates to model and tool backends with the configured connection.

Microsoft's current preview documentation notes that runtime access keys are gateway-scoped. A key can reach every published model and tool in that gateway rather than being limited to one asset.

That is an important design constraint.

Issue separate keys per application and environment, rotate them, monitor their use, and avoid placing unrelated trust zones behind one gateway. Where supported, use managed identity for backend access so provider secrets do not need to be stored and rotated in the gateway configuration.

Preview Scope and Availability

The AI Gateway tier is currently available in public preview in:

• East US 2

• Sweden Central

Microsoft says the tier is available at no cost during the preview and that pricing will be announced separately.

The documentation also states that the preview has no production SLA and that APIs, quotas, regions, limits, telemetry, portal workflows, and pricing can change before general availability.

Private Link and outbound virtual network integration are included in the preview, but teams still need to validate DNS, routing, backend connectivity, telemetry destinations, and the complete data-residency path.

The location of the gateway alone does not determine where model requests, tool data, or telemetry are processed.

AI Gateway Tier Versus Existing API Management Tiers

The dedicated tier is not automatically the right choice for every API Management customer.

Use the AI Gateway tier when the main requirement is a focused, managed boundary for models and MCP tools.

An existing API Management tier may remain the better fit when the organization also needs broad, general-purpose API management capabilities or features not yet included in the preview.

Teams with an established API Management AI gateway should compare capabilities and migration impact rather than assuming the new tier replaces the current service.

Who Should Care?

AI platform teams should care because the tier provides a shared catalog, governance boundary, and runtime for models and tools.

Application teams should care because they can consume approved assets without managing every backend credential.

Security teams should care because the gateway becomes a concentrated control point for model traffic and agent tools.

Operations teams should care because gateway availability, capacity, telemetry, and backend failures affect every dependent AI workload.

FinOps teams should care because token metrics, quotas, model routing, and consumer attribution can improve cost control.

Practical Cloud Engineer Takeaway

Begin with one noncritical application in a supported region.

Publish one Foundry-hosted model and one low-risk, read-only MCP tool. Issue a dedicated runtime key for the pilot and configure the backend with managed identity where supported.

Then test:

• Authentication and key rotation

• Least-privilege backend access

• Request and token rate limits

• Content-safety behavior

• Primary and fallback model differences

• OpenTelemetry export

• Application Insights dashboards and alerts

• Private networking and DNS

• Provider and tool failure modes

• Cost attribution

• Configuration deployment through code

• Gateway deletion and rollback

Do not begin with an agent that can make destructive changes. Prove the access, policy, telemetry, and recovery model with read-only tools first.

Bottom Line

The Azure API Management AI Gateway tier brings a dedicated, AI-first gateway experience to public preview.

It gives platform teams one place to publish and govern supported models, MCP servers, and tools while offering application teams stable endpoints and a self-service path to approved assets.

The architecture can reduce duplicated credential, policy, and telemetry work across AI applications. It also creates a critical shared dependency with broad access to models and enterprise tools.

For now, the right approach is a controlled preview pilot with isolated trust boundaries, dedicated runtime keys, least-privilege backend identities, validated telemetry, and a clear rollback plan.

Sources

Microsoft announcement: https://techcommunity.microsoft.com/blog/integrationsonazureblog/ai-gateway-tier-of-api-management-now-in-public-preview/4540170

AI Gateway tier overview: https://learn.microsoft.com/azure/api-management/ai-gateway-overview

AI Gateway governance and security: https://learn.microsoft.com/azure/api-management/ai-gateway-govern-secure-assets

---

Stay radical, stay curious, and keep pushing the boundaries of what is possible in the cloud.

Chriz Beyond Cloud with Chriz