If your product has a public API, there's a good chance an API gateway already sits in front of it, checking keys and enforcing rate limits before traffic reaches your services. Adding LLM features puts a second kind of traffic into the same stack: calls leaving your app for OpenAI, Anthropic, or Google, priced per token and often taking seconds instead of milliseconds.
API management vendors have responded by adding AI plugins to the gateways teams already run, while a separate category of AI gateways sits only in front of model providers. Both promise to handle model traffic, so it's fair to ask whether you need a second gateway at all.
The two overlap less than the names suggest. An API gateway is built to know whether a request succeeded, while an LLM app also needs to know what each model call cost, which step of an agent made it, and whether the answer was any good.
The AI gateway vs API gateway decision comes down to where each one sits in your stack and what it can see about a model call.
AI gateway vs API gateway
An API gateway manages traffic coming into your own services, with authentication, routing by path, and request-based rate limits. An AI gateway manages traffic going out from your app to model providers, with token-based limits, fallback between models, and cost on every call. Most production LLM apps end up running both, one at each edge.
The differences follow from that split in direction:
| API gateway | AI gateway | |
|---|---|---|
| Traffic | Into your services | Out to model providers |
| Unit of control | Requests | Tokens and dollars |
| Routing | Host, path, headers | Model, provider, cost |
| Failure handling | Healthy service instance | Another model or provider |
| Caching | HTTP responses | Request body, exact or semantic |
| Security | Who is calling | What the prompt contains |
| Observability | Status, latency, throughput | Tokens, cost, prompts, responses |
Both gateways route, retry, cache, and log, so the categories sound alike on paper. Each one does those jobs against a different unit, and that's where the practical differences come from.
What is an API gateway?
An API gateway is the entry point for requests coming into your services. Clients call one endpoint, and the gateway authenticates them, applies rate limits, routes each request to the right backend by host, path, or header, and returns the response. AWS API Gateway, or NGINX configured as a gateway, are common examples, and in a microservices setup the gateway is usually the only part of the system a client talks to directly.
Its unit of work is the HTTP request. Rate limits count requests per second or minute, health checks decide which instance of a service gets traffic, and logs record status codes, latency, and throughput. That model fits traffic where requests are roughly similar in size and cost, which describes most REST and gRPC APIs.
What is an AI gateway?
An AI gateway sits between your application and the model providers it calls. Your app sends requests to one API, usually OpenAI-compatible, and the gateway forwards each one to the right model and handles the work around the call. That includes falling back to another provider when one fails, enforcing token and spend limits, caching repeat requests, screening prompts, and logging tokens and cost for every request.
The term overlaps almost entirely with "LLM gateway," and how an LLM gateway works applies to both: policy checks run before the provider is called, and logging runs after.
Its unit of work is the token. A 50-token classification call and a 100,000-token document summary can hit the same endpoint, with very different latency and cost, so the controls are built around tokens, models, and dollars instead of request counts.
Where each gateway sits in your stack
An API gateway faces inward, between your clients and your services. An AI gateway faces outward, between your services and the model providers they call. In an app with both, a single user request passes through each one:
User → API gateway → Your app or agent → AI gateway → OpenAI / Anthropic / Google
(auth, rate limits) (model routing, token limits, cost)
Agents make the second hop much busier than the first. One inbound request to an agent can turn into a planning call, a model call after each tool result, a retry, and a final answer, so the API gateway sees one request while the AI gateway sees a dozen model calls, each with its own model, token count, and cost. Because of that, request-based rate limits at the API gateway don't protect your model budget, since a single request that reaches the agent can still set off a loop of expensive calls behind it.
Where an MCP gateway fits
Agents add a third kind of traffic: calls to tools and data sources over the Model Context Protocol. An MCP gateway governs that path, controlling which MCP servers an agent can reach and with which credentials. It sits beside the AI gateway rather than replacing it, since tool calls and model calls carry different risks, and some AI gateways now bundle MCP support so both paths share one set of keys and logs.
Key differences
The overlap in vocabulary hides differences in what each gateway measures and acts on, and they show up across most of the request path:
- Requests vs tokens - Rate limits on an API gateway count requests, which works when requests cost about the same. An AI gateway counts tokens and dollars instead, so you can set token budgets per key and spend caps per customer, and see which feature drives the bill the same day instead of on the provider's monthly invoice.
- Routing - A typical API gateway sends traffic to a fixed backend based on host, path, or header. An AI gateway routes on the model named in the request body, so switching models becomes a config change, and it can split traffic across providers or weigh cost and latency. Choosing a model per task is a narrower job, and dedicated LLM routers handle that inside or alongside the gateway.
- Failure handling - When a provider returns 429s or overloaded errors, an AI gateway falls back to a different model or provider, often the same model on another cloud, where an API gateway would retry or move traffic to a healthy instance of the same service. Fallback happens between requests, though, so a stream that fails halfway still reaches your app as a partial answer.
- Streaming - Server-sent events aren't new to API gateways, and most can proxy them. What changes with LLMs is duration and accounting: a streamed response can stay open for tens of seconds and may report token usage only at the end, so timeouts, retries, and cost logging all have to allow for it.
- Caching - Model calls are POST requests with the prompt in the body, which is why caches built around HTTP responses to GET requests rarely help with them. An AI gateway keys its cache on the request itself, by exact match or by semantic similarity. Semantic matching raises the hit rate but can serve a wrong answer to a slightly different question, so it helps to know when a semantic cache is worth it before turning it on.
- Security - Authentication answers who is calling, and that's where an API gateway's checks end. An AI gateway also looks at what the request says, since prompt injection ranks first on the OWASP Top 10 for LLM Applications, so it can block jailbreak and injection attempts and mask personal data before a prompt reaches the provider.
- Observability - A 200 from the upstream tells an API gateway the call succeeded, and its logs stop at status, latency, and throughput. An AI gateway records the model, tokens, cost, prompt, and response for each call, along with metrics like time to first token. Agents push past even that, because one run spans many calls, and connecting them takes LLM tracing that follows the run step by step.
Put side by side, an API gateway tells you your services are up and who is using them, while an AI gateway tells you what your model traffic is doing and what it costs.
Can your API gateway act as an AI gateway?
For the policy side, often yes. API management platforms have added AI plugins to their gateways, so a platform team already running one can put token limits, semantic caching, and prompt guards in front of model traffic, using the same keys, policies, and dashboards as the rest of its APIs. If governing model calls is all you need, that route avoids bringing in a new vendor.
What the plugins aren't built to do is follow the model call into the rest of your app. They run on the same request-and-policy model as the gateway underneath, so a gateway log shows one request with its tokens and cost. In an agent run, the useful view is the trace: which step called which model, what each tool returned, and where the run went wrong.
Output quality is the other gap. Where plugins screen responses, they're typically checking for unsafe content, which is different from measuring whether the answer was correct, and scoring answers on live traffic usually means adding an eval tool beside the gateway. During an incident, that leaves you matching records between two systems to connect a bad score to the request behind it.
When to use each
Since the two gateways sit at different edges, the choice is rarely one or the other. What you're deciding is whether you need an AI gateway yet, and what form it should take.
When an API gateway is enough
If your app makes no model calls, or makes them from one service to one provider at low volume, an API gateway in front of your app and the provider's SDK behind it is a reasonable setup. That covers prototypes, internal tools, and early pilots, where an outage or a surprise bill wouldn't hurt much and another network hop isn't worth adding. You'll still want the API gateway for your public endpoints whatever you decide about model traffic.
When to add an AI gateway
An AI gateway starts paying for itself once model calls are in front of real users and an outage or a runaway bill would be a problem. A few signs you've reached that point:
- More than one provider - Each of the LLM APIs you might combine has its own SDK, keys, error types, and rate limits, and a gateway gives you one API and one fallback chain across all of them.
- Spend you can't attribute - When the invoice grows and you can't tell which customer or feature drove it, per-call cost tracking answers that the same day.
- Agents in production - An agent loop with no token or spend ceiling can burn through a budget before anyone checks a dashboard.
- Compliance requirements - Masking personal data before it reaches a provider and keeping a central record of every prompt are easier to enforce in one place than in every service.
What you give up is an extra network hop on each request (Respan's hosted gateway typically adds 30-40ms) and one more dependency in the request path, so the gateway's own uptime matters as much as your provider's.
When you need both
In a typical production LLM app, the API gateway handles clients coming in and the AI gateway handles model calls going out. The open question is whether they live in one product. If your platform team already runs an API management platform and mainly needs to govern token use, adding its AI plugins keeps everything under one control plane. If the engineers building the AI features need to trace agent runs and score output, a dedicated AI gateway puts that work next to the routing, while the API gateway stays where it is.
Beyond the 200: an AI gateway that scores the answer
An API gateway's job ends when the 200 goes back to the client. For an LLM app, that's when the questions start: what the call cost, which step of the agent made it, and whether the answer was right. Respan routes every model call through one API, and each request it routes is already a trace, already priced, and ready to score, so there's no second tool to wire in after the gateway.
Pointing an existing OpenAI client at Respan is a base URL and key change:
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.respan.ai/api/",
api_key=os.environ["RESPAN_API_KEY"],
)
response = client.chat.completions.create(
model="gpt-5.5",
messages=[{"role": "user", "content": "Summarize this support ticket."}],
)Instead of reading logs after the fact, use Respan to run observability in production, know when production shifts, and act before it spreads:
- One API for 1,000+ models - Point the OpenAI SDK at Respan and switch models by changing one word.
- Fallbacks across providers - Define the fallback chain once, at the org level or per request, and traffic moves to the next model when a provider errors or rate-limits.
- Token and spend limits - Cap requests, tokens, or spend per API key and set budgets per customer, with hard caps that reject a request before it reaches a provider.
- Cost by model, request, and customer - See which customer or feature drives the bill, with no processing fee on Respan credits.
- Exact-repeat caching - Serve requests with identical messages from cache and skip the model call, with separate cache entries per model or per customer if you need them.
- Guardrails and PII masking - Block jailbreaks, prompt injection, and unsafe content on incoming requests, and mask personal data before it reaches the model provider.
- Traces for agent runs - Every routed call is logged as a span, and with the Respan SDK, tool runs and agent turns join it in one trace, each with its own input, output, latency, and cost.
- Scores on live traffic - Behaviors score every new span for patterns like refusals, incomplete answers, and missed instructions, on the Team plan and up, and you can define your own in plain language.
- Run the gateway yourself - Respan's open-source gateway is a single binary that lets any SDK call any model, with routing, fallbacks, and MCP turned on in one config file. It's still pre-alpha, and when you want every request traced, priced, and scored, the Respan plugin sends them to the hosted platform.
Respan sits beside your existing API gateway rather than replacing it, so the platform team keeps its policies while the engineers building AI features get traces and scores on the same path. It's SOC 2, GDPR, and ISO 27001 compliant, and supports HIPAA workloads with a BAA.
FAQ
Is Apigee an AI gateway?
Apigee is an API management platform, and it can act as an AI gateway for token control through its LLM token policies. PromptTokenLimit throttles tokens from user prompts, and LLMTokenQuota enforces quotas per API product, developer, or app, though neither is available on Apigee hybrid. Teams that also need agent traces and scores on live traffic can run a dedicated AI gateway like Respan alongside it, between their app and the model providers.
Is OpenRouter an AI gateway?
OpenRouter works as a hosted AI gateway for model access, giving you one OpenAI-compatible endpoint to hundreds of models with automatic fallbacks between providers. Respan offers 1,000+ models through one API as well, with tracing, spend limits, and scoring on every routed call. For a wider comparison, see the OpenRouter alternatives that cover routing alongside observability.
What are the most popular AI gateways?
Commonly compared AI gateways include Respan, LiteLLM, Portkey, OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, and Kong AI Gateway. They split into dedicated gateways that sit only in front of model providers, like Respan and LiteLLM, and API management platforms with AI features, like Kong. Respan adds tracing and evals to the routing itself, and the best LLM gateways comparison breaks down the rest by features and pricing.
