An AI agent rarely makes one model call. A single user action can fan out into a dozen model and tool calls, often across more than one provider. That fan-out makes a provider outage, a rate limit, or a loop with no spending ceiling hurt faster than it would in a simple chat app.
That's why production LLM stacks tend to add a layer between the application and the model providers, so failover, spend limits, and logging live in one place instead of in every service that calls a model.
That layer is also the natural starting point for observability and evals, since every request already passes through it. An LLM gateway is worth understanding at the architecture level before you decide whether your stack needs one.
What is an LLM gateway?
An LLM gateway is a service that sits between your application and the model providers it calls, so your app makes requests to one OpenAI-compatible API instead of maintaining a separate integration for each provider. The gateway forwards each request to the right model and handles the operational work around it:
- Unified API - One interface across providers.
- Provider fallback - A backup model when one fails.
- Caching - Stored responses for repeated requests.
- Rate limits and budgets - Caps per user, feature, or dollar.
- Guardrails - Unsafe requests blocked before the model.
- Observability - Cost, latency, and model for each call.
Your application keeps one client and one set of credentials, while the routing and policy logic moves into the gateway, where you can change it without redeploying the app.
LLM gateway vs. calling providers directly
Calling a provider directly is a reasonable place to start, since it means one SDK, one key, and no extra network hop. The tradeoffs show up once there's a second provider, real traffic, or an agent turning one user action into many calls.
| Calling providers directly | Through a gateway | |
|---|---|---|
| Switching models | Edit code, redeploy | Change the model string |
| Provider outage | Requests fail | Fails over to backup |
| Cost attribution | Monthly provider invoice | Per request, key, customer |
| Rate limits | Per provider account | Per feature, key, customer |
| Audit trail | Whatever each app logs | Central record of calls |
Picture a provider returning 503s for 25 minutes during peak traffic. Called directly, every request in that window fails, and your agent is down until someone writes a fallback under pressure. Through a gateway with Bedrock configured as the backup, the same model answers from a different cloud and the request still succeeds.
Cost works the same way. When one feature or customer starts spending 10× the average, per-call attribution shows it the same day instead of on the month-end invoice. And if you want to test a new model on a feature, a gateway turns that into a config change and a small traffic split rather than a code change.
How an LLM gateway works
An LLM gateway's architecture is a request path with policy at both ends, with checks that run before the provider is called and logging that runs after. A typical request moves through these steps:
- Your app sends an OpenAI-format request to the gateway's endpoint, authenticated with a gateway key instead of a provider key.
- The gateway checks the key against its limits and runs any guardrails on the incoming request.
- If caching is on and the request is an exact repeat, the stored response goes straight back.
- The gateway resolves the route: which model, which provider, and which fallback chain applies.
- It translates the request into the provider's native format, calls the provider, and retries or falls back if the provider errors or rate-limits.
- The response is normalized back into the OpenAI shape and returned to your app.
- The call is logged with tokens, cost, latency, model, and the key or customer that sent it.
Steps 2 and 7 are where a production team gets most of the value from this architecture. Policy runs before any money is spent on the provider, and logging happens on each call whether or not the app that made it remembered to instrument anything.
Core capabilities of an LLM gateway
Unified API across providers
Many gateways expose OpenAI's Chat Completions API as the common interface, so your app keeps using openai-python or openai-node pointed at the gateway's base URL. The gateway translates each request into the provider's native API on the back end.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.respan.ai/api/",
api_key=os.environ["RESPAN_API_KEY"],
)
response = client.chat.completions.create(
model="gpt-5.5",
messages=[{"role": "user", "content": "Summarize this support ticket."}],
)For a standard chat call, switching models means changing the model string, though provider-specific parameters and capabilities like image input still depend on the model you switch to.
Fallbacks and retries
When the primary provider returns a server error, a rate limit, or drops the connection, the gateway retries and then moves to the next model in a configured chain. The backup is usually the same model on a different cloud:
- Anthropic Sonnet → Bedrock Sonnet
- OpenAI GPT-5.5 → Azure OpenAI GPT-5.5
- Google Gemini → Vertex AI Gemini
If you've handled Anthropic 429 and 529 errors in application code, you've already written a small piece of this. A gateway with no fallback configured is half a gateway, so set the chain up on day one rather than after the first outage. One limit to plan for is that a response that fails partway through a stream can't be handed to a backup model cleanly, so your app still needs a way to handle a partial output.
Caching
Gateway caches come in two forms. An exact-match cache returns a prior response when the same request arrives again, which is predictable and gets a good hit rate when settings are deterministic, such as temperature 0. A semantic cache embeds the prompt and returns a response for a similar enough match, which raises the hit rate, but a false positive ships a wrong answer to a user who asked something slightly different.
A cache hit skips the model call and its cost entirely. Start with exact-match only, and add semantic matching for specific routes once you know where the false positives would land. The gateway cache is also only one of the cache layers an LLM app can run, alongside provider-side prompt caching.
Rate limits and spend controls
Provider rate limits are blunt, since they apply per account rather than per feature. A gateway lets you set finer budgets per user, per feature, per workspace, per minute, per day, or per dollar. With the right setup, a free-tier feature can run out of budget without taking your premium feature down with it.
Spend caps matter more once agents are involved, because a loop with no ceiling can burn through a budget before anyone checks a dashboard. A soft cap that sends an alert is useful, while a hard cap that blocks requests is what stops a leaked key. If several teams share one gateway, give each its own key or workspace with its own limits, so one team's burst of traffic doesn't throttle everyone else.
Observability
Since every request passes through the gateway, it can record the model, prompt, response, tokens, latency, and cost of each call, plus the key and any customer ID you attach, without extra instrumentation in your app. That makes it the cheapest place to start answering questions like which feature is driving cost or which model is erroring.
A log of individual calls has a ceiling, though. In an agent run, the useful view is the full trace: which step called which model, what each tool returned, and where the run went wrong.
LLM gateway security
The gateway's position in the request path makes it the natural place to screen inputs before they reach a model. Prompt injection sits at the top of the OWASP Top 10 for LLM applications, and a gateway can block it, along with jailbreaks and unsafe content, before the provider ever sees the request.
Respan Guardrails run inside the gateway and block incoming requests in eight categories, from jailbreaks to illegal instructions. You pick the categories in the platform, and they're free with the gateway with no code changes to enable them, though your app should handle the HTTP 400 a blocked request returns.
The same position helps keep personal data away from providers. With Respan's PII masking in gateway mode, personal data in a request is masked before it reaches the model provider. Provider API keys also live in the gateway instead of in each service, so rotating one is a single change, and each app holds a gateway key you can cap or revoke on its own. When compliance asks where prompts go, the gateway's log answers for each call, not only the ones an app chose to record.
LLM gateway vs. API gateway, AI gateway, proxy, and router
These terms overlap, and vendors use them loosely. The difference is mostly in what each layer sits in front of and what it understands about the traffic.
| Sits in front of | Understands | Main job | |
|---|---|---|---|
| API gateway | Your own services | HTTP requests | Auth, routing, throttling |
| LLM gateway | Model providers | Models, tokens, cost | Reliability and spend control |
| AI gateway | Models and AI APIs | Text, image, audio, embeddings | LLM gateway, broader scope |
| LLM proxy | Model providers | Request forwarding | Pass calls through |
| LLM router | Candidate models | The request's task | Pick a model |
| MCP gateway | MCP servers | Tool calls | Govern agent tool access |
A traditional API gateway manages HTTP traffic to your own services, but it doesn't know what a token is, which model served a request, or what that request cost, which is the core of the AI gateway vs API gateway difference. An LLM gateway adds that model awareness. "AI gateway" is the broader term, often used by API-management vendors whose gateways cover image, audio, and embedding APIs as well as text models, so the capabilities overlap heavily and the right term is usually whichever one your stack already uses.
An LLM proxy forwards requests to providers, and in practice the word is often used for a gateway with fewer controls around it. A router is narrower still, since it's a single policy (which model should answer this request) that usually runs inside a gateway, and dedicated LLM routers pick the model per task. An MCP gateway sits on a different path entirely, controlling which tools and data sources an agent can reach over the Model Context Protocol.
When to use an LLM gateway, and when you don't need one
A gateway earns its place once an LLM feature is in front of real users and an outage or a surprise bill would be a problem. It also tends to pay off when you call more than one provider, when you need to know cost per customer or per feature, or when agents are making calls on a user's behalf with no human checking each one.
For a prototype, an internal tool, or a single app calling a single provider at low volume, calling the provider directly is simpler, and you can add a gateway later. Adding one does come with costs worth weighing:
- Latency - Each request takes an extra network hop.
- A dependency in the request path - If the gateway is down, your calls are too, so its uptime matters as much as your provider's.
- Mid-stream failures - Fallback works between requests, not halfway through a streamed response.
- Cache judgment - Responses built on private data or current information shouldn't be served to a different user or reused once they're stale.
- Agent tools - A gateway governs model calls, while the tools an agent can reach need their own controls.
For most production teams the reliability and spend control outweigh the extra hop, especially once caching takes some calls off the provider entirely. If you're at the point of comparing products, the best LLM gateways comparison covers the options and how they're priced.
Respan: an LLM gateway with observability and evals built in
A standalone gateway gives you routing and request logs, and tracing and evals usually come from separate tools wired on afterward. Respan's LLM gateway puts all three in one place, so every request it routes is already a trace, already priced, and ready to score, with no second integration. When a score drops, you click into the trace behind it instead of matching timestamps across tools.
Instead of reading logs after the fact, use Respan to run observability in production, know when production shifts, and act before it spreads:
- One API for 1,000+ models - Point the OpenAI SDK at Respan and switch models by changing one word.
- Fallback chains - Define the chain once, at the org level or per request, and when retries on one model are exhausted, Respan moves to the next.
- Spend limits that block - Soft caps alert you, and hard caps stop requests before they reach the provider, per API key or per customer.
- Costs by model, request, and customer - See which customer or feature drives the bill, with no processing fee on Respan credits and provider discounts passed straight through.
- Exact-repeat caching - Serve requests with identical messages from cache and skip the model call and its cost, with the option to keep separate cache entries per model or per customer.
- Guardrails - Block jailbreaks, prompt injection, and unsafe content on incoming requests, free with the gateway.
- Tracing for agents - Every routed call is logged as a span automatically, and with the Respan SDK, tool runs and agent turns join it in one trace, each with its own input, output, latency, and cost.
- Online evals - Score live production traffic with the same evaluators you run before shipping, so quality regressions surface before users report them.
- Task-aware routing - Respan Router (beta) picks the model for each task when you set the model to
span-router, running on Respan credits as an alternative to a fixed fallback chain. - Run it on your own infrastructure - Respan's open-source gateway runs as a single binary under MIT or Apache-2.0. Any OpenAI, Anthropic, or Gemini SDK can call any model through it, with routing, fallbacks, and MCP as plugins, and the Respan plugin sends each request into the platform as a trace. It is still pre-alpha.
Respan is SOC 2, GDPR, and ISO 27001 compliant, and supports HIPAA workloads with a BAA, so the gateway can sit in front of regulated data from the start.
FAQ
Does an LLM gateway add latency?
Yes, a small amount, because each request makes an extra network hop. Respan's gateway typically adds 30-40ms. A cache hit skips the model call entirely, so for repeated requests the gateway makes responses faster, not slower.
Should I build or buy an LLM gateway?
Buy, for most teams. A proxy that forwards calls to providers is only a few hundred lines of code, but the work is everything around it: fallback rules for each provider's error types, cost attribution per customer, limits that block before the provider is called, logs you can query, and keeping up as provider APIs change. Respan covers that layer along with tracing and evals, so the time goes into your product instead. If you need the gateway inside your own network but don't want to build one, Respan's open-source gateway is a middle option you run yourself, with the hosted platform there when you want tracing and evals on that traffic.
Is there an open-source LLM gateway?
Yes. Respan's open-source gateway is MIT or Apache-2.0 licensed and runs as a single binary on your own machine or server. Any OpenAI, Anthropic, or Gemini SDK can call any model through it, routing, fallbacks, retries, and MCP are plugins you turn on in one config file, and the Respan plugin sends each request to the hosted platform as a trace when you want tracing and evals on top. It is currently pre-alpha. LiteLLM and Bifrost are other open-source gateways, and with any of them, running it means owning the deployment, upgrades, and on-call.
Do I need an LLM gateway if I already have LLM observability?
Usually, yes, once you're in production with more than one provider or real spend. Observability tools record what happened, while a gateway changes what happens next, by failing over, blocking a request at a spend cap, or serving a cached response. With Respan both run in the same system, so each routed call lands in your traces without a second integration.
How does an LLM gateway reduce costs?
A gateway reduces costs mainly by not calling the model when it doesn't have to and by stopping spend before it runs away. Cache hits skip the provider call, hard spend caps block requests past a budget, and per-customer attribution shows which feature to fix first. Task-aware routing can also send simpler steps to cheaper models, and the same gateway-level tactics show up in most guides to reduce OpenAI API costs.
