Fallbacks

When a call fails, the gateway tries the next attempt: another key or provider for the model, then the next model of your route.

Fallback happens at three levels:

• Keys and providers. A model name such as gpt-5.5 resolves to every offering of it on every key that can pay for it, your own keys first. If one attempt fails, the next is tried. You get this without setting anything.
• Models. A route you configure in the platform lists models to try in order. A fallback route is a Fallback node: the next entry runs only when the one before fails.
• Respan’s keys. With managed fallback, Respan’s keys take over when your own provider key is rate limited.

Create a fallback route once:

curl -X POST https://api.respan.ai/api/routes/ \
  -H "Authorization: Bearer $RESPAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "smart",
    "mode": "fallback",
    "entries": [
      {"name": "gpt-5.5"},
      {"name": "claude-sonnet-4-5-20250929"},
      {"name": "gemini-2.5-pro"}
    ]
  }'

Then send its name as model:

curl https://gateway.respan.ai/v1/chat/completions \
  -H "Authorization: Bearer $RESPAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "smart", "messages": [{"role": "user", "content": "Hello!"}]}'

import os
from openai import OpenAI

client = OpenAI(base_url="https://gateway.respan.ai/v1", api_key=os.environ["RESPAN_API_KEY"])

response = client.chat.completions.create(
    model="smart",
    messages=[{"role": "user", "content": "Hello!"}],
)

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://gateway.respan.ai/v1", apiKey: process.env.RESPAN_API_KEY });

const response = await client.chat.completions.create({
  model: "smart",
  messages: [{ role: "user", content: "Hello!" }],
});

A request for smart tries every attempt of gpt-5.5, then every attempt of claude-sonnet-4-5-20250929, then every attempt of gemini-2.5-pro. Your code doesn’t change when you change the route.

Settings

How failover works

1. Checks. The gateway authenticates the request and applies your limits and credit check. A request refused here never reaches a provider and doesn’t fall back.
2. Plan. model resolves to an ordered list of attempts, each one model offering on one key. A route contributes its entries in order; each model contributes its offerings, on your own keys first. See How attempts are built.
3. First attempt. The gateway sends it. If the provider answers with a success status, that answer is returned, streamed or not.
4. Next attempt. A rate limit, server error, timeout, or dropped connection moves to the next attempt: another of your keys, another provider of the model, then the next entry of the route.
5. Respan’s keys, if opted in. Attempts on Respan’s keys that stand in for your own key run only if you sent x-respan-managed-fallback-enabled: true and your key’s attempt answered 429.
6. Result. The first success is returned. If every attempt fails, you get the last error.

Two errors end the request at once, because no other attempt can fix them: input larger than the model’s context window, and 401 from one of your own provider keys. Once an answer has started streaming, it isn’t switched to another attempt. Each provider call has a fixed upstream timeout of 600 seconds.

Attempts for one model

Even without a route, one model name gives the gateway several attempts. For gpt-5.5 the order is:

1. Your own provider keys (BYOK) saved in the platform that serve the model, the model’s maker first (OpenAI before a reseller such as Azure OpenAI). Several keys for one provider are load balanced by weight.
2. Keys your workspace shares with the project.
3. Respan’s keys, on your credits, offering by offering: the model’s default provider first, then the rest from cheapest.

When any of your own keys can serve the model, step 3 is left out unless you opt in to managed fallback. See Provider order.

Fallback routes

A fallback route is a Fallback node: its entries are tried in order, and each entry expands into its own attempts as above. An entry can be a model name, a provider offering such as azure/gpt-5.5, or a Respan alias.

{
  "name": "smart",
  "mode": "fallback",
  "entries": [{"name": "azure/gpt-5.5"}, {"name": "gpt-5.5"}, {"name": "claude-sonnet-4-5-20250929"}]
}

• Weights are ignored in a fallback route.
• To pick by weight instead, use a split route, a LoadBalance node. See Load balancing.
• Change the chain with one PATCH to /api/routes/{name}/. No deploy is needed.

Creating, listing, and deleting routes is covered in Routes and aliases.

Managed fallback

When one of your own provider keys can serve a model, the gateway uses only your keys for it. To let Respan’s keys take over when your key is rate limited, send this header:

curl https://gateway.respan.ai/v1/chat/completions \
  -H "Authorization: Bearer $RESPAN_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-respan-managed-fallback-enabled: true" \
  -d '{"model": "gpt-5.5", "messages": [{"role": "user", "content": "Hello!"}]}'

• Respan’s keys are tried only after your key’s attempt answers 429. Any other error from your key doesn’t move the request to Respan’s keys.
• Calls on Respan’s keys use your credits, and only for offerings credits can pay for.
• Without the header, a request on your keys never moves to Respan’s keys.

What you’re billed

You pay for the attempt that answered, not the model you asked for: that offering, on the key that served it. A call on Respan’s key is charged to your credits from the provider’s reported usage. A call on your own key is billed by your provider, and is still logged and counted against your limits.

To see which attempt answered, open the request’s X-Respan-Log-Id in Gateway › Logs. The log names the model, provider, and deployment that served it, and earlier failed attempts are rows in the same trace.

Next steps

• Routes and aliases: create and manage routes.
• Load balancing: split traffic by weight, and nest it with fallbacks.
• Provider routing: how attempts are ordered.