---
title: "Load balancing"
subtitle: "Split traffic across your own provider keys, or across models, by weight."
---

Load balancing decides which key or model a request tries first. Use key weights to spread traffic across several keys for one provider, for example to stay under each key's rate limit. Use a split route to divide traffic between models, for example to send a quarter of requests to a model you're evaluating.

Both work the same way. A **LoadBalance** node picks one member by weight. If that member fails, it picks again from the remaining members, by their weights, until one answers or none are left.

| What to balance | Where you set it | How a request picks |
|---|---|---|
| [Your provider keys](#balance-across-your-own-keys) | **Load balance weight** on each deployment | One key per request by weight; on failure, the next pick from the rest. |
| [Models](#balance-across-models) | A route with `mode: "split"` | One entry per request by weight; on failure, the next pick from the rest. |

## Balance across your own keys

Each provider key you add is a deployment. When more than one of your active deployments can serve a model, the gateway picks one per request by its weight.

<Steps>
<Step title="Open the provider">

Go to **Gateway** › **API keys**, click **Bring your own key**, and click the provider, for example **OpenAI**. See [Bring your own key](/docs/ai-gateway/spend/credits-vs-byok#byok) for adding the first key.

</Step>
<Step title="Add or open a deployment">

Click **Add deployment** and confirm with **I understand** to add another key, or click an existing deployment to edit it.

</Step>
<Step title="Set the weight">

Enter a whole number in **Load balance weight** and click **Save**. The hint under it shows this deployment's share of traffic.

<img src={"/docs/images/routing/deployment-weight-light.png"} width="480" className="block dark:hidden" alt="An OpenAI deployment named Acme primary with Load balance weight 3 and a hint of 75%, above a second deployment, Acme overflow, at 25%" />
<img src={"/docs/images/routing/deployment-weight-dark.png"} width="480" className="hidden dark:block" alt="An OpenAI deployment named Acme primary with Load balance weight 3 and a hint of 75%, above a second deployment, Acme overflow, at 25%" />

</Step>
</Steps>

- A deployment's share is its weight divided by the total weight of the deployments that serve the model. The default weight is 1.
- Turn off **Active**, or set the weight to 0, to stop sending it traffic.
- If the picked key fails, the gateway picks again from the remaining keys by weight, so every key gets a turn before the request moves to another provider or model.
- For deployments such as Azure OpenAI, **Advanced options** has **Available models** and **Exclude models**, so a deployment only serves the models it has.

Weights split traffic within one pool. Your project's own keys always come before keys your workspace shares, and both come before Respan's keys. If your keys are rate limited, [managed fallback](/docs/ai-gateway/routing/fallbacks#managed-fallback) lets Respan's keys take over. See [Provider order](/docs/ai-gateway/routing/provider-routing#provider-order).

## Balance across models

Create a route with `mode: "split"`. It's a LoadBalance node over its entries: each request picks one entry by weight, and if it fails, the gateway picks again from the rest.

```bash
curl -X POST https://api.respan.ai/api/routes/ \
  -H "Authorization: Bearer $RESPAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "chat-split",
    "mode": "split",
    "entries": [
      {"name": "gpt-5.5", "weight": 3},
      {"name": "claude-sonnet-4-5-20250929", "weight": 1}
    ]
  }'
```

Then send the route's name as `model`:

<Tabs>
<Tab title="cURL">
```bash
curl https://gateway.respan.ai/v1/chat/completions \
  -H "Authorization: Bearer $RESPAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "chat-split", "messages": [{"role": "user", "content": "Hello!"}]}'
```
</Tab>
<Tab title="Python">
```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://gateway.respan.ai/v1", api_key=os.environ["RESPAN_API_KEY"])

response = client.chat.completions.create(
    model="chat-split",
    messages=[{"role": "user", "content": "Hello!"}],
)
```
</Tab>
<Tab title="TypeScript">
```typescript
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://gateway.respan.ai/v1", apiKey: process.env.RESPAN_API_KEY });

const response = await client.chat.completions.create({
  model: "chat-split",
  messages: [{ role: "user", content: "Hello!" }],
});
```
</Tab>
</Tabs>

About 75% of requests start on `gpt-5.5` and 25% on `claude-sonnet-4-5-20250929`. If the first pick fails, the other entry runs.

- A split route needs a positive total weight. An entry with weight 0 never runs.
- To change the split, send `PATCH /api/routes/chat-split/` with new `entries`. No deploy is needed.
- To compare the models' quality and cost after a split, filter **Gateway** › **Logs** by model.

Routes are covered in full in [Routes and aliases](/docs/ai-gateway/routing/routes-and-aliases).

## Nest load balancing and fallback

A request's plan is a tree of Fallback nodes (try in order) and LoadBalance nodes (pick by weight, on failure pick again from the rest). The gateway builds it from your route and your keys:

- A **split route** is a LoadBalance node. Each entry it picks is a model, and a model is a Fallback node over its offerings and keys. So when `gpt-5.5` is picked, all of its attempts run before the split moves on to `claude-sonnet-4-5-20250929`.
- A **fallback route** is a Fallback node. Each model entry in it includes a LoadBalance node across your own keys for that provider, picked by **Load balance weight**.
- Either route can list a [Respan alias](/docs/ai-gateway/routing/routes-and-aliases#respan-aliases) as an entry, which brings in that alias's own node.

For a split route `chat-split` with two OpenAI keys weighted 3 and 1, a request runs as `LoadBalance[ Fallback[ LoadBalance[key A, key B], ... ], claude-sonnet-4-5-20250929 ]`: first the split picks a model, then that model's own keys are picked by weight.

{/* REVIEW: the brief describes routes nesting freely, such as Fallback[LoadBalance[a, b], c]. In respan-backend origin/main (routing/serializers.py, rust/crates/model-resolver/src/plan.rs, Oct 27) a project route's entries may name models, provider offerings and Respan routes only, not another project route, so a user can't build that shape from two of their own routes. Confirm whether route-in-route nesting (or a tree editor in the platform) ships before documenting that example. */}

## See where a request went

Open the request's `X-Respan-Log-Id` in **Gateway** › **Logs**. The log names the model, the provider, and the deployment that served it. To see every request one key served, filter Logs by **Deployment**.

## Next steps

- [Routes and aliases](/docs/ai-gateway/routing/routes-and-aliases): manage split and fallback routes.
- [Fallbacks](/docs/ai-gateway/routing/fallbacks): what happens when the picked key or model fails.
- [Bring your own key](/docs/ai-gateway/spend/credits-vs-byok#byok): add provider keys.
- [Provider routing](/docs/ai-gateway/routing/provider-routing)