> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://respan.ai/docs/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://respan.ai/docs/_mcp/server.

# Load balancing

Load balancing decides where each request goes before it's sent. Use it to spread traffic across several keys for one provider, for example to stay under each key's rate limit, or to split traffic between models.

## Balance across your provider keys

Each provider key you add is a deployment. When a provider has more than one active deployment, Respan picks one for each request, weighted by its **Load balance weight**.

#### Open the provider

Go to **Gateway** › **API keys**, click **Bring your own key**, and click the provider, for example **OpenAI**. See [Provider keys](/docs/documentation/admin/llm-provider-keys) for adding the first key.

#### Add or open a deployment

Click **Add deployment** and confirm with **I understand** to add another key, or click an existing deployment to edit it.

#### Set the weight

Enter a whole number in **Load balance weight**. The hint under it shows this deployment's share of traffic. Click **Save**.

![An OpenAI deployment named Acme primary with Load balance weight 3 and a hint of 75%, above a second deployment, Acme overflow, at 25%](/docs/_fern-img/c29f53c4e0363bfe1498e1f58f7d03f3d76cc0d401c8703ae78db3dbe6d60595.webp)![An OpenAI deployment named Acme primary with Load balance weight 3 and a hint of 75%, above a second deployment, Acme overflow, at 25%](/docs/_fern-img/c110b86bc4efab75901e4bd511cea2b487dc68e9aa293dd911ab70aeb7d645c4.webp)

A deployment's share is its weight divided by the total weight of the provider's active deployments. Turn off **Active**, or set the weight to 0, to stop sending it traffic.

For Azure OpenAI, Vertex AI, and Bedrock deployments, **Advanced options** has **Available models** and **Exclude models**, so a deployment only serves the models it has.

## Balance across models

Send `load_balance_models` to split requests between models. Respan picks one model per request by weight and uses it in place of `model`. Each item needs `model` and `weight`.

#### OpenAI (Python)

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.respan.ai/api/",
    api_key=os.environ["RESPAN_API_KEY"],
)

response = client.chat.completions.create(
    model="gpt-5.5",  # replaced by the model Respan picks
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={
        "load_balance_models": [
            {"model": "gpt-5.5", "weight": 3},
            {"model": "claude-sonnet-4-5-20250929", "weight": 1},
        ]
    },
)
```

#### cURL

```bash
curl https://api.respan.ai/api/chat/completions \
  -H "Authorization: Bearer $RESPAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.5",
    "messages": [{"role": "user", "content": "Hello!"}],
    "load_balance_models": [
      {"model": "gpt-5.5", "weight": 3},
      {"model": "claude-sonnet-4-5-20250929", "weight": 1}
    ]
  }'
```

An item can also carry `credentials` for that model only, such as an Azure deployment's `api_key`, `api_base`, and `api_version`. The picked model then follows your [retry and fallback](/docs/documentation/features/gateway/retries) settings, and its log shows which model served the request.

If your organization already has a saved load balancing group, send `load_balance_group: {"group_id": "YOUR_GROUP_ID"}` instead. It takes precedence over `load_balance_models`.