OpenAI Chat Completions

Call any chat model in the catalog with the OpenAI Chat Completions format.

Chat Completions is the gateway’s main format. Send an OpenAI Chat Completions request for any chat model, from OpenAI, Anthropic, Google, or another provider, and get an OpenAI-shaped answer back. Every gateway feature works with it.

POST https://gateway.respan.ai/v1/chat/completions

curl https://gateway.respan.ai/v1/chat/completions \
  -H "Authorization: Bearer $RESPAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.5",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is an AI gateway?"}
    ]
  }'

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.respan.ai/v1",
    api_key=os.environ["RESPAN_API_KEY"],
)

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "What is an AI gateway?"},
    ],
)
print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.respan.ai/v1",
  apiKey: process.env.RESPAN_API_KEY,
});

const response = await client.chat.completions.create({
  model: "gpt-5.5",
  messages: [
    { role: "system", content: "You are a concise assistant." },
    { role: "user", content: "What is an AI gateway?" },
  ],
});
console.log(response.choices[0].message.content);

Authentication

Send your Respan API key as Authorization: Bearer $RESPAN_API_KEY. Your provider keys never go in the request; save them once with BYOK.

Request body

The body is OpenAI’s. model and messages are required. A field you don’t send is left out, so the provider’s own default applies. A field the answering provider has no equivalent for is dropped for that provider, and one it names differently is renamed.

Other providers’ models

Change model to call another provider’s model. The request is translated into that provider’s native API, and the answer comes back as an OpenAI chat.completion.

response = client.chat.completions.create(
    model="claude-sonnet-4-5-20250929",
    messages=[{"role": "user", "content": "What is an AI gateway?"}],
)

Gemini models are called on Gemini’s native API, not its OpenAI-compatible endpoint, so PDF and other file parts work. A {"type": "json_object"} response format without a schema can’t be expressed for Anthropic models and is refused with 400; send a json_schema instead.

Streaming

Set stream: true and the gateway returns the answer as server-sent events (SSE) as the model writes it. Streams come back in OpenAI’s chunk format, whichever provider answers, including a provider that only answers whole.

curl -N https://gateway.respan.ai/v1/chat/completions \
  -H "Authorization: Bearer $RESPAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.5",
    "messages": [{"role": "user", "content": "Write a haiku about routers."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

stream = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Write a haiku about routers."}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
    if chunk.usage:
        print("\n", chunk.usage)

const stream = await client.chat.completions.create({
  model: "gpt-5.5",
  messages: [{ role: "user", content: "Write a haiku about routers." }],
  stream: true,
  stream_options: { include_usage: true },
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  if (chunk.usage) console.log("\n", chunk.usage);
}

The X-Respan-Log-Id and X-Respan-Gateway-Request-Id headers are on the response head, so you have the log ID before the first token.

The other formats stream in their own vendor’s shape: see Responses, Anthropic Messages, and Gemini.

SSE basics

The response has Content-Type: text/event-stream. Each event is one or more field: value lines followed by a blank line. Most SDKs parse this for you. If you parse it yourself, read data: lines, and ignore lines that start with :, which are comments some servers send to keep the connection open.

Each event is a data: {chat.completion.chunk}, with text in choices[0].delta.content and tool calls in delta.tool_calls. The stream ends with data: [DONE]:

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","model":"gpt-5.5","choices":[{"index":0,"delta":{"content":"Packets"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","model":"gpt-5.5","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Usage in the final chunk

Send stream_options: {"include_usage": true} to get token usage. The gateway then sends one more chunk before [DONE], with an empty choices list and a usage object:

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","model":"gpt-5.5","choices":[],"usage":{"prompt_tokens":14,"completion_tokens":17,"total_tokens":31}}

data: [DONE]

The gateway always collects usage from the provider for billing and logging, whether or not you ask. Without include_usage, it doesn’t add usage for you, though some providers send their own usage on the last chunk anyway; don’t rely on that.

Errors before and during a stream

Where an error lands depends on whether the stream has started.

• Before the stream starts, you get a normal HTTP error status and error body, the same as a non-streamed request. Every gateway refusal (an invalid key, limits, budgets, credits) happens here, before any provider is called. Fallbacks also happen here: if the first attempt fails before sending anything, the next attempt is tried and you see one clean stream.
• During the stream, the status is already 200. If the provider breaks off, the gateway closes out anything the provider finished, then sends the error as the last event in the stream’s own format: a data: event holding {"error": {...}} on Chat Completions, an event: error on Anthropic Messages. A provider stream that stops before its end is reported this way with the code truncated, not as a normal finish.

data: {"error":{"message":"...","type":"...","code":"truncated"}}

So check each event for an error, not only the status code. A stream that ends with an error event did not finish, even if some content arrived.

Timeouts

The gateway waits up to 600 seconds for the provider. This limit is fixed and can’t be changed per request.

Cancellation and billing

To stop a stream, close the connection, for example by breaking out of the loop and closing the stream in your SDK. The gateway stops reading from the provider.

• What you’re billed: the usage the provider reports, or the tokens generated up to the point you left. Input tokens are always charged, because the provider already processed them.
• What’s logged: the call is logged with what was streamed before you left, and its usage and cost.
• Non-streamed requests run to completion even if your client disconnects, because the provider charges for them anyway. They are logged and billed in full.

Structured outputs

Set response_format to a JSON schema and the model answers with JSON that matches it. The gateway translates the schema into each provider’s structured output setting, so the same request works for OpenAI, Anthropic and Gemini models.

Pick a model: check a model’s supported features on the model catalog , or read Models.

curl https://gateway.respan.ai/v1/chat/completions \
  -H "Authorization: Bearer $RESPAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.5",
    "messages": [{"role": "user", "content": "Ada Lovelace was born in London in 1815."}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "person",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "birth_year": {"type": "integer"},
            "city": {"type": "string"}
          },
          "required": ["name", "birth_year", "city"],
          "additionalProperties": false
        }
      }
    }
  }'

import json

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Ada Lovelace was born in London in 1815."}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "person",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "birth_year": {"type": "integer"},
                    "city": {"type": "string"},
                },
                "required": ["name", "birth_year", "city"],
                "additionalProperties": False,
            },
        },
    },
)
print(json.loads(response.choices[0].message.content))

const response = await client.chat.completions.create({
  model: "gpt-5.5",
  messages: [{ role: "user", content: "Ada Lovelace was born in London in 1815." }],
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "person",
      strict: true,
      schema: {
        type: "object",
        properties: {
          name: { type: "string" },
          birth_year: { type: "integer" },
          city: { type: "string" },
        },
        required: ["name", "birth_year", "city"],
        additionalProperties: false,
      },
    },
  },
});
console.log(JSON.parse(response.choices[0].message.content!));

response_format values

json_schema takes name, schema, strict and an optional description.

Model support

Structured outputs depend on the provider:

Anthropic and Gemini always enforce the schema strictly and have no field for the wrapper’s description. So when the target is Anthropic or Gemini, a json_schema with a description or with strict: false is refused with 400 rather than sent with those settings dropped. For requests that may route to any provider, through a route with fallbacks for example, leave out description and keep strict: true (or unset).

Providers that serve OpenAI’s format may still differ in how much of JSON Schema they enforce. On a model without native schema support, the schema guides the answer but isn’t guaranteed.

Invalid JSON

The gateway returns the model’s answer as the provider sent it. It does not repair or re-validate JSON. An answer can still fail to parse when:

• the model stops at max_tokens (finish_reason: "length") before the JSON closes,
• the model refuses (message.refusal is set on OpenAI models),
• the model doesn’t enforce the schema natively, or you used json_object.

Parse the answer in a try block, check finish_reason, and retry or fall back on failure. When streaming, the JSON is only complete at the last chunk.

Pricing and logs

A structured output call is billed and logged as a normal chat call: one chat row per request, with the JSON answer as the reply’s content. To keep schema versions apart in Logs, tag calls with metadata, such as {"schema": "person_v2"}. See Respan params.

Tool calling

Tool calling lets a model ask for a function call instead of answering. You define the functions in the request’s tools field and run them in your own code: the model asks for a call, you run it and send back the result.

The gateway translates tool definitions, the model’s tool calls and your tool results for every chat provider, so the same request works for OpenAI, Anthropic and Gemini models.

Pick a model: check a model’s tool support on the model catalog , or read Models.

How a tool call works

A tool call takes two requests:

1. You send the messages and your tools. The model answers with tool_calls instead of text, and finish_reason is tool_calls.
2. You run each call, append the assistant message and one tool message per call (matched by tool_call_id), and send the conversation again. The model answers with text, or asks for more calls.

curl https://gateway.respan.ai/v1/chat/completions \
  -H "Authorization: Bearer $RESPAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.5",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

import json

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
messages = [{"role": "user", "content": "What is the weather in Paris?"}]

response = client.chat.completions.create(model="gpt-5.5", messages=messages, tools=tools)
message = response.choices[0].message

const tools: OpenAI.ChatCompletionTool[] = [{
  type: "function",
  function: {
    name: "get_weather",
    description: "Get the current weather for a city",
    parameters: {
      type: "object",
      properties: { city: { type: "string" } },
      required: ["city"],
    },
  },
}];
const messages: OpenAI.ChatCompletionMessageParam[] = [
  { role: "user", content: "What is the weather in Paris?" },
];

const response = await client.chat.completions.create({ model: "gpt-5.5", messages, tools });
const message = response.choices[0].message;

{
  "role": "assistant",
  "content": null,
  "tool_calls": [{
    "id": "call_abc123",
    "type": "function",
    "function": { "name": "get_weather", "arguments": "{\"city\": \"Paris\"}" }
  }]
}

call = message.tool_calls[0]
result = {"temp_c": 18, "sky": "clear"}  # your function's output

messages.append(message)
messages.append({
    "role": "tool",
    "tool_call_id": call.id,
    "content": json.dumps(result),
})

final = client.chat.completions.create(model="gpt-5.5", messages=messages, tools=tools)
print(final.choices[0].message.content)

Agent loop

An agent repeats those two steps until the model answers with text. Cap the number of turns so a model that keeps calling tools can’t run forever.

TOOLS = {"get_weather": lambda city: {"temp_c": 18, "sky": "clear"}}

def run_agent(messages, max_turns=10):
    for _ in range(max_turns):
        response = client.chat.completions.create(
            model="gpt-5.5",
            messages=messages,
            tools=tools,
            extra_body={"thread_identifier": "agent_run_42"},
        )
        message = response.choices[0].message
        messages.append(message)

        if not message.tool_calls:
            return message.content

        for call in message.tool_calls:
            args = json.loads(call.function.arguments)
            output = TOOLS[call.function.name](**args)
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": json.dumps(output),
            })
    raise RuntimeError("Agent did not finish within max_turns")

print(run_agent([{"role": "user", "content": "Is it warmer in Paris or Rome?"}]))

The model may ask for several calls in one turn (parallel tool calls). Answer every call before you send the next request; a missing tool message is refused by most providers.

Each turn of the loop is its own request and its own trace. Pass the same thread_identifier (as above) to see all turns together in Logs, or see Threads and traces to nest them in one trace.

tool_choice

tool_choice is translated where the target provider has a form for it.

Routing by tool support

When a request can go to several models, such as through a route with fallbacks, the gateway drops the candidates that can’t meet it: models without tool support when you send tools or tool_choice, and models that can’t force a call when tool_choice is required or names a function. The first model is always tried as you named it.

Pricing and logs

• Each request is billed and logged as a chat call: one chat row per request, with your tools, the model’s tool_calls and your tool messages in its messages.
• Tool definitions and results count as input tokens, so long tool lists and large results raise the cost of every turn.

Prompt caching

A cache_control marker on a message part reaches Anthropic models, as in Anthropic’s own API. Cached tokens are priced at the provider’s cache rates. See Prompt caching.

Respan params

Respan params go at the top level of the body (with the OpenAI SDK, in extra_body), in a respan_params object, or in the X-Data-Respan-Params header as base64 JSON.

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={
        "customer_identifier": "user_8842",
        "metadata": {"feature": "support-chat"},
        "thread_identifier": "ticket_1042",
    },
)

On Chat Completions, metadata holds your own properties, so the gateway doesn’t read a respan_params key inside it. See Respan params.

Response

The body is an OpenAI chat.completion, whichever provider answered. model names the model that answered, which can differ from the one you sent when you call a route. usage holds the provider’s token counts, including cached and reasoning tokens where the provider reports them.

Errors

A refusal from the gateway itself, such as an invalid key or a spend limit, comes back as an OpenAI-style error with its status and code. A provider error is passed through with its status, except that a provider refusing Respan’s own key is answered in Respan’s words (a provider 401 on Respan’s key becomes a 502).

For an error that arrives after a stream has started, see Errors before and during a stream.

Next steps

• Threads and traces: group an agent’s turns.
• Usage accounting.
• Fallbacks.
• Responses, if your code uses OpenAI’s Responses API.