--- title: "OpenAI Chat Completions" subtitle: "Call any chat model in the catalog with the OpenAI Chat Completions format." --- Chat Completions is the gateway's main format. Send an OpenAI Chat Completions request for any chat model, from OpenAI, Anthropic, Google, or another provider, and get an OpenAI-shaped answer back. Every gateway feature works with it. `POST https://gateway.respan.ai/v1/chat/completions` ```bash curl https://gateway.respan.ai/v1/chat/completions \ -H "Authorization: Bearer $RESPAN_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-5.5", "messages": [ {"role": "system", "content": "You are a concise assistant."}, {"role": "user", "content": "What is an AI gateway?"} ] }' ``` ```python import os from openai import OpenAI client = OpenAI( base_url="https://gateway.respan.ai/v1", api_key=os.environ["RESPAN_API_KEY"], ) response = client.chat.completions.create( model="gpt-5.5", messages=[ {"role": "system", "content": "You are a concise assistant."}, {"role": "user", "content": "What is an AI gateway?"}, ], ) print(response.choices[0].message.content) ``` ```typescript import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://gateway.respan.ai/v1", apiKey: process.env.RESPAN_API_KEY, }); const response = await client.chat.completions.create({ model: "gpt-5.5", messages: [ { role: "system", content: "You are a concise assistant." }, { role: "user", content: "What is an AI gateway?" }, ], }); console.log(response.choices[0].message.content); ``` ## Authentication Send your Respan API key as `Authorization: Bearer $RESPAN_API_KEY`. Your provider keys never go in the request; save them once with [BYOK](/docs/ai-gateway/spend/credits-vs-byok#byok). ## Request body The body is OpenAI's. `model` and `messages` are required. A field you don't send is left out, so the provider's own default applies. A field the answering provider has no equivalent for is dropped for that provider, and one it names differently is renamed. | Field | Type | What the gateway does | |---|---|---| | `model` | string, required | A model ID from the [catalog](https://www.respan.ai/models), a provider offering such as `azure/gpt-5.5`, or a [route](/docs/ai-gateway/routing/routes-and-aliases). The response's `model` names the model that answered. | | `messages` | array, required | `system`, `user`, `assistant`, and `tool` messages. Content parts `text`, `image_url`, `file`, and `input_audio` are accepted where the model takes them. See [Vision and files](/docs/ai-gateway/modalities/vision-and-files). | | `stream`, `stream_options` | boolean, object | Server-sent events in OpenAI's chunk format, whichever provider answers. `include_usage` adds a final usage chunk. See [Streaming](#streaming). | | `temperature`, `top_p` | number | Sent to every provider. Dropped for models that refuse them, such as OpenAI's gpt-5.4 and later. | | `max_tokens`, `max_completion_tokens` | integer | Either name works. Sent as `max_tokens` to Anthropic and `max_output_tokens` to Gemini. | | `stop` | string or array | Sent as `stop_sequences` to Anthropic and Gemini. Dropped for OpenAI reasoning models. | | `n` | integer | Sent as `candidate_count` to Gemini, dropped for Anthropic. Each choice's output is billed. | | `seed` | integer | OpenAI-compatible providers. Dropped for Anthropic. | | `presence_penalty`, `frequency_penalty` | number | OpenAI-compatible providers (not reasoning models) and Gemini. Dropped for Anthropic. | | `logprobs`, `top_logprobs` | boolean, integer | OpenAI-compatible providers and Gemini. Dropped for Anthropic. | | `logit_bias` | object | OpenAI-compatible providers (not reasoning models). Dropped for Anthropic and Gemini. | | `tools`, `tool_choice`, `parallel_tool_calls` | array, string or object, boolean | Translated for every provider. `parallel_tool_calls: false` becomes `disable_parallel_tool_use` for Anthropic. See [Tool calling](#tool-calling). | | `response_format` | object | `json_schema` works on every provider with structured output. `json_object` without a schema is refused with `400` for Anthropic models. See [Structured outputs](#structured-outputs). | | `reasoning_effort` | string | Mapped to each provider's setting: thinking for Claude and Gemini, `thinking` for DeepSeek. Dropped for non-reasoning models. Reasoning tokens bill as output. | | `user` | string | Sent to OpenAI-compatible providers, as `metadata.user_id` to Anthropic, dropped for Gemini. To track a customer in Respan, use `customer_identifier`. | | `safety_identifier` | string | OpenAI's end-user ID for abuse checks. See [Safety identifiers](/docs/ai-gateway/security/safety-identifiers). | | `service_tier`, `prompt_cache_key`, `verbosity`, `modalities`, `web_search_options` | | Reach OpenAI models unchanged. Translated where another provider has an equivalent (`web_search_options` becomes Anthropic's web search tool), dropped where it doesn't. The `service_tier` that served the call is logged and priced. | | Respan params | | Customer, thread, trace, metadata, and keys. Never sent to the provider. See [Respan params](#respan-params). | {/* REVIEW: confirm seed and top_logprobs reach Gemini models on gateway-next. */} ## Other providers' models Change `model` to call another provider's model. The request is translated into that provider's native API, and the answer comes back as an OpenAI `chat.completion`. ```python response = client.chat.completions.create( model="claude-sonnet-4-5-20250929", messages=[{"role": "user", "content": "What is an AI gateway?"}], ) ``` Gemini models are called on Gemini's native API, not its OpenAI-compatible endpoint, so PDF and other file parts work. A `{"type": "json_object"}` response format without a schema can't be expressed for Anthropic models and is refused with `400`; send a `json_schema` instead. ## Streaming Set `stream: true` and the gateway returns the answer as server-sent events (SSE) as the model writes it. Streams come back in OpenAI's chunk format, whichever provider answers, including a provider that only answers whole. ```bash curl -N https://gateway.respan.ai/v1/chat/completions \ -H "Authorization: Bearer $RESPAN_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-5.5", "messages": [{"role": "user", "content": "Write a haiku about routers."}], "stream": true, "stream_options": {"include_usage": true} }' ``` ```python stream = client.chat.completions.create( model="gpt-5.5", messages=[{"role": "user", "content": "Write a haiku about routers."}], stream=True, stream_options={"include_usage": True}, ) for chunk in stream: if chunk.choices and chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="") if chunk.usage: print("\n", chunk.usage) ``` ```typescript const stream = await client.chat.completions.create({ model: "gpt-5.5", messages: [{ role: "user", content: "Write a haiku about routers." }], stream: true, stream_options: { include_usage: true }, }); for await (const chunk of stream) { process.stdout.write(chunk.choices[0]?.delta?.content ?? ""); if (chunk.usage) console.log("\n", chunk.usage); } ``` The `X-Respan-Log-Id` and `X-Respan-Gateway-Request-Id` headers are on the response head, so you have the log ID before the first token. The other formats stream in their own vendor's shape: see [Responses](/docs/ai-gateway/requests/responses#streaming), [Anthropic Messages](/docs/ai-gateway/requests/anthropic-messages#streaming), and [Gemini](/docs/ai-gateway/requests/gemini#streaming). ### SSE basics The response has `Content-Type: text/event-stream`. Each event is one or more `field: value` lines followed by a blank line. Most SDKs parse this for you. If you parse it yourself, read `data:` lines, and ignore lines that start with `:`, which are comments some servers send to keep the connection open. Each event is a `data: {chat.completion.chunk}`, with text in `choices[0].delta.content` and tool calls in `delta.tool_calls`. The stream ends with `data: [DONE]`: ```text data: {"id":"chatcmpl-123","object":"chat.completion.chunk","model":"gpt-5.5","choices":[{"index":0,"delta":{"content":"Packets"},"finish_reason":null}]} data: {"id":"chatcmpl-123","object":"chat.completion.chunk","model":"gpt-5.5","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} data: [DONE] ``` ### Usage in the final chunk Send `stream_options: {"include_usage": true}` to get token usage. The gateway then sends one more chunk before `[DONE]`, with an empty `choices` list and a `usage` object: ```text data: {"id":"chatcmpl-123","object":"chat.completion.chunk","model":"gpt-5.5","choices":[],"usage":{"prompt_tokens":14,"completion_tokens":17,"total_tokens":31}} data: [DONE] ``` The gateway always collects usage from the provider for billing and logging, whether or not you ask. Without `include_usage`, it doesn't add usage for you, though some providers send their own usage on the last chunk anyway; don't rely on that. ### Errors before and during a stream Where an error lands depends on whether the stream has started. - **Before the stream starts,** you get a normal HTTP error status and error body, the same as a non-streamed request. Every gateway refusal (an invalid key, limits, budgets, credits) happens here, before any provider is called. Fallbacks also happen here: if the first attempt fails before sending anything, the next attempt is tried and you see one clean stream. - **During the stream,** the status is already `200`. If the provider breaks off, the gateway closes out anything the provider finished, then sends the error as the last event in the stream's own format: a `data:` event holding `{"error": {...}}` on Chat Completions, an `event: error` on Anthropic Messages. A provider stream that stops before its end is reported this way with the code `truncated`, not as a normal finish. ```text data: {"error":{"message":"...","type":"...","code":"truncated"}} ``` So check each event for an error, not only the status code. A stream that ends with an error event did not finish, even if some content arrived. ### Timeouts The gateway waits up to 600 seconds for the provider. This limit is fixed and can't be changed per request. ### Cancellation and billing To stop a stream, close the connection, for example by breaking out of the loop and closing the stream in your SDK. The gateway stops reading from the provider. - **What you're billed:** the usage the provider reports, or the tokens generated up to the point you left. Input tokens are always charged, because the provider already processed them. - **What's logged:** the call is logged with what was streamed before you left, and its usage and cost. - **Non-streamed requests** run to completion even if your client disconnects, because the provider charges for them anyway. They are logged and billed in full. ## Structured outputs Set `response_format` to a JSON schema and the model answers with JSON that matches it. The gateway translates the schema into each provider's structured output setting, so the same request works for OpenAI, Anthropic and Gemini models. **Pick a model:** check a model's supported features on the [model catalog](https://www.respan.ai/models), or read [Models](/docs/ai-gateway/models/overview). ```bash curl https://gateway.respan.ai/v1/chat/completions \ -H "Authorization: Bearer $RESPAN_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-5.5", "messages": [{"role": "user", "content": "Ada Lovelace was born in London in 1815."}], "response_format": { "type": "json_schema", "json_schema": { "name": "person", "strict": true, "schema": { "type": "object", "properties": { "name": {"type": "string"}, "birth_year": {"type": "integer"}, "city": {"type": "string"} }, "required": ["name", "birth_year", "city"], "additionalProperties": false } } } }' ``` ```python import json response = client.chat.completions.create( model="gpt-5.5", messages=[{"role": "user", "content": "Ada Lovelace was born in London in 1815."}], response_format={ "type": "json_schema", "json_schema": { "name": "person", "strict": True, "schema": { "type": "object", "properties": { "name": {"type": "string"}, "birth_year": {"type": "integer"}, "city": {"type": "string"}, }, "required": ["name", "birth_year", "city"], "additionalProperties": False, }, }, }, ) print(json.loads(response.choices[0].message.content)) ``` ```typescript const response = await client.chat.completions.create({ model: "gpt-5.5", messages: [{ role: "user", content: "Ada Lovelace was born in London in 1815." }], response_format: { type: "json_schema", json_schema: { name: "person", strict: true, schema: { type: "object", properties: { name: { type: "string" }, birth_year: { type: "integer" }, city: { type: "string" }, }, required: ["name", "birth_year", "city"], additionalProperties: false, }, }, }, }); console.log(JSON.parse(response.choices[0].message.content!)); ``` ### response_format values | Value | What it does | | --- | --- | | `{"type": "text"}` | Plain text. The default. | | `{"type": "json_object"}` | Any valid JSON, without a schema. Tell the model to answer in JSON in your prompt too. | | `{"type": "json_schema", "json_schema": {...}}` | JSON that matches `json_schema.schema`. | `json_schema` takes `name`, `schema`, `strict` and an optional `description`. ### Model support Structured outputs depend on the provider: | Target | `json_schema` | `json_object` | | --- | --- | --- | | OpenAI-format providers | Sent as is | Sent as is | | Anthropic | Sent as `output_config.format` | Refused (`400`): supply a schema | | Gemini | Sent as `responseJsonSchema` with `responseMimeType: application/json` | Sent as `responseMimeType: application/json` | Anthropic and Gemini always enforce the schema strictly and have no field for the wrapper's description. So when the target is Anthropic or Gemini, a `json_schema` with a `description` or with `strict: false` is refused with `400` rather than sent with those settings dropped. For requests that may route to any provider, through a [route](/docs/ai-gateway/routing/routes-and-aliases) with fallbacks for example, leave out `description` and keep `strict: true` (or unset). Providers that serve OpenAI's format may still differ in how much of JSON Schema they enforce. On a model without native schema support, the schema guides the answer but isn't guaranteed. {/* REVIEW: translation is from the OSS gateway's Anthropic and Gemini formats; confirm gateway-next behaves the same and how response_format is handled for other non-OpenAI formats. */} ### Invalid JSON The gateway returns the model's answer as the provider sent it. It does not repair or re-validate JSON. An answer can still fail to parse when: - the model stops at `max_tokens` (`finish_reason: "length"`) before the JSON closes, - the model refuses (`message.refusal` is set on OpenAI models), - the model doesn't enforce the schema natively, or you used `json_object`. Parse the answer in a `try` block, check `finish_reason`, and retry or fall back on failure. When streaming, the JSON is only complete at the last chunk. {/* REVIEW: confirm the gateway does no JSON repair (no response-healing equivalent) on chat answers. */} ### Pricing and logs A structured output call is billed and logged as a normal chat call: one `chat` row per request, with the JSON answer as the reply's content. To keep schema versions apart in Logs, tag calls with `metadata`, such as `{"schema": "person_v2"}`. See [Respan params](#respan-params). ## Tool calling Tool calling lets a model ask for a function call instead of answering. You define the functions in the request's `tools` field and run them in your own code: the model asks for a call, you run it and send back the result. The gateway translates tool definitions, the model's tool calls and your tool results for every chat provider, so the same request works for OpenAI, Anthropic and Gemini models. **Pick a model:** check a model's tool support on the [model catalog](https://www.respan.ai/models), or read [Models](/docs/ai-gateway/models/overview). ### How a tool call works A tool call takes two requests: 1. You send the messages and your `tools`. The model answers with `tool_calls` instead of text, and `finish_reason` is `tool_calls`. 2. You run each call, append the assistant message and one `tool` message per call (matched by `tool_call_id`), and send the conversation again. The model answers with text, or asks for more calls. ```bash curl https://gateway.respan.ai/v1/chat/completions \ -H "Authorization: Bearer $RESPAN_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-5.5", "messages": [{"role": "user", "content": "What is the weather in Paris?"}], "tools": [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } } }], "tool_choice": "auto" }' ``` ```python import json tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] messages = [{"role": "user", "content": "What is the weather in Paris?"}] response = client.chat.completions.create(model="gpt-5.5", messages=messages, tools=tools) message = response.choices[0].message ``` ```typescript const tools: OpenAI.ChatCompletionTool[] = [{ type: "function", function: { name: "get_weather", description: "Get the current weather for a city", parameters: { type: "object", properties: { city: { type: "string" } }, required: ["city"], }, }, }]; const messages: OpenAI.ChatCompletionMessageParam[] = [ { role: "user", content: "What is the weather in Paris?" }, ]; const response = await client.chat.completions.create({ model: "gpt-5.5", messages, tools }); const message = response.choices[0].message; ``` The answer carries the call instead of text: ```json { "role": "assistant", "content": null, "tool_calls": [{ "id": "call_abc123", "type": "function", "function": { "name": "get_weather", "arguments": "{\"city\": \"Paris\"}" } }] } ``` `arguments` is a JSON string. Parse it before you use it, and validate it: the model can produce arguments that don't match your schema. ```python call = message.tool_calls[0] result = {"temp_c": 18, "sky": "clear"} # your function's output messages.append(message) messages.append({ "role": "tool", "tool_call_id": call.id, "content": json.dumps(result), }) final = client.chat.completions.create(model="gpt-5.5", messages=messages, tools=tools) print(final.choices[0].message.content) ``` Send `tools` again on every request in the conversation, so the model can read earlier calls and ask for more. ### Agent loop An agent repeats those two steps until the model answers with text. Cap the number of turns so a model that keeps calling tools can't run forever. ```python TOOLS = {"get_weather": lambda city: {"temp_c": 18, "sky": "clear"}} def run_agent(messages, max_turns=10): for _ in range(max_turns): response = client.chat.completions.create( model="gpt-5.5", messages=messages, tools=tools, extra_body={"thread_identifier": "agent_run_42"}, ) message = response.choices[0].message messages.append(message) if not message.tool_calls: return message.content for call in message.tool_calls: args = json.loads(call.function.arguments) output = TOOLS[call.function.name](**args) messages.append({ "role": "tool", "tool_call_id": call.id, "content": json.dumps(output), }) raise RuntimeError("Agent did not finish within max_turns") print(run_agent([{"role": "user", "content": "Is it warmer in Paris or Rome?"}])) ``` The model may ask for several calls in one turn (parallel tool calls). Answer every call before you send the next request; a missing `tool` message is refused by most providers. Each turn of the loop is its own request and its own trace. Pass the same `thread_identifier` (as above) to see all turns together in Logs, or see [Threads and traces](/docs/ai-gateway/customers/threads-and-traces) to nest them in one trace. ### tool_choice | Value | What it does | | --- | --- | | `"auto"` | The model decides. The default when `tools` is set. | | `"none"` | The model answers with text. | | `"required"` | The model must call at least one tool. | | `{"type": "function", "function": {"name": "get_weather"}}` | The model must call that tool. | `tool_choice` is translated where the target provider has a form for it. ### Routing by tool support When a request can go to several models, such as through a [route](/docs/ai-gateway/routing/routes-and-aliases) with [fallbacks](/docs/ai-gateway/routing/fallbacks), the gateway drops the candidates that can't meet it: models without tool support when you send `tools` or `tool_choice`, and models that can't force a call when `tool_choice` is `required` or names a function. The first model is always tried as you named it. {/* REVIEW: this capability filtering comes from the old gateway; confirm gateway-next filters a route's attempts by tool support. */} ### Pricing and logs - Each request is billed and logged as a chat call: one `chat` row per request, with your `tools`, the model's `tool_calls` and your `tool` messages in its messages. - Tool definitions and results count as input tokens, so long tool lists and large results raise the cost of every turn. ## Prompt caching A `cache_control` marker on a message part reaches Anthropic models, as in Anthropic's own API. Cached tokens are priced at the provider's cache rates. See [Prompt caching](/docs/ai-gateway/features/caching#prompt-caching). ## Respan params Respan params go at the top level of the body (with the OpenAI SDK, in `extra_body`), in a `respan_params` object, or in the `X-Data-Respan-Params` header as base64 JSON. ```python response = client.chat.completions.create( model="gpt-5.5", messages=[{"role": "user", "content": "Hello!"}], extra_body={ "customer_identifier": "user_8842", "metadata": {"feature": "support-chat"}, "thread_identifier": "ticket_1042", }, ) ``` On Chat Completions, `metadata` holds your own properties, so the gateway doesn't read a `respan_params` key inside it. See [Respan params](/docs/ai-gateway/customers/respan-params). ## Response The body is an OpenAI `chat.completion`, whichever provider answered. `model` names the model that answered, which can differ from the one you sent when you call a [route](/docs/ai-gateway/routing/routes-and-aliases). `usage` holds the provider's token counts, including cached and reasoning tokens where the provider reports them. | Header | Description | |---|---| | `X-Respan-Log-Id` | The ID of this request's log. | | `X-Respan-Gateway-Request-Id` | The ID of the whole request, on every log row it writes. | ## Errors A refusal from the gateway itself, such as an invalid key or a spend limit, comes back as an OpenAI-style error with its status and `code`. A provider error is passed through with its status, except that a provider refusing Respan's own key is answered in Respan's words (a provider `401` on Respan's key becomes a `502`). For an error that arrives after a stream has started, see [Errors before and during a stream](#stream-errors). ## Next steps - [Threads and traces](/docs/ai-gateway/customers/threads-and-traces): group an agent's turns. - [Usage accounting](/docs/ai-gateway/spend/usage-accounting). - [Fallbacks](/docs/ai-gateway/routing/fallbacks). - [Responses](/docs/ai-gateway/requests/responses), if your code uses OpenAI's Responses API.