Providers and models

Add model providers with a key, point the gateway at any compatible endpoint, and see how a model name picks a provider.

A provider is where the gateway sends model calls. It is either a built-in vendor plus a key, or any endpoint that speaks a supported format.

[providers.openai]
vendor = "openai"
key = "env:OPENAI_API_KEY"

[providers.anthropic]
vendor = "anthropic"
key = "env:ANTHROPIC_API_KEY"

The table name (openai, anthropic) is the provider’s name. You use it in routes and in provider/model names. Write keys as env:NAME to keep them out of the file; a literal string also works.

Built-in vendors

A vendor already knows its base URL, how it takes a key, and which formats it speaks.

OpenAI, Anthropic, Gemini, Respan and TypeSafe are verified end to end against the live API. The others are maintained but not yet verified that way. azure and vllm need a base_url. cloudflare needs CLOUDFLARE_ACCOUNT_ID or a base_url.

No config file

With no gateway.toml, every vendor whose key is set in the environment becomes a provider, named after the vendor:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
respan-gateway

Ollama is added if OLLAMA_HOST is set. RESPAN_<VENDOR>_BASE_URL moves a vendor’s base URL, for example RESPAN_OPENAI_BASE_URL=https://my-proxy.example/v1. Both only apply without a config file. With one, only the file’s base_url counts. respan-gateway init writes a config file from what it finds in the environment.

Base URLs

base_url replaces a vendor’s URL. It is the URL the vendor’s own SDK takes: OpenAI-compatible ones include the version (https://api.openai.com/v1), Anthropic’s does not (https://api.anthropic.com).

[providers.openai]
vendor = "openai"
key = "env:OPENAI_API_KEY"
base_url = "https://my-proxy.example/v1"

A vendor whose formats sit at different URLs, such as Gemini, moves them with a table of formats, one base URL per format:

[providers.gemini]
vendor = "gemini"
key = "env:GEMINI_API_KEY"
formats = { gemini-generate = "https://gemini.internal/v1beta", openai-chat = "https://gemini.internal/v1beta/openai" }

Only the formats in the table are served. A single base_url on such a vendor is refused, since it can’t say where each format is.

OpenAI-compatible endpoints

Anything that speaks a supported format is a provider. Give it formats and a base_url instead of a vendor:

[providers.local]
formats = ["openai-chat"]
base_url = "http://localhost:8000/v1"
key = "env:LOCAL_KEY"          # optional

formats is one format name or a list. The chat formats are openai-chat, openai-responses, anthropic-messages and gemini-generate; see API formats for the rest. key is sent the way the format expects. To send it in a different header, use auth instead:

[providers.internal]
formats = ["openai-chat"]
base_url = "https://llm.internal/v1"
auth = { type = "header", name = "x-internal-key", key = "env:INTERNAL_KEY" }
# or: auth = { type = "bearer", key = "env:INTERNAL_KEY" }

A provider with no key, no auth and no vendor key passes on the client’s own credentials.

Local models

[providers.ollama]
vendor = "ollama"              # http://localhost:11434/v1 by default
proxy = false                  # skip [network]'s proxy, if you set one

[providers.vllm]
vendor = "vllm"
base_url = "http://localhost:8000/v1"
# key = "env:VLLM_API_KEY"     # only if the server was started with one

How a model name picks a provider

The model in each request decides where it goes. The gateway tries, in order:

1. A route with exactly that name, such as smart.
2. provider/model: the provider named before the first / gets the rest as the model. anthropic/claude-haiku-5-5 goes to the provider named anthropic; groq/openai/gpt-oss-20b sends openai/gpt-oss-20b to groq.
3. The first glob route that matches, such as [routes."claude-*"].
4. The provider whose model list has it. The gateway reads every provider’s model list at start and every 10 minutes, so claude-haiku-5-5 goes to Anthropic and gpt-6-luna to OpenAI with no routes at all.
5. Otherwise, the default provider for that kind of request: the first provider in the file that serves it.

The provider can speak a different format from the client. An Anthropic SDK asking for a GPT model is translated to OpenAI’s format and back.

To set the default yourself, or to turn off model lists:

[plugins.routing]
default = "openai"
listings = false     # route by routes and provider/model names only

GET /ui/api/routes/explain?model=M shows where a model would go and why. See Routes and fallback.

List models

GET /v1/models merges every provider’s model list into one OpenAI-shaped list. Each model is named the way a client should ask for it: as is, if routing sends that name to its provider, otherwise as provider/model.

curl http://localhost:8080/v1/models
respan-gateway models             # the same list from the command line

Each entry has id, owned_by (the provider) and kind (chat, embedding, image, transcription, speech, video or realtime). GET /v1beta/models returns the same list in Gemini’s shape, and /v1/models with an anthropic-version header returns Anthropic’s.

Respan as a provider

A Respan API key reaches the models in Respan’s gateway catalog with one key:

[providers.respan]
vendor = "respan"
key = "env:RESPAN_API_KEY"

Respan speaks OpenAI Chat, OpenAI Responses, Anthropic Messages and Gemini’s generateContent. The gateway sends each model in its maker’s own format (Claude models as Anthropic Messages, Gemini models as generateContent) and anything else as Chat Completions. Its model list is Respan’s public catalog.

To trace requests in Respan rather than route through it, see the respan plugin in Plugins.