Plugins

The plugins built into the gateway, what each one does, and how to configure it.

Everything the gateway does beyond translating formats is a plugin: routing, the tool loop, pricing, tracing. The built-in plugins use the same public Rust API you can use to write your own.

Some plugins are always on. The others are off until you add their section to gateway.toml. Each is configured under [plugins.<name>], and the gateway runs them in a fixed order, whatever order the file lists them in.

A misspelled key, or a section the gateway doesn’t know, is refused when the config loads. respan-gateway check shows it before you start.

Routing

Picks the provider for each request from its model name, then sends it to the route’s candidates: in turn with retries and fallback, or raced with hedge. Routes are top-level [routes.<model>] tables; Routes and fallback covers them.

[plugins.routing] holds two settings:

[plugins.routing]
default = "openai"     # where a request goes when nothing else routes it
listings = true        # read each provider's model list at start and every 10 minutes

Routing can’t be turned off.

Tool loop

Offers the tools of your tool servers to the model and runs the calls itself, then returns one ordinary reply. It works for every chat format and provider, and in realtime sessions. A client can skip it for one request with the header x-respan-tool-loop: off.

It is off until you add the section:

[tools.search]
vendor = "linkup"
key = "env:LINKUP_API_KEY"

[plugins.tool_loop]
servers = ["search"]
allow = ["search__*"]
max_steps = 8

A request the tool loop runs goes to its route’s first candidate only, with no fallback, retry or hedge.

To turn it off, remove [plugins.tool_loop].

MCP router

Serves your tool servers to MCP clients. /mcp/{name} reaches one server as it is. /mcp fronts every server as one, with tools and prompts named server__name and resources under mcp://server/.... Servers with auth = "forward" are left out of /mcp.

It has no section of its own. It serves whatever [tools.<name>] tables the config has; with none, there is no /mcp. See Tools and MCP.

Pricing

Prices every call to a provider as its usage is reported, at the price of the model that call sent. Retries, fallbacks, a race’s loser and each tool-loop step are separate calls, so a request’s cost is the sum of its calls. The request log and the web UI show it, and pick = "cheapest" routes use the same prices.

Prices come from each provider’s model list. Declare or override one with a model card, in US dollars per million tokens:

[models."openai/my-finetune"]
pricing = { input = 1.0, output = 2.0, cached_input = 0.5 }

pricing also takes cache_write and per_second (for audio). Pricing is always on; it only measures and never blocks a request.

Respan tracing

Sends each request to Respan  as a trace over OTLP. A request is a root span, and each call it made is a child span: a chat span for a model call, with model, usage, messages and reply, and a tool span for each tool the tool loop ran. Spans are sent in batches off the request path. If Respan is slow or down, at most 4096 traces wait and the rest are dropped, so requests never slow down.

It is off until you add the section:

[plugins.respan]
key = "env:RESPAN_API_KEY"
content = false              # metadata only: no messages or replies

Only your clients’ requests are traced, not the gateway’s own calls to tool servers. To turn it off, remove [plugins.respan].

Conversation state

Keeps conversation state that a client or provider can’t keep:

• A Responses API client sent to a provider that keeps no state (Anthropic, Gemini, Chat Completions). Its previous_response_id keeps working: the gateway gives each response an id and sends the whole conversation on the next turn. A request with store: false is not kept.
• A Chat Completions client whose call goes to OpenAI’s Responses API. The reasoning behind a turn’s tool calls is kept and put back on the next turn, so a reasoning model doesn’t lose it.

It is on by default:

[plugins.responses]
keep = "1h"          # how long after it was last read
# enabled = false    # turn it off

State lives in one gateway’s memory, up to about 256 MiB. It doesn’t survive a restart and isn’t shared between gateways, so a client whose requests reach several gateways must reach the one that kept its conversation.

Client credentials

Sends a client’s own Authorization, x-api-key or x-goog-api-key to providers that have no key of their own, and to tool servers with auth = "forward". It only does this when the client speaks the provider’s own format: a key never crosses a translation.

It has no section. A provider gets the client’s credentials when it has no key of its own: no key or auth, and either no vendor or a vendor that takes no key, such as ollama or vllm. A provider with a key never does. Client keys from client_keys are removed before this, so a provider never sees them.

Relay

Used only by respan-gateway proxy. It passes the vendor’s response headers and error bodies back to the client as they came, so a tool such as Claude Code sees the vendor’s own answers.

Request log

Records every finished request for the web UI, the admin API and the log lines on stderr: client key, model, provider, status, tokens, cost and latency. It is always on, and holds requests in memory.

[plugins.request_log]
history = 1000       # finished requests kept for the web UI

The top-level log setting (text, json or off) controls the stderr lines. See Config file.

Video jobs

Sends a video job’s follow-ups (status, content, delete) to the provider that started the job, by the job’s id. It is always on and has no settings.

Retired settings

These older keys are refused with a message that says what replaced them: