> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://www.respan.ai/docs/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://www.respan.ai/docs/_mcp/server. # Configure plugins Everything the gateway does beyond relaying a request is set in one file, `gateway.toml`. Each block of the file turns on one capability. The gateway reads `gateway.toml` from the directory it starts in, or the file you pass with `-c`. ## A complete example This file connects three providers, protects the gateway with a key, routes a model name to two providers that race, gives every model web search, and traces every request in Respan. ```toml client_keys = ["env:GATEWAY_KEY"] # Who may call the gateway [providers.openai] # A vendor and its key vendor = "openai" key = "env:OPENAI_API_KEY" [providers.gemini] vendor = "gemini" key = "env:GEMINI_API_KEY" [providers.respan] # 1000+ models with one key vendor = "respan" key = "env:RESPAN_API_KEY" [routes.smart] # Clients ask for "smart" to = ["respan/claude-sonnet-4-5-20250929", "openai/gpt-5.5"] hedge = "800ms" # Race them; the first answer wins [mcp.search] # An MCP server url = "https://mcp.exa.ai/mcp" [plugins.tool_loop] # Its tools, for every model servers = ["search"] [plugins.respan] # A trace in Respan per request key = "env:RESPAN_API_KEY" ``` `env:NAME` reads a value from the environment, or from a `.env` file in the current directory or a parent. A literal value works too. Settings that aren't in a `[table]`, such as `client_keys`, go at the top of the file. Check the file before you start the gateway. `check` prints what the gateway will serve, and `--online` also calls each provider and MCP server with its key: ```bash respan-gateway check --online respan-gateway ``` To apply an edit without a restart, send the process `SIGHUP`, start it with `--watch`, or click **Reload** on the web UI's **Config** page. Requests already in flight finish on the old config, and an invalid file is refused while the old one keeps serving. ## Providers Each `[providers.NAME]` block adds one provider. `vendor` brings the vendor's URLs, formats and model list, so a key is all you add. ```toml [providers.groq] vendor = "groq" key = "env:GROQ_API_KEY" ``` Built-in vendors: `openai`, `anthropic`, `gemini`, `respan`, `groq`, `deepseek`, `openrouter`, `xai`, `mistral`, `together`, `fireworks`, `ollama`, `azure`, `vllm`, `cloudflare` and `typesafe`. Any other endpoint that speaks an OpenAI-, Anthropic- or Gemini-style API works with a `base_url` and its format: ```toml [providers.lmstudio] base_url = "http://localhost:1234/v1" formats = ["openai-chat"] ``` | Field | What it does | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `vendor` | A built-in vendor. Sets the base URLs, formats and model list. | | `key` | The provider's API key. With no key, a client's own credentials are forwarded, but only when the client speaks the provider's format. | | `base_url` | Where the provider is. Required for `azure` and `vllm`, and for a provider with no `vendor`. | | `formats` | The formats the provider speaks, or a map of format to base URL when they live at different URLs. | A request's model name picks its provider: the provider whose model list has the model, or the one you name with `provider/model`. ## Client keys Set `client_keys` to make every request carry a gateway key. The gateway strips it before calling a provider, so providers never see it. Without client keys the gateway answers only on your machine, and it refuses to start on a public address. ```toml client_keys = [ { name = "ops", key = "env:OPS_KEY", admin = true }, { name = "app", key = "env:APP_KEY", tools = ["search__*"] }, ] ``` | Field | What it does | | ------- | ----------------------------------------------------------------------------------------------- | | `name` | Shows in the request log as the request's client. | | `key` | The key clients send as `Authorization: Bearer`, `x-api-key` or `x-goog-api-key`. | | `admin` | Whether the key can open the web UI and admin API. Every key can until one sets `admin = true`. | | `tools` | The MCP tools this key's requests may use, as `server__tool` globs. | `respan-gateway init --client-key` writes a strong key into a new `gateway.toml`. You can also pass keys as `RESPAN_CLIENT_KEYS`, separated by commas. ## Routing A route gives a model name to one or more candidates. Clients ask for the route's name as the model. ```toml [routes.cheap-and-good] to = ["openai/gpt-5.5", "respan/claude-sonnet-4-5-20250929", "gemini/gemini-3.6-flash"] pick = "cheapest" needs = ["tools"] tries = 2 ``` A route's name can be a glob, such as `[routes."llama-*"]`, to catch a family of models. `to = "groq"` sends them to a provider under the name asked for. | Field | What it does | Default | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | -------------- | | `to` | One candidate or a list: `provider`, `provider/model`, or `{ target = "...", weight = 3 }`. | Required | | `pick` | The order candidates are tried in: `first`, `cheapest` (input price plus three times output), `fastest`, `weighted` or `random`. | `first` | | `needs` | Capabilities a candidate's model must list, such as `tools`, `reasoning`, `vision` or `pdf`. | None | | `fallback` | Whether a failed candidate falls back to the next one. | `true` | | `tries` | Tries per candidate, from 1 to 10. | `1` | | `backoff`, `max_backoff` | The first wait before a retry, doubled after each, and its cap. | `250ms`, `10s` | | `hedge` | Race the candidates instead: start each this long after the one before. The first answer wins. | Off | | `when` | Limit the route to some clients or kinds: `{ clients = ["app"], kinds = ["chat"] }`. | Every request | A request goes to the first match of: a route of that exact name, a `provider/model` prefix, the first glob route that matches, the provider whose model list has the model, and finally the default provider. Set the default with: ```toml [plugins.routing] default = "openai" ``` To see where a model goes and why, open `http://localhost:8080/ui/api/routes/explain?model=smart`. To price models the gateway can't find a price for, declare it in US dollars per million tokens. `cheapest` and the request log's cost use it: ```toml [models."lmstudio/qwen3-8b"] pricing = { input = 0.05, output = 0.2 } ``` ## MCP servers Each `[mcp.NAME]` block puts an MCP server behind the gateway at `/mcp/NAME`. Its fields match an MCP client's `mcp.json`, so you can copy an entry across. The server's credentials stay on the gateway; clients use a gateway key. #### Remote (HTTP) ```toml [mcp.github] url = "https://api.githubcopilot.com/mcp/" auth = { bearer = "env:GITHUB_TOKEN" } ``` #### Local (stdio) ```toml [mcp.files] command = "npx" args = ["-y", "@modelcontextprotocol/server-filesystem", "/data"] ``` | Field | What it does | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `url`, `headers` | A remote server and any headers it needs. Header values are treated as secrets. | | `command`, `args`, `env`, `cwd` | A local server the gateway starts as a child process, one per client session. | | `auth` | `{ bearer = "env:KEY" }`, `{ oauth = { token_url = "...", client_id = "...", client_secret = "..." } }`, `"login"` for a server a person must authorize (run `respan-gateway mcp login NAME` once), or `"forward"` to pass each client's own credentials. | To serve every MCP server at one endpoint, `/mcp`, turn on the aggregate. Tools are named `server__tool`: ```toml [plugins.mcp_aggregate] deny = ["files__write_*"] ``` ## Tool loop The tool loop gives every chat request the tools of your MCP servers. When the model calls one, the gateway runs it, sends the result back to the model, and returns only the final answer. It works with any chat format and any provider, and in realtime voice sessions. A client needs no MCP support. ```toml [plugins.tool_loop] servers = ["search", "github"] allow = ["search__*", "github__search_*"] deny = ["github__delete_*"] ``` | Field | What it does | Default | | --------------------------- | --------------------------------------------------------------------- | ----------------- | | `servers` | The `mcp` servers whose tools are offered. | Required | | `allow`, `deny` | Which tools, as `server__tool` globs. `deny` wins. | All tools | | `models` | Model name prefixes the loop applies to, such as `["gpt", "claude"]`. | All models | | `max_steps` | Model calls per request, from 1 to 64. | `8` | | `timeout`, `tool_timeout` | Limits for the whole loop and for one tool call. | `5m`, `60s` | | `parallel` | Tool calls run at once in one step. | `4` | | `max_result`, `max_results` | Size limits for one tool's answer and for all of them in a request. | `64KiB`, `256KiB` | | `tool_list_ttl` | How long a server's tool list is cached. | `5m` | A tool that fails becomes a result the model reads, not a failed request. A client skips the loop for one request with the header `x-respan-tool-loop: off`. > **Warning** > > Every client key can make the model call every tool you offer, with the server's credentials. Offer only the tools you would let a client use, and list destructive ones in `deny`. ## Respan tracing Turn on the `respan` plugin to send each request to Respan as a trace, with its model, provider, tokens, cost and latency. Find them in [Logs](/docs/documentation/features/observe/explore-spans). ```toml [plugins.respan] key = "env:RESPAN_API_KEY" content = false # Send metadata only, without messages or replies ``` | Field | What it does | Default | | --------- | ---------------------------------------------------------------------------- | ---------------- | | `key` | Your [Respan API key](https://platform.respan.ai/platform/gateway/api-keys). | Required | | `content` | Whether messages and replies are sent. | `true` | | `service` | The service name on the traces. | `respan-gateway` | ## Request log The web UI's request history is always on. Add a file to keep it across restarts: ```toml [plugins.request_log] history = 1000 file = "./requests.jsonl" ``` | Field | What it does | Default | | ---------- | ------------------------------------------------------------------------------------- | ------- | | `history` | Finished requests kept in memory for the web UI. | `1000` | | `file` | A file each finished request is appended to as one JSON line, without bodies or keys. | None | | `file_max` | The size at which the file rotates to `.1`. | `64MiB` | ## Conversation state When a Responses API client is translated to another provider, the gateway keeps each response in memory so `previous_response_id` keeps working. It is on by default. ```toml [plugins.responses] keep = "1h" # How long after it was last read # enabled = false ``` State lives in one gateway's memory: a client whose requests reach several gateways must reach the one that kept its conversation. ## Server settings These go at the top of the file, above every `[table]`. | Setting | What it does | Default | | ------------------ | -------------------------------------------------------------------------------------------- | ---------------- | | `listen` | The address and port, such as `127.0.0.1:8080`. Beyond loopback, `client_keys` are required. | `127.0.0.1:8080` | | `log` | Request logs on stderr: `text`, `json` or `off`. | `text` | | `ui` | Whether the web UI and admin API are served. | `true` | | `cors` | Browser origins allowed to call the gateway: `{ origins = ["https://app.example.com"] }`. | None | | `body_limit` | The largest request body. | `32MiB` | | `ws_message_limit` | The largest WebSocket message. | `16MiB` | | `shutdown_grace` | How long open requests may finish after a stop signal. | `30s` | The full list of options is in [`gateway.example.toml`](https://github.com/respanai/respan-gateway/blob/main/gateway.example.toml).