---
title: "Write a plugin"
subtitle: "Add your own behavior to the gateway in Rust, on the same API the built-in plugins use."
---

A plugin can edit a request, refuse it, answer it itself, send it somewhere else, call models of its own, meter usage, or rewrite a response as it streams. Routing, the tool loop, pricing and Respan tracing are all plugins on this API.

Plugins are Rust and compiled in. There is no dynamic loading, and `gateway.toml` can't load a plugin. To run yours, you build a gateway binary that registers it, either your own binary built from the gateway's crates, or a fork of `respan-gateway` with your plugin added. Both are covered [below](#run-your-plugin).

## How a request flows

```text
 client ──bytes──▶ client format ──Input──▶ plugins ──▶ provider format ──bytes──▶ provider
   ▲               (decode)                 on_client   (encode)
   │                                        on_provider
   └──bytes──── client format ◀──Output─── filters ◀── provider format ◀──bytes──┘
                (encode)                                (decode)
```

The client's format decodes the request into a typed `Input`. Plugins see that value, never bytes, headers or keys. The provider's format encodes it for the provider, and the reply comes back as a stream of typed `Output` events that plugins can filter before the client's format writes them out.

So a plugin doesn't need to know which vendor is involved. A filter on a chat `Input` works the same whether the client spoke OpenAI and the provider is Anthropic, or the other way round.

Each kind of traffic has its own `Input` and `Output` types:

| Kind | Crate | Types |
| --- | --- | --- |
| Chat | `respan-gateway-conversation` | `Input` (model, messages, options, tools), `Output` (stream events) |
| Images, speech, transcription, embeddings, video | `respan-gateway-media` | `image::Input`, `speech::Input`, … with their `Output`s |
| Realtime sessions | `respan-gateway-realtime` | `Input` and `Output` session events |
| MCP | `respan-gateway-mcp` | `Input` and `Output` JSON-RPC envelopes |
| Decisions | `respan-gateway-decision` | `Input` and `Output` |

A chat reply is a sequence of `Output` events, the same for a stream and a whole body: `Start`, then for each content block a `BlockStart`, its `Delta`s (`Text`, `Arguments` or `Reasoning`) and a `BlockEnd`, with `Usage` and `Finish` among them.

## The Plugin trait

A plugin implements one trait, `Plugin` from `respan-gateway-core`, with four hooks:

```rust
#[async_trait]
pub trait Plugin: Send + Sync + 'static {
    async fn on_client(&self, cx: &mut ClientContext) -> Result<Option<Response>>;  // once per request
    async fn on_client_end(&self, cx: &ClientContext, end: &Result<()>);            // after it ends
    async fn on_provider(&self, px: &mut ProviderContext) -> Result<()>;            // before each call
    async fn on_provider_end(&self, px: &ProviderContext, end: &Result<()>);        // after each call
}
```

Every hook has a default that does nothing, so implement only the ones you need.

| Hook | When it runs | What it returns |
| --- | --- | --- |
| `on_client` | Once per request, before anything is sent. Add filters, change the target, or make calls here. | `Ok(None)` to pass, `Ok(Some(response))` to answer the request yourself (later plugins are skipped), `Err(fault)` to refuse it. |
| `on_provider` | Before every call to a provider. One request can make several calls: retries, fallbacks, each tool-loop step, your own calls. `px.input()` is what this call sends and `px.provider()` is where it goes. | `Ok(())`, or `Err(fault)` to fail the call. |
| `on_provider_end` | Once per call, when it ends, fails or is abandoned. | nothing |
| `on_client_end` | Once per request, after the response ended and every call's end hook ran. `end` is `Ok(())` if the client read the whole response. | nothing |

End hooks run after the client has its answer, so they add no latency. `gateway.idle().await` waits for them.

Every plugin sees all traffic. A plugin for one kind checks the kind first and lets the rest pass:

```rust
if !cx.is_kind::<Input>() {
    return Ok(None);
}
```

## A minimal plugin

This plugin refuses chat requests for preview models, and sends every other chat request at temperature 0. It works for every chat format, translated or not.

```rust
use respan_gateway_conversation::Input;
use respan_gateway_core::{
    ClientContext, ErrorCode, Fault, Plugin, Response, Result, async_trait,
};

pub struct Guard;

#[async_trait]
impl Plugin for Guard {
    async fn on_client(&self, cx: &mut ClientContext) -> Result<Option<Response>> {
        let Ok(chat) = cx.input::<Input>() else {
            return Ok(None); // Not chat: let it pass.
        };
        if chat.model.as_deref().is_some_and(|m| m.contains("preview")) {
            return Err(Fault::bad_request("preview models are not allowed here")
                .with(ErrorCode("model_blocked".into())));
        }
        cx.filter(|chat: &mut Input| {
            chat.options.temperature = Some(0.0);
            Ok(())
        })?;
        Ok(None)
    }
}
```

The client gets the refusal in its own format. An OpenAI client receives:

```json
{"error":{"message":"preview models are not allowed here","type":"model_blocked","param":null,"code":"model_blocked"}}
```

## What a plugin can do

These are the main calls on `ClientContext` (`cx`) and `ProviderContext` (`px`). The rustdoc on `Plugin`, `ClientContext`, `ProviderContext`, `Call` and `Response` in `respan-gateway-core` has the full contract.

| Call | What it does |
| --- | --- |
| `cx.is_kind::<T>()`, `cx.input::<T>()` | Check the request's kind, and read it. |
| `cx.filter(\|x: &mut Input\| ..)` | Edit the request before any provider sees it. |
| `cx.filter(\|x: &mut Output\| ..)` | Edit each response event. |
| `cx.filter(MyFilter)` | A filter with state, a type that implements `Filter<Output>`: for each event it pushes the event, an edited copy, several events or none. |
| `cx.wrap(..)` | Replace the response stream with your own, for work that must wait (filters are synchronous). |
| `cx.set_target("anthropic")` | Choose the provider the request goes to. |
| `cx.call("openai").with_input(..)` | Make a call of your own. It passes through every plugin's `on_provider`, so it is logged, priced and guarded like any other call. |
| `Response::stream(..)` | Answer the request yourself with a stream of `Output` events, such as a cache hit or a canned reply. |
| `px.filter(..)` | Filter one call's events at the provider edge, before any client-edge filter. A meter goes here. |
| `cx.extensions_mut().insert(x)` | Keep typed state for the request, and read it back in the end hooks with `cx.extensions().get::<T>()`. |

A stateful response filter, which masks a word in every text delta:

```rust
use respan_gateway_conversation::{Delta, Output};
use respan_gateway_core::{Filter, Result};

struct Mask(&'static str);

impl Filter<Output> for Mask {
    fn edit(&mut self, mut event: Output, out: &mut Vec<Output>) -> Result<()> {
        if let Output::Delta { delta: Delta::Text(text), .. } = &mut event {
            *text = text.replace(self.0, "****");
        }
        out.push(event);
        Ok(())
    }
}
```

A word can be split across deltas (`"sec"`, `"ret"`). A real redactor holds back text that might start a match, and overrides `Filter::end` to flush it when the stream ends. The repository's [guardrail](https://github.com/respanai/respan-gateway/blob/main/examples/guardrail.rs) and [PII](https://github.com/respanai/respan-gateway/blob/main/examples/pii.rs) examples do this.

## Faults

`Fault` is the one error type. Its constructor says whose fault it is, and the client's format turns it into a status and an error body:

| Constructor | Meaning | HTTP status |
| --- | --- | --- |
| `Fault::bad_request(msg)` | The client asked for something wrong | 400 |
| `Fault::not_found(msg)` | Nothing serves what the client asked for | 404 |
| `Fault::upstream(msg)` | A provider failed | 502 |
| `Fault::config(msg)` | The gateway is misconfigured | 500 |
| `Fault::internal(msg)` | A bug | 500 |

`.with(ErrorCode("your_code".into()))` sets the error code clients see, `.with(RetryAfter(..))` a retry hint, and `.with(respan_gateway_http_faults::Status(StatusCode::FORBIDDEN))` the HTTP status. A panic in a hook or filter becomes an `internal` fault for that request only.

## Order

Plugins run in the order they are registered, and the order changes behavior:

- `on_client` runs first to last. The first plugin that answers or refuses stops the rest, so register a guard that must see everything before a plugin that answers, such as a cache.
- Input filters apply first-added first. Output filters apply last-added first, like layers of middleware.
- `on_provider` runs for the gateway's plugins in order, then for plugins registered on that one provider.
- `on_provider_end` runs before `on_client_end`.

## Run your plugin

<Tabs>
<Tab title="Your own binary">

Build a gateway from the crates and register your plugin. The crates aren't on crates.io yet, so depend on them from git:

```toml
[dependencies]
respan-gateway-core = { git = "https://github.com/respanai/respan-gateway" }
respan-gateway-conversation = { git = "https://github.com/respanai/respan-gateway" }
respan-gateway-conversation-http = { git = "https://github.com/respanai/respan-gateway" }
respan-gateway-http = { git = "https://github.com/respanai/respan-gateway" }
respan-gateway-http-transport = { git = "https://github.com/respanai/respan-gateway" }
respan-gateway-serve = { git = "https://github.com/respanai/respan-gateway" }
axum = "0.8"
tokio = { version = "1", features = ["full"] }
```

A gateway that speaks OpenAI Chat and Anthropic Messages to clients, sends everything to OpenAI, and runs `Guard`:

```rust
use respan_gateway_conversation::Input;
use respan_gateway_conversation_http::anthropic::AnthropicMessages;
use respan_gateway_conversation_http::openai::OpenAiChat;
use respan_gateway_core::{Fault, Gateway, Provider, Result};
use respan_gateway_http::http::{HeaderMap, HeaderValue};
use respan_gateway_http_transport::HttpTransport;

fn gateway(api_key: &str) -> Result<Gateway> {
    let mut headers = HeaderMap::new();
    let bearer = HeaderValue::from_str(&format!("Bearer {api_key}"))
        .map_err(Fault::from_error)?;
    headers.insert("authorization", bearer);
    let openai = HttpTransport::new("https://api.openai.com/v1")?
        .with_headers(headers)?
        .connect();
    Gateway::new()
        .provider(Provider::new("openai", OpenAiChat, openai))?
        .client::<Input, _, _>(OpenAiChat)?
        .client::<Input, _, _>(AnthropicMessages)?
        .plugin(Guard)
        .target("openai")
}

#[tokio::main]
async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
    let gateway = gateway(&std::env::var("OPENAI_API_KEY")?)?;
    let app = respan_gateway_serve::router(gateway.clone());
    let listener = tokio::net::TcpListener::bind("127.0.0.1:8080").await?;
    axum::serve(listener, app).await?;
    gateway.idle().await; // Let end hooks finish.
    Ok(())
}
```

Add more providers in any format, and route between them with `cx.set_target`, or register the built-in `Routing` plugin from `respan-gateway-plugin-routing`. This binary doesn't read `gateway.toml`: you compose it in code.

</Tab>
<Tab title="In respan-gateway">

To get your plugin into the `respan-gateway` binary, with its config file, web UI and CLI, add it to a fork of the repository and rebuild:

1. Create `crates/plugins/<name>`, a crate named `respan-gateway-plugin-<name>`. Copy the `[package]` fields and `[lints]` from a sibling such as `crates/plugins/respan/Cargo.toml`. Depend only on core, the kinds you read, and what the plugin needs.
2. Add it to `[workspace.dependencies]` in the root `Cargo.toml` and to `apps/gateway/Cargo.toml`.
3. Give it config: add a field to `PluginsConfig` in `apps/gateway/src/config.rs`, as `Option<YourConfig>` so it is off unless configured. Mark the struct `#[serde(deny_unknown_fields)]`, and read secrets through `env:` with `compose::secret`.
4. Register it in `with_plugins` in `apps/gateway/src/app/built.rs`. The binary fixes the order: plugins that observe everything (tracing, the request log) first, then plugins that route or edit the request, then plugins that answer it (the tool loop, then `Dispatch`, which runs a route's fallbacks), and pricing last.
5. Build it with `cargo build --release -p respan-gateway`. Build the web UI first (`cd apps/ui && pnpm install && pnpm build`) if you want it embedded; without it the binary serves a placeholder page. `just run` does both and starts the gateway.

Then turn it on in `gateway.toml` under `[plugins.<name>]`. To contribute it upstream, also add its settings to `docs/configuration.md`, a row to `crates/plugins/README.md`, a `README.md` in the crate, and tests; see [CONTRIBUTING.md](https://github.com/respanai/respan-gateway/blob/main/CONTRIBUTING.md).

</Tab>
</Tabs>

## Test it

Test against [`mockllm`](https://github.com/lihe07/mockllm), a scripted fake of the OpenAI, Anthropic and Gemini APIs over real HTTP. It needs no keys and costs nothing:

```rust
#[tokio::test]
async fn preview_models_are_refused() {
    let fake = mockllm::FakeOpenAi::start().await;
    fake.answer(mockllm::Answer::text("hello"));
    let gateway = gateway_at(&fake.base_url()).unwrap(); // `gateway`, at the fake's URL

    let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
    let url = format!("http://{}/v1/chat/completions", listener.local_addr().unwrap());
    let app = respan_gateway_serve::router(gateway.clone());
    tokio::spawn(async move { axum::serve(listener, app).await });

    let status = reqwest::Client::new()
        .post(url)
        .header("content-type", "application/json")
        .body(r#"{"model":"gpt-preview","messages":[{"role":"user","content":"hi"}]}"#)
        .send().await.unwrap()
        .status();
    assert_eq!(status, 400);
}
```

Also cover streamed and whole responses, a client of another format (send to `/v1/messages` too), a provider error, a client that leaves mid-stream, and the order relative to the plugins you expect beside yours.

## Further reading

- The built-in plugins, smallest first: [`credentials`](https://github.com/respanai/respan-gateway/tree/main/crates/plugins/credentials), [`relay`](https://github.com/respanai/respan-gateway/tree/main/crates/plugins/relay), [`pricing`](https://github.com/respanai/respan-gateway/tree/main/crates/plugins/pricing), [`respan`](https://github.com/respanai/respan-gateway/tree/main/crates/plugins/respan), [`routing`](https://github.com/respanai/respan-gateway/tree/main/crates/plugins/routing), [`tool-loop`](https://github.com/respanai/respan-gateway/tree/main/crates/plugins/tool-loop).
- [`examples/`](https://github.com/respanai/respan-gateway/tree/main/examples): guardrails, PII redaction across streams, billing, admission control, an LLM judge, realtime policies. Each is one runnable file.
- The [plugin guide](https://github.com/respanai/respan-gateway/blob/main/docs/plugins.md) in the repository, with more recipes.
- [Plugins](/docs/oss-gateway/plugins): what each built-in plugin does and how to configure it.