Write a plugin

Add your own behavior to the gateway in Rust, on the same API the built-in plugins use.

A plugin can edit a request, refuse it, answer it itself, send it somewhere else, call models of its own, meter usage, or rewrite a response as it streams. Routing, the tool loop, pricing and Respan tracing are all plugins on this API.

Plugins are Rust and compiled in. There is no dynamic loading, and gateway.toml can’t load a plugin. To run yours, you build a gateway binary that registers it, either your own binary built from the gateway’s crates, or a fork of respan-gateway with your plugin added. Both are covered below.

How a request flows

 client ──bytes──▶ client format ──Input──▶ plugins ──▶ provider format ──bytes──▶ provider
   ▲               (decode)                 on_client   (encode)
   │                                        on_provider
   └──bytes──── client format ◀──Output─── filters ◀── provider format ◀──bytes──┘
                (encode)                                (decode)

The client’s format decodes the request into a typed Input. Plugins see that value, never bytes, headers or keys. The provider’s format encodes it for the provider, and the reply comes back as a stream of typed Output events that plugins can filter before the client’s format writes them out.

So a plugin doesn’t need to know which vendor is involved. A filter on a chat Input works the same whether the client spoke OpenAI and the provider is Anthropic, or the other way round.

Each kind of traffic has its own Input and Output types:

A chat reply is a sequence of Output events, the same for a stream and a whole body: Start, then for each content block a BlockStart, its Deltas (Text, Arguments or Reasoning) and a BlockEnd, with Usage and Finish among them.

The Plugin trait

A plugin implements one trait, Plugin from respan-gateway-core, with four hooks:

#[async_trait]
pub trait Plugin: Send + Sync + 'static {
    async fn on_client(&self, cx: &mut ClientContext) -> Result<Option<Response>>;  // once per request
    async fn on_client_end(&self, cx: &ClientContext, end: &Result<()>);            // after it ends
    async fn on_provider(&self, px: &mut ProviderContext) -> Result<()>;            // before each call
    async fn on_provider_end(&self, px: &ProviderContext, end: &Result<()>);        // after each call
}

Every hook has a default that does nothing, so implement only the ones you need.

End hooks run after the client has its answer, so they add no latency. gateway.idle().await waits for them.

Every plugin sees all traffic. A plugin for one kind checks the kind first and lets the rest pass:

if !cx.is_kind::<Input>() {
    return Ok(None);
}

A minimal plugin

This plugin refuses chat requests for preview models, and sends every other chat request at temperature 0. It works for every chat format, translated or not.

use respan_gateway_conversation::Input;
use respan_gateway_core::{
    ClientContext, ErrorCode, Fault, Plugin, Response, Result, async_trait,
};

pub struct Guard;

#[async_trait]
impl Plugin for Guard {
    async fn on_client(&self, cx: &mut ClientContext) -> Result<Option<Response>> {
        let Ok(chat) = cx.input::<Input>() else {
            return Ok(None); // Not chat: let it pass.
        };
        if chat.model.as_deref().is_some_and(|m| m.contains("preview")) {
            return Err(Fault::bad_request("preview models are not allowed here")
                .with(ErrorCode("model_blocked".into())));
        }
        cx.filter(|chat: &mut Input| {
            chat.options.temperature = Some(0.0);
            Ok(())
        })?;
        Ok(None)
    }
}

The client gets the refusal in its own format. An OpenAI client receives:

{"error":{"message":"preview models are not allowed here","type":"model_blocked","param":null,"code":"model_blocked"}}

What a plugin can do

These are the main calls on ClientContext (cx) and ProviderContext (px). The rustdoc on Plugin, ClientContext, ProviderContext, Call and Response in respan-gateway-core has the full contract.

A stateful response filter, which masks a word in every text delta:

use respan_gateway_conversation::{Delta, Output};
use respan_gateway_core::{Filter, Result};

struct Mask(&'static str);

impl Filter<Output> for Mask {
    fn edit(&mut self, mut event: Output, out: &mut Vec<Output>) -> Result<()> {
        if let Output::Delta { delta: Delta::Text(text), .. } = &mut event {
            *text = text.replace(self.0, "****");
        }
        out.push(event);
        Ok(())
    }
}

A word can be split across deltas ("sec", "ret"). A real redactor holds back text that might start a match, and overrides Filter::end to flush it when the stream ends. The repository’s guardrail  and PII  examples do this.

Faults

Fault is the one error type. Its constructor says whose fault it is, and the client’s format turns it into a status and an error body:

.with(ErrorCode("your_code".into())) sets the error code clients see, .with(RetryAfter(..)) a retry hint, and .with(respan_gateway_http_faults::Status(StatusCode::FORBIDDEN)) the HTTP status. A panic in a hook or filter becomes an internal fault for that request only.

Order

Plugins run in the order they are registered, and the order changes behavior:

• on_client runs first to last. The first plugin that answers or refuses stops the rest, so register a guard that must see everything before a plugin that answers, such as a cache.
• Input filters apply first-added first. Output filters apply last-added first, like layers of middleware.
• on_provider runs for the gateway’s plugins in order, then for plugins registered on that one provider.
• on_provider_end runs before on_client_end.

Run your plugin

[dependencies]
respan-gateway-core = { git = "https://github.com/respanai/respan-gateway" }
respan-gateway-conversation = { git = "https://github.com/respanai/respan-gateway" }
respan-gateway-conversation-http = { git = "https://github.com/respanai/respan-gateway" }
respan-gateway-http = { git = "https://github.com/respanai/respan-gateway" }
respan-gateway-http-transport = { git = "https://github.com/respanai/respan-gateway" }
respan-gateway-serve = { git = "https://github.com/respanai/respan-gateway" }
axum = "0.8"
tokio = { version = "1", features = ["full"] }

use respan_gateway_conversation::Input;
use respan_gateway_conversation_http::anthropic::AnthropicMessages;
use respan_gateway_conversation_http::openai::OpenAiChat;
use respan_gateway_core::{Fault, Gateway, Provider, Result};
use respan_gateway_http::http::{HeaderMap, HeaderValue};
use respan_gateway_http_transport::HttpTransport;

fn gateway(api_key: &str) -> Result<Gateway> {
    let mut headers = HeaderMap::new();
    let bearer = HeaderValue::from_str(&format!("Bearer {api_key}"))
        .map_err(Fault::from_error)?;
    headers.insert("authorization", bearer);
    let openai = HttpTransport::new("https://api.openai.com/v1")?
        .with_headers(headers)?
        .connect();
    Gateway::new()
        .provider(Provider::new("openai", OpenAiChat, openai))?
        .client::<Input, _, _>(OpenAiChat)?
        .client::<Input, _, _>(AnthropicMessages)?
        .plugin(Guard)
        .target("openai")
}

#[tokio::main]
async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
    let gateway = gateway(&std::env::var("OPENAI_API_KEY")?)?;
    let app = respan_gateway_serve::router(gateway.clone());
    let listener = tokio::net::TcpListener::bind("127.0.0.1:8080").await?;
    axum::serve(listener, app).await?;
    gateway.idle().await; // Let end hooks finish.
    Ok(())
}

Test it

Test against mockllm, a scripted fake of the OpenAI, Anthropic and Gemini APIs over real HTTP. It needs no keys and costs nothing:

#[tokio::test]
async fn preview_models_are_refused() {
    let fake = mockllm::FakeOpenAi::start().await;
    fake.answer(mockllm::Answer::text("hello"));
    let gateway = gateway_at(&fake.base_url()).unwrap(); // `gateway`, at the fake's URL

    let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
    let url = format!("http://{}/v1/chat/completions", listener.local_addr().unwrap());
    let app = respan_gateway_serve::router(gateway.clone());
    tokio::spawn(async move { axum::serve(listener, app).await });

    let status = reqwest::Client::new()
        .post(url)
        .header("content-type", "application/json")
        .body(r#"{"model":"gpt-preview","messages":[{"role":"user","content":"hi"}]}"#)
        .send().await.unwrap()
        .status();
    assert_eq!(status, 400);
}

Also cover streamed and whole responses, a client of another format (send to /v1/messages too), a provider error, a client that leaves mid-stream, and the order relative to the plugins you expect beside yours.

Further reading

• The built-in plugins, smallest first: credentials, relay, pricing, respan, routing, tool-loop.
• examples/: guardrails, PII redaction across streams, billing, admission control, an LLM judge, realtime policies. Each is one runnable file.
• The plugin guide in the repository, with more recipes.
• Plugins: what each built-in plugin does and how to configure it.