Skip to Content
Launching the open-source gateway

Rust library

Run the gateway inside your own Rust service, from a gateway.toml or from the crates directly.

The gateway is a Cargo workspace of crates, published on crates.io. There are two ways to embed it:

  • respan-gateway, the binary’s own crate, also a library. It builds the whole application (providers, routes, tools, plugins, the web UI) from the same config as gateway.toml. Use this to run the gateway inside your service.
  • respan-gateway-core and the crates around it. You build a Gateway by hand from providers, client formats and plugins of your own. Use this to write a custom gateway or a plugin.

From a config

[dependencies]
respan-gateway = "0.2"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "net"] }

To track main instead of the release, depend on git:

respan-gateway = { git = "https://github.com/respanai/respan-gateway" }

Config is the parsed config and App is the gateway built from it. This serves it the way the binary does:

use respan_gateway::{App, Config};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = Config::parse(r#"
        [providers.openai]
        vendor = "openai"
        key = "env:OPENAI_API_KEY"
    "#)?;
    let app = App::new(config)?;
    app.listed().await; // wait for the providers' model lists

    let addr = app.listen(); // the config's `listen`, or 127.0.0.1:8080
    if let Some(why) = app.open_refusal(addr) {
        return Err(why.into());
    }
    let listener = tokio::net::TcpListener::bind(addr).await?;
    println!("{}", app.banner(listener.local_addr()?));
    app.serve(listener, respan_gateway::shutdown_signal()).await?;
    Ok(())
}

Ways to get a Config:

CallReads
Config::parse(text)TOML text
Config::load(path)a TOML file
Config::from_env()the environment: a provider for every vendor whose key is set
Config::discover(None)./gateway.toml if it exists, else the environment

App::serve serves until the shutdown future resolves, then drains open responses for up to shutdown_grace (30 s by default). shutdown_signal() resolves on Ctrl-C or SIGTERM.

The binary refuses to listen on a non-loopback address without client keys. App::serve does not check this for you: call app.open_refusal(addr) first, as above.

Without a socket

app.router() is an Axum Router. Hand it requests directly to use the gateway in-process:

axum = "0.8"
tower = { version = "0.5", features = ["util"] }
use tower::ServiceExt;

let app = respan_gateway::App::new(respan_gateway::Config::from_env()?)?;
let body = r#"{"model":"claude-haiku-5-5","messages":[{"role":"user","content":"hi"}]}"#;
let request = axum::http::Request::post("/v1/chat/completions")
    .header("content-type", "application/json")
    .body(axum::body::Body::from(body))?;
let response = app.router().oneshot(request).await?;
let bytes = axum::body::to_bytes(response.into_body(), usize::MAX).await?;
println!("{}", String::from_utf8_lossy(&bytes));
app.gateway().idle().await; // let end hooks finish

You can also mount app.router() inside your own Axum app.

Reloading

Give the app a config source, and app.reload() rebuilds the gateway from it. New requests go to the new gateway; open requests and sessions finish on the old one. An invalid config is an Err, and the old one keeps serving.

let app = App::new(Config::load("gateway.toml")?)?
    .reload_from(|| Config::load("gateway.toml"));
// later, e.g. on SIGHUP:
app.reload()?;

From the crates

A Gateway from respan-gateway-core is built by value from providers, client formats and plugins. This one speaks OpenAI Chat and Anthropic Messages to clients and sends everything to OpenAI:

[dependencies]
respan-gateway-core = "0.1"
respan-gateway-conversation = "0.1"
respan-gateway-conversation-http = "0.1"
respan-gateway-http = "0.1"
respan-gateway-http-transport = "0.2"
respan-gateway-serve = "0.2"
axum = "0.8"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "net"] }
use respan_gateway_conversation::Input;
use respan_gateway_conversation_http::anthropic::AnthropicMessages;
use respan_gateway_conversation_http::openai::OpenAiChat;
use respan_gateway_core::{Fault, Gateway, Provider, Result};
use respan_gateway_http::http::{HeaderMap, HeaderValue};
use respan_gateway_http_transport::HttpTransport;

fn gateway(api_key: &str) -> Result<Gateway> {
    let mut headers = HeaderMap::new();
    let bearer = HeaderValue::from_str(&format!("Bearer {api_key}")).map_err(Fault::from_error)?;
    headers.insert("authorization", bearer);
    let openai = HttpTransport::new("https://api.openai.com/v1")?
        .with_headers(headers)?
        .connect();
    Gateway::new()
        .provider(Provider::new("openai", OpenAiChat, openai))?
        .client::<Input, _, _>(OpenAiChat)?
        .client::<Input, _, _>(AnthropicMessages)?
        .target("openai")
}

#[tokio::main]
async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
    let gateway = gateway(&std::env::var("OPENAI_API_KEY")?)?;
    let app = respan_gateway_serve::router(gateway.clone());
    let listener = tokio::net::TcpListener::bind("127.0.0.1:8080").await?;
    axum::serve(listener, app).await?;
    gateway.idle().await;
    Ok(())
}

Add your own plugins with .plugin(...) before .target(...). Routing, fallback, the tool loop and the rest are first-party plugin crates (respan-gateway-plugin-routing, respan-gateway-plugin-tool-loop, …) that you register the same way. See Write a plugin.

Where to look

In respanai/respan-gateway :