---
title: "Rust library"
subtitle: "Run the gateway inside your own Rust service, from a gateway.toml or from the crates directly."
---

The gateway is a Cargo workspace of crates, published on crates.io. There are two ways to embed it:

- **`respan-gateway`**, the binary's own crate, also a library. It builds the whole application (providers, routes, tools, plugins, the web UI) from the same config as `gateway.toml`. Use this to run the gateway inside your service.
- **`respan-gateway-core`** and the crates around it. You build a `Gateway` by hand from providers, client formats and plugins of your own. Use this to write a custom gateway or a plugin.

## From a config

```toml
[dependencies]
respan-gateway = "0.2"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "net"] }
```

To track `main` instead of the release, depend on git:

```toml
respan-gateway = { git = "https://github.com/respanai/respan-gateway" }
```

`Config` is the parsed config and `App` is the gateway built from it. This serves it the way the binary does:

```rust
use respan_gateway::{App, Config};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = Config::parse(r#"
        [providers.openai]
        vendor = "openai"
        key = "env:OPENAI_API_KEY"
    "#)?;
    let app = App::new(config)?;
    app.listed().await; // wait for the providers' model lists

    let addr = app.listen(); // the config's `listen`, or 127.0.0.1:8080
    if let Some(why) = app.open_refusal(addr) {
        return Err(why.into());
    }
    let listener = tokio::net::TcpListener::bind(addr).await?;
    println!("{}", app.banner(listener.local_addr()?));
    app.serve(listener, respan_gateway::shutdown_signal()).await?;
    Ok(())
}
```

Ways to get a `Config`:

| Call | Reads |
| --- | --- |
| `Config::parse(text)` | TOML text |
| `Config::load(path)` | a TOML file |
| `Config::from_env()` | the environment: a provider for every vendor whose key is set |
| `Config::discover(None)` | `./gateway.toml` if it exists, else the environment |

`App::serve` serves until the shutdown future resolves, then drains open responses for up to `shutdown_grace` (30 s by default). `shutdown_signal()` resolves on Ctrl-C or SIGTERM.

<Warning>
The binary refuses to listen on a non-loopback address without client keys. `App::serve` does not check this for you: call `app.open_refusal(addr)` first, as above.
</Warning>

### Without a socket

`app.router()` is an Axum `Router`. Hand it requests directly to use the gateway in-process:

```toml
axum = "0.8"
tower = { version = "0.5", features = ["util"] }
```

```rust
use tower::ServiceExt;

let app = respan_gateway::App::new(respan_gateway::Config::from_env()?)?;
let body = r#"{"model":"claude-haiku-5-5","messages":[{"role":"user","content":"hi"}]}"#;
let request = axum::http::Request::post("/v1/chat/completions")
    .header("content-type", "application/json")
    .body(axum::body::Body::from(body))?;
let response = app.router().oneshot(request).await?;
let bytes = axum::body::to_bytes(response.into_body(), usize::MAX).await?;
println!("{}", String::from_utf8_lossy(&bytes));
app.gateway().idle().await; // let end hooks finish
```

You can also mount `app.router()` inside your own Axum app.

### Reloading

Give the app a config source, and `app.reload()` rebuilds the gateway from it. New requests go to the new gateway; open requests and sessions finish on the old one. An invalid config is an `Err`, and the old one keeps serving.

```rust
let app = App::new(Config::load("gateway.toml")?)?
    .reload_from(|| Config::load("gateway.toml"));
// later, e.g. on SIGHUP:
app.reload()?;
```

## From the crates

A `Gateway` from `respan-gateway-core` is built by value from providers, client formats and plugins. This one speaks OpenAI Chat and Anthropic Messages to clients and sends everything to OpenAI:

```toml
[dependencies]
respan-gateway-core = "0.1"
respan-gateway-conversation = "0.1"
respan-gateway-conversation-http = "0.1"
respan-gateway-http = "0.1"
respan-gateway-http-transport = "0.2"
respan-gateway-serve = "0.2"
axum = "0.8"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "net"] }
```

```rust
use respan_gateway_conversation::Input;
use respan_gateway_conversation_http::anthropic::AnthropicMessages;
use respan_gateway_conversation_http::openai::OpenAiChat;
use respan_gateway_core::{Fault, Gateway, Provider, Result};
use respan_gateway_http::http::{HeaderMap, HeaderValue};
use respan_gateway_http_transport::HttpTransport;

fn gateway(api_key: &str) -> Result<Gateway> {
    let mut headers = HeaderMap::new();
    let bearer = HeaderValue::from_str(&format!("Bearer {api_key}")).map_err(Fault::from_error)?;
    headers.insert("authorization", bearer);
    let openai = HttpTransport::new("https://api.openai.com/v1")?
        .with_headers(headers)?
        .connect();
    Gateway::new()
        .provider(Provider::new("openai", OpenAiChat, openai))?
        .client::<Input, _, _>(OpenAiChat)?
        .client::<Input, _, _>(AnthropicMessages)?
        .target("openai")
}

#[tokio::main]
async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
    let gateway = gateway(&std::env::var("OPENAI_API_KEY")?)?;
    let app = respan_gateway_serve::router(gateway.clone());
    let listener = tokio::net::TcpListener::bind("127.0.0.1:8080").await?;
    axum::serve(listener, app).await?;
    gateway.idle().await;
    Ok(())
}
```

Add your own plugins with `.plugin(...)` before `.target(...)`. Routing, fallback, the tool loop and the rest are first-party plugin crates (`respan-gateway-plugin-routing`, `respan-gateway-plugin-tool-loop`, ...) that you register the same way. See [Write a plugin](/docs/oss-gateway/write-a-plugin).

## Where to look

In [respanai/respan-gateway](https://github.com/respanai/respan-gateway):

- [`apps/gateway/src/lib.rs`](https://github.com/respanai/respan-gateway/blob/main/apps/gateway/src/lib.rs): what the `respan-gateway` library exports.
- [`examples/`](https://github.com/respanai/respan-gateway/tree/main/examples): runnable examples, one file each. [`embedded.rs`](https://github.com/respanai/respan-gateway/blob/main/examples/embedded.rs) runs a hand-built gateway in-process with no HTTP server. Run one with `cargo run -p respan-gateway-examples --example embedded`.
- [`docs/plugins.md`](https://github.com/respanai/respan-gateway/blob/main/docs/plugins.md): the plugin API, and how to test a gateway against a fake provider.
- [`crates/core/README.md`](https://github.com/respanai/respan-gateway/blob/main/crates/core/README.md): the `Gateway`, `Plugin` and `Provider` contract.
- [`crates/serve/README.md`](https://github.com/respanai/respan-gateway/blob/main/crates/serve/README.md): hosting a `Gateway` with Axum.