Skip to Content
Launching the open-source gateway

RubyLLM (gateway)

RubyLLM does not have a Ruby-side tracing instrumentor. Route all calls through the Respan gateway to capture every request as a trace.

Paste this into your coding agent:

Help me set up the Respan gateway with the RubyLLM integration.
Follow https://www.respan.ai/docs/integrations/gateway/ruby-llm.md

Setup

Install RubyLLM

Terminal
gem install ruby_llm

Or add it to your Gemfile.

Ruby
gem "ruby_llm"

Set environment variables

Terminal
export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"

No provider key needed. The Respan gateway handles provider authentication.

Configure RubyLLM with Respan

Ruby
RubyLLM.configure do |config|
  config.openai_api_key = ENV["RESPAN_API_KEY"]
  config.openai_api_base = "https://gateway.respan.ai"
end

Make your first request

Ruby
chat = RubyLLM.chat(model: "gpt-5.5")
response = chat.ask("Hello, world!")
puts response.content

View your trace

Open the Traces page  to see your gateway-routed calls with prompts, tokens, and cost.

Switch models

Use another OpenAI model ID that Respan exposes through the same OpenAI-compatible endpoint.

Ruby
chat = RubyLLM.chat(model: "gpt-5.5")
chat = RubyLLM.chat(model: "gpt-5-mini")

response = chat.ask("Tell me about artificial intelligence")
puts response.content

RubyLLM’s OpenAI-compatible adapter does not provide a provider-neutral way to show Claude or Gemini model switches in this setup. Use the Respan API or OpenAI SDK gateway pages for provider-neutral Claude and Gemini examples.

See the full model list .

Streaming

Ruby
chat = RubyLLM.chat(model: "gpt-5.5")
chat.ask("Explain quantum computing") do |chunk|
  print chunk.content
end

Responses API (reasoning, tools, and prompts)

RubyLLM’s default OpenAI provider uses the Chat Completions API. OpenAI blocks reasoning together with function tools on that endpoint (Function tools with reasoning_effort are not supported ... use /v1/responses instead). To combine reasoning and tool calls, route through the OpenAI Responses API with the ruby_llm-responses_api provider, pointed at the Respan gateway.

Install the provider gem

Ruby
# Gemfile
gem "ruby_llm"
gem "ruby_llm-responses_api"

Point the Responses provider at Respan

Ruby
require "ruby_llm"
require "ruby_llm-responses_api"

RubyLLM.configure do |config|
  config.openai_api_key  = ENV["RESPAN_API_KEY"]
  config.openai_api_base = "https://gateway.respan.ai"
end

# provider: :openai_responses routes to https://gateway.respan.ai/responses
chat = RubyLLM.chat(model: "gpt-5.5", provider: :openai_responses)
puts chat.ask("Hello!").content

OpenAI reasoning models (gpt-5.x, o-series) only accept temperature = 1 and reject top_p. Set with_temperature(1) on Responses chats that use these models, otherwise RubyLLM’s default temperature is rejected.

Reasoning with tool calls

Define a tool as usual, then combine with_thinking and with_tools. RubyLLM runs the tool loop for you.

Ruby
class GetOrderStatus < RubyLLM::Tool
  description "Look up an order's status, carrier, and tracking by order_id."
  param :order_id, desc: "The order ID, e.g. 'A-48217'."

  def execute(order_id:)
    { order_id:, status: "delayed", carrier: "UPS", tracking: "1Z999AA10123456784" }
  end
end

chat = RubyLLM.chat(model: "gpt-5.5", provider: :openai_responses)
              .with_temperature(1)
              .with_thinking(effort: :low)
              .with_tools(GetOrderStatus)

puts chat.ask("Where is my order A-48217?").content

Prompt management

Pass a Respan-managed prompt under respan_params with with_params. Do not use a top-level prompt key on the Responses path: the OpenAI Responses API has its own native prompt parameter, so a top-level prompt collides with it and the request fails.

Ruby
chat = RubyLLM.chat(model: "gpt-5.5", provider: :openai_responses)
              .with_params(
                respan_params: {
                  prompt: {
                    prompt_id: "YOUR_PROMPT_ID",
                    variables: { store_name: "Lumen Electronics" },
                    schema_version: 2,
                    patch: { top_p: nil }
                  }
                }
              )

puts chat.ask("Hi, where is my order A-48217?").content

The patch drops the prompt’s stored top_p, which reasoning models reject on the Responses API.

Prompt, tools, and reasoning together

The Responses path forwards the prompt’s stored parameters. When the prompt is used with a reasoning model, drop the stored top_p (which reasoning models reject) with a schema v2 patch.

Ruby
chat = RubyLLM.chat(model: "gpt-5.5", provider: :openai_responses)
              .with_temperature(1)          # reasoning models accept only temperature = 1
              .with_thinking(effort: :low)  # reasoning
              .with_tools(GetOrderStatus)   # tools (client-side)
              .with_params(                 # Respan-managed prompt
                respan_params: {
                  prompt: {
                    prompt_id: "YOUR_PROMPT_ID",
                    variables: { store_name: "Lumen Electronics" },
                    schema_version: 2,
                    patch: { top_p: nil }   # reasoning models reject top_p
                  }
                }
              )

puts chat.ask("Hi, where is my order A-48217? It still hasn't arrived.").content

respan_params also accepts every other gateway field (customer_identifier, metadata, and so on). Prompt evaluation reruns stay on the Chat Completions path, but production traffic sent through the Responses API is still logged and can be evaluated as production data.

Multi-tenancy with contexts

Use RubyLLM contexts to isolate per-tenant configuration.

Ruby
tenant_ctx = RubyLLM.context do |config|
  config.openai_api_key = tenant.respan_api_key
  config.openai_api_base = "https://gateway.respan.ai"
end

chat = tenant_ctx.chat(model: "gpt-5.5")
response = chat.ask("Hello!")

Rails integration

Set your Respan config in an initializer.

Ruby
# config/initializers/ruby_llm.rb
RubyLLM.configure do |config|
  config.openai_api_key = ENV["RESPAN_API_KEY"]
  config.openai_api_base = "https://gateway.respan.ai"
end

Use acts_as_chat as normal, and all LLM calls will be routed through Respan.