GatewayLLM SDKs

Ollama (gateway)

Route Ollama model calls through the Respan gateway after the Ollama model is available in your Respan model list. For direct Ollama SDK tracing, see Ollama tracing setup.

Setup

Live verification with the tested gateway key returned model unavailable for the previous ollama/llama3.1 example. Use a model ID that is available in your Respan model list.

1

Install packages

pip install openai python-dotenv
2

Set environment variables

export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"
3

Point an OpenAI-compatible client to the Respan gateway

import os
from dotenv import load_dotenv
from openai import OpenAI
load_dotenv()
client = OpenAI(
    api_key=os.environ["RESPAN_API_KEY"],
    base_url=os.getenv("RESPAN_BASE_URL", "https://api.respan.ai/api"),
)
response = client.chat.completions.create(
    model="YOUR_OLLAMA_MODEL_ID",
    messages=[{"role": "user", "content": "Say hello in three languages."}],
)
print(response.choices[0].message.content)

Switch models

Change the model parameter to use 1000+ models from different providers through the same gateway.

response = client.chat.completions.create(model="YOUR_OLLAMA_MODEL_ID", messages=messages)
response = client.chat.completions.create(model="gpt-5.5", messages=messages)
response = client.chat.completions.create(model="claude-sonnet-4-5-20250929", messages=messages)

See the full model list.

Respan parameters

Pass additional Respan parameters via extra_body for gateway features.

response = client.chat.completions.create(
    model="YOUR_OLLAMA_MODEL_ID",
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={
        "customer_identifier": "user_123",
        "fallback_models": ["gpt-5.5"],
        "metadata": {"session_id": "abc123"},
        "thread_identifier": "conversation_456",
    },
)

See Respan params & metadata for the full list.