Skip to main content
  1. Sign up — Create an account at platform.respan.ai
  2. Create an API key — Generate one on the API keys page
  3. Add credits or a provider key — Add credits on the Credits page or connect your own provider key on the Integrations page
Add the Docs MCP to your AI coding tool to get help building with Respan. No API key needed.
{
  "mcpServers": {
    "respan-docs": {
      "url": "https://docs.respan.ai/mcp"
    }
  }
}

​
What is LiteLLM?

LiteLLM provides a unified Python interface for calling 100+ LLM providers using the OpenAI format. Respan can instrument all LiteLLM calls for tracing, route them through the Respan gateway, or both.

​
Setup

1

Install packages

pip install respan-ai openinference-instrumentation-litellm litellm python-dotenv
2

Set environment variables

# Set the provider key(s) you want to use
export OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
# export ANTHROPIC_API_KEY="YOUR_ANTHROPIC_API_KEY"
export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"
3

Initialize and run

import os
from dotenv import load_dotenv

load_dotenv()

import litellm
from respan import Respan
from openinference.instrumentation.litellm import LiteLLMInstrumentor

# Initialize Respan with LiteLLM instrumentation
respan = Respan(instrumentations=[LiteLLMInstrumentor()])

# Calls go directly to the provider, auto-traced by Respan
response = litellm.completion(
    model="gpt-4.1-nano",
    messages=[{"role": "user", "content": "Say hello in three languages."}],
)
print(response.choices[0].message.content)
respan.flush()
4

View your trace

Open the Traces page to see your auto-instrumented LLM spans.
This step applies to Tracing and Both setups. The Gateway-only setup does not produce traces.

​
Configuration

ParameterTypeDefaultDescription
api_keystr | NoneNoneFalls back to RESPAN_API_KEY env var.
base_urlstr | NoneNoneFalls back to RESPAN_BASE_URL env var.
instrumentationslist[]Plugin instrumentations to activate (e.g. LiteLLMInstrumentor()).
customer_identifierstr | NoneNoneDefault customer identifier for all spans.
metadatadict | NoneNoneDefault metadata attached to all spans.
environmentstr | NoneNoneEnvironment tag (e.g. "production").

​
Attributes

Attach customer identifiers, thread IDs, and metadata to spans.

​
In Respan()

Set defaults at initialization — these apply to all spans. 
from respan import Respan
from openinference.instrumentation.litellm import LiteLLMInstrumentor

respan = Respan(
    instrumentations=[LiteLLMInstrumentor()],
    customer_identifier="user_123",
    metadata={"service": "chat-api", "version": "1.0.0"},
)

​
With propagate_attributes

Override per-request using a context manager. 
import litellm
from respan import Respan, workflow, propagate_attributes
from openinference.instrumentation.litellm import LiteLLMInstrumentor

respan = Respan(
    instrumentations=[LiteLLMInstrumentor()],
    metadata={"service": "chat-api", "version": "1.0.0"},
)

@workflow(name="handle_request")
def handle_request(user_id: str, question: str):
    with propagate_attributes(
        customer_identifier=user_id,
        thread_identifier="conv_001",
        metadata={"plan": "pro"},
    ):
        response = litellm.completion(
            model="gpt-4.1-nano",
            messages=[{"role": "user", "content": question}],
        )
        print(response.choices[0].message.content)
AttributeTypeDescription
customer_identifierstrIdentifies the end user in Respan analytics.
thread_identifierstrGroups related messages into a conversation.
metadatadictCustom key-value pairs. Merged with default metadata.

​
Decorators

Use @workflow and @task to create structured trace hierarchies. 
import litellm
from respan import Respan, workflow, task
from openinference.instrumentation.litellm import LiteLLMInstrumentor

respan = Respan(instrumentations=[LiteLLMInstrumentor()])

@task(name="generate_outline")
def outline(topic: str) -> str:
    response = litellm.completion(
        model="gpt-4.1-nano",
        messages=[
            {"role": "user", "content": f"Create a brief outline about: {topic}"},
        ],
    )
    return response.choices[0].message.content

@workflow(name="content_pipeline")
def pipeline(topic: str):
    plan = outline(topic)
    response = litellm.completion(
        model="gpt-4.1-nano",
        messages=[
            {"role": "user", "content": f"Write content from this outline: {plan}"},
        ],
    )
    print(response.choices[0].message.content)

pipeline("Benefits of API gateways")
respan.flush()

​
Examples

​
Basic completion

response = litellm.completion(
    model="gpt-4.1-nano",
    messages=[{"role": "user", "content": "Say hello in three languages."}],
)
print(response.choices[0].message.content)

​
Multiple providers

LiteLLM’s unified interface lets you switch between providers by changing the model string. 
# OpenAI
response = litellm.completion(
    model="gpt-4.1-nano",
    messages=[{"role": "user", "content": "Hello"}],
)

# Anthropic
response = litellm.completion(
    model="claude-sonnet-4-5-20250929",
    messages=[{"role": "user", "content": "Hello"}],
)

# Together AI
response = litellm.completion(
    model="together_ai/meta-llama/Llama-3-70b-chat-hf",
    messages=[{"role": "user", "content": "Hello"}],
)

​
Gateway features

The features below require the Gateway or Both setup from Step 3.

​
Switch models

Change the model parameter to use 250+ models from different providers through the same gateway. 
# OpenAI
response = litellm.completion(api_key=RESPAN_KEY, api_base=RESPAN_URL, model="gpt-4.1-nano", messages=messages)

# Anthropic
response = litellm.completion(api_key=RESPAN_KEY, api_base=RESPAN_URL, model="claude-sonnet-4-5-20250929", messages=messages)

# Google
response = litellm.completion(api_key=RESPAN_KEY, api_base=RESPAN_URL, model="gemini-2.5-flash", messages=messages)
See the full model list.

​
Respan parameters

Pass additional Respan parameters via extra_body for gateway features. 
response = litellm.completion(
    api_key=os.getenv("RESPAN_API_KEY"),
    api_base="https://api.respan.ai/api",
    model="gpt-4.1-nano",
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={
        "customer_identifier": "user_123",
        "fallback_models": ["gpt-3.5-turbo"],
        "metadata": {"session_id": "abc123"},
        "thread_identifier": "conversation_456",
    },
)
See Respan parameters for the full list.