Groq (gateway)

Setup

Install packages

OpenInference (Recommended)

Traceloop

$ pip install respan-ai openinference-instrumentation-groq groq openai python-dotenv

Set environment variables

$ export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"

No GROQ_API_KEY needed — the Respan gateway handles provider authentication.

Initialize and run

OpenInference (Recommended)

Traceloop

1 import os
2 from dotenv import load_dotenv
3 
4 load_dotenv()
5 
6 from openai import OpenAI
7 from respan import Respan
8 from openinference.instrumentation.groq import GroqInstrumentor
9 
10 # Initialize Respan with Groq instrumentation
11 respan = Respan(instrumentations=[GroqInstrumentor()])
12 
13 # Route all LLM calls through the Respan gateway
14 client = OpenAI(
15     api_key=os.getenv("RESPAN_API_KEY"),
16     base_url=os.getenv("RESPAN_BASE_URL", "https://api.respan.ai/api"),
17 )
18 
19 response = client.chat.completions.create(
20     model="llama-3.1-70b-versatile",
21     messages=[{"role": "user", "content": "Say hello in three languages."}],
22 )
23 print(response.choices[0].message.content)
24 respan.flush()

Gateway features

The features below require routing requests through the Respan gateway. Use the setup steps above to configure that flow.

Switch models

Change the model parameter to use 250+ models from different providers through the same gateway.

1 # Groq (Llama)
2 response = client.chat.completions.create(model="llama-3.1-70b-versatile", messages=messages)
3 
4 # OpenAI
5 response = client.chat.completions.create(model="gpt-4.1-nano", messages=messages)
6 
7 # Anthropic
8 response = client.chat.completions.create(model="claude-sonnet-4-5-20250929", messages=messages)

See the full model list.

Respan parameters

Pass additional Respan parameters via extra_body for gateway features.

1 response = client.chat.completions.create(
2     model="llama-3.1-70b-versatile",
3     messages=[{"role": "user", "content": "Hello"}],
4     extra_body={
5         "customer_identifier": "user_123",
6         "fallback_models": ["gpt-4.1-nano"],
7         "metadata": {"session_id": "abc123"},
8         "thread_identifier": "conversation_456",
9     },
10 )

See Respan params & metadata for the full list.

Setup

Install packages

OpenInference (Recommended)

Traceloop

$ pip install respan-ai openinference-instrumentation-groq groq openai python-dotenv

Set environment variables

$ export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"

No GROQ_API_KEY needed — the Respan gateway handles provider authentication.

Initialize and run

OpenInference (Recommended)

Traceloop

1 import os
2 from dotenv import load_dotenv
3 
4 load_dotenv()
5 
6 from openai import OpenAI
7 from respan import Respan
8 from openinference.instrumentation.groq import GroqInstrumentor
9 
10 # Initialize Respan with Groq instrumentation
11 respan = Respan(instrumentations=[GroqInstrumentor()])
12 
13 # Route all LLM calls through the Respan gateway
14 client = OpenAI(
15     api_key=os.getenv("RESPAN_API_KEY"),
16     base_url=os.getenv("RESPAN_BASE_URL", "https://api.respan.ai/api"),
17 )
18 
19 response = client.chat.completions.create(
20     model="llama-3.1-70b-versatile",
21     messages=[{"role": "user", "content": "Say hello in three languages."}],
22 )
23 print(response.choices[0].message.content)
24 respan.flush()

Gateway features

The features below require routing requests through the Respan gateway. Use the setup steps above to configure that flow.

Switch models

Change the model parameter to use 250+ models from different providers through the same gateway.

1 # Groq (Llama)
2 response = client.chat.completions.create(model="llama-3.1-70b-versatile", messages=messages)
3 
4 # OpenAI
5 response = client.chat.completions.create(model="gpt-4.1-nano", messages=messages)
6 
7 # Anthropic
8 response = client.chat.completions.create(model="claude-sonnet-4-5-20250929", messages=messages)

See the full model list.

Respan parameters

Pass additional Respan parameters via extra_body for gateway features.

1 response = client.chat.completions.create(
2     model="llama-3.1-70b-versatile",
3     messages=[{"role": "user", "content": "Hello"}],
4     extra_body={
5         "customer_identifier": "user_123",
6         "fallback_models": ["gpt-4.1-nano"],
7         "metadata": {"session_id": "abc123"},
8         "thread_identifier": "conversation_456",
9     },
10 )

See Respan params & metadata for the full list.

1	import os
2	from dotenv import load_dotenv
3
4	load_dotenv()
5
6	from openai import OpenAI
7	from respan import Respan
8	from openinference.instrumentation.groq import GroqInstrumentor
9
10	# Initialize Respan with Groq instrumentation
11	respan = Respan(instrumentations=[GroqInstrumentor()])
12
13	# Route all LLM calls through the Respan gateway
14	client = OpenAI(
15	api_key=os.getenv("RESPAN_API_KEY"),
16	base_url=os.getenv("RESPAN_BASE_URL", "https://api.respan.ai/api"),
17	)
18
19	response = client.chat.completions.create(
20	model="llama-3.1-70b-versatile",
21	messages=[{"role": "user", "content": "Say hello in three languages."}],
22	)
23	print(response.choices[0].message.content)
24	respan.flush()

1	# Groq (Llama)
2	response = client.chat.completions.create(model="llama-3.1-70b-versatile", messages=messages)
3
4	# OpenAI
5	response = client.chat.completions.create(model="gpt-4.1-nano", messages=messages)
6
7	# Anthropic
8	response = client.chat.completions.create(model="claude-sonnet-4-5-20250929", messages=messages)

1	response = client.chat.completions.create(
2	model="llama-3.1-70b-versatile",
3	messages=[{"role": "user", "content": "Hello"}],
4	extra_body={
5	"customer_identifier": "user_123",
6	"fallback_models": ["gpt-4.1-nano"],
7	"metadata": {"session_id": "abc123"},
8	"thread_identifier": "conversation_456",
9	},
10	)