Groq (gateway)

Route Groq model calls through the Respan gateway to use Respan credentials, request logs, routing, fallbacks, and metadata. For direct Groq SDK tracing, see Groq tracing setup.

Setup

Install packages

$ pip install openai python-dotenv

Set environment variables

$ export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"

No GROQ_API_KEY is required for gateway calls when your Groq provider credentials are configured in Respan.

Point an OpenAI-compatible client to the Respan gateway

1 import os
2 
3 from dotenv import load_dotenv
4 from openai import OpenAI
5 
6 load_dotenv()
7 
8 client = OpenAI(
9     api_key=os.environ["RESPAN_API_KEY"],
10     base_url=os.getenv("RESPAN_BASE_URL", "https://api.respan.ai/api"),
11 )
12 
13 response = client.chat.completions.create(
14     model="groq/llama-3.1-8b-instant",
15     messages=[{"role": "user", "content": "Say hello in three languages."}],
16 )
17 print(response.choices[0].message.content)

Switch models

Change the model parameter to use 250+ models from different providers through the same gateway.

1 response = client.chat.completions.create(model="groq/llama-3.1-8b-instant", messages=messages)
2 response = client.chat.completions.create(model="gpt-4.1-nano", messages=messages)
3 response = client.chat.completions.create(model="claude-sonnet-4-5-20250929", messages=messages)

See the full model list.

Respan parameters

Pass additional Respan parameters via extra_body for gateway features.

1 response = client.chat.completions.create(
2     model="groq/llama-3.1-8b-instant",
3     messages=[{"role": "user", "content": "Hello"}],
4     extra_body={
5         "customer_identifier": "user_123",
6         "fallback_models": ["gpt-4.1-nano"],
7         "metadata": {"session_id": "abc123"},
8         "thread_identifier": "conversation_456",
9     },
10 )

See Respan params & metadata for the full list.

1	import os
2
3	from dotenv import load_dotenv
4	from openai import OpenAI
5
6	load_dotenv()
7
8	client = OpenAI(
9	api_key=os.environ["RESPAN_API_KEY"],
10	base_url=os.getenv("RESPAN_BASE_URL", "https://api.respan.ai/api"),
11	)
12
13	response = client.chat.completions.create(
14	model="groq/llama-3.1-8b-instant",
15	messages=[{"role": "user", "content": "Say hello in three languages."}],
16	)
17	print(response.choices[0].message.content)

1	response = client.chat.completions.create(
2	model="groq/llama-3.1-8b-instant",
3	messages=[{"role": "user", "content": "Hello"}],
4	extra_body={
5	"customer_identifier": "user_123",
6	"fallback_models": ["gpt-4.1-nano"],
7	"metadata": {"session_id": "abc123"},
8	"thread_identifier": "conversation_456",
9	},
10	)