Together AI (gateway)

Route Together AI model calls through the Respan gateway to use Respan credentials, request logs, routing, fallbacks, and metadata. No TOGETHER_API_KEY is needed in application code when TogetherAI is configured in Respan.

Setup

Install packages

$ pip install openai python-dotenv

Set environment variables

$ export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"

Optional:

$ export RESPAN_BASE_URL="https://api.respan.ai/api"

Configure TogetherAI provider credentials or credits in Respan. Keep TOGETHER_API_KEY out of your application runtime.

Point an OpenAI-compatible client to the Respan gateway

1 import os
2 
3 from dotenv import load_dotenv
4 from openai import OpenAI
5 
6 load_dotenv()
7 
8 client = OpenAI(
9     api_key=os.environ["RESPAN_API_KEY"],
10     base_url=os.getenv("RESPAN_BASE_URL", "https://api.respan.ai/api"),
11 )
12 
13 response = client.chat.completions.create(
14     model="together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo",
15     messages=[{"role": "user", "content": "Say hello in three languages."}],
16 )
17 print(response.choices[0].message.content)

Switch models

Change the model parameter to use 250+ models from different providers through the same gateway.

1 response = client.chat.completions.create(model="together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo", messages=messages)
2 response = client.chat.completions.create(model="together_ai/Qwen/Qwen2.5-72B-Instruct-Turbo", messages=messages)
3 response = client.chat.completions.create(model="gpt-5.5", messages=messages)

See the full model list.

Respan parameters

Pass additional Respan parameters via extra_body for gateway features.

1 response = client.chat.completions.create(
2     model="together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo",
3     messages=[{"role": "user", "content": "Hello"}],
4     extra_body={
5         "customer_identifier": "user_123",
6         "fallback_models": ["gpt-5.5"],
7         "metadata": {"session_id": "abc123"},
8         "thread_identifier": "conversation_456",
9     },
10 )

See Respan params & metadata for the full list.

Bring your own Together AI key

If you want to bill Together AI directly, attach your provider key in Respan. The provider ID for request-level credentials is togetherai and the credential field is api_key.

1 response = client.chat.completions.create(
2     model="together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo",
3     messages=[{"role": "user", "content": "Hello"}],
4     extra_body={
5         "customer_credentials": {
6             "togetherai": {
7                 "api_key": "YOUR_TOGETHER_API_KEY"
8             }
9         }
10     },
11 )