Provider: Custom / Self-hosted

Call your own self-hosted or custom LLM endpoint through Respan Gateway.

This page is for Respan LLM Gateway users wiring up a custom or self-hosted model.

Bring your own OpenAI-compatible endpoint and call it through Respan Gateway to get unified observability (logs, cost, latency, reliability) alongside the rest of your models.

Paste this into your coding agent:


Help me set up the Respan gateway to call Custom / Self-hosted models.
Follow https://www.respan.ai/docs/integrations/gateway/model-providers/custom.md

Quick setup

Go to Gateway › API keys › Bring your own key and click New custom provider. Fill in Provider ID, Provider name, and Base URL, click Add provider, then add your endpoint’s key with Add deployment.

Create a custom model

Go to Gateway › Models, click New custom model, and pick your custom provider. Use the model name your endpoint expects.

Call your custom model

The base URL is https://api.respan.ai/api and the only key needed is your RESPAN_API_KEY. Replace MY_CUSTOM_MODEL_ID with the model ID you created and verified in your project. This placeholder is intentional: custom model IDs are project-specific and are not entries in the public model catalog.

Point the OpenAI SDK at the Respan gateway and call your custom model.

Python


from openai import OpenAI

client = OpenAI(
    api_key="YOUR_RESPAN_API_KEY",
    base_url="https://api.respan.ai/api",
)

response = client.chat.completions.create(
    model="MY_CUSTOM_MODEL_ID",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

More integrations

Custom models work with every Respan gateway integration over the OpenAI-compatible API:

Override credentials per model (Optional)

Use credential_override when a single request should route to a different endpoint or API key than the one configured on your custom provider.

JSON


{
  "credential_override": {
    "MY_CUSTOM_MODEL_ID": {
      "api_key": "YOUR_CUSTOM_API_KEY",
      "api_base": "https://your-endpoint.example.com/v1"
    }
  }
}

Log without proxying (Optional)

Already calling your endpoint directly? Send logs to Respan asynchronously to track cost, latency, and performance for those external calls.

Python


import requests

requests.post(
    "https://api.respan.ai/api/request-logs/create/",
    headers={
        "Authorization": "Bearer YOUR_RESPAN_API_KEY",
        "Content-Type": "application/json",
    },
    json={
        "model": "MY_CUSTOM_MODEL_ID",
        "prompt_messages": [{"role": "user", "content": "Hello!"}],
        "completion_message": {"role": "assistant", "content": "Hello from my custom provider."},
        "cost": 0.001,
        "generation_time": 1.2,
        "customer_params": {"customer_identifier": "user_123"},
    },
)

See the logging guide for the full setup.

Custom Providers API

Create, list, update, and delete custom providers programmatically.