GatewayReliability

Retries

Automatically retry failed requests with configurable attempts and backoff.

For the complete list of all request parameters, see API reference.

When an LLM call fails, the system detects the error and retries the request to prevent failovers.

Go to the Retries page and enable retries and set the number of retries and the initial retry time.

Retries Page
Something went wrong!

Respan will automatically retry failed requests if the failure is a rate limit issue from the upstream provider:

1model # User requested model
2model_params = respan_models_data[model]
3# Exponential backoff retry logic
4for i in range(0, fallback_retries):
5    try:
6        response = respan_response_with_load_balance(model)
7        return response
8        break
9    except RateLimitError:
10        if model_params["fallback_models"]:
11            for fallback_model in model_params["fallback_models"]:
12                response = respan_response_with_load_balance(fallback_model)
13                return response
14        sleep(2 ** i)
15    except Exception as e:
16        raise e