Haystack (gateway) | Respan Docs

Route Haystack’s OpenAI-compatible LLM calls through the Respan gateway to use 250+ models from different providers. No separate OpenAI provider key is required.

Setup

Install packages

$ pip install haystack-ai

Set environment variables

$ export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"

Point Haystack to the Respan gateway

1 import os
2 
3 from haystack import Pipeline
4 from haystack.components.builders import PromptBuilder
5 from haystack.components.generators import OpenAIGenerator
6 
7 os.environ["OPENAI_API_KEY"] = os.environ["RESPAN_API_KEY"]
8 os.environ["OPENAI_BASE_URL"] = os.getenv("RESPAN_BASE_URL", "https://api.respan.ai/api")
9 
10 pipeline = Pipeline()
11 pipeline.add_component(
12     "prompt_builder",
13     PromptBuilder(template="Answer the following question: {{question}}"),
14 )
15 pipeline.add_component("generator", OpenAIGenerator(model="gpt-5-mini"))
16 pipeline.connect("prompt_builder", "generator")
17 
18 result = pipeline.run(
19     {"prompt_builder": {"question": "What is the capital of France?"}}
20 )
21 print(result["generator"]["replies"][0])

Switch models

Change the model parameter on OpenAIGenerator to use another OpenAI model through the same gateway-backed endpoint.

1 pipeline.add_component(
2     "generator",
3     OpenAIGenerator(model="gpt-5.5"),
4 )

OpenAIGenerator is Haystack’s OpenAI-compatible generator. This page avoids showing Claude or Gemini inside that OpenAI-named component; use the Respan API or OpenAI SDK gateway pages for provider-neutral Claude and Gemini examples.

See the full model list.

Prompt management

Use Respan prompt management to store the prompt template on the platform, then pass the managed prompt_id and variables through Haystack’s generation_kwargs.extra_body.prompt. Use schema_version: 2 for all new integrations.

1 import os
2 
3 from haystack import Pipeline
4 from haystack.components.generators.chat import OpenAIChatGenerator
5 from haystack.dataclasses import ChatMessage
6 
7 PROMPT_ID = "YOUR_PROMPT_ID"
8 
9 os.environ["OPENAI_API_KEY"] = os.environ["RESPAN_API_KEY"]
10 os.environ["OPENAI_BASE_URL"] = os.getenv("RESPAN_BASE_URL", "https://api.respan.ai/api")
11 
12 pipeline = Pipeline()
13 pipeline.add_component(
14     "generator",
15     OpenAIChatGenerator(model="gpt-5-mini"),
16 )
17 
18 result = pipeline.run(
19     {
20         "generator": {
21             "messages": [
22                 ChatMessage.from_user("Who created Python?"),
23             ],
24             "generation_kwargs": {
25                 "temperature": 0.0,
26                 "extra_body": {
27                     "prompt": {
28                         "prompt_id": PROMPT_ID,
29                         "schema_version": 2,
30                         "variables": {
31                             "question": "Who created Python?",
32                             "context": "Python was created by Guido van Rossum and first released in 1991.",
33                         },
34                         "override": True,
35                     }
36                 },
37             },
38         }
39     }
40 )
41 
42 print(result["generator"]["replies"][0].text)

Haystack still requires a runtime message for OpenAIChatGenerator. The Respan gateway reads extra_body.prompt, renders the managed prompt from the platform, and uses the prompt variables for the final model request.

See the full Haystack examples.