> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://respan.ai/docs/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://respan.ai/docs/_mcp/server.

# Haystack (gateway)

> Set up Haystack with the Respan gateway.

Route Haystack's OpenAI-compatible LLM calls through the Respan gateway to use 1000+ models from different providers. No separate OpenAI provider key is required.

## Setup

#### Install packages

```bash
pip install haystack-ai
```

#### Set environment variables

```bash
export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"
```

#### Point Haystack to the Respan gateway

```python
import os

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

os.environ["OPENAI_API_KEY"] = os.environ["RESPAN_API_KEY"]
os.environ["OPENAI_BASE_URL"] = "https://api.respan.ai/api"

pipeline = Pipeline()
pipeline.add_component(
    "prompt_builder",
    ChatPromptBuilder(
        template=[ChatMessage.from_user("Answer the following question: {{question}}")],
        required_variables=["question"],
    ),
)
pipeline.add_component("generator", OpenAIChatGenerator(model="gpt-5-mini"))
pipeline.connect("prompt_builder", "generator")

result = pipeline.run(
    {"prompt_builder": {"question": "What is the capital of France?"}}
)
print(result["generator"]["replies"][0].text)
```

> **Note**
>
> Haystack's OpenAI-compatible generators read OpenAI-shaped environment variables. The example maps those values from `RESPAN_API_KEY`; users do not need an OpenAI provider key.

## Switch models

Change the `model` parameter on `OpenAIChatGenerator` to use another OpenAI model through the same gateway-backed endpoint.

```python
pipeline.add_component(
    "generator",
    OpenAIChatGenerator(model="gpt-5.5"),
)
```

> **Note**
>
> `OpenAIChatGenerator` is Haystack's OpenAI-compatible chat generator. Haystack 3 removed the older `OpenAIGenerator`, so use the chat generator with `ChatPromptBuilder` and `ChatMessage` as shown above. This page avoids showing Claude or Gemini inside that OpenAI-named component; use the Respan API or OpenAI SDK gateway pages for provider-neutral Claude and Gemini examples.

See the [full model list](https://platform.respan.ai/platform/models).

## Prompt management

Use [Respan prompt management](/docs/documentation/features/prompt-management/prompt-management-quickstart) to store the prompt template on the platform, then pass the managed `prompt_id` and variables through Haystack's `generation_kwargs.extra_body.prompt`. Use `schema_version: 2` for all new integrations.

```python
import os

from haystack import Pipeline
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

PROMPT_ID = "YOUR_PROMPT_ID"

os.environ["OPENAI_API_KEY"] = os.environ["RESPAN_API_KEY"]
os.environ["OPENAI_BASE_URL"] = "https://api.respan.ai/api"

pipeline = Pipeline()
pipeline.add_component(
    "generator",
    OpenAIChatGenerator(model="gpt-5-mini"),
)

result = pipeline.run(
    {
        "generator": {
            "messages": [
                ChatMessage.from_user("Who created Python?"),
            ],
            "generation_kwargs": {
                "temperature": 0.0,
                "extra_body": {
                    "prompt": {
                        "prompt_id": PROMPT_ID,
                        "schema_version": 2,
                        "variables": {
                            "question": "Who created Python?",
                            "context": "Python was created by Guido van Rossum and first released in 1991.",
                        },
                        "override": True,
                    }
                },
            },
        }
    }
)

print(result["generator"]["replies"][0].text)
```

> **Note**
>
> Haystack still requires a runtime message for `OpenAIChatGenerator`. The Respan gateway reads `extra_body.prompt`, renders the managed prompt from the platform, and uses the prompt variables for the final model request. The prompt must have a deployed version, otherwise the gateway returns a 404.

See the [full Haystack examples](https://github.com/respanai/respan-example-projects/tree/main/python/tracing/haystack).