Experiments V2 (Beta)

Experiments lets you run repeatable evaluations over a dataset and inspect outputs, evaluator scores, and run status. Choose a workflow type based on your use case.

Prompt workflow

Render a saved prompt template with dataset variables, then run LLM calls automatically.

Via UI
Via API

Click New experiment

Go to Experiments and click New experiment.

Select a dataset

Choose the dataset you want to run on.

Select task = Prompt

Pick Prompt as the task type, then choose the prompt and version.

Select evaluators and create

Select evaluators to score outputs, then click Create. Inspect outputs and scores once the run finishes.

Dataset entries must include the variables your prompt template expects.

Dataset format

{
  "input": {
    "name": "John Doe",
    "issue": "Damaged product",
    "order_id": "ORD-12345"
  }
}

Create a prompt, deploy a version, then create the experiment:

import requests

API_KEY = "YOUR_API_KEY"
url = "https://api.respan.ai/api/v2/experiments/"

payload = {
  "name": "Prompt Workflow Experiment",
  "dataset_id": "YOUR_DATASET_ID",
  "workflows": [{"type": "prompt", "config": {"prompt_id": "YOUR_PROMPT_ID"}}],
  "evaluator_slugs": ["your_evaluator_slug"]
}

res = requests.post(url, headers={"Authorization": f"Bearer {API_KEY}"}, json=payload)
print(res.json())

List results with List experiment logs. Each result includes a span tree with prompt rendering, LLM call, and evaluator spans.

Completion workflow

Run direct LLM completions on dataset messages automatically — no prompt templates needed.

Via UI
Via API

Click New experiment

Go to Experiments and click New experiment.

Select a dataset

Choose the dataset you want to run on.

Select task = LLM generation

Pick LLM generation (chat completion), then configure the model and parameters (temperature, max tokens).

Select evaluators and create

Select evaluators, then click Create. Inspect outputs and scores once the run finishes.

Dataset entries should have input as an OpenAI-style messages array.

Dataset format

{
  "input": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "What is AI?" }
  ]
}

import requests

API_KEY = "YOUR_API_KEY"
url = "https://api.respan.ai/api/v2/experiments/"

payload = {
  "name": "Completion Workflow Experiment",
  "dataset_id": "YOUR_DATASET_ID",
  "workflows": [{
    "type": "completion",
    "config": {
      "model": "gpt-4o-mini",
      "temperature": 0.7,
      "max_tokens": 200
    }
  }],
  "evaluator_slugs": ["your_evaluator_slug"]
}

res = requests.post(url, headers={"Authorization": f"Bearer {API_KEY}"}, json=payload)
print(res.json())

List results with List experiment logs. Multiple completion workflows run sequentially (output of workflow N becomes input to workflow N+1).

Custom workflow

Fetch inputs, run your own code/model, then submit outputs back for automatic evaluation.

Via UI
Via API

Click New experiment

Go to Experiments and click New experiment.

Select a dataset

Choose the dataset you want to run on.

Select task = Custom

Pick Custom as the task type, select evaluators, then click Create.

Submit outputs via API

The system creates placeholder rows. Use the API to submit outputs — evaluators run automatically.

The UI is used to monitor progress and review results. Outputs are submitted via API.

Custom workflow accepts any JSON-serializable input/output.

How it works

1. Create experiment → 2. List placeholders → 3. Process externally → 4. Submit output → 5. Evaluators run

Create the experiment:

import requests

API_KEY = "YOUR_API_KEY"
url = "https://api.respan.ai/api/v2/experiments/"

payload = {
  "name": "Custom Workflow Experiment",
  "dataset_id": "YOUR_DATASET_ID",
  "workflows": [{
    "type": "custom",
    "config": {
      "allow_submission": True,
      "timeout_hours": 24
    }
  }],
  "evaluator_slugs": ["your_evaluator_slug"]
}

res = requests.post(url, headers={"Authorization": f"Bearer {API_KEY}"}, json=payload)
print(res.json())

List placeholders, then PATCH each log with your output. Evaluators run automatically.Reference: List logs | Update log

Custom and built-in workflows are mutually exclusive. Only one custom workflow per experiment.

Troubleshooting

Logs list is empty after creation

The experiment may still be processing — wait 5–10 seconds and retry. Also check that your dataset is not empty.

Evaluator spans never appear

Confirm the evaluator slug exists and is accessible. Evaluators run asynchronously — poll the log detail endpoint after submission.

Inputs/outputs look truncated

Use the detail endpoint to retrieve the full span tree and untruncated fields.

Get started

Features

Admin

Security

Resources

Help & Community

Experiments V2 (Beta)

Prompt workflow

Completion workflow

Custom workflow

Troubleshooting

Get started

Features

Admin

Security

Resources

Help & Community

​Prompt workflow

​Completion workflow

​Custom workflow

​Troubleshooting

Prompt workflow

Completion workflow

Custom workflow

Troubleshooting