Run an evaluator | Respan Docs

curl -X POST https://api.respan.ai/api/evaluators/evaluator_id/run/ \
     -H "Authorization: Bearer <respanApiKey>" \
     -H "Content-Type: application/json" \
     -d '{
  "inputs": {
    "input": null,
    "output": null,
    "expected_output": null,
    "metrics": {
      "latency": 0.45,
      "cost": 0.0023
    },
    "metadata": {
      "model": "gpt-4o-mini"
    }
  }
}'

{
  "id": "string",
  "created_at": "2024-01-15T09:30:00Z",
  "type": "string",
  "environment": "string",
  "numerical_value": 1.1,
  "string_value": "string",
  "boolean_value": true,
  "categorical_value": [
    "string"
  ],
  "json_value": "string",
  "is_passed": true,
  "cost": 1.1,
  "evaluator_id": "string",
  "evaluator_slug": "string",
  "scorer": "string",
  "log_id": "string",
  "prompt_id": "string",
  "prompt_version_number": 1,
  "dataset_id": "string",
  "automation_id": "string",
  "status": "string",
  "error_message": "string",
  "inputs": {
    "input": null,
    "output": null,
    "expected_output": null,
    "metrics": {},
    "metadata": {},
    "llm_input": "string",
    "llm_output": "string"
  },
  "evaluator": {
    "id": "evl_550e8400",
    "name": "Response Quality",
    "type": "llm",
    "score_value_type": "numerical",
    "version_id": "evlv_550e8400",
    "version": 1,
    "is_read_only": false,
    "version_description": "Tightened rubric wording",
    "evaluator_slug": "response_quality",
    "eval_class": "keywordsai_custom_llm",
    "description": "Grades whether the response is accurate and complete.",
    "score_config": {
      "min_score": 1,
      "max_score": 5,
      "choices": [
        {
          "name": "Excellent",
          "value": "excellent"
        }
      ]
    },
    "passing_conditions": {
      "primary_score": {
        "operator": "gte",
        "value": 3
      }
    },
    "llm_config": {
      "model": "gpt-4o-mini",
      "evaluator_definition": "Rate the response quality.\n<input>{{input}}</input>\n<output>{{output}}</output>",
      "scoring_rubric": "1=Poor, 5=Excellent",
      "temperature": 0.1,
      "max_tokens": 200,
      "top_p": 1.1,
      "frequency_penalty": 1.1,
      "presence_penalty": 1.1,
      "stop": [
        "string"
      ],
      "response_format": {},
      "tools": [
        {}
      ],
      "verbosity": "string"
    },
    "code_config": {
      "eval_code_snippet": "def main(eval_inputs):\n    return 1 if eval_inputs.get('output') else 0"
    },
    "configurations": {},
    "categorical_choices": [
      {
        "name": "Excellent",
        "value": "excellent"
      }
    ],
    "starred": false,
    "created_at": "2024-01-15T09:30:00Z",
    "updated_at": "2024-01-15T09:30:00Z",
    "created_by": {
      "id": 1,
      "first_name": "string",
      "last_name": "string",
      "email": "string"
    },
    "updated_by": {
      "id": 1,
      "first_name": "string",
      "last_name": "string",
      "email": "string"
    },
    "editor": {
      "id": 1,
      "first_name": "string",
      "last_name": "string",
      "email": "string"
    },
    "tags": [
      {
        "id": 1,
        "name": "string",
        "color": "string"
      }
    ]
  }
}

Run an evaluator against raw unified inputs. The evaluator ID may include a version suffix such as evl_abc123:2 to run a specific version.

Authentication

AuthorizationBearer

Use your Respan API key for Respan API authentication. Enter only the Respan API key value; clients send Authorization: Bearer <RESPAN_API_KEY>. For /api/responses, provider credentials such as Perplexity, OpenAI, or Azure OpenAI go in Settings -> Providers or respan_params.credential_override in the request body, not in this authentication field.

AuthorizationBearer

Use a dashboard JWT only for dashboard-authenticated endpoints. Respan API-key endpoints use the respanApiKey auth field instead.

Path parameters

evaluator_idstringRequired

Evaluator ID. To run a specific version, pass an ID with a version suffix where supported, for example evl_abc123:2.

Request

This endpoint expects an object.

inputsobjectRequired

Unified evaluator inputs.

generation_methodenumOptionalDefaults to auto

Optional method override for evaluators that support multiple execution modes.

Allowed values:

evaluation_idstringOptional

Legacy evaluator ID field. Prefer the path parameter or evaluator_id.

evaluator_idstringOptional

Optional evaluator ID override. Supports version suffixes such as evl_abc123:2.

Response

Evaluation result.

idstringOptional

Score/result ID.

created_atdatetimeOptional

typestringOptional

environmentstringOptional

numerical_valuedouble or nullOptional

string_valuestring or nullOptional

boolean_valueboolean or nullOptional

categorical_valuelist of strings or nullOptional

json_valuestring or nullOptional

is_passedboolean or nullOptional

costdouble or nullOptional

evaluator_idstringOptional

evaluator_slugstringOptional

scorerstring or nullOptional

log_idstring or nullOptional

prompt_idstring or nullOptional

prompt_version_numberinteger or nullOptional

dataset_idstring or nullOptional

automation_idstring or nullOptional

statusstring or nullOptional

error_messagestring or nullOptional

inputsobjectOptional

Unified evaluator inputs.

evaluatorobjectOptional

Errors

400

Bad Request Error

401

Unauthorized Error

404

Not Found Error