For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
DiscordPlatform
DocumentationIntegrationsAPI referenceSDKsChangelog
DocumentationIntegrationsAPI referenceSDKsChangelog
  • API Reference
      • POSTCreate an evaluator
      • POSTList evaluators with filters
      • GETRetrieve an evaluator
      • PATCHUpdate an evaluator
      • DELDelete an evaluator
      • POSTRun an evaluator
      • GETList evaluators
      • GETGet evaluators summary
      • POSTGet filtered evaluators summary
      • PUTReplace an evaluator
      • GETList evaluator versions
      • POSTCreate evaluator version
      • GETRetrieve evaluator version
      • PUTReplace evaluator version
      • PATCHUpdate evaluator version
  • Reference
    • Filters API Reference
LogoLogo
DiscordPlatform
API ReferenceEvaluators

Run an evaluator

POST
https://api.respan.ai/api/evaluators/:evaluator_id/run/
POST
/api/evaluators/:evaluator_id/run/
$curl -X POST https://api.respan.ai/api/evaluators/evaluator_id/run/ \
>     -H "Authorization: Bearer sk_live_xxxxx" \
>     -H "Content-Type: application/json" \
>     -d '{
>  "inputs": {
>    "input": "What is the capital of France?",
>    "output": "The capital of France is Paris.",
>    "expected_output": "Paris",
>    "metrics": {
>      "latency": 0.45,
>      "cost": 0.0023
>    },
>    "metadata": {
>      "model": "gpt-4o-mini"
>    }
>  }
>}'
1{
2  "id": "res_123e4567-e89b-12d3-a456-426614174000",
3  "created_at": "2024-01-15T09:30:00Z",
4  "type": "numerical",
5  "environment": "production",
6  "numerical_value": 4.5,
7  "string_value": "Good response",
8  "boolean_value": true,
9  "categorical_value": [
10    "accurate"
11  ],
12  "json_value": "{\"score\":4.5,\"comments\":\"Well done\"}",
13  "is_passed": true,
14  "cost": 0.0023,
15  "evaluator_id": "evl_550e8400",
16  "evaluator_slug": "response_quality",
17  "scorer": "gpt-4o-mini",
18  "log_id": "log_987654321",
19  "prompt_id": "prm_123456789",
20  "prompt_version_number": 2,
21  "dataset_id": "ds_abc123",
22  "automation_id": "auto_456def",
23  "status": "completed",
24  "error_message": "",
25  "inputs": {
26    "input": "What is the capital of France?",
27    "output": "The capital of France is Paris.",
28    "expected_output": "Paris",
29    "metrics": {},
30    "metadata": {},
31    "llm_input": "What is the capital of France?",
32    "llm_output": "The capital of France is Paris."
33  },
34  "evaluator": {
35    "id": "evl_550e8400",
36    "name": "Response Quality",
37    "type": "llm",
38    "score_value_type": "numerical",
39    "version_id": "evlv_550e8400",
40    "version": 1,
41    "is_read_only": false,
42    "version_description": "Tightened rubric wording",
43    "evaluator_slug": "response_quality",
44    "eval_class": "keywordsai_custom_llm",
45    "description": "Grades whether the response is accurate and complete.",
46    "score_config": {
47      "min_score": 1,
48      "max_score": 5,
49      "choices": [
50        {
51          "name": "Excellent",
52          "value": "excellent"
53        }
54      ]
55    },
56    "passing_conditions": {
57      "primary_score": {
58        "operator": "gte",
59        "value": 3
60      }
61    },
62    "llm_config": {
63      "model": "gpt-4o-mini",
64      "evaluator_definition": "Rate the response quality.\n<input>{{input}}</input>\n<output>{{output}}</output>",
65      "scoring_rubric": "1=Poor, 5=Excellent",
66      "temperature": 0.1,
67      "max_tokens": 200,
68      "top_p": 1.1,
69      "frequency_penalty": 1.1,
70      "presence_penalty": 1.1,
71      "stop": [
72        "\n"
73      ],
74      "response_format": {},
75      "tools": [
76        {}
77      ],
78      "verbosity": "detailed"
79    },
80    "code_config": {
81      "eval_code_snippet": "def main(eval_inputs):\n    return 1 if eval_inputs.get('output') else 0"
82    },
83    "configurations": {},
84    "categorical_choices": [
85      {
86        "name": "Excellent",
87        "value": "excellent"
88      }
89    ],
90    "starred": false,
91    "created_at": "2024-01-15T09:30:00Z",
92    "updated_at": "2024-01-15T09:30:00Z",
93    "created_by": {
94      "id": 101,
95      "first_name": "Alice",
96      "last_name": "Johnson",
97      "email": "alice.johnson@respan.ai"
98    },
99    "updated_by": {
100      "id": 102,
101      "first_name": "Bob",
102      "last_name": "Smith",
103      "email": "bob.smith@respan.ai"
104    },
105    "editor": {
106      "id": 103,
107      "first_name": "Carol",
108      "last_name": "Davis",
109      "email": "carol.davis@respan.ai"
110    },
111    "tags": [
112      {
113        "id": 1,
114        "name": "quality",
115        "color": "#4CAF50"
116      }
117    ]
118  }
119}

Run an evaluator against raw unified inputs. The evaluator ID may include a version suffix such as evl_abc123:2 to run a specific version.

Was this page helpful?
Previous

List evaluators

Next
Built with

Path parameters

evaluator_idstringRequired

Evaluator ID. To run a specific version, pass an ID with a version suffix where supported, for example evl_abc123:2.

Headers

AuthorizationstringRequired

Bearer token. Use Bearer YOUR_API_KEY for API key auth or Bearer <JWT> for dashboard auth.

Request

This endpoint expects an object.
inputsobjectRequired
Unified evaluator inputs.
generation_methodenumOptionalDefaults to auto
Optional method override for evaluators that support multiple execution modes.
Allowed values:
evaluation_idstringOptional

Legacy evaluator ID field. Prefer the path parameter or evaluator_id.

evaluator_idstringOptional

Optional evaluator ID override. Supports version suffixes such as evl_abc123:2.

Response

Evaluation result.
idstring

Score/result ID.

created_atdatetime
typestring
environmentstring
numerical_valuedouble or null
string_valuestring or null
boolean_valueboolean or null
categorical_valuelist of strings or null
json_valuestring or null
is_passedboolean or null
costdouble or null
evaluator_idstring
evaluator_slugstring
scorerstring or null
log_idstring or null
prompt_idstring or null
prompt_version_numberinteger or null
dataset_idstring or null
automation_idstring or null
statusstring or null
error_messagestring or null
inputsobject
Unified evaluator inputs.
evaluatorobject

Errors

400
Bad Request Error
401
Unauthorized Error
404
Not Found Error