For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
DiscordPlatform
DocumentationIntegrationsAPI referenceSDKsChangelog
DocumentationIntegrationsAPI referenceSDKsChangelog
  • API Reference
      • POSTCreate an evaluator
      • POSTList evaluators with filters
      • GETRetrieve an evaluator
      • PATCHUpdate an evaluator
      • DELDelete an evaluator
      • POSTRun an evaluator
      • GETList evaluators
      • GETGet evaluators summary
      • POSTGet filtered evaluators summary
      • PUTReplace an evaluator
      • GETList evaluator versions
      • POSTCreate evaluator version
      • GETRetrieve evaluator version
      • PUTReplace evaluator version
      • PATCHUpdate evaluator version
  • Reference
    • Filters API Reference
LogoLogo
DiscordPlatform
API ReferenceEvaluators

Create an evaluator

POST
https://api.respan.ai/api/evaluators/
POST
/api/evaluators/
$curl -X POST https://api.respan.ai/api/evaluators/ \
>     -H "Authorization: Bearer sk_live_xxxxx" \
>     -H "Content-Type: application/json" \
>     -d '{
>  "name": "Response Quality",
>  "type": "llm",
>  "score_value_type": "numerical",
>  "evaluator_slug": "response_quality",
>  "eval_class": "keywordsai_custom_llm",
>  "score_config": {
>    "min_score": 1,
>    "max_score": 5
>  },
>  "passing_conditions": {
>    "primary_score": {
>      "operator": "gte",
>      "value": 3
>    }
>  },
>  "llm_config": {
>    "model": "gpt-4o-mini",
>    "evaluator_definition": "Rate the quality.\n<input>{{input}}</input>\n<output>{{output}}</output>",
>    "scoring_rubric": "1=Poor, 5=Excellent"
>  }
>}'
1{
2  "id": "evl_550e8400",
3  "name": "Response Quality",
4  "type": "llm",
5  "score_value_type": "numerical",
6  "version_id": "evlv_550e8400",
7  "version": 1,
8  "is_read_only": false,
9  "version_description": "Tightened rubric wording",
10  "evaluator_slug": "response_quality",
11  "eval_class": "keywordsai_custom_llm",
12  "description": "Grades whether the response is accurate and complete.",
13  "score_config": {
14    "min_score": 1,
15    "max_score": 5,
16    "choices": [
17      {
18        "name": "Excellent",
19        "value": "excellent"
20      }
21    ]
22  },
23  "passing_conditions": {
24    "primary_score": {
25      "operator": "gte",
26      "value": 3
27    }
28  },
29  "llm_config": {
30    "model": "gpt-4o-mini",
31    "evaluator_definition": "Rate the response quality.\n<input>{{input}}</input>\n<output>{{output}}</output>",
32    "scoring_rubric": "1=Poor, 5=Excellent",
33    "temperature": 0.1,
34    "max_tokens": 200,
35    "top_p": 0.9,
36    "frequency_penalty": 0.5,
37    "presence_penalty": 0.3,
38    "stop": [
39      "###"
40    ],
41    "response_format": {},
42    "tools": [
43      {}
44    ],
45    "verbosity": "detailed"
46  },
47  "code_config": {
48    "eval_code_snippet": "def main(eval_inputs):\n    return 1 if eval_inputs.get('output') else 0"
49  },
50  "configurations": {},
51  "categorical_choices": [
52    {
53      "name": "Excellent",
54      "value": "excellent"
55    }
56  ],
57  "starred": false,
58  "created_at": "2024-01-15T09:30:00Z",
59  "updated_at": "2024-01-15T09:30:00Z",
60  "created_by": {
61    "id": 101,
62    "first_name": "Alice",
63    "last_name": "Johnson",
64    "email": "alice.johnson@respan.ai"
65  },
66  "updated_by": {
67    "id": 102,
68    "first_name": "Bob",
69    "last_name": "Smith",
70    "email": "bob.smith@respan.ai"
71  },
72  "editor": {
73    "id": 103,
74    "first_name": "Carol",
75    "last_name": "Davis",
76    "email": "carol.davis@respan.ai"
77  },
78  "tags": [
79    {
80      "id": 10,
81      "name": "Quality",
82      "color": "#4CAF50"
83    }
84  ]
85}
Create a grader/evaluator. Current public evaluator types are `llm`, `human`, and `code`; legacy human-specific types remain readable for older evaluators. New clients should use the flat config fields (`score_config`, `passing_conditions`, `llm_config`, `code_config`) instead of relying on legacy `configurations`.
Was this page helpful?
Previous

List evaluators with filters

Next
Built with

Create a grader/evaluator. Current public evaluator types are llm, human, and code; legacy human-specific types remain readable for older evaluators. New clients should use the flat config fields (score_config, passing_conditions, llm_config, code_config) instead of relying on legacy configurations.

Headers

AuthorizationstringRequired

Bearer token. Use Bearer YOUR_API_KEY for API key auth or Bearer <JWT> for dashboard auth.

Request

This endpoint expects an object.
namestringRequired
typeenumRequired
Allowed values:
score_value_typeenumRequired
evaluator_slugstringOptional

Organization-scoped evaluator slug.

eval_classenumOptional

Optional pre-built evaluator template.

Allowed values:
descriptionstring or nullOptional
score_configobjectOptional

Score configuration. For numerical/percentage scores, use min_score and max_score. For select scores, use choices.

passing_conditionsobject or nullOptional
Passing conditions in the standard Respan filter format.
llm_configobject or nullOptional
LLM grader configuration. The backend validates this against the selected evaluator form.
code_configobject or nullOptional
Code grader configuration.
configurationsmap from strings to anyOptional

Legacy user-facing configuration object. New clients should prefer llm_config, code_config, score_config, and passing_conditions.

categorical_choiceslist of objectsOptional
starredbooleanOptional

Response

Created evaluator.
idstring
Stable evaluator ID shared by all versions.
namestring
typeenum
Allowed values:
score_value_typeenum
version_idstring
Unique ID for this evaluator version.
versioninteger
Version number. The first draft is version 0.
is_read_onlyboolean
Whether this version is committed and immutable.
version_descriptionstring
Commit message for this version.
evaluator_slugstring

Organization-scoped evaluator slug.

eval_classenum

Optional pre-built evaluator template.

Allowed values:
descriptionstring or null
score_configobject

Score configuration. For numerical/percentage scores, use min_score and max_score. For select scores, use choices.

passing_conditionsobject or null
Passing conditions in the standard Respan filter format.
llm_configobject or null
LLM grader configuration. The backend validates this against the selected evaluator form.
code_configobject or null
Code grader configuration.
configurationsmap from strings to any

Legacy user-facing configuration object. New clients should prefer llm_config, code_config, score_config, and passing_conditions.

categorical_choiceslist of objects or null
starredboolean
created_atdatetime
updated_atdatetime
created_byobject
updated_byobject
editorobject
tagslist of objects

Errors

400
Bad Request Error
401
Unauthorized Error