API ReferenceEvaluations

Eval Sets Create

POST handler with superadmin-only field protection.

Strips superadmin-only fields from non-superadmin requests before delegating to OrganizationInjectionMixin.post() for org injection.

Authentication

AuthorizationBearer
JWT access token or Respan API key

Path parameters

idstringRequired

Request

This endpoint expects an object.
namestringRequired1-255 characters
organizationintegerRequired
initial_log_filtersobjectOptional

Pydantic model for FilterParamDict. A dictionary that maps metric names to their filter parameters.

Each key is a metric name (str), and each value can be:

  • A single MetricFilterParamPydantic (one condition)
  • A List[MetricFilterParamPydantic] (multiple conditions for same metric)
  • A FilterBundlePydantic (nested filter bundle with connector)

Note: Uses extra=“allow” for dynamic metric name fields. The pydantic_extra annotation tells Pydantic what types to expect for extra fields, and generates typed additionalProperties in JSON Schema.

projectstring or nullOptional>=1 character
typeenumOptional
  • llm - Llm
  • human - Human
  • dataset - Dataset
  • sampling - Sampling
  • reference - Reference
descriptionstringOptional
running_progressdoubleOptional
running_statusenumOptional
  • draft - Draft
  • pending - Pending
  • paused - Paused
  • running - Running
  • waiting_for_annotations - Waiting For Annotations
  • completed - Completed
  • failed - Failed
running_atdatetime or nullOptional
unique_organization_idslist of stringsOptional
timestampslist of datetimesOptional
ingest_workflow_idstring or nullOptional<=255 characters
starredbooleanOptional
evaluatorstring or nullOptional>=1 character

Response

idstringRead-only
completed_annotation_countintegerRead-only
updated_byobject
tagslist of objectsRead-only
namestring<=255 characters
granularityenum

Eval unit for this dataset: ‘logs’ (one span per row) or ‘traces’ (one root row per trace). Chosen at creation and immutable.

  • logs - Logs
  • traces - Traces
  • threads - Threads
created_atdatetimeRead-only
updated_atdatetimeRead-only
log_countintegerRead-only
statusenum
  • initializing - Initializing
  • ready - Ready
  • failed - Failed
  • loading - Loading
organizationinteger
initial_log_filtersobject

Pydantic model for FilterParamDict. A dictionary that maps metric names to their filter parameters.

Each key is a metric name (str), and each value can be:

  • A single MetricFilterParamPydantic (one condition)
  • A List[MetricFilterParamPydantic] (multiple conditions for same metric)
  • A FilterBundlePydantic (nested filter bundle with connector)

Note: Uses extra=“allow” for dynamic metric name fields. The pydantic_extra annotation tells Pydantic what types to expect for extra fields, and generates typed additionalProperties in JSON Schema.

projectstring or null
typeenum
  • llm - Llm
  • human - Human
  • dataset - Dataset
  • sampling - Sampling
  • reference - Reference
descriptionstring
running_progressdouble
running_statusenum
  • draft - Draft
  • pending - Pending
  • paused - Paused
  • running - Running
  • waiting_for_annotations - Waiting For Annotations
  • completed - Completed
  • failed - Failed
running_atdatetime or null
unique_organization_idslist of strings
timestampslist of datetimes
ingest_workflow_idstring or null<=255 characters
starredboolean
evaluatorstring or null