Deterministic code checks

Not every check needs an LLM. When the rule is deterministic, for example “is this valid JSON?” or “is the response under 280 characters?” or “does it match this regex?”, a code grader is faster, free, and perfectly consistent.

This walkthrough builds a code grader that checks whether a support-ticket classifier returns well-formed JSON. It is the running example for the customer-support evaluators in Weighted-average scoring and Route failures to human review.

When to use a code grader

Use a code grader when	Use an LLM grader when
The rule is exact (format, length, schema, regex, exact match)	The judgment is subjective (quality, tone, relevance)
You want zero cost and zero latency	Some reasoning about meaning is required
The same input must always score the same	Natural-language understanding is required

The grader function

A code grader is a Python main(eval_inputs) function that returns a score. eval_inputs["output"] is always available; input, expected_output, metadata, and metrics are there when the run supplies them (see Where grader inputs come from).

1 import json
2 
3 
4 def main(eval_inputs):
5     """Pass when the model output is valid JSON with the required keys."""
6     output = eval_inputs["output"]
7     if isinstance(output, str):
8         try:
9             output = json.loads(output)
10         except (ValueError, TypeError):
11             return False
12     if not isinstance(output, dict):
13         return False
14     required = {"category", "priority", "sentiment"}
15     return required.issubset(output.keys())

The function returns a boolean, so the grader’s output type is Boolean: true passes, false fails.

eval_inputs["output"] can arrive either as a JSON string or as an already-parsed object, depending on how the run captured it. The check above handles both: it parses strings, and treats anything that is not a JSON object as a fail.

Walkthrough

Create the code grader

In the Graders section, add a grader named Valid classification JSON and set the output type to Boolean. Switch to Code evaluation and paste the main(eval_inputs) function above. Use Templates if you want starter code for a common pattern.

Code grader editor — The code grader editor with the main(eval_inputs) function that validates the classification JSON.

Test both outcomes

Click Test run with a passing output and a failing one to confirm both branches.

Output {"category": "damaged_item", "priority": "high", "sentiment": "negative"} returns true.
Output Sorry to hear that! I'll help you sort this out. returns false.

Code grader test run showing true and false — Test run: valid classification JSON scores true, free-form prose scores false.

Build the workflow and deploy

On the canvas, connect Original input to the Valid classification JSON grader to Final result. Test run the flow, then Deploy.

Code grader evaluator workflow — The evaluator canvas: Original input to the code grader to Final result.

What’s next

Weighted-average scoring. Combine this format check with an LLM quality score.
Route failures to human review. Escalate low-scoring outputs to a person.
Back to the Evaluators overview.