Before you start

Large language models are non-deterministic. The same prompt can return a great answer today and a wrong one tomorrow, and every change you make (a new model, a reworded prompt, an extra tool) can quietly improve one thing while breaking another. Checking a few outputs by hand does not catch this, and it does not scale. Evaluation is how you replace “looks good to me” with a number you can defend.

A quick example of why this matters

We ran one agent across three models on the same 20 questions, and measured accuracy against the cost the gateway actually billed:

The results were counterintuitive:

• The most capable model was not the best value. claude-opus-4-8 got one more question right than gpt-5.4, but cost roughly 3x more per report and ran twice as slow.
• The framework’s own cost numbers were fiction. The agent self-reported nearly the same cost for every model, while the real billed spread was about 20x (from $0.011 to$0.227 per report).

None of that is visible without evals. You cannot pick the right model, or even trust your own cost numbers, until you measure. Read the full benchmark.

The eval workflow

Most eval setups, including Respan’s, are built from four pieces.

Datasets

A dataset is a collection of cases, each an input and, ideally, a known-good reference output. You build datasets from curated examples or by sampling real production traces, so you are testing against inputs your system actually sees, not toy prompts.

Evaluators

An evaluator scores each output. Three common kinds:

• LLM as a judge. Another LLM grades an output against a rubric or a reference answer. Best for open-ended, subjective quality where exact match fails: faithfulness, tone, helpfulness, “did it actually answer the question.” Keep the judge model and prompt fixed across runs so scores stay comparable, and spot-check it against a few human labels so you trust the grade.
• Code and rule-based. Deterministic checks: exact match, regex, JSON-schema and format validation, latency or cost thresholds. Cheap, fast, and unambiguous.
• Human review. Sampled expert judgment for cases too nuanced to automate, and for calibrating your automated evaluators.

Experiments (offline evals)

An experiment runs a dataset through your system across variants (prompt A vs prompt B, model X vs model Y) and compares the scores. This is offline evaluation: you run it against a fixed dataset before you ship, to regression-test every prompt or model change and to block a release when a score drops. See Experiments.

Online evals

Online evals are a separate thing. Instead of a fixed dataset, they score your live production traffic in real time, so you catch drift, edge cases, and real-world failures you could never fully anticipate offline. Offline experiments tell you whether a change is safe to ship; online evals tell you how it behaves once it is actually live.

What you can evaluate

Evals are not one thing. What you measure depends on what you are building. The common use cases:

• Model selection. Compare models on quality, cost, and latency to pick the right one for your agent. See the benchmark guide.
• Prompts. Regression-test every prompt change, so an “improvement” does not quietly break something else.
• Agent trajectories. Score the whole multi-step run, not just the final message: did the agent finish the task, how many turns did it take, was the reasoning sound.
• Tool use. Did the agent call the right tools, with the right parameters, in a sensible order.
• Retrieval and RAG. Is the retrieved context relevant, and is the answer grounded in it rather than hallucinated.
• Output quality and safety. Correctness, format compliance, tone, and guardrails like toxicity, PII, and jailbreaks.

Most real systems need a few of these at once. Start with the one failure that is hurting you most, then grow coverage from there.