Evaluate a multi-turn assistant

Everything so far graded one output. A chat assistant is different: it has to carry context across turns, stay consistent with what it already said, respect constraints the user gave three messages ago, and actually move toward a resolution. A reply can be perfect on its own and still be wrong in context, because it forgot something or contradicted itself. So a single-call eval is not enough here.

What breaks in a conversation

The failure modes that only show up across turns:

Memory loss. The user said “I’m on the Pro plan” earlier; five turns later the assistant asks which plan they are on.
Contradiction. It says one thing, then says the opposite later in the same chat.
Constraint drift. The user asked for answers under 50 words, or in Spanish, and the assistant quietly stops honoring it.
No progress. Every reply is individually fine, but the conversation loops and never resolves the request.
Ignoring corrections. The user says “no, I meant the annual plan,” and the assistant keeps answering about the monthly one.

None of these are visible if you grade a single reply in isolation.

Two lenses: the turn and the outcome

Evaluate a conversation at two levels:

Turn level. Given the conversation so far, was this reply good: relevant, correct, consistent with earlier turns, still honoring the constraints? This catches the drift and contradiction failures at the moment they happen.
Conversation level. Taking the whole exchange, did it reach the goal: was the user’s request resolved, in how many turns, without contradicting itself? This is the one that matters most.

Lead with the outcome. Grade whether the conversation accomplished the task, not whether it followed a particular script. There are many good ways to help a user; pinning the eval to one exact path punishes assistants that took a different but valid route. Score the result, then use turn-level checks to explain a bad result.

The dataset is a conversation, not a row

Your cases are whole conversations, and you can build them two ways:

Recorded transcripts. Real (or replayed) conversations from production, each labeled with whether it succeeded. Best for regression testing against things that actually happened.
Scenarios. A goal and a starting situation (“user wants to downgrade their plan but keep their data”), with the conversation generated at eval time. Best for coverage, since you can write the hard cases you have not seen yet.

For scenarios you need something to play the user. The common approach is a simulated user: an LLM given the goal and persona, talking to your assistant until the task is done or a turn limit is hit. You then grade the resulting transcript. It is the only practical way to test branchy conversations without hand-scripting every path.

What to score

Task completion (outcome). Did the conversation resolve the user’s actual request? Usually an LLM judge reading the full transcript against the goal.
Efficiency. How many turns did it take? A resolution in three turns beats the same resolution in nine.
Consistency. Does the assistant ever contradict itself or forget a stated constraint across the transcript?
Per-turn quality. The single-call checks (relevance, correctness, tone, groundedness) applied at each step, to localize where a bad conversation went wrong.

What this gives you

You catch the failures that only exist across turns, memory loss and contradiction, which single-reply evals cannot see.
You measure the thing that matters, whether the user got helped, not whether the bot followed a script.
A bad outcome is explainable. The conversation score tells you it failed; the turn-level scores tell you which turn dropped the ball.

Next steps

Run task-completion and consistency checks on live conversations, so real-world failures surface in production. That is the online evals page.
When a turn is the weak link, improve that prompt with the single-call loop from earlier in this guide.