What Is LLM-as-a-Judge? Method, Examples, and Best Practices

An LLM feature in production generates more output in a week than a team will read in a quarter. Before release someone spot-checks thirty responses, decides it looks fine, and ships. Everything after that goes out unread.

LLM-as-a-judge is the method most teams reach for to close that distance. One model scores another system's output against a criterion you write, returns a number and a reason for it, and does that on every output instead of the handful a person had time for.

What it buys is coverage. A judge runs continuously, costs cents per thousand outputs, handles the subjective qualities that no regex can express, and changes when you edit a paragraph rather than when you retrain something. What it costs is certainty. Scores vary between runs, judges carry documented preferences for longer answers and for text from their own model family, and a judge nobody has checked against human labels produces confident numbers that may point the wrong way.

That tradeoff is workable, and the teams who get value out of it are the ones who treat the judge as a system to validate rather than a setting to switch on. Getting there means knowing how the LLM-as-a-judge method actually works, what a real example looks like in practice, and which best practices keep a score worth acting on.

What Is LLM-as-a-Judge?

LLM-as-a-judge is an evaluation method that uses one language model to score another system's output against criteria you define in writing. The judge receives an evaluation prompt containing the criterion, the output under review, and whatever context that criterion needs. It returns a score, a label, or a preference, usually with a reason attached.

The idea sounds circular at first. If a model produced the output, asking a model to grade it seems like asking the same question twice.

It works because the two tasks are not the same size. Generating a response means holding a system prompt, a user request, retrieved context, and a set of tools in mind at once. Grading one property of a finished response is closer to classification, and a model given one narrow question to answer tends to answer it better than a model juggling everything at once.

A judge score is not a metric in the sense that accuracy or precision are metrics. Those are defined independently of whoever computes them, and two teams measuring accuracy are measuring the same thing. A judge score is defined by the words in your evaluation prompt, so two teams measuring "helpfulness" are measuring whatever each of them wrote down. The number is a proxy for human labeling that you specify and then have to validate.

Three forms of the method came out of the original research, and they answer different questions:

• Single-output, reference-free - The judge scores one output against a written criterion with no correct answer supplied. Best for open-ended qualities like tone, relevance, or whether the response addressed the question asked.
• Single-output, reference-based - The judge also receives a known-good answer and scores the output against it. Best for correctness, where the reference anchors the judgment and keeps scoring consistent between runs.
• Pairwise comparison - The judge sees two outputs for the same input and picks the better one. Best for A/B decisions between prompts, models, or fine-tunes, where relative quality is easier to establish than an absolute score.

The reference-based form has a practical limit worth knowing early, because production traffic rarely arrives with a golden answer attached. That pushes correctness scoring toward offline test sets and leaves the reference-free form to handle live traffic.

The technique got its name from Zheng et al., 2023, which introduced MT-Bench and Chatbot Arena and found that strong judge models matched human preferences at over 80% agreement, about the rate at which human annotators agree with each other.

LLM-as-a-Judge Example

Consider an internal tool that drafts release notes from merged pull requests. The failure that matters is not a clumsy sentence. It is a release note describing a change that is not in the diff, which ships to customers as a feature they do not have.

The criterion, written as a question with a fixed answer set: does every claim in the release note appear in the diff it was generated from? That is faithfulness to a source, and no regex catches it.

The evaluation prompt sets out the scale and defines every point on it:

You are checking a generated release note against the code diff it was written from.
 
Diff:
{{input}}
 
Release note:
{{output}}
 
Score 1 to 5 on whether every claim in the release note is supported by the diff.
 
5 = every claim maps to a change in the diff
3 = claims are supported but one is overstated in scope or impact
1 = the note describes a change that does not appear in the diff
 
Think through each claim in the note and say which lines of the diff support it.
Then reply with the number on its own line.

Run that against a note whose claims all trace back to the diff and the judge returns a 5, with reasoning that names the supporting change for each claim. Run it against a note announcing a performance improvement when the diff only renamed a variable and it returns a 1, with reasoning pointing at the unsupported claim.

Now strip the definitions out and ask the same judge to "rate this release note from 1 to 5 on quality." The scores that come back are not wrong exactly, they are unrepeatable, because nothing in the instruction says whether a well-written note about the wrong change beats a plain note about the right one. The rubric is the part doing the work.

One thing the score does not tell you: whether the judge is right. A 5 means the judge believes every claim is supported. Establishing that the judge's 5 matches a person's 5 is a separate exercise, and it is the difference between a number you can act on and a number you can only look at.

Is LLM-as-a-Judge Reliable?

A judge is reliable when it agrees with the people whose judgment it is standing in for, on your task, at a rate you have actually measured. That makes reliability a property of a specific judge against a specific set of labels rather than a property of the method, and it is why two teams can run the same technique and end up with numbers worth very different amounts.

Published agreement figures are a poor substitute for measuring your own. Work by Thakur et al. evaluating thirteen judge models found that percent agreement obscures more than it reveals, since judges with similar agreement rates can assign substantially different scores. The same work found that judges are easiest to trust in the direction you least need them: true positives stayed stable across judges, while true negatives fell away as judge quality dropped. Confirming that a correct answer is correct is the easy half of the job.

The failure modes worth knowing before you trust a number:

• Verbosity preference - Judges tend to rate longer responses higher, so a prompt change that produces more output can lift a score without improving anything.
• Family preference - Judges tend to favor text from their own model family, which puts a thumb on the scale whenever the judge and the system under test share a lineage.
• Position preference - In pairwise comparisons, the ordering of the two candidates influences which one wins.
• Leniency - Judges skew toward passing, which means the scores that matter most, the ones catching a bad output, are the ones arriving least reliably.
• Prompt sensitivity - The length and quality of the evaluation prompt changes the scores it produces, so rubric edits are score changes.
• Under-specified answers - Outputs that are incomplete rather than wrong are the hardest class for a judge to call, and judges sometimes miss even an answer matching the reference word for word.

Drift is the failure that separates a judge you built from a judge you run. The judge model gets updated underneath you, traffic shifts while the criterion stays fixed, and the rubric that described good output last quarter quietly stops describing it. None of that announces itself in the score, which is why the labeled set is worth re-running on a schedule rather than once at the start.

A judge is good enough to ship when its disagreements with your labels are ones you understand and can live with, and when you know which direction it errs in. Perfect agreement is not the bar, since human reviewers do not reach it with each other either.

LLM-as-a-Judge Pros and Cons

The method earns its place on economics as much as on quality. What it gives you and what it costs you both scale with volume, which is why the tradeoff looks different at a thousand outputs a day than it does at a million.

What works in its favor:

• Scale and turnaround - A judge scores continuously and returns results in seconds, which makes scoring every deploy and a slice of production traffic a realistic ambition rather than a staffing question.
• No reference answer required - Reference-free criteria score an output on its own merits, which is what makes production monitoring possible at all when nobody knows the correct response.
• Criteria in plain language - Changing what you measure means editing a paragraph, so the definition of quality can keep pace with the product instead of lagging a release behind.
• A reason with every score - Asking for reasoning leaves an audit trail, and reading a handful of those explanations is the fastest way to find out your criterion says something you did not intend.
• Domain experts can participate - The evaluation prompt is prose, so the people who know what a good output looks like can write and revise it without going through an engineer.

What works against it:

• Scores are not deterministic - The same output can score differently across runs, which is manageable at low temperature and with coarse labels, though it does mean small deltas between conditions are often noise.
• Cost scales with traffic and with depth - Each score is an extra model call, and attaching four criteria to every output multiplies that by four before traffic grows at all.
• Too slow for inline enforcement - A judge adds a full model round trip, so blocking a response on a judge verdict pushes latency into territory most user-facing paths cannot absorb.
• Data leaves your environment - Scoring through a third-party provider means outputs go to that provider, which is a question worth answering in advance for regulated workloads.
• Setup and upkeep are real work - Writing criteria, labeling a set, measuring agreement, and re-checking on a schedule is a recurring commitment rather than a one-time integration.

The tradeoff, in the end, is coverage against certainty. A judge gives you a defensible number on every output instead of a confident number on the few somebody had time to read, and whether that trade is worth making depends on how much of your traffic currently gets read by nobody.

LLM-as-a-Judge vs Human Evaluation

Human labels are what a judge gets validated against, so a team running judges is already running human review, just on a much smaller share of the volume. The question is rarely which one to use. It is how much of the traffic each one covers, and which cases get routed from the first to the second.

The division of labor follows from the table rather than from a preference about which one is better.

• Humans - Produce the labels the judge is measured against, settle cases where a wrong call is expensive, and catch failures that depend on knowing things the judge was never told, such as whether a claim about your product is true.
• Judges - Score the outputs nobody would otherwise read, hold a release gate open or shut without a person in the loop, and run continuously against production instead of in batches whenever someone has capacity.

The routing rule between them deserves more thought than it usually gets. Sending every low-scoring output to a human routes on the judge's own confidence, and a judge that under-scores cautious, hedged answers will fill the review queue with careful responses while fluent wrong ones sail through. Routing on disagreement instead, either between two judges or between a judge and a code check, surfaces the cases where something is genuinely unresolved.

Most production systems land in the same split: deterministic checks on anything with a verifiable answer, a judge across all or a sampled share of traffic, and human review on a small percentage drawn from both flagged and random cases. Working through the full evaluation method walks that out against a real system.

How to Use the LLM-as-a-Judge Method

Building a judge is closer to a small machine learning project than to writing a prompt. There is a labeled dataset, a model that approximates those labels, and a measurement of how well it does.

1. Start with one criterion, written as a question - Not "quality," but something with a yes and a no, like whether the response answered the question that was asked. If two properties matter, that is two judges, because a single score covering both moves for reasons you cannot recover afterward.

2. Pull the evaluation set from real traffic - Twenty to fifty outputs is enough to start, weighted toward the cases that are hard to call rather than the obvious passes. Cases your users actually sent beat cases you invented, which encode what you expect to happen rather than what does.

3. Label the set yourself before writing the prompt - This is the step people skip and the one that pays, because labeling forces the edge decisions into the open. The disagreements you have with yourself while doing it are the ambiguities the judge will hit.

4. Write the evaluation prompt from those labels - Define every point on the scale, keep the scale coarse, and ask for reasoning before the verdict so the reasoning shapes the answer instead of justifying it.

5. Measure, then iterate - Run the judge over your labeled set and look at every disagreement, since most early ones trace to the rubric rather than to the model. Revise, re-run, and keep a held-out slice for a final check.

6. Fix the judge configuration - Pick the judge model deliberately, avoiding the family that produced the outputs where you can. Set temperature to zero and pin both, so a score from this month is comparable to one from last month.

7. Deploy it in both places - The same criterion belongs in CI as a gate on changes and in production as a monitor on live traffic, because a fixed test set cannot see a model update or a shift in how users phrase things.

8. Sample and filter before you scale it - Filtering to the traffic that matters and sampling within it will surface a systematic regression nearly as fast as scoring everything, and filtering first keeps a uniform sample from mostly covering the safe path.

9. Close the loop - The failing case becomes a row in the dataset, the fix runs against that row and everything else in the set, and the number either moves or it does not.

Reference-based criteria are the exception to step seven, since production traffic rarely arrives with a golden answer attached, so correctness checks stay offline while reference-free criteria run in both places. The loop in step nine is also where most setups stall, because exporting spans, reformatting them into test cases, and loading them into a second product is work somebody has to prioritize against shipping.

Run LLM Judges on Production Traffic With Respan

Respan is an AI evaluation tool that runs the same judge offline against a dataset and online against live production spans, and links every score back to the run that produced it. Click a failing score and land on the exact execution behind it, with the input, the output, the tool calls, the latency, and the cost attached.

From there the loop closes inside one product. The failing case becomes a dataset row, the fix becomes an experiment, and the score moves or it does not.

Here's how Respan helps:

• Judges you write, on models you choose - Define the criterion in plain language, pick the judge model and temperature, set the score range and the passing threshold.
• Composable evaluators - Chain an LLM judge, a deterministic code check, and a human reviewer into one workflow that returns a single score per output.
• Conditional routing - Send only low-scoring outputs to a person, or gate an expensive judge on cost or latency so it runs where it matters.
• Online evals - Deploy the same evaluator on live spans, filtered by status, customer, or thread and sampled to control spend, so regressions surface in real time.
• Datasets from real traffic - Pull requests from your logs by filter and sampling rate, or upload a CSV, so every test case is one your users actually sent.
• Experiments - Run a prompt version, a model, or a dataset through your evaluators at scale, with per-row and average scores and a click into the trace behind any row.
• Versioned evaluators - Every deploy publishes an immutable version with history and rollback, so a scoring change never silently invalidates a trend.
• Monitors and alerts - Watch scores, cost, errors, or latency over any window, with alerts to Slack, email, or a webhook the moment a threshold breaks.
• Span-level tracing - Every LLM call, tool run, retrieval, and agent turn becomes a span with its own input, output, latency, and cost.
• Prompt management - Commit and compare prompt versions and deploy instantly, with no code change and no redeploy.

Respan is SOC 2, HIPAA, GDPR, and ISO 27001 compliant, with PII masking and omit-logs for teams handling regulated data. The free tier includes two LLM-as-judge evaluators, 100k logs, 1k scores, and five datasets, which is enough to run a judge against real production traffic rather than a demo.

LLM-as-a-Judge Best Practices

Most of what separates a judge you can act on from one you cannot comes down to a handful of decisions made when the judge is written, plus a habit of re-checking it afterward.

• Prefer coarse labels - Binary or three-way categories repeat across runs in a way that fine-grained numeric scales do not, and a scale nobody can define point by point will not produce comparable numbers.
• One criterion per judge - Split completeness, accuracy, and tone into separate graders and combine the results deterministically afterward, which keeps each score interpretable and tells you which property actually moved.
• Define every label, including the middle one - If you cannot articulate what separates a 3 from a 4, the judge cannot either, and that is usually a sign the scale should be shorter.
• Ask for reasoning before the verdict - Chain-of-thought scoring leaves an explanation you can audit, and the G-Eval work built a whole framework around generating evaluation steps first and scoring against them.
• Keep the judge out of the family under test - A judge from the same model family as the system it grades has a structural preference for one of the candidates, and using a different family costs nothing to arrange.
• Swap positions on pairwise comparisons - Run each comparison in both orderings and count a win only when the same output wins twice, which controls for position bias directly.
• Re-run the labeled set on a schedule - Judge models change, traffic changes, and criteria age, so treat agreement as something you re-measure quarterly rather than establish once.
• Push verifiable checks into code - Schema validity, required fields, forbidden strings, and citation counts are exact, and spending a model call to confirm that JSON parses costs money and latency for a worse answer.
• Sample deliberately in production - Filter to the traffic that carries risk before you sample within it, since a uniform sample of a system where a small share of traffic is the risky path will mostly score the safe path.

None of these make a judge correct. They make it consistent enough that a change in the score reflects a change in the system, which is the only property that makes the number worth tracking. Getting there is mostly a matter of narrowing what you ask and then checking the answer against people, and both are cheaper than they look next to the cost of shipping a regression you could have caught.

Frequently Asked Questions

What is G-Eval?

G-Eval is a framework for making judge scores more stable, introduced by Liu et al. in 2023. It asks the judge model to generate detailed evaluation steps from your criterion first, then scores the output against those generated steps in a form-filling format, and derives the final number from a probability-weighted summation of the possible scores rather than a single sampled token.

The practical takeaway survives even if you never implement it directly. Making the reasoning explicit before the verdict, and avoiding a single sampled number as the final answer, are both available in any judge you write.

What is the difference between single-output and pairwise LLM-as-a-judge?

Single-output scoring grades one output at a time against a rubric and returns an absolute score, with or without a reference answer. Pairwise comparison shows the judge two outputs for the same input and asks which is better, returning a preference rather than a number.

Use single-output when you need a value you can track over time, which covers regression gates and production monitoring, since an absolute score is comparable across runs and a preference is not. Use pairwise when you are choosing between two candidates, such as a prompt revision or a model swap, where relative quality is easier to judge than an absolute one. Pairwise carries position bias, so run both orderings.

Which model should you use as a judge?

Start with a capable current model, since alignment with human judgment generally improves with model strength, then test whether a smaller and cheaper one holds up on your specific criterion before scaling it across production traffic. Many narrow criteria do not need a frontier model.

Two constraints matter more than the choice itself. Avoid a judge from the same family as the system under test, because self-preference is well documented across current models. And pin the model and temperature once you have validated the judge, because a model update underneath a fixed rubric will move your scores without anything about your system changing.

How much does LLM-as-a-judge cost to run in production?

Two costs run in parallel. The inference cost is a model call per score, which is cents per thousand outputs with a small judge model and considerably more with a frontier model scoring long inputs. The platform cost depends on how scoring is metered, and score-based billing scales with how many criteria you attach per output, so evaluating more deeply raises the bill with no change in traffic.

Respan's free tier covers 1k scores, 100k logs, and two LLM-as-judge evaluators, which is enough to run a judge against live traffic and measure the real cost curve before committing. Sampling is the larger lever either way, since filtering to the traffic that carries risk and scoring a fraction of it surfaces systematic regressions at a fraction of the spend.

Can an LLM judge replace human review?

Not entirely, and the goal is usually not replacement. Human labels are the reference a judge is validated against, so removing human review removes the only evidence that the judge is measuring what you think it is.

What a judge replaces is the reading of every output by hand, which does not scale past a certain volume and was never happening consistently anyway. The workable split is a judge across all or a sampled share of traffic, with human review concentrated on calibration, on high-stakes cases, and on outputs where two automated scorers disagree.

How do you run LLM judges on live production traffic?

The mechanics are the same criterion applied to spans as they arrive rather than to rows in a test set, filtered to the traffic you care about and sampled to control cost. Reference-free criteria work here because they score an output on its own merits, while correctness checks that need a golden answer stay offline.

In Respan, an evaluator built for offline experiments deploys against live spans without rewriting it, with filtering by status, customer, or thread and sampling to control spend. Every score links back to the span that produced it, and alerts fire to Slack, email, or a webhook when a score crosses a threshold, so a regression surfaces from the platform rather than from a customer. The full evaluation stack covers how the online half fits alongside metrics and offline gates.