LLM Evaluation in Production: Metrics, Methods, and Tools

A prompt edit lifts the average score from 4.1 to 4.4, and the change ships. The average moved because most cases got slightly better. Underneath it, the twelve cases that required citing a source went from correct to confidently wrong, and the mean absorbed them.

That happens whenever quality collapses into one number. A single score averages over criteria that move independently, so a gain on tone can hide a loss on faithfulness, and nothing in the number tells you which one you are looking at.

Scoring is also the part most teams stop doing after launch. The suite runs in CI, the build goes green, and once the change is live the only remaining signal is a support ticket. The failures that cost money, however, tend to appear in traffic that no fixed dataset contains.

So evaluation has to work in two places, gating changes before they ship and scoring live traffic after they do. LLM evaluation in production depends on the metrics you choose and the methods that produce them, and on whether your tools keep scoring once the traffic is real.

What Is LLM Evaluation?

LLM evaluation is the practice of scoring model or agent output against defined quality criteria, using deterministic checks, model-based judges, human reviewers, or some combination. It replaces spot-checking with a number that repeats, which is what makes a prompt change comparable to the prompt it replaced.

The term stretches across two jobs that get confused with each other: ranking a base model against other models, and measuring your own system on your own traffic.

Model evaluation vs application evaluation

Model evaluation measures a base model's general capability. MMLU, SWE-bench, GPQA, and the rest exist to rank models against each other on tasks nobody's users are actually performing. They are useful once, when you are picking a starting model.

Application evaluation measures your system on your data. The same model scores differently inside a retrieval pipeline than it does on a benchmark, because the failure surface includes your prompt, your context, your tools, and your users' phrasing. A model that tops a leaderboard can still be the wrong choice for a workload where the deciding factor is how it behaves when retrieval returns nothing useful.

Nearly everything worth measuring after the model selection decision is application evaluation.

Offline and online evaluation

Offline evaluation scores a fixed dataset before a change ships. It answers whether the new prompt beats the old one, whether the cheaper model holds up, whether the refactor broke something that used to work. Because the dataset is fixed, the comparison is clean.

Online evaluation scores production spans as they arrive. It answers whether the thing you shipped is still working, which is a different question and the one that fixed datasets structurally cannot address. A provider ships a point release, the phrasing of user queries drifts through a season, a retrieval index goes stale, and none of it registers against a test set written in March.

Teams that run only the offline half learn about regressions from customers. Teams that run only the online half catch problems after they have reached users. The two are worth wiring to the same evaluators so a failure caught in production can become a test case without a translation step.

LLM Evaluation Methods

A score comes from somewhere, and the source determines what it can and cannot see. Cost, latency, and trustworthiness all vary by an order of magnitude across the options below, which is why production systems mix them rather than standardizing on one.

Deterministic code checks

Anything with a verifiable answer belongs in code. Schema validity, required fields, numeric ranges, forbidden strings, citation counts, and tool-call structure are all exact, and running a language model to check whether JSON parses wastes money and adds latency for a worse answer.

Code checks are also the ones that never drift. A regex written a year ago still means what it meant, which is not true of a judge prompt evaluated by a model that has since been updated. Start here, and push as much as will fit.

LLM-as-a-judge

Subjective criteria need a model. Faithfulness to retrieved context, tone, whether an answer addressed the question actually asked rather than an adjacent one, and whether a refusal was correct all resist encoding as rules, and a judge model scoring against a written rubric handles them at a cost per call low enough to run at scale.

The catch is calibration. Judges carry systematic biases, including preferences for longer responses, for particular writing styles, and for outputs from their own model family. An uncalibrated judge produces confident numbers in the wrong direction, and nothing about the output signals that it is happening.

Calibrating means scoring a set of human-labeled examples with the judge, measuring agreement, and revising the rubric wherever the two diverge. Budget for it before you budget for the judge itself, because a judge nobody has checked against human labels is an opinion with a decimal point on it.

Human review

Humans are the tiebreaker and the calibration source. Route the cases where automated scoring is low-confidence, where the stakes make a wrong score expensive, and where the judge and the code check disagree with each other.

Random sampling matters as much as targeted review. If reviewers only ever see cases the automated layer already flagged, you learn about the failures you can already detect and nothing about the ones you cannot.

Benchmarks and where they stop being useful

Public benchmarks are the cheapest way to narrow a model shortlist and a poor way to decide anything after that. Contamination is part of the reason, since a benchmark published long enough ago has likely appeared in training data somewhere. The larger reason is that a benchmark measures a task that is not your task.

There is a subtler failure worth knowing about. Final-answer scoring credits a correct answer regardless of how it was reached, so a model that arrives at the right result through enumeration or guesswork scores identically to one that reasoned its way there. On a benchmark that inflates the number. In your application it means a passing score sitting on top of a process that will not generalize.

Common LLM Evaluation Metrics for Production

Metrics group by what they measure and which systems they apply to. Every production system needs the task-agnostic set, and the rest follows from your architecture, since running RAG metrics against a system with no retrieval step adds cost and no signal.

Task-agnostic metrics

These apply to almost any system that produces text.

• Hallucination rate - Whether the output contains claims that are not grounded in the provided context or in fact.
• Instruction adherence - Whether the response respected what the prompt asked, including constraints on length, scope, and format.
• Answer relevance - Whether the response addressed the question asked.
• Format validity - Whether the output matches an expected structure such as a JSON schema. Deterministic, and cheap enough to run on everything.
• Refusal correctness - Whether the system declined when it should have, and answered when the refusal was unnecessary.

RAG metrics

Retrieval adds its own failure surface, and generation metrics alone cannot tell you whether the problem was the answer or the chunks behind it.

• Faithfulness - Whether every claim in the answer is supported by the retrieved context.
• Context precision - What share of retrieved chunks were relevant to the query.
• Context recall - Whether retrieval surfaced the chunks needed to answer at all.
• Answer correctness - Whether the final response matches ground truth on a labeled set.

Splitting retrieval quality from generation quality is what makes the number actionable, and RAG evaluation goes through instrumenting these against a live pipeline.

Agent and tool-use metrics

Agents fail in shapes that single-turn systems do not have, so scoring the final answer grades the destination and ignores the route.

• Tool selection accuracy - Whether the agent reached for the right tool at each step.
• Parameter validity - Whether the arguments passed to that tool were well-formed and semantically correct.
• Task completion - Whether the agent finished what it was asked to do, scored binary or against a rubric.
• Trajectory efficiency - Whether the path was reasonable, or whether the agent looped and backtracked its way to the answer.
• Error recovery - Whether a failed tool call was handled or cascaded.

An agent that called the wrong API, caught the error, retried against the right one, and returned a correct answer passes a final-output check while hiding a bug that surfaces under load. Agent evaluation covers the span schema these metrics attach to.

Safety metrics

• Toxicity and bias - Whether outputs contain harmful or policy-violating content.
• PII and PHI leakage - Whether sensitive data appears in outputs or in the logs behind them.
• Prompt injection resistance - Whether adversarial input can override system instructions.
• Jailbreak susceptibility - Whether crafted prompts push the system outside its intended behavior.

The last two are measured by attacking the system rather than by scoring normal traffic, which is why they usually sit alongside AI red teaming tools rather than inside the regular eval suite.

Operational metrics

• P95 latency - Tail latency by span, which is where user-visible slowness lives.
• Cost per request - Token spend broken down by model, by request, and by end customer.
• Error rate - Provider errors, rate limits, and tool failures, classified by cause.
• Throughput - Requests sustained under load, and where the ceiling sits.

Quality metrics without operational ones describe a system that scores well and costs too much to run. Keeping both on one dashboard is the practical argument for not separating evaluation from AI observability across two tools.

Why Evaluation Breaks After Deploy

Most teams get the offline half working and stall on the online half. The gap is rarely a missing metric, since the metric is usually already defined and running in CI. It is that the score arrives in production detached from everything needed to act on it.

• The score has no trace behind it - A number saying faithfulness dropped to 0.71 last Tuesday is a notification, not a finding. Acting on it needs the input, the retrieved context, the tool calls, the model version, and the prompt version that produced it, and when the scoring system and the tracing system are different products, assembling that takes the first hour of the incident. The reverse fails the same way: a trace with no score attached leaves you reading logs hoping to notice a pattern, which works until traffic exceeds what a person can read.
• The failure never becomes a test case - Catching a production failure is only worth the effort if it prevents the next one, which means the failing run becomes a dataset row, the fix runs as an experiment against that row and everything already in the set, and the score either moves or it does not. Where that loop crosses a tool boundary it tends not to happen, because exporting spans, reformatting them into test cases, and loading them into a second product is work somebody has to prioritize against shipping.
• Scoring everything costs too much to sustain - Running an LLM judge across 100% of production traffic gets expensive enough that teams either skip online evaluation or switch it off after the first invoice. Sampling solves it, and a well-chosen 5% filtered by status, customer, or thread will surface a systematic regression about as fast as scoring everything would. Filter before you sample, though, since a uniform random sample of a system where 2% of traffic is the risky path will mostly score the safe path.

None of these are solved by picking better metrics. They are solved by keeping the score, the run behind it, and the dataset it feeds in the same place, which is what separates the tools below more than their metric libraries do.

10 LLM Evaluation Tools

Our list of the best LLM evaluation tools below runs platforms first and libraries second, which reflects where the scoring lives. A platform stores results, runs judges against live traffic, and keeps the history that makes a trend readable, and a library produces a score and leaves storage, orchestration, and production scheduling to you.

1. Respan

Respan scores production traffic and connects every score to the run that produced it. Click a failing eval and land on the exact agent execution behind it, with the input, the output, the tool calls, the latency, and the cost attached. From there the failing case becomes a dataset row, the fix becomes an experiment, and the number moves or it does not.

The same evaluator definition runs offline against a dataset and online against live spans, so the gate in CI and the monitor in production measure the same thing.

Respan features:

• Online evals - The same evaluator deployed on live production spans, filtered by status, customer, or thread and sampled to control cost, so regressions surface in real time.
• Composable evaluators - An LLM judge, a deterministic code check, and a human reviewer composed into one evaluator that turns any output into a single score.
• Datasets from real traffic - Pull requests from logs by filter and sampling rate, or upload a CSV, so every test case is one your users actually sent.
• Experiments - Run a prompt version, a model, or a dataset through your evaluators at scale, with per-row and average scores, side-by-side distributions, and a click into the full trace behind any row.
• Human review - Route failures to a person for scoring when automated evaluators cannot settle them.
• Span-level tracing - Every LLM call, tool run, retrieval, and agent turn becomes a span nested parent to child, each with its own input, output, latency, and cost.
• Prompt management - Commit and compare prompt versions and deploy instantly, with no code change and no redeploy to change a prompt.
• Red teaming - Adversarial campaigns probe for prompt injection, system-prompt leakage, secret disclosure, and goal hijacking, graded by severity, against the same agent you already trace.
• Gateway - One endpoint for 1,000+ models with automatic failover, retries, and load balancing, with every request logged as a span.
• Monitors - Watch cost, errors, latency, or tokens over any window, with alerts to Slack, email, or a webhook the moment a threshold breaks.

Respan is SOC 2, HIPAA, GDPR, and ISO 27001 compliant, with PII masking and omit-logs for teams handling regulated data.

Pricing: Free covers 100k logs, 1k scores, 5 datasets, and 2 evaluators at 7-day retention, which is enough to run online evals against real traffic rather than a demo. Team is $199/mo billed yearly and lifts datasets, evaluators, and prompts to unlimited with 10k scores, 30-day retention, and a 99.9% uptime SLA. Enterprise adds a HIPAA BAA and custom retention.

2. Braintrust

Braintrust organizes evaluation around the experiment. Change a prompt, run it against a dataset, and read per-scorer deltas against the baseline before the branch merges.

With tools like Braintrust, experiments are immutable snapshots, which is what separates them from playground runs that overwrite results on each iteration and makes a quality trend readable over months.

Scoring comes from the Autoevals library of prebuilt functions, from LLM-as-judge scorers written against natural-language criteria, or from custom code. Note that Braintrust's documentation notes that autoevals score individual spans rather than whole traces, so trajectory-level scoring on a multi-step agent means writing that logic yourself.

Pricing: Starter is free with 1 GB and 10k scores, and Pro is $249/mo for 5 GB and 50k scores. Billing meters scores rather than traces, so a team running five scorers per output reaches the cap five times faster than a team running one, and the bill tracks how deeply you evaluate rather than how much traffic you serve.

3. Langfuse

Langfuse is MIT-licensed and self-hostable on every tier, including the paid ones, which is unusual in a category where deployment is normally the upgrade trigger.

Judges come from a managed catalog that Langfuse maintains with partners including Ragas, covering hallucination, context relevance, toxicity, and helpfulness, or you write the prompt yourself and preview it against recent project data.

Langfuse never proxies your LLM calls and observes asynchronously outside the request path, so failover, spend caps, and retries stay in whatever gateway you pair it with.

Pricing: Core is $29/mo and Pro is $199/mo on the same 100k units, meaning the difference buys retention and compliance paperwork rather than volume. A unit is a trace, an observation, or a score, so one agent request with twenty spans and three scorers meters as twenty-four.

4. LangSmith

LangSmith puts evaluation directly on the tracing layer, with online evaluators applied to production runs at a configurable sampling rate and datasets assembled from those same runs. Beyond the usual code and judge evaluators tools like LangSmith support pairwise comparison, which scores two candidate outputs against each other rather than each against a rubric, and composite evaluators that roll several feedback keys into one weighted score.

Two meters run at once, and both climb faster than you might expect. Trace counts include every run inside a chain, so a pipeline with retrieval, reranking, and generation bills three times per user query. There is no read-only seat either, so a product manager who only reads results costs the same as an engineer writing scorers.

Pricing: Plus is $39/seat/mo with 10k base traces, then $2.50 per 1k base traces and $5.00 per 1k at extended retention. Cost scales on headcount and instrumentation granularity simultaneously, so a chatty agent traced at span level on a ten-person team compounds both at once.

5. Arize

Two products ship under the Arize name and the split decides what you get. Phoenix is the open-source platform, free and self-hostable, covering tracing, span-level evals, prompt playgrounds, datasets, and experiments. Arize AX is the managed platform built on top of it, adding longer retention, custom dashboards, production monitoring, and compliance.

Where the two diverge for production work is orchestration. Arize's documentation describes online evaluators as running on the AX platform, where Arize handles triggering as traces arrive, execution, and joining results back to the originating spans. The Phoenix path is the off-platform one, meaning you download the spans, score them with a pipeline you build, and write the results back yourself.

Pricing: AX Free covers 25k spans, AX Pro is $50/mo for 50k spans and 10 GB, then $10 per million spans and $3/GB. Nothing is published between Pro and Enterprise, so a SOC 2 or HIPAA requirement moves you from $50/mo straight to a sales conversation.

6. Galileo

Galileo built its evaluation layer on small language models rather than frontier judges. Luna-2 is fine-tuned specifically for scoring, which brings per-evaluation latency and cost down far enough to make monitoring a large share of production traffic practical, and the model can be further tuned on your own criteria.

Worth knowing before you plan around it: Galileo's own documentation states that Luna-2 is available only in the Enterprise tier, and runtime guardrails sit there too, so the free and Pro tiers give you scores without the cost profile or the enforcement that make the platform distinctive.

Pricing: Free covers 5k traces and Pro is $100/mo for 50k, with Enterprise carrying unlimited traces plus VPC and on-premise deployment. The capability most teams evaluate Galileo for is on the tier with no published price.

7. Promptfoo

Promptfoo is declarative and CLI-first. Test cases, prompt variants, providers, and assertions live in a YAML file, promptfoo eval runs the matrix, and the whole thing drops into CI without a service to stand up.

The product has moved considerably toward security. Its own navigation now leads with red teaming, guardrails, model security, and an MCP proxy, with evaluations listed last and pointing at documentation rather than a product page. The Community tier is free permanently and includes all evaluation features, all providers, and red teaming capped at 10k probes per month, though there is no managed UI and no production-traffic scoring, so continuous monitoring means adding a second tool.

Pricing: Community is free forever and runs locally or on your own infrastructure. Enterprise and On-Premise are both custom with nothing published, and they are where team sharing, continuous monitoring, SSO, API access, and managed cloud deployment live.

8. DeepEval

DeepEval models evaluation as unit testing. Assertions run under pytest against LLMTestCase objects, deepeval test run executes them, and a failing threshold fails the build, which puts evaluation in the place engineers already look. It is Apache 2.0 and runs metrics through any judge model you point it at, or through local NLP models when you would rather not send data to a provider.

G-Eval scores against criteria written in plain language, and DAG composes deterministic decision steps into a judge, which gives repeatable scoring for criteria that have a definite structure. Storage, dashboards, and production scoring are not part of the framework, so results live wherever you put them until you connect the hosted platform.

Pricing: The library is free. The hosted path is Confident AI, where Starter is $200/mo and Team is $2,000/mo, both with unlimited seats and traces and storage billed at $1/GB-month. There is nothing between those tiers, so outgrowing Starter is a tenfold jump.

9. Ragas

Ragas started as a RAG evaluation toolkit and the metric design still reflects that origin, with faithfulness, context precision, context recall, and answer relevancy as the core set. It is a pip install and a function call, which makes it the fastest way to put a defensible number on a retrieval pipeline.

Scope is the tradeoff with Ragas. The library generates synthetic evaluation data and supports production monitoring, but it is a metric implementation rather than a platform, so datasets, run history, trace context, and scheduling are yours to build around it. For a system whose primary failure mode is retrieval quality, that may be all you need.

Pricing: The framework is open source and free.

10. TruLens

TruLens instruments an application with feedback functions that score outputs as they flow through it, then aggregates results into a leaderboard for comparing app versions against each other.

Instrumentation is OpenTelemetry-based, so spans cover LLM generations, retrievals, and tool calls, and a Selector API targets any span attribute for scoring rather than only the final output. The RAG Triad of context relevance, groundedness, and answer relevance is the framework's best-known metric set.

Teams outside the Snowflake ecosystem that maintains it get a capable library and a dashboard, with retention, alerting, and access control left to their own infrastructure.

Pricing: The library is free, and no standalone pricing exists. Production monitoring at scale means either building around it or arriving through a Snowflake account.

Frequently Asked Questions

What is the best LLM evaluation tool?

Respan, for teams that need evaluation to keep working after deployment. Online evals run on live production spans, every score links to the trace behind it, failing cases become datasets, and experiments prove the fix moved the number, all inside one product rather than across three.

Most of this category treats production scoring as the thing you upgrade into. Free-tier retention runs 14 days or less in several places, online evaluation sits behind a per-seat plan in others, and runtime enforcement is commonly an Enterprise line item. Respan includes online evals, tracing, datasets, and human review on the free tier at 100k logs and 1k scores, so the loop is testable against real traffic before anything is spent. If your bottleneck is offline rigor rather than production, the production evaluation roster works through the same platforms on that axis.

Is DeepEval a good LLM evaluation framework?

For putting evaluation into CI, yes. The pytest interface means engineers write assertions in a form they already know, thresholds fail builds without custom glue, and G-Eval and DAG cover criteria the built-in metrics miss. Apache 2.0 licensing and support for local judge models also make it workable where data cannot leave your environment.

It is a framework rather than a platform, which is the boundary to plan around. Scores exist wherever your test runner puts them, there is no production-traffic scoring, and there is no trace behind a failing metric to open. Teams generally pair it with something that handles the production half, or move to a platform that runs the same evaluator definition in both places.

What is confirmation hacking in LLM evaluation?

Confirmation hacking is a failure mode where an evaluating model, agent, or pipeline seeks and interprets evidence to confirm a conclusion it already holds, such as the system under test being safe or correct, instead of running a falsification-driven audit. The judge is not lying. It is looking for agreement and finding it, which is what a judge asked to verify rather than to break will tend to do.

It shows up in automated auditing, where an auditor agent settles on a hypothesis early and then gathers support for it rather than evidence against it, a pattern documented in Anthropic's work on automated auditing and in OpenAI's write-up on validating public evals.

The mitigations are the same ones that make any judge trustworthy. Write rubrics that ask what would make this output wrong rather than whether it looks right, score a human-labeled set and measure agreement before trusting the judge at scale, and keep the failing cases in the dataset so a later run has to face them again.

How many metrics should you evaluate on?

Three to five criteria for most systems, each scored by its own evaluator. One score conflates everything and moves for reasons you cannot recover. Fifteen scores produce a dashboard nobody reads and a bill that scales with the count.

Pick them from how your system actually fails rather than from a metric catalog. Read the last month of complaints and bug reports, group them, and turn each group into a criterion with its own threshold. A system that invents policy, resolves the wrong request, and escalates things it should have handled needs three different scorers, and faithfulness alone catches one of them.

Can you evaluate an LLM without a labeled dataset?

Yes, with reference-free scoring. Faithfulness compares an answer against the context it was given rather than against ground truth. Format checks, instruction adherence, and toxicity need no labels at all. Pairwise comparison ranks two outputs against each other, which sidesteps the question of what the correct answer was.

Labels still earn their cost for anything measuring correctness, and the practical path is to build them from production rather than writing them up front. Sample real traces, have someone record expected behavior as notes rather than exact strings, and the set grows out of traffic you already served. Working through the full method takes about a week the first time.

How much does LLM evaluation cost?

Code checks are free. Judge scoring costs cents per thousand requests with a small judge model, and human review costs whatever your reviewers cost. The larger variable is usually the platform, and there the meter matters more than the sticker price.

Score-based billing scales with how many evaluators you attach per output, so evaluating more deeply raises the bill without any change in traffic. Unit-based billing counts every span, so instrumenting at finer granularity does the same. Seat-based billing scales with headcount independent of usage. Work out your spans-per-request and scorers-per-output ratios against your actual traffic before comparing published floors, because the same workload can land in a different order across vendors depending on which meter is running.