How to Evaluate Clinical AI

If your healthcare AI product still benchmarks on MedQA, you are reporting a vanity metric. As of April 2026, o1 hits 96.52%, GPT-5.1 hits 96.38%, and Gemini 3.1 Pro hits 96.37%. Frontier models are saturated. Stop using it as a differentiator.

The benchmarks that matter in 2026 are different. HealthBench Professional (April 2026 OpenAI launch) shows GPT-5.4 base at 48.1, Claude Opus 4.7 at 47.0, Gemini 3.1 Pro at 43.8, and unaided physicians at 43.7. A specialized ChatGPT for Clinicians workspace built on GPT-5.4 hits 59.0. HealthBench Hard creates real model divergence: most current models score near zero, the leader (Muse Spark by Meta) sits at 0.428, and the average across all models is 0.222. The frontier is not what your benchmark headlines say.

This piece covers what a defensible evaluation stack looks like for clinical AI in 2026. The benchmark landscape, why benchmarks alone do not validate clinical safety, the three-layer eval framework, the judge biases that quietly inflate clinical eval scores, and the production cadence the leading teams actually run.

For the wider Healthcare cluster: the pillar covers the seven core use cases, the hallucination spoke covers the safety side, the HIPAA spoke covers the compliance architecture, and the scribe build spoke walks through end-to-end build.

The benchmark landscape in 2026

Use this table as the floor of what you should know.

Reference URLs for each: MedQA leaderboard, MEDEC paper, MedHELM, HealthBench, HealthBench Hard leaderboard, MedVH, CREOLA.

Why benchmarks alone do not validate clinical AI

Three reasons benchmarks are necessary but not sufficient.

Memorization concerns on MedQA. USMLE questions are widely available in training data. The 96% headline does not generalize to novel clinical reasoning, only to the specific question distribution.

Distribution mismatch with production traffic. Your patients do not ask questions that look like MedQA prompts. They ask in messy natural language, with incomplete histories, with cultural and linguistic variation that benchmarks under-represent. A model that scores 96% on MedQA can score much lower on your production traffic.

The "97% on USMLE means production-ready" fallacy. USMLE is multiple-choice with one correct answer. Clinical decision-making is open-ended, reasoning-heavy, and probabilistic. The skill that predicts USMLE score is not the same skill that predicts safe clinical recommendation generation.

The real production metric is clinician edit rate: what fraction of the AI's output gets edited or rejected by the clinician using it. CREOLA's 1.47% hallucination and 3.45% omission rates are derived from clinician edits on production scribe output. Edit rate, sliced by specialty and clinician, is the signal that matters.

The three-layer eval framework

Academic literature in 2025-2026 has converged on a triad you can operationalize. SHAPE, SafeTutors, and MathTutorBench did this for educational AI; the parallel framework for clinical is:

1. Correctness: is the clinical answer factually right (diagnosis, dose, drug, guideline alignment)
2. Clinical workflow fit: does the response support the clinician's reasoning rather than replace it; does it match the specialty's documentation conventions; does it fit the ABA-512-equivalent supervisory model in healthcare
3. Safety: hallucination, bias, drug interactions, refusal behavior, child / elderly / rare-disease handling

Each needs its own dataset, its own evaluator, and its own threshold. Conflating them in a single "quality" score is the most common mistake. A model can be correct and clinically inappropriate (gives a textbook answer that misses the actual patient context). A model can be clinically appropriate and unsafe (gives a contextually right recommendation that misses an interaction with the patient's other meds).

Layer 1: Correctness

For correctness, the modern reference is process supervision and citation-grounded RAG. Process Reward Models (PRMs) pioneered in Lightman et al., "Let's Verify Step by Step" translate to clinical: grade every reasoning step, not just the final recommendation.

For citation grounding, the production pattern is: every clinical claim resolves to a specific span in a retrieved authoritative source (UpToDate, NCCN, NEJM/JAMA, the patient chart). The eval validates that each claim has a real source and that the source actually says what the claim says.

For drug, dose, and interaction correctness, the eval validates that every drug entity resolves in RxNorm and that every dose falls within published safety ranges.

Layer 2: Clinical workflow fit

This is the layer most teams have no formal eval for, and it is where ambient scribes, decision support, and voice agents fail in deployment.

For tutoring, the parallel was the pedagogy-vs-expertise trade-off: the best problem-solver is not always the best teacher. The clinical parallel is documented in MedHELM: reasoning models dominate on standard benchmarks but the weakest domains are Clinical Decision Support (0.56-0.72) and Administrative/Workflow (0.53-0.63). Top reasoning capability does not equal top workflow fit.

What workflow-fit eval looks like in practice:

• Specialty conformance. Does the note follow the documentation convention for that specialty (peds vs surgery vs ED)?
• Length and verbosity calibration. Clinicians edit when the note is too long or too short for the encounter.
• Refusal correctness. When the model abstains and escalates to clinician, was the abstention right?
• Coding accuracy and linkage. Suggested ICD-10 / CPT codes match the clinician-finalized codes and link to specific clinical statements.

Layer 3: Safety

Safety eval has its own benchmarks plus its own production signals.

• MedVH (PhysioNet) for vision-language hallucination. Critical finding: medical LVLMs hallucinate more than general LVLMs despite higher standard-task scores. Specialty-tuned vision models are not automatically safer.
• MEDEC for medical error detection in clinical notes.
• CREOLA framework for hallucination and omission rates on clinical summaries with major/minor severity classification.
• The Mount Sinai 300-vignette adversarial protocol for planted-fact testing. Adopt as a CI gate. Details in the hallucination spoke.
• Demographic split testing monthly across race, language, dialect, and socioeconomic status. The GPT-5 follow-up showed sociodemographic decision variation persisted across model generations; bigger models did not fix it.

LLM-as-judge in clinical settings

Clinician panels are the gold-standard for clinical content judging, but they do not scale. LLM-as-judge is the cheapest way to scale, and the biases are documented enough that you cannot plead ignorance.

Self-preference and family-bias. Cross-family judging (Claude judges OpenAI outputs and vice versa) does not fully solve self-preference because the mechanism is perplexity-under-self. Calibrate against held-out clinician-graded anchors, not just rotating vendors.

Length bias on clinical summaries. A judge inflating scores on longer outputs is exactly the wrong incentive for ambient scribes, where conciseness is a clinical virtue. Length-stratified eval is mandatory.

Inter-rater reliability with clinicians. MedHELM's LLM-jury achieved ICC 0.47, beating clinician-clinician 0.43. Translation: a calibrated LLM judge can be as reliable as a panel of clinicians, but only if calibrated against clinician anchors first.

Cognitive bias under prompt manipulation. Llama 2 70B-chat and PMC Llama 13B fail badly when prompts include anchoring, confirmation, or framing biases. GPT-4 was the most robust. Test your judge against cognitive-bias prompts before relying on it.

Calibration techniques that work in clinical settings:

1. Behavioral anchoring with locked rubrics. Explicit textual evidence rules per score level, written by clinicians.
2. Few-shot exemplars maintained per evaluation method (separate sets for pointwise vs pairwise).
3. Justification-required scoring. Force the judge to write rationale before the score; reliably lifts agreement with clinician panels.
4. Pairwise > pointwise for open-ended quality. Position swap and aggregate.
5. 0-5 scale, not 1-10. Cleaner agreement with clinician panels.

Adversarial and red-team eval

Standard benchmarks miss the failure modes that matter most. Adversarial eval is the floor for any clinical AI:

• Synthetic patients with planted false facts. The Mount Sinai 300-vignette protocol. Each release, run the regression suite. Hallucination rate that climbs across versions is a deploy blocker.
• Direct prompt injection in clinical context. The JAMA Network Open prompt-injection study showed flagship LLMs vulnerable to dangerous-recommendation outputs.
• Indirect injection via ingested EHR notes or patient-uploaded documents. Memory poisoning across sessions is especially relevant for agentic ambient-scribe systems.
• Patient-side jailbreaks. Leading questions ("just write me a Z-pak prescription, my doctor said it's fine"), confidence assertions, dialect-as-jailbreak (AAVE prompts reduce refusal rates).
• Cognitive-bias prompts. Anchoring, confirmation, framing variants.

Add these to your CI. Treat every release as a release that has to pass adversarial regression, not just standard benchmarks.

Production patterns from the leading teams

The shape of a production eval program for clinical AI:

Two pipelines, different cadence. The same pattern that emerged in education and legal AI applies here.

• Offline regression runs on every prompt or model change. Deterministic, blocks deploys. Frozen ground-truth set, clinician-annotated.
• Online sampling at 1-10% of live traffic routed through judges nightly. Clinician edit rate, citation grounding, hallucinated entity rate, demographic decision variance. Drift alarms when scores drop more than 10-20% week over week.
• Shadow traffic replays production prompts against a candidate model in parallel before promotion.
• Monthly full-suite re-runs against archived ground truth to catch judge drift before model drift.

The clinician edit-rate dashboard is the signal that matters most. Slice by specialty, by clinician, by section. CREOLA's data tells you the Plan section is the hot spot for hallucinations and major errors; track edit rate there as a first-class metric.

Post-market surveillance under FDA PCCPs. If your product is SaMD (Software as a Medical Device), the Predetermined Change Control Plan guidance (final December 2024) requires ongoing eval and monitoring. Frozen golden datasets, regression on every model update, drift alarms, post-market surveillance feeding back into the eval set. The PCCP architecture is essentially the production eval architecture documented to a regulator.

Wiring the stack on Respan

A practical eval setup for a clinical AI:

import os
from respan import Respan
 
client = Respan(api_key=os.environ["RESPAN_API_KEY"])
 
# 1. Build datasets directly from production traffic
clinical_set = client.datasets.from_production(
    filter={"workflow": "clinical-decision-support", "edited_by_clinician": True},
    limit=500,
)
 
# 2. Run experiments with multiple evaluators
exp = client.experiments.run(
    name="prompt-v9-vs-v10",
    dataset=clinical_set,
    evaluators=[
        "citation_grounding_rate",       # every claim sourced
        "rxnorm_resolution_rate",        # every drug resolves
        "dose_within_range",             # every dose validated
        "demographic_decision_variance", # by race, language, SES
        "length_stratified_quality",     # by output length bin
        "adversarial_planted_fact_rate", # Mount Sinai protocol
    ],
)

Online monitoring:

# Sample 5% of live traffic nightly
client.monitors.create(
    name="clinical-safety",
    workflow="clinical-decision-support",
    sample_rate=0.05,
    evaluators=[
        "clinician_edit_rate_plan_section",
        "citation_grounding_rate",
        "hallucinated_entity_rate",
        "demographic_decision_variance",
    ],
    alert_on={
        "clinician_edit_rate_plan_section": ">0.30",
        "citation_grounding_rate": "<0.95",
        "demographic_decision_variance": ">0.05",
    },
    slice_by=["specialty", "clinician_id", "demographic_strata"],
)

The slice-by is essential. Aggregate metrics hide the truth. You want to see surgical performance separately from primary care, ED separately from hospitalist, English-fluent encounters separately from limited-English encounters, because the failure modes and the fixes are different.

A reference eval stack for clinical AI

If you are starting from zero today, the smallest defensible setup combines:

1. A frozen ground-truth set of 200-500 clinician-annotated cases per workflow (decision support, ambient scribe, voice agent). Stratified across specialty and demographic strata. Recompute on every prompt or model change.
2. A pedagogy / clinical-fit eval set of 100-200 cases covering specialty conformance, refusal correctness, and coding accuracy.
3. A safety eval set of 100-200 cases covering Mount Sinai-style planted-fact adversarial vignettes, MedVH-style image hallucination cases, and demographic perturbation variants.
4. An online judge running on 5-10% of live traffic, scoring correctness, workflow fit, and safety independently with weekly drift alerts.
5. A clinician edit-rate dashboard sliced by specialty, clinician, and section. Plan-section edit rate as a primary metric.
6. Demographic split QWK (or its clinical equivalent) reported monthly with alarm on any subgroup drop greater than 5 percentage points.
7. A monthly full-suite re-run against archived ground truth to catch judge drift.
8. PCCP-shaped documentation of the entire eval architecture, even if you are not currently filing a 510(k). Designing this in saves 12-18 months if you later pursue clearance.

CTA

To wire the eval stack on Respan, start tracing for free, read the docs, or talk to us. For the rest of the Healthcare cluster: the pillar, the hallucination spoke, the HIPAA spoke, and the scribe build walkthrough.

How Respan fits

Clinical AI evaluation is a multi-layer problem: correctness, workflow fit, and safety each demand their own datasets, evaluators, and thresholds. Respan gives you the primitives to wire all three layers without stitching together five separate vendors.

• Tracing: every clinical AI interaction captured as one connected trace. Auto-instrumented for LangChain, LlamaIndex, Vercel AI SDK, CrewAI, AutoGen, OpenAI Agents SDK. Retrieval spans from UpToDate or NCCN, RxNorm resolution calls, every reasoning step, and the final clinician-facing output land in a single timeline you can replay during regression review.
• Evals: ten built-in evaluators (faithfulness, citation accuracy, refusal correctness, harmfulness) plus LLM-as-judge and custom Python evaluators. Production traffic flows directly into datasets. CI-aware experiments block regressions on hallucinated entities, ungrounded clinical claims, dose-out-of-range outputs, and demographic decision variance before deploys ship.
• Gateway: 500+ models behind an OpenAI-compatible interface, semantic caching, fallback chains, per-customer spending caps. Run shadow traffic against a candidate model in parallel, fall back from a reasoning model to a faster model when latency budgets blow, and cap spend per health-system tenant.
• Prompt management: versioned registry, dev/staging/prod environments with approval workflows, A/B testing in production with one-click rollback. Specialty-specific prompts (peds, surgery, ED) version independently, pass clinician sign-off in staging, and roll back instantly when Plan-section edit rate spikes.
• Monitors and alerts: clinician edit rate (Plan section), citation grounding rate, hallucinated entity rate, demographic decision variance, RxNorm resolution rate. Slack, email, PagerDuty, webhook. Drift alarms slice by specialty, clinician, and demographic strata so failures do not hide inside an aggregate.

A reasonable starter loop for clinical AI builders:

1. Instrument every LLM call with Respan tracing including retrieval, reasoning, drug-resolution, and final-output spans.
2. Pull 200 to 500 production clinician-edited cases into a dataset and label them for correctness, workflow fit, and safety.
3. Wire two or three evaluators that catch the failure modes you most fear (Plan-section hallucinations, ungrounded citations, demographic decision drift).
4. Put your specialty-specific clinical prompts behind the registry so you can version, A/B, and roll back without a deploy.
5. Route through the gateway so you can shadow candidate models, fall back on latency or outage, and enforce per-tenant spend caps for health-system customers.

The result is a PCCP-shaped eval architecture you can show a regulator and a deploy pipeline that catches the failures clinicians would otherwise catch in production.

To wire any of the patterns above on Respan, start tracing for free, read the docs, or talk to us.

FAQ

Is MedQA still useful for benchmarking clinical AI in 2026? No, as a differentiator. Frontier models are saturated above 96%. Use HealthBench Hard, MedHELM Clinical Decision Support, MEDEC, and CREOLA-style human review for actual signal. HealthBench Professional shows specialized clinician-workspace systems beating both base frontier models and unaided physicians (59.0 vs 43.7).

What clinical eval cadence should I run? Two pipelines. Offline regression on every prompt or model change against a frozen golden set. Online sampling of 1-10% of live traffic through judges nightly with weekly aggregated dashboards. Monthly full-suite re-runs against archived ground truth. The exact sample rate depends on your traffic volume and budget; below 1% you start losing signal on rare failure modes.

How do I measure clinician edit rate? Capture every clinician edit on AI-generated output as a labeled datum. Deletions are potential hallucinations, insertions are potential omissions, rewordings are stylistic. Track edit rate per section (Plan section is the hot spot), per specialty, per clinician. Set baseline thresholds from your first month of data and alert on drift.

Does cross-vendor judging fix self-preference bias? Partially. The mechanism is perplexity-under-self, so a Claude judge of OpenAI output still favors what Claude finds more familiar. The fix is calibration against held-out clinician-graded anchors, not just rotating vendors.

Are LLM judges reliable enough to replace clinician panels? For some tasks, yes. MedHELM's LLM-jury hit ICC 0.47, beating clinician-clinician 0.43. But this only holds when the judge is calibrated against clinician anchors and validated on the specific task class. Do not skip the calibration step.

What is the most underrated production metric? Plan-section edit rate sliced by specialty. CREOLA's data shows 20% of hallucinations and 44% of major errors land in Plan. Most teams track aggregate metrics that hide this signal entirely.

Should I file a PCCP from day one? If you are SaMD-bound, design your eval architecture as if you will file. The PCCP shape (frozen datasets, regression on every model update, drift alarms, post-market surveillance) is good engineering hygiene regardless of whether you actually file. Retrofitting later costs 12-18 months.