TypeSafe's Jev answers a question with a probability instead of a paragraph, and within a few weeks of its release OpenAI and Microsoft had shipped their own versions, with open-source projects close behind. Developers building agents started asking which of these models belongs in their stack, and whether they should replace the LLM calls that currently make routing and gating decisions.
Those decisions run on every request, so even a small error rate adds up to a lot of wrong routes and missed escalations. Classifier models are cheap enough to make per-request decisions practical, which also means more of your agent's control flow ends up depending on them.
A classifier model always hands back an answer in the shape you asked for, but the shape says nothing about whether the answer is right. The model you pick matters less than how you measure it on your own traffic, before and after it ships.
Here's what LLM classifier models are and how to evaluate them before you trust one in production.
What are LLM classifier models?
An LLM classifier model takes an input and returns an answer from a fixed set, usually with a probability attached, instead of generating free text. The output might be one label out of ten, a yes or no, or a position on a scale you define, and your code branches on it directly.
In an agent, these calls tend to sit at decision points rather than in the response itself: choosing which model or tool handles a request, gating a tool call, flagging a prompt injection, deciding when to hand off to a human, or scoring an output before it goes out.
The difference from a generator is what you get back. A chat model asked to classify will usually comply, but its answer arrives as text you have to parse, and its confidence isn't part of the response. An LLM-as-a-judge setup sits close to this since it also scores outputs, though a judge typically generates its verdict and reasoning as text, while a classifier model returns the probability as the answer.
Because they return probabilities instead of text, these models are often described as non-generative AI, though the label fits how they answer more than what they're built from. Encoder-based classifiers were never designed to generate text, while many decision models start from a generative LLM and are post-trained to return probabilities in place of writing anything.
How does LLM-as-a-classifier work?
There are a few ways to get a classification out of a language model, and they differ mostly in where the label set lives: in the prompt, in the token probabilities, or in the model's weights.
Prompted classification with constrained outputs
The simplest version puts the labels and their definitions in the prompt and constrains the response to an enum through structured outputs, so the model can't return a label that isn't on the list. It needs no training data, and changing the taxonomy means editing a prompt.
What you give up is cost and a usable confidence score. Every classification is a full generation call on a general-purpose model, and the label comes back without a probability unless you ask the model to rate its own confidence, which is a self-report rather than a measured number.
Logprob scoring over label tokens
If the API exposes log probabilities, you can skip parsing and read the probability the model assigns to each label's first token. Normalizing those across the label set gives you a distribution you can threshold.
This works best when every label starts with a distinct token. It also depends on the provider returning logprobs for the model you're using, which not every API does.
Classification heads on a fine-tuned LLM
The more involved approach replaces the model's vocabulary output layer, which scores every token it could generate next, with a smaller layer that scores only your classes. You then fine-tune on labeled examples so the model learns to map inputs onto that layer.
As a result, you get a real probability per class from a single forward pass with nothing generated, which makes each call fast and cheap. However, it needs labeled data up front, and adding or renaming a class means retraining.
How the Jev release changed how we think about LLM classifiers
None of the techniques above are new, and teams have run classifiers next to their LLMs for years. What changed with Jev is the interface. Instead of training a classifier, you send TypeSafe a state, which can be text or JSON, along with typed questions, and get back typed answers with probabilities.
A Jev question can be a choice between options, a score on an ordered scale, or a noul, which is a yes-or-no probability. Nothing is generated as text, and TypeSafe's docs say each question is evaluated in parallel and in isolation against the same state, so adding questions barely changes response time.
That interface, more than any accuracy number, is what the Jev AI model, as well as new similar models to Jev, brought to the category: classification you call inline like any other function, instead of a model you train and host. Other vendors adopted the same typed-question shape quickly, and Microsoft says Microsoft-Decision-1 is built on Qwen3.5-9B post-trained for decision scoring, while many of the open-weight versions also start from a small existing model.
Microsoft's documentation shows what a request looks like in practice, with the state and a named choice question whose options are spelled out as criteria:
{
"model": "<deployment-name>",
"state": "The API returns 500 on every call.",
"questions": {
"team": {
"type": "choice",
"instructions": "Which team should handle this ticket?",
"criteria": {
"billing": "Charges, invoices, and refunds",
"engineering": "Bugs, errors, and outages",
"support": "How-to and account questions"
}
}
}
}The answers come back keyed by the same question names, with the selected option and a probability for each one, so team maps straight onto a routing branch in your code.
For developers, the practical change is that a classifier is now a line item next to your LLM calls rather than a separate ML project, and comparing the best AI decision models has become its own buying decision.
Types of LLM classifier models
Many LLM classifier models in use today are either hosted decision models that answer typed questions over an API or small encoders you run on your own hardware.
Decision models
Respan launched Span-01 to apply the decision-model approach to agent traces. Instead of typed questions about a single input, it takes behaviors written in plain language and returns the probability that each one is present, absent, or not observable across the whole trace.
Jev is TypeSafe's hosted decision model, priced at $0.042 per million input tokens with output free. Because you pay only for input, cost tracks how much state you send per call and how often you call it, rather than how long the answer is.
The OpenAI Decisions API is in public beta and runs on gpt-6-luna, the only model it supports. It takes text, or text with images passed inline as base64, and answers predicate, choice, and score questions that map closely onto Jev's types. Input costs $0.10 per million tokens and output is free, so the same input-driven cost logic applies at a higher rate.
Teams already on Azure can deploy Microsoft-Decision-1 from Microsoft Foundry, and it's also available through OpenRouter. It takes text or JSON with noul, choice, and score questions, and costs $0.042 per million input tokens with free output. Deploying through Foundry means working inside Azure's deployment and quota model, which suits existing Azure users more than anyone starting fresh.
Jev, the OpenAI Decisions API, and Microsoft-Decision-1 are all hosted, so every state you classify goes to the vendor's infrastructure, which is worth checking against your data requirements before you route regulated inputs through them.
Encoder classifiers
ConvAI Innovations released Laya as an open-weight, non-generative model built on a 421M-parameter ModernBERT-large encoder under Apache 2.0, with a 322M multilingual version. It answers the same kind of typed questions as the hosted decision models in a single forward pass, and you can fine-tune it on your own data. The model card notes that base checkpoints score near chance zero-shot on its typed-decisions benchmark and ship overconfident until recalibrated. What you take on is the tuning and calibration work along with the hosting, which is worth weighing when you look at how Jev and Laya compare.
Llama Prompt Guard 2 takes a different approach. Meta's BERT-based classifiers come in 86M and 22M sizes and return a single label, benign or malicious, for prompt injection and jailbreak attempts, with the 86M model trained on non-English attacks as well.
The label set is fixed, so Prompt Guard does one job and you can't ask it new questions, and its 512-token window means longer inputs have to be split and scanned in segments. It fits as one layer of AI guardrails in front of a model rather than as a general-purpose classifier.
Side by side, the approaches trade training effort against what you get back:
| Approach | Training data | Probability | Main tradeoff |
|---|---|---|---|
| Prompted classification | None | Self-reported only | Full generation per call |
| Logprob scoring | None | From label tokens | Needs exposed logprobs |
| Classification heads | Labeled examples | Per class | Retrain to change classes |
| Decision models | None | Per answer | Input-token cost per call |
| Encoder classifiers | Optional fine-tuning | Per label | Hosting and tuning on you |
Which row fits usually depends less on headline accuracy than on whether you have labeled data and where your inputs are allowed to go.
How to evaluate LLM classifier models
Vendor benchmarks are a starting point, but most published numbers for these models come from the vendors themselves. What decides whether a classifier works for you is how it performs on your inputs and at the thresholds you'll actually use.
Build the test set from production traffic
Pull real requests from your logs instead of writing examples by hand, and label them against the definitions the classifier will use. Make sure rare classes have enough examples to measure, and include borderline cases on purpose, since those are where models disagree and where a wrong answer costs you.
Measure per-class precision, recall, and calibration
Overall accuracy hides the failures that matter, because a model can score well while missing most examples of a rare class like escalation requests. Per-class precision and recall show where it breaks, and the per-class F1 and confusion-matrix approach used for intent classification with LLMs carries over directly.
Calibration matters as much as accuracy here, since the probability is the product. If a model says 0.9, it should be right about nine times in ten, otherwise any threshold you set on it is a guess.
Test prompt and policy wording separately for each model
How you phrase a question changes the answer, and not in a way that transfers between models. Research on prompt formatting found that meaning-preserving changes moved few-shot accuracy by up to 76 points on LLaMA-2-13B, and that format performance correlated only weakly from one model to the next.
That study covered prompted LLMs, but decision models also read natural-language question and option text. Because of that, tune the wording for each candidate before comparing them, rather than reusing one prompt across all of them.
Check how injected input moves the answer
Classifiers often read untrusted text, like tool results or user messages, and that text can try to push the decision. When researchers at Nanyang Technological University tested 510 injection cases against jev-1.13.0, injected content shifted the probabilities but changed the selected answer in 1.8% of cases, rising to 3.5% with an optimized attack.
The flips clustered where the original margin between options was already small. The useful test is to inject into your own near-boundary cases, and to treat low-margin decisions as candidates for a fallback.
Keep scoring decisions after deployment
A test set is a snapshot, while production inputs keep changing and vendors keep shipping new model versions. Sample live decisions, score them against the same definitions you tested with, and alert when the distribution of answers or the error rate moves, because a classifier that drifts still returns a valid label every time.
Classify agent behavior in production with Span-01
Scoring live decisions takes a classifier of its own, one that reads the full trace rather than a single message. Span-01 is Respan's classifier for agent traces: you write a behavior in plain language, and it scores that behavior against a trace in one forward pass, without generating text.
Because behaviors are definitions rather than a fixed taxonomy, you can check for whatever matters to your app, and the pricing is low enough to run it across live traffic instead of a hand-picked sample.
- Plain-language behaviors - Describe a behavior like "the agent called a tool the user didn't ask for," and Span-01 scores it, including definitions it hasn't seen before.
- Many behaviors, one pass - Each definition runs as its own branch over the same trace, with no per-request limit on how many you check.
- Probabilities you can act on - Every behavior comes back as present, absent, or not observable, so a trace without enough evidence reads as unknown instead of a false negative.
- Built-in and custom behaviors - Respan's Behaviors feature ships built-in behaviors such as refusals, instruction misses, and incomplete answers, and you can add your own, all scored by Span-01 on live traffic.
- Free to start - Span-01 Lite is free for input and output, and Span-01 costs $0.02 per million input tokens with free output.
Since Span-01 runs on the traces Respan already captures, a bad score opens straight onto the trace behind it, and those traces can become the test set for the next round of evaluation.
FAQ
Which LLM is best for classification?
For agent behavior across a full trace, Span-01 scores plain-language behavior definitions without generating text, and Span-01 Lite is free to start. For typed questions about a single input, hosted decision models like Jev or the OpenAI Decisions API work without training, and with a stable label set and labeled data, a fine-tuned encoder is cheap and fast per call.
What is a classifier in AI?
A classifier is a model that assigns an input to one of a set of predefined categories, usually with a probability for each. Spam filters and sentiment models are classic examples, and LLM classifier models apply the same idea with a language model underneath, often to make decisions inside agents.
Is an LLM classifier better than a fine-tuned BERT classifier?
Not by default. A fine-tuned BERT-style classifier is cheap and fast when you have labeled data and the labels rarely change. LLM classifiers and decision models earn their place when you don't have labeled data or the questions change often, and the way to know for your task is to test both on the same production test set.
Can you run an LLM classifier on every request?
Yes, if the per-call cost and latency fit your budget. Decision models price input only, at $0.042 per million input tokens for both Jev and Microsoft-Decision-1, and encoders like Laya can run on your own hardware. To score agent traces in production, Span-01 Lite is free within a daily cap and Span-01 costs $0.02 per million input tokens, with output free on both.
