IntroducingBenchmark & Span-1
User frustration
Laziness
Unsafe
Jailbreak
Win
Escalation
Refund
Span-1 is a 4B model that flags behaviors in your AI traces, so you don't have to send every check to a frontier model.
It's small enough to run on every trace. Below is how it compares to GPT-6-luna, Jev, and six other models on an open benchmark, plus a playground where you can test it yourself.
Most teams check their traces by prompting a large model on every one. It works, but it gets expensive, and most of those checks are simple.
You shouldn't need a frontier model to spot a refund request.
So we trained a small model that only does behavior detection. Give it a trace and a behavior written in plain English, and it tells you whether that behavior happened.
Who does the judging?
Most eval tools prompt a general-purpose LLM. A few have trained their own models for it.
4B behavior judge
Custom classifiers + in-house reasoning
3B/8B evaluation models
8B · RAG hallucination
Jev available, not default
Prompt-based scoring
Prompt-based scoring
OpenAI · Anthropic · Google
Galileo's Luna-2 and OpenAI's GPT-6-luna are unrelated. This shows who uses what, not how they rank.
Scores across both test sets.
Reported F1 scores for nine models on the core and multilingual sets. "All" is the provided overall score.
| System | Core | Multilingual | All |
|---|---|---|---|
| 01Span-1 | 0.784 | 0.902 | 0.843 |
| 02GPT-5.6-terra | 0.756 | 0.917 | 0.837 |
| 03deepseek-v4-flash | 0.743 | 0.885 | 0.814 |
| 04GPT-6-luna | 0.728 | 0.901 | 0.815 |
| 05Span-1 Lite | 0.673 | 0.848 | 0.761 |
| 06qwen3-235b | 0.584 | 0.772 | 0.678 |
| 07Sonnet 5 | 0.576 | 0.858 | 0.610 |
| 08jev | 0.574 | 0.856 | 0.715 |
| 09laya | 0.196 | 0.250 | 0.223 |
Scores live trafficas it comes in.
Traces get labeled when they arrive, not in a batch job a day later.
LLM-as-a-judgeSpan-1 is the judge.
No judge prompt to write for each check. Keep the frontier models for the hard cases.
No eval pipelineto maintain.
Pick a behavior and see where it shows up in your real traffic.
Span-1 vs Jev.
Pick a behavior to load an example, or paste your own trace, and see what each model says.
Behavior specialist
TypeSafe judge
Same trace and behavior. Expected is our reading of the definition; live labels are model outputs.
Span-1 Lite is free!
View current model pricing ↗per 1M tokens
The benchmark is public.
The labels behind these F1 scores come from a consensus of models, not human annotators. The full dataset is on Hugging Face.
Browse the dataset ↗From traces to behavior scores.
Connect your traces
Send traces through the Respan gateway or tracing SDK, or run npx respan setup.
Pick a behavior
Use one of the built-in behaviors, or write your own in plain English.
See the results
Scores show up on your traces as new traffic comes in.
Try it on your own traffic.
A benchmark only tells you so much. Join the waitlist to run Span-1 on your own traces.