What Is a Small Language Model? SLM vs LLM Explained

Small models have had a big year. Alibaba's Qwen3.5-9B runs on a laptop GPU with a 262K-token context window, Google built Gemini Nano 4, Android's on-device model, on its Gemma 4 small sizes, and Apple now runs a 20-billion-parameter model on its newest iPhones by activating only a few billion parameters per request.

That changes the default for anyone building with language models. Sending every request to a frontier model means paying frontier rates and waiting on frontier latency for tasks like classifying a ticket or formatting a tool call, which a model a fraction of the size can often handle.

The more useful question is which tasks a small model can take, since production apps and agents usually end up running small and large models side by side. The hard part is proving where that line sits for your own traffic, because public benchmarks don't measure your prompts.

Below is what a small language model is, how SLM vs LLM compares, where small models fit in agents, and five worth testing.

What Are Small Language Models?

A small language model (SLM) is a language model compact enough to run on a phone, a laptop, or a single GPU, usually with somewhere between a few hundred million and around 10 billion parameters. It works the same way a large language model does, predicting the next token with a transformer, but with far fewer parameters, so it needs less memory, answers faster, and costs less per request.

There's no official cutoff, and definitions range from a few million parameters to anything below 10 billion. Parameter counts have also gotten harder to read, since Google's Gemma 4 E2B stores 5.1B parameters but computes like a 2.3B model, and mixture-of-experts models activate only a fraction of their weights per token. Among the types of AI models developers build with, SLMs are the ones defined by where they run rather than by the job they do.

Small models reach useful quality at that size through a handful of techniques. Knowledge distillation trains a small "student" model on the outputs of a larger "teacher," pruning removes weights that contribute little, and quantization stores the remaining weights at lower precision so the model fits in less memory. Training data also gets curated more aggressively than for large models, since a small model has less room to spend on noise.

SLM vs LLM: Small Language Models vs Large Language Models

SLMs win on cost, speed, and where they can run, and LLMs win on broad knowledge and hard, open-ended reasoning. Nearly every other difference follows from size, so the choice usually comes down to how narrow and repetitive the task is.

How SLMs and LLMs Differ

The largest open models and the hosted frontier AI models from OpenAI, Anthropic, and Google sit at the far end of that table. Their capability ceiling is higher, but you pay for the full model on every call, whether the task needed it or not.

Hosted "small" tiers sit somewhere in between. Claude Haiku 5.5, Gemini 3.5 Flash-Lite, Meta's Muse Ember, and OpenAI's smaller GPT tiers are priced and positioned as small models, but their vendors don't publish parameter counts, and you can't run them on your own hardware. Whether they count as SLMs depends on the definition you use, and in practice they behave like fast, inexpensive LLMs reached over an API.

The cost gap is large even when you don't run anything yourself. Through Respan's gateway, Qwen3.5-9B costs $0.09 per 1M input tokens and $0.13 per 1M output tokens, Claude Haiku 5.5 costs $0.10 and $0.50, and GPT-6 Astra costs $10 and $50. Self-hosting removes the per-token bill but adds GPU, idle capacity, and the time it takes to operate the servers, so it tends to pay off only at steady, high volume or when data can't leave your network.

When to Use an SLM vs an LLM

A small model is usually the better fit when the work has a predictable shape:

• The task is narrow and repeated - Classifying, extracting fields, rewriting, or formatting the same kind of input thousands of times a day.
• Latency matters - Autocomplete, voice, and agent steps where a user waits on every hop.
• Data has to stay put - On-device features, offline use, or workloads that can't leave your network.
• Volume is high - Per-request savings compound when a step runs millions of times.

An LLM is usually the better fit when the task is open-ended, needs broad world knowledge, involves long multi-step reasoning or very long context, or is new enough that you don't yet know what a correct answer looks like. For the hardest of those, reasoning models add billed thinking time on top of an LLM's size, which is worth paying for on a planning step and wasted on a lookup.

If you don't need on-device inference or tight data control, the cheapest first test is usually a hosted small model, either a small tier like Haiku, Flash-Lite, or Muse Ember, or an open SLM served by an inference provider. Self-hosting can wait until volume justifies the GPUs.

Small Language Model Examples and Use Cases

Small models show up wherever a task is narrow enough that a general-purpose model is overkill, or where the model has to run somewhere a large one can't fit. Use cases tend to group by those two questions: how narrow the job is, and where the model runs.

On-Device and Edge Apps

Phones are where many people already use an SLM without knowing it. Apple Foundation Models now include two on-device models: AFM 3 Core, a 3B dense model, and AFM 3 Core Advanced, a 20B model that activates 1 to 4 billion parameters per request so it fits in phone memory. Apps can call the on-device model through Apple's Foundation Models framework, offline and with no per-request cost.

On Android, Gemini Nano runs inside the AICore system service, and its fourth generation is built on Gemma 4's small sizes. What you give up on-device is reach, since Gemini Nano 4 requires recent flagship hardware with at least 12GB of RAM, which leaves out a large share of the phones people carry today.

Laptops, kiosks, cars, and factory equipment follow the same logic. When the network is slow, expensive, or off-limits, the model has to fit on the device, and that usually means a small one.

High-Volume, Narrow Tasks

Inside cloud apps, SLMs earn their place on steps that run constantly and vary little: intent classification, pulling fields out of documents, PII detection, moderation, short summaries, and formatting structured output for the next step. Handing those to a small model is a direct LLM cost optimization lever, and the savings grow with volume.

The tradeoff is consistency on unusual inputs. A small model that handles the common cases well can fail badly on the ones it hasn't seen, so these deployments usually keep a larger model as a fallback for low-confidence or out-of-pattern requests. The handoff needs a signal, such as low token log probabilities, an output that fails schema validation, or a low score from a separate verifier model.

Domain-Specific Models

Because small models are inexpensive to fine-tune, they're the usual choice for a model specialized on one domain, such as medical coding, legal clause classification, a support team's ticket categories, or a company's internal terminology. Fine-tuning an 8B open model with LoRA fits on a single GPU, while fine-tuning a frontier model is either expensive or limited to whatever the vendor exposes.

A specialized small model can outperform a general large one on its narrow task. However, it falls behind quickly once inputs drift outside what it was tuned on, so the scope of the task has to stay as narrow as the training data.

Are Small Language Models the Future of Agentic AI?

A 2025 position paper from NVIDIA Research argues yes for many agent steps, based on estimates rather than production results, and with a large model still in the loop. Agents break a goal into steps like picking a tool, filling in its arguments, parsing the result, and deciding what to do next, and most of those steps are narrow and repetitive in a way open conversation isn't.

The Case for Small Models in Agents

The paper is titled "Small Language Models are the Future of Agentic AI," and its authors define an SLM as a model that fits on a common consumer device with practical latency for a single user, and they treat most models under 10B parameters as SLMs. Their case is that agent steps are scoped, repetitive, and non-conversational, so a small model fine-tuned for each step can do the job reliably at a lower cost.

The paper puts numbers behind that position. It estimates that serving a 7B model is 10 to 30 times cheaper in latency, energy, and compute than serving a 70B to 175B model, and in case studies of three open-source agents (MetaGPT, Open Operator, and Cradle), the authors estimate that roughly 40% to 70% of LLM calls could be handled reliably by specialized small models. Format-bound steps also get easier to pin down, since a fine-tuned SLM can be trained to produce the exact JSON shape a tool expects.

Where Small Models Fall Short in Agents

The same paper lists the counterarguments, and they hold real weight. Large models keep an edge in general language understanding, which matters for open-ended planning and for recovering when a step goes somewhere unexpected. Centralized LLM inference also benefits from economies of scale that can erode an SLM's per-token savings once you count the infrastructure and the people needed to run it.

Tool use is one place the gap shows up in published numbers. Google reports Gemma 4 E4B at 42.2% on the Tau2 agentic tool-use benchmark, against 76.9% for the 31B model in the same family, so a small model on a tool-calling step needs testing before it replaces a larger one. The NVIDIA authors also point out that SLMs are still trained and judged on generalist benchmarks, which say little about how a model handles the one narrow step you'd give it.

Mixing Small and Large Models in One Agent

The paper's answer to those limits is the heterogeneous agent, where small models handle routine steps by default and a large model takes the steps that need broad reasoning, long context, or open conversation. In practice, that might mean a planner on a frontier or reasoning model, small models on classification, extraction, and tool-call formatting, and a fallback to the large model when the small one is unsure. Vendors now build lineups around that split: Meta released Muse Ember on November 9 as the high-volume executor under its larger Muse Wonder, summed up in its launch post as "Muse Wonder plans, Muse Ember handles the volume."

Making that work takes routing, whether that's rules you write for each step or an LLM router that picks the model per request. Either way, the routing is only as good as your evidence about which steps the small model can handle, and that evidence is easy to skip until something breaks in production.

Best Small Language Models

Where a model can run is usually the first constraint you'll hit, and these five open-weight models span that range, from phones to a single workstation GPU. The list stops at 15B parameters, a little past the usual 10B line, because a 15B model still runs on one workstation GPU, which is the practical test most SLM definitions are getting at. Each one is current as of November 2026.

If you want a starting point by task:

• Classification or extraction in the cloud - Qwen3.5-9B or Ministral 3 8B.
• On-device apps with image or audio input - Gemma 4 E4B, or E2B on phones with less memory.
• Agents that read screens, charts, or documents - Phi-4-reasoning-vision-15B.
• Fine-tuning on a well-documented base - SmolLM3-3B.

Treat any of these as the first model to test rather than a final answer, since the right pick depends on how each one scores on your own requests.

Small Language Models at a Glance

All five publish their weights, so you can run them on your own hardware with no per-token bill, and every one except Phi uses Apache 2.0. That control is the main appeal of open-weight models, and if a task needs more than 15B parameters can offer, larger open source LLMs trade some of that portability for a higher ceiling.

The memory figures cover the weights only, before the context window's KV cache, so plan for more headroom on long prompts. Google publishes Gemma's figures, the others are approximate from parameter counts, and Microsoft publishes Phi-4-reasoning-vision-15B only as 16-bit weights. Most of these models run through Ollama, llama.cpp, LM Studio, MLX, or vLLM, and 4-bit quantization is what brings them down to laptop and phone memory, usually at some cost in quality, so it's worth re-running your evals on the quantized build you'll actually ship.

Gemma 4 E2B and E4B

Gemma 4 E4B and the smaller E2B are the phone-sized end of Google's Gemma 4 family, released in April 2026. The "E" stands for effective parameters, since per-layer embeddings raise the stored size while each model computes like a 2B or 4B model. Both accept text, images, and audio, support function calling and a thinking mode, and serve as the base for Gemini Nano 4. Choosing between running Gemma yourself and calling Google's hosted models is its own decision, covered in Gemma vs Gemini.

• Parameters - 2.3B effective (E2B) and 4.5B effective (E4B)
• Memory - 2.9 GB (E2B) and 4.5 GB (E4B) at 4-bit, per Google
• Runs on - Phones and laptops
• License - Apache 2.0
• Context window - 128K tokens
• Best for - On-device apps that need image or audio input

Limits: Google reports E4B at 69.4% on MMLU Pro against 85.2% for the 31B model in the same family, so it fits narrow tasks better than open-ended ones.

SmolLM3-3B

SmolLM3 is a 3B model from Hugging Face, released in July 2025 along with its pretraining data, training code, and intermediate checkpoints. It switches between think and no-think modes within one model, supports six European languages, and handles up to 128K tokens of context with YaRN extension. With that much of its training documented, it works well as a base for teams that want to know exactly what they're fine-tuning on top of.

• Parameters - 3B
• Memory - About 2 GB at 4-bit
• Runs on - Laptops and edge devices
• License - Apache 2.0
• Context window - 128K tokens (with YaRN)
• Best for - Fine-tuning on a well-documented base model

Limits: It's text-only and supports six languages natively (English, French, Spanish, German, Italian, and Portuguese), so multimodal or broadly multilingual products will need one of the other picks.

Ministral 3 8B

Ministral 3 8B is the middle size of Mistral's Ministral 3 edge family, which also comes in 3B and 14B and was released in December 2025. It pairs an 8.4B language model with a 0.4B vision encoder, so it reads images as well as text, and its 256K context window is large for a model at this size. Mistral publishes base, instruct, and reasoning variants of each size, which gives you a clean starting point for fine-tuning.

• Parameters - 8.4B language model plus a 0.4B vision encoder
• Memory - About 6 GB at 4-bit
• Runs on - Laptop or single GPU
• License - Apache 2.0
• Context window - 256K tokens
• Best for - Edge deployments that read documents or images

Limits: Reasoning lives in a separate checkpoint rather than a mode you toggle, so moving between fast answers and step-by-step reasoning means serving a second model.

Qwen3.5-9B

Alibaba's Qwen3.5-9B is the largest of four small Qwen3.5 models released in early 2026, alongside 0.8B, 2B, and 4B sizes, with weights on Hugging Face. It takes text, images, and video, switches between thinking and non-thinking modes, and handles 262K tokens natively, extendable to about 1M. Alibaba's newer generation skips this size, since the Qwen 3.8 models start at 27B, which leaves Qwen3.5 as the family's current small option.

• Parameters - About 9B
• Memory - About 6 to 7 GB at 4-bit
• Runs on - Laptop GPU or single GPU
• License - Apache 2.0
• Context window - 262K tokens
• Best for - Multimodal and long-context tasks on modest hardware

Limits: Artificial Analysis found that the Qwen3.5 small models produce far more output tokens than their larger siblings when reasoning, so thinking mode can eat into the latency and cost savings that made a small model appealing.

Phi-4-reasoning-vision-15B

Built on Microsoft's Phi-4-Reasoning language model and a SigLIP-2 vision encoder, Phi-4-reasoning-vision-15B arrived in March 2026. The model learns when to reason step by step and when to answer directly, which keeps simple perception tasks like OCR and captioning fast while still working through math, charts, and dense documents. Microsoft aims it at UI screenshots and document understanding, which makes it a reasonable fit for agents that read screens.

• Parameters - 15B
• Memory - About 30 GB at 16-bit, with no official 4-bit build
• Runs on - Single workstation GPU
• License - MIT
• Context window - 16K tokens
• Best for - Charts, documents, and UI screenshots

Limits: A 16K context window is short for 2026, which rules out long documents and long agent histories without chunking.

How to Test a Small Language Model Before You Switch

Benchmarks and parameter counts can narrow the list, but they can't tell you whether a small model handles your prompts, your users' phrasing, and your edge cases. The reliable way to find out is to run it against your own traffic and compare it with the model it would replace.

That usually starts with a few hundred real production inputs for the step you want to move, including the unusual ones. Both models get the same inputs and the same scoring, with exact-match checks where outputs are structured and LLM-as-a-judge where they aren't. Cost and latency belong in the comparison too, since a small quality drop can be worth it on a high-volume background step and not on a customer-facing one. It's the same approach you'd use to test AI models before any launch, narrowed to a single step.

The loop is much easier to run behind an LLM gateway, where every model sits behind one API. In Respan, which puts 1,000+ models behind its AI gateway and adds tracing and evals on top, the setup looks like this:

1. Point your existing client at Respan - If your app uses the OpenAI SDK, change the base URL and the API key. Every call is then logged with its input, output, latency, and cost.
2. Swap the model for one step - Change the model string on the step you want to test, and leave the rest of the call alone. Expect to tighten the prompt for the small model too, and keep both prompt versions in the experiment.

import os
from openai import OpenAI
 
client = OpenAI(
    api_key=os.getenv("RESPAN_API_KEY"),
    base_url="https://api.respan.ai/api",
)
 
# The ticket-classification step on its current model
response = client.chat.completions.create(
    model="gpt-6-astra",
    messages=[{"role": "user", "content": "Classify this support ticket: ..."}],
)
 
# The same step on a small model: only the model string changes
response = client.chat.completions.create(
    model="qwen3.5-9b",
    messages=[{"role": "user", "content": "Classify this support ticket: ..."}],
)

3. Build a dataset from production logs - Sample that step's real requests into a dataset, using filters and time ranges so the unusual inputs come along with the common ones.
4. Run an experiment - Score the current model and the small model on the same dataset with the same automated evals, whether that's an LLM judge, a code check on structured output, or human review, and compare the scores side by side.
5. Move traffic and keep scoring - Once the small model holds up, switch the step and keep scoring live output with online evals, currently in beta. A fallback route covers provider errors: you list the models once, send the route name as the model, and Respan moves to the next entry on a rate limit, server error, or timeout.

If you'd rather not choose the model for each step yourself, Respan Router, also in beta, picks one per request behind a single model ID after checking that a candidate can handle the request's context length, images, tools, and structured output. It can't be combined with fallbacks, so steps that need a fallback route stay on a fixed model.

Some steps don't need text at all, only a decision, like which team should handle a ticket or whether an output passes a check. Respan's Span-02 is built for that kind of step: it answers typed questions (a choice, yes/no, a score, or a number) with a probability for each answer, so you get a confidence signal along with the decision. It runs about 4B active parameters, and its open weights run on a single NVIDIA Blackwell GPU.

FAQ

Is ChatGPT an LLM or an SLM?

ChatGPT is an LLM product. It's an app built on OpenAI's GPT models, which are large language models served from OpenAI's infrastructure. OpenAI also offers smaller, cheaper GPT tiers through its API, but it doesn't publish their parameter counts, and none of them run on your own device.

Is DeepSeek an SLM or an LLM?

DeepSeek's current models are LLMs. DeepSeek V4.1 Flash is a mixture-of-experts model, and V4 Flash, which it replaced in September 2026, already had 284B total parameters with 13B active per token. DeepSeek did release small distilled versions of its R1 reasoning model in 2025, starting at 1.5B parameters, and those count as SLMs.

What Is the Smallest Language Model?

There isn't a single smallest model, since new tiny releases keep appearing. Current small models go down to a few hundred million parameters, such as SmolLM's 135M version and Liquid AI's LFM2.5-230M. Models that small are mostly useful once they're fine-tuned for one narrow job, like extracting fields from a known document type.

How Are Small Language Models Trained?

Small language models are trained the same way large ones are, with pretraining on a large text corpus followed by instruction tuning and often reinforcement learning, but their training data is curated more tightly. Many also learn from a larger model through distillation, where the small model trains on the larger model's outputs, and they're often pruned or quantized afterward to fit on smaller hardware.

What Are the Limitations of Small Language Models?

Small language models know less, so they hallucinate more on facts outside their training data and fall behind on long, multi-step reasoning and open-ended planning. Some also have short context windows, and tool-use accuracy drops noticeably at small sizes. Because of that, a small model needs testing on the specific task you'd give it.