Skip to main content
<- Articles

AI Token Cost: Why Price per Token Isn’t Your Real Bill

The rate card tells you what a token costs. Your bill depends on how many tokens each task actually uses, and that's the number to forecast and budget around.

Respan · November 13, 2026 · Updated November 19, 2026 · 15 min read

Respan Gateway: up to 55% off tokens on Gemini, GPT, Claude and 1,000+ more models. One API key, 0% fee.

The price of a million tokens keeps falling with each model generation, and yet the monthly bill for an LLM feature in production tends to go the other way. Agents make several model calls per task, reasoning models generate output nobody reads, and every request re-sends a system prompt, tool definitions, and a growing conversation history.

For an engineering team or a founder, that makes the invoice hard to predict, because the number of tokens each task consumes moves with every prompt change, model swap, and new agent step.

The way to plan for it is to measure and budget on cost per task, then cap spend at the request, agent run, and customer level before a call reaches the provider. AI token cost starts with the price per token, but the price per token isn't your real bill.

How much does an AI token cost?

In November 2026, one million AI tokens cost between $0.10 and $8 for input and between $0.50 and $40 for output on the major model APIs, depending on the model. Output tokens cost five to six times more than input tokens on most current OpenAI, Anthropic, and Google models, so the length of a response usually matters more to the bill than the length of the prompt.

Each provider sells a small tier, a mid tier, and a frontier tier, and the spread between them is wide. At standard-tier list prices from Anthropic's pricing, OpenAI's API pricing, and Google's Gemini API pricing in November 2026, a million tokens costs:

TierProviderModelInput (per 1M tokens)Output (per 1M tokens)
SmallAnthropicClaude Haiku 5.5$0.10$0.50
SmallOpenAIGPT-6 Luna$0.10$0.50
SmallGoogleGemini 3.1 Flash-Lite$0.25$1.50
MidAnthropicClaude Sonnet 5.5$2$10
MidOpenAIGPT-6.1 Terra$1$5
MidOpenAIGPT-6.2 Sol$2$10
MidGoogleGemini 3.5 Flash$1.50$9
FrontierAnthropicClaude Opus 5.5$4$20
FrontierAnthropicClaude Fable 5.2$8$40
FrontierOpenAIGPT-6.2 Astra$8$40
FrontierGoogleGemini 4 Argon$4$20
FrontierGoogleGemini 4 Neon$3$15

Two of these prices come with conditions. Gemini 4 Argon is listed at the standard rate Google charges from December 1, and through November 30 it runs at an introductory $2 input and $10 output. Claude Haiku 5.5 moves to $0.50 input and $2.50 output once a prompt passes 100,000 tokens, which matters for long-context work like document analysis.

How to calculate AI token cost

The cost of a single request is the sum of each token type multiplied by its price, divided by one million:

cost per request = (uncached input tokens × input price
                  + cached input tokens × cached input price
                  + output tokens × output price) / 1,000,000

Output tokens include any reasoning the model does before it answers. Take a request to Claude Sonnet 5.5 with a 3,000-token system prompt served from cache, 2,000 tokens of new input, and an 800-token response. The new input costs $0.004, the cached prompt costs $0.0003, and the response costs $0.008, for a total of about $0.0123. Without caching, the same request costs $0.018.

You can get exact counts on both sides of the call. Before sending, provider token-counting tools, such as Anthropic's token counting endpoint or OpenAI's tiktoken library, return the input count for a given model. After the call, the usage object in each response reports the billed input, cached, and output tokens, and those billed numbers are the ones a cost estimate should be checked against.

Input and output tokens

Input tokens are everything you send: the system prompt, tool definitions, retrieved documents, conversation history, and the user's message. Output tokens are everything the model generates in return.

Because output carries the higher price, a feature that writes long answers can cost more than one that reads long documents, even when the documents are far larger than the answers. That is also why output length is usually the first number to check when a feature's cost looks off.

Cached tokens

Providers discount input that repeats across requests, such as a long system prompt or a fixed set of tool definitions. Anthropic bills cache reads at a tenth of the input price on most models, 5% on Claude Opus 5.5 and Sonnet 5.5, and 2.5% on Claude Fable 5.2, while writing to the cache costs 1.25 times the input price for a five-minute cache or twice the input price for a one-hour cache. OpenAI's cached input runs 5% to 10% of the standard input price across its current GPT-6, GPT-6.1, and GPT-6.2 models, and Google charges a cached-token price plus an hourly storage fee for the cached content.

Caching pays off only when the same prefix comes back within the cache's lifetime, so your hit rate belongs in any cost estimate, and how providers decide what counts as a hit is the core mechanic of LLM prompt caching.

Reasoning tokens

Reasoning models think before they answer, and that thinking is billed. OpenAI states that reasoning tokens stay hidden from the API response but are still billed as output tokens, Google labels its Gemini output prices as including thinking tokens, and Anthropic reports thinking as part of a response's billed output tokens.

If the Sonnet request above spends 2,000 tokens reasoning before its 800-token answer, the output side grows from $0.008 to $0.028, and the request costs about $0.032, more than double the original estimate.

Reasoning can also show up on the input side. Anthropic keeps prior turns' thinking blocks in context on Claude Opus 4.5 and models numbered 4.6 and higher, which includes the Claude 5.x models in the table, and bills them as input, so a long multi-turn conversation with a reasoning model keeps paying for its earlier thinking.

Why price per token isn't your real bill

The formula above is accurate per request, but the token counts that go into it are often larger than a quick estimate assumes. Several things inflate token usage in production without showing up on any pricing page, and some costs sit outside token pricing entirely: Google charges twice as much for audio input as for text on Gemini 3.1 Flash-Lite, and OpenAI bills its built-in web search at $10 per 1,000 calls on top of the tokens the search results add.

Tokenizers count the same text differently

Each provider splits text into tokens with its own tokenizer, so a token is a different size from one model to the next. Anthropic's docs note that Claude 4.7 and later models, including the Claude 5.x models in the table, use a newer tokenizer that produces about 30% more tokens for the same text than earlier Claude models.

As a result, a model that costs 20% less per token can still cost more per task if it turns your prompt into 30% more tokens. Running the same real prompts through each candidate model and comparing the billed token counts gives a truer picture than comparing rate cards.

System prompts and tool definitions ride on every call

Every request re-sends the system prompt and the full schema of every tool the model can call, whether or not the model uses them. An agent with a 2,500-token system prompt and 20 tools at 150 tokens each starts every call with 5,500 input tokens before the user has typed anything. Caching softens that cost, but only on the calls that hit the cache.

Conversation history and retrieved context grow over time

Chat-style features send the whole conversation back on each turn, so input tokens grow with every message, and a late turn in a long conversation can cost several times as much as the first. Retrieval adds the same pressure from another direction, since every chunk you retrieve is billed as input whether or not it shapes the answer.

Retries and failed calls are billed too

When a response fails to parse or gets rejected by a validation step, the tokens it consumed are still billed, and the retry pays for the whole request again. A 5% retry rate adds roughly 5% to that call's cost, and an agent that restarts a failed run from the beginning pays again for every step it had already completed.

Agents multiply calls per task

A chat reply is one model call, while an agent completing a task might plan, call tools, read the results, and revise across a dozen calls, each one carrying the fixed overhead above plus everything the run has produced so far. Because of that, cost per task for an agent depends as much on how many steps it takes as on which model it uses, and the step count can vary widely between two runs of the same task.

Tracing each run step by step with agent observability is how you find out which step is spending the tokens.

How to forecast token spend per task and per month

Forecasting works best from measured usage, because estimates built from prompt lengths miss most of the hidden tokens above. Once AI observability is recording tokens and cost on every call, a forecast comes down to a few steps:

  1. Measure cost per task - Average the cost of a complete task (a chat turn, a processed document, an agent run) from real traffic, including retries and reasoning tokens.
  2. Count tasks per user - Take how many tasks an active user triggers per day from the same data.
  3. Multiply it out - Cost per task × tasks per user per day × active users × 30 days.
  4. Add headroom - Usage moves with launches, new features, and heavy customers, so set the budget above the forecast rather than at it, and redo the forecast when the product changes.

As a simplified example, take a support agent that makes six Claude Sonnet 5.5 calls per run and assume each one costs the $0.0123 from earlier. That comes to about $0.074 per run, or about $0.078 with a 5% retry rate, and at three runs per user per day across 2,000 active users, it adds up to 180,000 runs and roughly $14,000 a month before headroom.

The estimate treats every call as the same size and leaves out cache writes and history growth, so the cost per run you measure from real traces will usually come in higher, which is why the first step works from real traffic. The same arithmetic also shows where to look when a forecast misses: a jump in cost per task points at a prompt, model, or agent change, while a jump in tasks per user points at product usage, and the two call for different fixes.

LLM token budget strategies

A token budget is a limit on how many tokens or dollars a unit of work can consume, and in production it works at several levels at once. Each level catches a different failure, from a single runaway response to a customer whose usage outgrows what they pay.

Per-request budgets

The simplest budget is a cap on output length per call. OpenAI's max_output_tokens limits all generated tokens, reasoning included, and on Claude, thinking tokens count toward max_tokens, so the cap bounds the most expensive part of the request.

On the input side, a fixed context budget, such as keeping the last few turns plus a summary in place of the full history, stops long conversations from growing without limit.

Per-agent-run budgets

Agents need a budget on the whole run as well as on each call, because a loop of reasonable calls can still add up to an unreasonable task. Capping the number of steps and the cumulative tokens or cost per run, then ending the run or handing it to a person when it hits the cap, keeps a stuck agent from retrying the same tool call all night.

Per-customer and per-feature budgets

If you sell an AI product, each customer's token spend has to stay below what that customer pays you, and a per-customer budget is how you hold that margin. It depends on attribution first: every request needs a customer ID and a feature tag so spend can be totaled per customer, which is a core job of AI observability tools.

With attribution in place, you can set monthly budgets per customer or per plan tier, and per feature, so one expensive feature doesn't eat the margin of the rest.

Soft limits vs hard limits

A soft limit sends an alert when spend crosses a threshold and lets requests continue, while a hard limit refuses requests once the cap is reached. Soft limits suit internal teams and early warnings, since blocking a production feature has its own cost, and hard limits suit anything that can run away, such as leaked keys, contractor keys, free-tier users, and agents stuck in loops.

Whichever you use, the check has to happen before the request reaches the provider, because a limit enforced after the fact only tells you how much was already spent. Tools differ on this point, and the gap between alerting and enforcing is a useful filter when you compare LLM cost tracking options.

Lowering the bill once you can see it

Once cost per task is measured and budgeted, reducing it becomes a matter of testing changes against those numbers. Routing simple steps to cheaper models, caching stable prefixes (cache reads run 2.5% to 10% of the input price on Anthropic's current models), moving work that can wait to batch APIs (Anthropic, OpenAI, and Google all discount batch requests by 50%), and trimming context are the usual levers.

Each of those changes can also move output quality, so LLM cost optimization only counts when the cheaper setup still gives good answers. Measuring cost per task and quality before and after each change is what tells you whether it saved money in practice.

Set and enforce token budgets with Respan

Respan is the AI router with built-in observability and automated evals. Every model call goes through one API to 1,000+ models, and every request is traced and priced as it passes through, so cost per request, per agent run, and per customer comes from the same place you route traffic.

Limits are checked in Respan's AI gateway before any provider is called. Cost is counted when a call ends, so the call that crosses a hard cap completes, and every request after it is refused at once and costs nothing. That puts a token budget in the request path, where it can stop spend, instead of on a dashboard that reports it afterward.

  • Cost where you need it - Spend broken down by model, by request, and by end customer, so you can see which customer or feature drives the bill.
  • Cost per agent run - Each run lands as one trace with its total cost, so you can find the runs that went over budget and set your in-code run caps from real numbers.
  • Soft and hard limits - Spend and usage caps per API key, so each app, feature, or environment can get its own key and its own cap, plus org-wide spend caps and monthly budgets per customer. Soft thresholds alert and let requests through, and hard thresholds block.
  • Monitors on cost - Alerts to Slack, email, or a webhook when cost, tokens, or errors cross a threshold over any window.
  • Cheaper calls once traffic runs through the gateway - Switch models by changing the model name, serve exact repeat requests from cache, and try Respan Router (beta), which picks a model per task and weighs cache costs before switching.

Route, observe, and evaluate every LLM call, and start on Respan's free plan with no credit card required.

FAQ

What is an AI token?

An AI token is the unit of text a language model reads and writes, usually a word, part of a word, or a punctuation mark. Providers count your prompt and the model's response in tokens and bill per million of them, with separate prices for input, cached input, and output.

How much text is 1,000 tokens?

About 750 English words, based on OpenAI's rule of thumb that 100 tokens come to roughly 75 words. Code and most non-English languages use more tokens for the same content, and each provider's tokenizer counts slightly differently, so check billed token counts on your own prompts before relying on the estimate.

Why are AI tokens so expensive?

Individual tokens are cheap, often a fraction of a cent per thousand, and the expense comes from volume. Output is priced at several times input, reasoning models bill their hidden thinking as output, and every call re-sends context the user never sees, so a single task can consume far more tokens than its visible text suggests. Tracking cost per request in Respan shows which of those is driving your bill.

Are AI tokens free?

API tokens are paid, though some providers offer free tiers. Google's Gemini API has a free tier on several models, including Gemini 3.5 Flash and Gemini 3.1 Flash-Lite, but not on Gemini 3.1 Pro Preview, and consumer chat subscriptions charge a flat monthly fee in place of per-token billing. Respan's free plan lets you route calls to 1,000+ models through one API with no credit card, using your own provider keys or Respan credits, with no processing fee on credits.

Related articles

Built for AI agents.
Break less.
Ship more.