Every AI chat interface has a small counter somewhere — a character limit, a “context used” bar, a warning that your conversation is getting long. That counter isn’t measuring words or characters. It’s counting tokens, and tokens are what you’re actually billed for, what determines how much text a model can hold in its head at once, and why the same paragraph can cost more in one language than another.
Most explanations of tokens stop at “it’s roughly a word.” That’s close enough to be useless the moment you’re trying to budget an API bill or figure out why your prompt got truncated. Here’s what’s actually going on, with real numbers.
What a token actually is
A tokenizer splits text into subword chunks called tokens, using a compression technique called byte-pair encoding (BPE). Common words usually stay whole; rarer or longer words get split into pieces the model has seen often enough during training to have learned as a unit.
Running actual text through the tokenizer OpenAI’s current models use (o200k_base, the encoding behind GPT-4o, GPT-4.1, and the GPT-5 family) makes this concrete:
| Text | Tokens |
|---|---|
Hello, world! | ["Hello", ",", " world", "!"] — 4 tokens |
tokenization | ["token", "ization"] — 2 tokens |
unbelievable | ["un", "bel", "ievable"] — 3 tokens |
Notice the leading space gets bundled into " world" — spacing is part of the token, not a separate character. That’s a detail people who’ve never actually looked at raw token output tend to get wrong.
Every provider tokenizes differently
OpenAI publishes its tokenizer (tiktoken) as open source, so you can run it locally and get an exact count before you ever call the API. Anthropic doesn’t publish Claude’s tokenizer — if you need an exact count, you call their count_tokens endpoint. Google’s Gemini models use a SentencePiece-based tokenizer with its own vocabulary.
The practical consequence: the same sentence produces a different token count depending on which model you send it to. There’s no universal “the text is X tokens” — only “the text is X tokens according to this specific tokenizer.” Any tool (ours included) that reports one number for every model is giving you an estimate for anything that isn’t the tokenizer it actually runs.
The “4 characters per token” rule, checked against real output
The commonly repeated heuristic is that one token is about 4 characters, or 0.75 words, in English. It’s a reasonable rule of thumb, but it comes from OpenAI’s older cl100k_base encoding. Running a normal paragraph of English prose through the current o200k_base encoding:
“Understanding how tokenization works helps developers estimate API costs accurately before shipping a production application. Modern language models process text as sequences of subword units rather than whole words…”
51 words, 375 characters → 62 tokens. That’s about 6 characters and 0.82 words per token — a bit more efficient than the classic rule, because the newer tokenizer has a larger vocabulary (200K entries vs. 100K) and captures more common phrases as single tokens.
The ratio also isn’t stable across languages. Take a short sentence and its Spanish translation:
The cat sat on the mat.→ 7 tokensEl gato se sentó en la alfombra.→ 10 tokens
Same meaning, noticeably more tokens. Accented characters and word forms that don’t appear as often in the (English-heavy) training data get split more aggressively. If you’re building for a non-English audience, budget more headroom than the English-based rule of thumb suggests — code behaves similarly, since punctuation-dense syntax splits into more pieces than prose does.
Why any of this matters
Cost. API pricing is per token, input and output priced separately, and rates vary a lot by model tier. As of July 2026:
| Model | Input ($/1M tokens) | Output ($/1M tokens) | Context window |
|---|---|---|---|
| Claude Fable 5 (Anthropic) | $10 | $50 | 1M |
| Claude Opus 5 (Anthropic) | $5 | $25 | 1M |
| Claude Sonnet 5 (Anthropic) | $3 | $15 | 1M |
| GPT-5.5 (OpenAI) | $5 | $30 | 1M (2x/1.5x beyond 272K) |
| GPT-5.4 (OpenAI) | $2.50 | $15 | 1.1M |
| Gemini 3.1 Pro (Google) | $2 (up to 200K) | $12 | 1M (higher rate past 200K) |
| Gemini 3.6 Flash (Google) | $1.50 | $7.50 | 1M |
Prices move fast in this market — Opus 5 and Gemini 3.6 Flash both shipped in the same week this article was written — so treat this table as a snapshot and check the provider’s pricing page before budgeting anything serious.
Two things worth noticing: GPT-5.5 charges more once a prompt crosses 272K tokens, so “1M context window” doesn’t always mean flat pricing across the whole window. And most providers now discount cached input heavily — Anthropic advertises up to 90% off tokens served from cache — which changes the math a lot for anything that resends the same system prompt or document on every turn. A 3,000-token conversation history costs $0.009 per request on Sonnet 5 at full price; with caching, that drops to under a tenth of a cent.
Context window. This is the total tokens (input plus output) the model can hold in a single conversation. Push past it and older messages get dropped or the request fails outright — which is why a document that “should fit” sometimes doesn’t once you count the system prompt, chat history, and the model’s own output against the same budget. It’s also worth knowing that models get measurably less reliable well before they reach that limit — what a context window really gives you covers the research on that.
Latency. More input tokens means more to process before the first output token arrives. Trimming a bloated prompt isn’t just cheaper, it’s faster.
Cutting your token bill
- Use prompt caching for anything static — system prompts, reference documents, few-shot examples — that gets sent on repeated calls. This is the single biggest lever available in 2026, well ahead of anything you can do by rewording your prompt.
- Don’t resend what the model already has. Summarize or drop old conversation turns instead of replaying the full history every time.
- Trim formatting overhead. Verbose markdown, repeated instructions, and unnecessary whitespace all consume tokens the model has to process.
- Match the model to the job. A smaller, cheaper model (Gemini Flash, GPT-5.4, Sonnet) is often good enough for routing, extraction, or classification — save the frontier models for the parts of the task that actually need them.
How to actually count tokens
Rules of thumb get you in the right ballpark. When you need a real number — before submitting a batch job, or to sanity-check an API bill — you need to run the text through an actual tokenizer.
Our free token counter uses OpenAI’s o200k_base encoding, the same one behind GPT-4o, GPT-4.1, and GPT-5. That means it’s exact for OpenAI models, and a close estimate for Claude and Gemini, since neither publishes a browser-usable tokenizer — if you need an exact Claude count, Anthropic’s count_tokens API is the source of truth. For everyday budgeting and prompt-length checks, the estimate is close enough to be useful, and everything runs locally in your browser, so nothing you paste is ever sent anywhere.
FAQ
Is a token a word?
Not exactly. Short, common words are often a single token, but longer or rarer words get split into two or more pieces. Punctuation, spacing, and even accented characters can each be their own token.
How many tokens is 1,000 words?
Roughly 1,200–1,300 tokens for English prose, depending on the tokenizer. The classic estimate is 1,300 (0.75 words per token); newer OpenAI models run slightly more efficient, closer to 1,200.
Do Claude and ChatGPT count tokens the same way?
No. Each provider trains and uses its own tokenizer, so the same text produces a different token count on each platform. There’s no single “correct” count outside the context of a specific model.
Why does non-English text use more tokens?
Tokenizer vocabularies are trained on text that skews heavily English, so English words are more likely to map to a single token. Other languages, especially ones with accented characters or different scripts, tend to get split into more, smaller pieces for the same amount of meaning.
Does formatting cost tokens?
Yes. Markdown syntax, extra line breaks, and repeated whitespace are all tokenized like any other text and count against your input.
Ready to check your own text? Try the free token counter — instant, accurate for OpenAI models, and 100% private since everything runs in your browser.
Keep reading:
- What Is a Context Window? Why 1M Tokens Doesn’t Mean 1M Usable Tokens
- PDF to Markdown for LLMs: Why Your PDF Costs 3x More Than It Should
Related Tools:
- Token Counter - Calculate tokens for ChatGPT, Claude, and Gemini