Quick answer:
A token is the unit a language model actually reads and writes: a chunk of text, usually a word, part of a word, or a punctuation mark. “Understanding” becomes roughly 1 token; “misunderstanding” might be 3. Models see the world as sequences of tokens, get priced per token, and have limits measured in tokens, which makes this the most financially consequential piece of jargon in AI.
Rule of thumb for English: 1 token is about 4 characters, so 1,000 tokens is roughly 750 words.
What is a token?
Models can’t work with raw letters efficiently, and a dictionary of whole words breaks on names, typos, and new words. The compromise is subword tokenization: build a vocabulary of maybe 50,000 to 200,000 frequent chunks, learned from data by repeatedly merging the most common character pairs (the classic algorithm is byte pair encoding, BPE).
Common words earn their own token. Rare words get assembled from pieces: “tokenization” might split into “token”, “ization”. Anything at all, including emoji and code, can be represented, because in the worst case it falls back to raw bytes. The tokenizer is the fixed translation layer between your text and the model’s number-world; each model family has its own, which is why the same paragraph is 210 tokens for one model and 194 for another.
You can see this live in OpenAI’s tokenizer playground, which is worth 2 minutes of anyone’s time. Watching “ChatGPT” split into pieces explains a lot of odd model behavior.
Why do tokens decide what AI costs?
Every API bill is tokens in plus tokens out, priced per million. Input tokens (your prompt, documents, conversation history) are cheaper; output tokens (the model’s reply) cost several times more, because generating is the expensive direction.
This turns prompt design into cost engineering. A RAG system stuffing 20 retrieved chunks into every request pays for those tokens on every single question. A chatbot that resends the full conversation history (which is how chat works under the hood) watches per-message cost grow as the conversation gets longer. Most “our AI feature is unprofitable” stories decompose into token arithmetic nobody did up front.
How do tokens explain weird model behavior?
A surprising amount of LLM weirdness is tokenization showing through:
- Counting letters: the famous “how many r’s in strawberry” failures happen because the model sees tokens, and it can’t easily peer inside them at individual letters.
- Arithmetic: numbers get chopped into inconsistent chunks (“2024” might be one token, “20241” two), which is part of why long arithmetic wobbles.
- Language inequality: tokenizers are trained mostly on English, so the same sentence in Thai or Tamil can cost several times the tokens, meaning higher price and less usable context for non-English users.
- Code formatting: indentation and whitespace tokenize differently than prose, which is why models occasionally mangle whitespace-sensitive code.
What should a practitioner remember?
Count before you ship. Every model API returns token counts, and libraries like tiktoken count offline. Know the token size of your average request, multiply by expected volume and the per-million price, and you have your unit economics in 10 minutes.
Also remember tokens are what fill the context window: the model’s working memory is a token budget, and every document you cram in spends it. Cost and capacity are the same number wearing 2 hats.
What should you know about tokens, good and bad?
What tokens get right
Any text at all can be represented
Names, typos, code, emoji, brand-new slang: subword pieces assemble anything, with raw bytes as the fallback. No “word not in dictionary” failures, ever.
Compression where it counts
Frequent words cost 1 token, rare ones a few pieces. That efficiency is why a 200,000-token context holds a novel instead of a chapter.
A precise, billable unit
Tokens made model usage meterable and comparable. You can price a feature to the fourth decimal before writing a line of it, which is more than most infrastructure lets you say.
Where tokens bite
The model can’t see inside them
Letter counting, anagrams, and character-level edits fight the representation itself. The strawberry r’s became a meme for exactly this reason.
Non-English pays a tax
Tokenizers trained mostly on English split other scripts into more pieces: same sentence, multiple times the tokens, multiple times the price, less effective context. Teams shipping globally should measure this on their own languages.
Counts vary by model
The same prompt is 194 tokens on one tokenizer and 210 on another, so cost comparisons between vendors need actual counting, never eyeballing.
Invisible until the invoice
Nothing in a chat UI shows tokens accumulating, and conversation history resends silently. The first surprising bill is a rite of passage; the second one is negligence.
Will tokens ever go away?
Researchers keep poking at byte-level and character-level models that skip the tokenizer, and multimodal models already treat images as their own kind of patch-tokens. But for text, subword tokens have proven annoyingly good value, so the pricing meters and the limits will read in tokens for the foreseeable future. Learn to estimate them the way ops people estimate gigabytes: roughly, quickly, and before signing anything.