On this page
Read
The useful mental model
A tokenizer is the model's packing rule. It converts content into token IDs before inference and converts generated IDs back into content. The rule depends on the selected model or encoding, so the same visible text can occupy a different number of tokens elsewhere.
In plain English
Before a model reads or writes text, a tokenizer cuts it into numbered pieces. Those pieces are tokens, and they do not line up one-for-one with words.
Where builders use it
- 01Estimating API usage
- 02Managing context-window capacity
- 03Reserving space for complete outputs
Why it matters
Models measure context capacity and many APIs measure usage in tokens. Counting the actual request helps builders avoid truncation, size work correctly, and estimate cost without relying on word-count guesses.
Why now
AI requests increasingly include message roles, tool definitions, images, files, and prior conversation state. Current token-counting APIs account for request structure that a simple text or character estimate can miss.
A common misconception
“One token is one word.”
A token can be a whole word, a word fragment, punctuation, whitespace, or another encoded unit. Segmentation varies by model, encoding, language, and content.
See
See the mental model
The visual is generated from the same structured knowledge as the written explanation.
A visible request is converted by a model-specific tokenizer into token IDs that consume a shared context budget, so the complete request should be counted with the selected model's method.
Watch
Watch the explainer
A short visual explanation built from the same reviewed knowledge object.
Token: the model's unit of context
See how a complete AI request becomes model-specific tokens, why tokens do not map one-to-one with words, and how builders protect enough context capacity for the answer.
Read transcript
A model reads tokens, not words. A tokenizer first cuts your sentence into numbered pieces called tokens. A common word may be one token. A long word may be several. Punctuation, spaces and other languages change where the cuts fall. Different models cut text differently, too. Your request holds more than the text you typed. Roles, tools, files and images all add tokens. A long file can use up the budget fast. Every token counts against one shared budget, with the answer included. So token counts and word counts differ. For a big request, count everything with the model's own method. Cut anything unneeded, and leave room for the answer.
Builder example
Fit a build session into one model request
A builder sends a long coding-session transcript, instructions, tool definitions, and source excerpts to a model that must produce a complete workbook section.
Failure
The workflow estimates size from word count alone, misses tokens added by request structure, and leaves too little capacity for the generated section.
Intervention
Count the complete request with the selected model's tokenizer or token-counting endpoint, remove irrelevant material, and reserve a defined portion of the context window for output.
Observable result
The request stays within the selected model's limit and the model has enough remaining capacity to return the complete workbook section.
Sources
Go deeper
Last researched 11 October 2026