# Token

> Part of the AI Garage Jargons glossary: https://aigarage.in/jargons (all terms: https://aigarage.in/jargons.md)

**Category:** Beginner
**Tags:** beginner, tokenization, pricing, context-window

A unit produced when a model's tokenizer converts input or output into model-readable pieces. A token can be a word, part of a word, punctuation, whitespace, or another encoded unit.

**In plain terms:** Before a model reads or writes text, a tokenizer cuts it into numbered pieces. Those pieces are tokens, and they do not line up one-for-one with words.

**Use cases:**
- Estimating API usage
- Managing context-window capacity
- Reserving space for complete outputs

**Why it matters:** Models measure context capacity and many APIs measure usage in tokens. Counting the actual request helps builders avoid truncation, size work correctly, and estimate cost without relying on word-count guesses.

**Why now:** AI requests increasingly include message roles, tool definitions, images, files, and prior conversation state. Current token-counting APIs account for request structure that a simple text or character estimate can miss.

**AI Garage example:** Before turning a long coding session into a build workbook, an AI Garage workflow could count the complete model input, reserve output space, and retrieve only the source passages needed for the current section.

**Mental model:** A tokenizer is the model's packing rule. It converts content into token IDs before inference and converts generated IDs back into content. The rule depends on the selected model or encoding, so the same visible text can occupy a different number of tokens elsewhere.

**Builder scenario: Fit a build session into one model request**

- **Context:** A builder sends a long coding-session transcript, instructions, tool definitions, and source excerpts to a model that must produce a complete workbook section.
- **Failure:** The workflow estimates size from word count alone, misses tokens added by request structure, and leaves too little capacity for the generated section.
- **Intervention:** Count the complete request with the selected model's tokenizer or token-counting endpoint, remove irrelevant material, and reserve a defined portion of the context window for output.
- **Outcome:** The request stays within the selected model's limit and the model has enough remaining capacity to return the complete workbook section.

**Common misconceptions:**
- **Claim:** One token is one word.
  **Correction:** A token can be a whole word, a word fragment, punctuation, whitespace, or another encoded unit. Segmentation varies by model, encoding, language, and content.

**Visual model (sequence):** Tokens are model-readable units, not a universal synonym for words. The selected model and full request shape determine the count.

**Visual description:** A visible request is converted by a model-specific tokenizer into token IDs that consume a shared context budget, so the complete request should be counted with the selected model's method.

**Related concepts:** [Context Window](https://aigarage.in/jargons/context-window), [Large Language Model](https://aigarage.in/jargons/llm), [API](https://aigarage.in/jargons/api)

**Video explainer:** [Token: the model's unit of context](https://nqqbubjesbyuwrdhqxnk.supabase.co/storage/v1/object/public/aigarage-public-media/jargons/token/v2/token.mp4) (47 seconds)

See how a complete AI request becomes model-specific tokens, why tokens do not map one-to-one with words, and how builders protect enough context capacity for the answer.

**Published:** 2026-10-12T11:59:37+05:30
**Captions:** [English](https://aigarage.in/jargons/token/captions.vtt)
**Narration:** generated

**Transcript:** A model reads tokens, not words.

A tokenizer first cuts your sentence into numbered pieces called tokens.

A common word may be one token. A long word may be several.

Punctuation, spaces and other languages change where the cuts fall.

Different models cut text differently, too.

Your request holds more than the text you typed.

Roles, tools, files and images all add tokens.

A long file can use up the budget fast.

Every token counts against one shared budget, with the answer included.

So token counts and word counts differ.

For a big request, count everything with the model's own method.

Cut anything unneeded, and leave room for the answer.

**Last researched:** 2026-10-11

**Sources:**
- [OpenAI API - Key concepts: tokens](https://developers.openai.com/api/docs/concepts)
- [OpenAI API - Counting tokens](https://developers.openai.com/api/docs/guides/token-counting)
- [OpenAI - tiktoken](https://github.com/openai/tiktoken)

**URL:** https://aigarage.in/jargons/token