What is a Token?
The basic unit of text that AI language models process, which may be a word, part of a word, or punctuation.
Pronunciation: /ˈtəʊkən/
Token explained in plain English
A token is a piece of text that a language model reads and generates. Tokens are not always whole words — common words like "the" might be one token, while uncommon words might be split into multiple tokens. Punctuation and spaces can also be tokens.\n\nModels have limits on how many tokens they can process at once (the context window). Both your input and the model's output count toward this limit.
Analogy
Tokens are like individual tiles in a mosaic. A simple image might use large tiles (whole words as tokens), while a detailed one uses smaller tiles (subword pieces) to represent the same picture more precisely.
Example
The phrase "Hello, world!" might be tokenised into ["Hello", ",", " world", "!"] — four tokens. A 1,000-word document might use roughly 1,300 tokens depending on the tokeniser.
How is Token used?
Developers monitor token usage because most AI APIs charge per token. Token counts determine how much text fits in a model's context window and affect response costs.
Common misconceptions about Token
A token is not always a word. One token does not equal one word, and token counts vary between models and languages.
History
Tokenisation approaches evolved from word-level to subword methods like BPE (Byte Pair Encoding) and SentencePiece, enabling models to handle any language and vocabulary efficiently.
Related terms
People also read
- LLM
A type of AI model trained on vast amounts of text to understand and generate human language.
- Context Window
The maximum amount of text a language model can consider at one time, measured in tokens.
- GPT
A family of large language models developed by OpenAI that generate human-like text.
- RAG
A technique that combines AI language models with external knowledge retrieval for more accurate answers.
- AI slop
Output from a generative AI system that favors quantity over quality.
- BERT
A model architecture for text representation.
- BLEU
A metric between 0.
- BLEURT
A metric for evaluating machine translations from one language to another, particularly to and from English.
- Chain-of-Thought Prompting
Asking an AI to show its reasoning step by step before giving a final answer, which often improves accuracy on complex tasks.
- Confabulation
When an AI produces a confident, fluent answer that sounds true but is factually wrong — generating plausible language without a reliable link to reality.