What is a Token?
The basic unit of text that AI language models process, which may be a word, part of a word, or punctuation.
Pronunciation: /ˈtəʊkən/
Token explained in plain English
A token is a piece of text that a language model reads and generates. Tokens are not always whole words — common words like "the" might be one token, while uncommon words might be split into multiple tokens. Punctuation and spaces can also be tokens.\n\nModels have limits on how many tokens they can process at once (the context window). Both your input and the model's output count toward this limit.
Analogy
Think of speaking to a doctor. You say, "I don't feel well." Before the doctor responds, they may also review your medical history, recent test results, and current medications. The doctor's advice is based on all of that information, not just your first sentence. AI works similarly—it uses both your request and any additional context it's given, and all of that counts toward token usage.
Example
Imagine you ask an AI, "Can you help me write a reply to this email?" Your question only uses a small number of tokens. But if the AI is also given the original email, previous messages, and some notes about the situation, all of that text is also counted as tokens. Token usage is based on everything the AI receives as context, not just what you type.
How is Token used?
Developers monitor token usage because most AI APIs charge per token. Token counts determine how much text fits in a model's context window and affect response costs.
Common misconceptions about Token
A token is not always a word. One token does not equal one word, and token counts vary between models and languages.
History
Related terms
People also read
- LLM
A type of AI model trained on vast amounts of text to understand and generate human language.
- Context Window
The maximum amount of text a language model can consider at one time, measured in tokens.
- GPT
A family of large language models developed by OpenAI that generate human-like text.
- RAG
A technique that combines AI language models with external knowledge retrieval for more accurate answers.
- AI slop
A colloquial term referring to the low-quality or unhelpful output generated by artificial intelligence systems
- BERT
A pre-trained language model developed by Google
- BLEU
A metric for evaluating the quality of machine translation
- BLEURT
A metric used to evaluate the quality of text generated by language models
- Chain-of-Thought Prompting
Asking an AI to show its reasoning step by step before giving a final answer, which often improves accuracy on complex tasks.
- Confabulation
When an AI produces a confident, fluent answer that sounds true but is factually wrong — generating plausible language without a reliable link to reality.