AI Basics Large Language Models Acronyms Beginner

Token

Pronunciation: /ˈtəʊkən/

The basic unit of text that AI language models process, which may be a word, part of a word, or punctuation.

Plain English Explanation

A token is a piece of text that a language model reads and generates. Tokens are not always whole words — common words like "the" might be one token, while uncommon words might be split into multiple tokens. Punctuation and spaces can also be tokens.\n\nModels have limits on how many tokens they can process at once (the context window). Both your input and the model's output count toward this limit.

Analogy

Tokens are like individual tiles in a mosaic. A simple image might use large tiles (whole words as tokens), while a detailed one uses smaller tiles (subword pieces) to represent the same picture more precisely.

How is it used?

Developers monitor token usage because most AI APIs charge per token. Token counts determine how much text fits in a model's context window and affect response costs.

Real-world Example

The phrase "Hello, world!" might be tokenised into ["Hello", ",", " world", "!"] — four tokens. A 1,000-word document might use roughly 1,300 tokens depending on the tokeniser.

Common Misconceptions

A token is not always a word. One token does not equal one word, and token counts vary between models and languages.

History

Tokenisation approaches evolved from word-level to subword methods like BPE (Byte Pair Encoding) and SentencePiece, enabling models to handle any language and vocabulary efficiently.

Related Terms

LLM Context Window