What is tokenization in LLMs? BPE, WordPiece and SentencePiece explained

Before a language model reads a word, a tokenizer chops text into pieces. How BPE, WordPiece and SentencePiece work, why tokens decide cost and context, and why Indian languages are affected most.

Tokenization is how a language model turns text into the units it actually reads, called tokens. Modern tokenizers split text into subword pieces using methods such as byte-pair encoding (BPE), WordPiece or SentencePiece. Tokens matter in practice because models are priced, limited and slowed by token count, and some languages use far more tokens than others.

Tokenization is the least glamorous part of a language model and one of the most consequential. It shapes what a model can represent, how much a request costs, how much fits in the context window, and how fairly a model treats different languages. Here is how it works.

Why not just use words or characters?

Whole words create a huge vocabulary and still miss rare words, names and typos. Single characters keep the vocabulary tiny but make sequences very long, which makes models slow and less able to capture meaning. Subword tokens are the compromise: common words become single tokens, while rare words are built from smaller, reusable pieces.

How does byte-pair encoding work?

BPE starts from individual characters or bytes and repeatedly merges the most frequent adjacent pair into a new token, until the vocabulary reaches a chosen size. Frequent sequences such as “ing” or “the” become single tokens. Its use for neural models was popularised by research on translating rare words with subword units. Many GPT-style models use byte-level BPE, which can represent any text without unknown tokens.

What are WordPiece and SentencePiece?

Why do tokens affect cost and speed?

Model APIs usually charge per input and output token, and generation time grows with output tokens. The context window, the maximum text a model can consider at once, is also measured in tokens. A prompt that tokenises efficiently is cheaper, faster and leaves more room for useful context.

Why are Indian languages affected?

Tokenizers learn their pieces from training data. If that data is mostly English, Devanagari, Bengali, Tamil and other scripts get split into many more, smaller tokens than equivalent English text. The result is higher cost, slower responses and less effective context for Indian-language users. Tokenizers trained on balanced, multilingual data reduce this gap. Our guide to building AI for Indian languages covers the wider challenges.

How do you build a tokenizer?

How does tokenization affect what models can do?

Tokenization quietly shapes model behaviour. If a number is split into awkward pieces, arithmetic gets harder. If a rare name is split into many fragments, the model may misspell it. If code indentation becomes many tokens, long programs fill the context window quickly. Engineers who understand this design prompts, chunk documents for retrieval and estimate costs much more accurately.

How should you count tokens in your own projects?

Use the tokenizer that matches the model you are calling, since different models use different vocabularies. Count tokens for typical prompts, retrieved context and expected outputs, then multiply by request volume to estimate monthly cost. Trimming repeated instructions, keeping retrieved chunks focused and asking for concise output are usually the cheapest wins.

What tokenization problems do developers hit?

What are special tokens?

Tokenizers also include special tokens that are not ordinary text: markers for the start and end of a sequence, padding, and in chat models, tokens that separate system instructions, user messages and assistant replies. Formatting a conversation with the wrong special tokens is a common reason a fine-tuned model behaves strangely, so chat templates must match exactly what the model was trained with.

Where does tokenization fit in learning LLMs?

It comes first. Before attention, embeddings or training, you need to know what the model is actually reading. Program Zero’s explainer on how LLMs like ChatGPT and Claude work starts with tokens for exactly this reason, and its NLP, transformers and LLM phase includes building a custom tokenizer with BPE, WordPiece and SentencePiece before training a small GPT. For the architecture that consumes tokens, see transformers explained without the maths.

Program Zero runs live online, so you can learn this from anywhere in India: metros such as Mumbai, Delhi NCR, Kolkata and Chennai, growing cities such as Chandigarh, Bhopal, Surat and Vadodara, and smaller towns such as Dehradun, Jammu, Gwalior and Jabalpur. Every learner joins the same evening classes, at the same fee.

Frequently asked questions

Related reading