Tokenization: The Oft Overlooked Appetizer
Language is a human skill. Computers were historically very bad at language…until LLMs came along.
Note: this story is still under construction!
For humans, language is made up of words. But an LLM has it’s own kind of word, called a “token”. Humans read words by considering what type of word it is (noun, verb, etc.) and what it means.
For example, we might see this sentence and group each word into a part of speech, which tells us something about the word’s meaning.
But LLMs do not understand language this way. They do not group words by their meaning, or by their part of speech.
Instead, they break words down into smaller pieces called tokens. These tokens are then grouped together in a sequence to form a sentence.
What is a token?
Let's explore how language models break down text into manageable pieces.
How big is a token?
We certainly do not forget you so soon as you forget us. It is perhaps our fate rather than our merit. We cannot help ourselves. We live at home, quiet, confined, and our feelings prey upon us. You are forced on exertion.
size of model
Many people think of LLMs as a sort of “black box”: you ask a question, and an answer magically appears! But there are ways to understand these models—and starting with tokens can help.
A token can be a single word, as shown in the diagram here. This is similar to how humans think of 'words'.
Or, tokens can be a series of words. If we increase the length of our tokens, then the number of tokens we need to represent a sentence decreases. But there's also a trade-off: the longer the token, the less flexibility we have in understanding the meaning of the sentence. And the larger our model becomes.
To increase efficiency, we could also break down words into smaller parts, like 'cat' into 'c', 'a', and 't'. This presents another tradeoff: the smaller the token, the more tokens we need to represent a sentence, but the more flexibility we have in understanding the meaning of the sentence.
How does a model make connections between words? How does it understand something?
The Distributional Hypothesis
The Distributional Hypothesis states that words that occur in the same contexts tend to have similar meanings. So how does an LLM group different instances of the same word?
We can examine the layers of the model to see how it processes tokens. In this graph, each dot is a sentence with the word 'bank' in it, grouped by similarity. Here is one layer of the model.
As we progress through the layers, we can see how the model's understanding of the word 'bank' evolves.
Here's another layer. Is the model starting to understand the different meanings of 'bank'? Hover over the dots to see the sentences.
And another layer!
Where do we go from here?
The world's a stage, and we are actors.
What role will LLM's play? Do they know something?
Time will tell dear ones!