Tokens
A model reads and writes tokens, not words
About 4 characters a token in English. Prices, speed and limits are all in tokens.
A language model never sees letters or words. A tokenizer cuts the text into common fragments. Here, tokenisation becomes two pieces, while short common words are one each. Spaces usually travel with the word after them. Each fragment is replaced by a number from a fixed vocabulary, and those numbers are all the model sees. In English a token averages about four characters, or three quarters of a word.
Byte-pair encoding · Going deeper
A vocabulary is built by merging the commonest pair, over and over
Byte-pair encoding learns merges from text; new words split into known pieces.
Next-token prediction
An LLM only ever predicts the next token, again and again
Score every token, pick one, append it, repeat. Fluent is not the same as true.
Here is the whole trick. Give the model some text. It returns a probability for every token in its vocabulary. Paris is by far the most likely. One token is chosen and appended, and the model runs again on the longer text, and again, one token at a time. Everything a chat assistant writes is made this way. It picks what is likely, not what is checked, which is why a model can state something false with complete fluency.
Embeddings
Embeddings place meanings as points, close when similar
Similar meanings sit close. Measure closeness with cosine similarity.
Inside the model, each token becomes a long list of numbers, an embedding, a point in a space of thousands of dimensions. Training arranges that space so that similar meanings land near each other: animals here, vehicles there, cities over here. Directions mean something too. The step from man to woman is roughly the same as the step from king to queen. Whole sentences and documents can be embedded the same way, which is how search by meaning works.
Cosine similarity · Going deeper
Cosine by hand: direction counts, length does not
Cosine similarity = dot product ÷ (length × length). For unit vectors it is the dot product.
Attention
Attention lets each token draw on every other token
Attention weights decide which words shape each word’s meaning.
What does it mean in this sentence? The animal, or the street? Attention answers that. For the word it, the model scores every other word, and turns the scores into weights that add up to one. Most of the weight goes to animal. So, in this layer, it takes on the meaning of the animal. Transformers do this for every word, in many heads at once, layer after layer.
Attention · Going deeper
The transformer, and why long context is costly
Attention compares every pair of tokens: cost grows with the square of the length.
Context window
The context window is the model’s entire working memory
Instructions, history, documents and the answer must all fit in the context window.
Everything the model can use for one answer must fit in its context window. The system prompt. The conversation so far. Any documents you added. The question. And room for the answer it will write. A model remembers nothing between requests. The application sends the whole history every time, and when it no longer fits, the oldest parts are cut or summarised.
Training stages
Pretraining teaches language; post-training teaches behaviour
Pretraining on trillions of tokens, then fine-tuning and RLHF for behaviour.
A chat model is made in stages. Pretraining: predict the next token across trillions of tokens of text and code. This is where language and knowledge come from, and nearly all the cost. The result can continue text, but does not follow instructions. Supervised fine-tuning shows it thousands of instructions with good answers. Preference tuning has people rank answers, and nudges the model towards the preferred ones. Then it is deployed behind an API.
Training stages · Going deeper
Foundation models: one pretrained model, many tasks
Pick a foundation model by modality, quality, latency, cost and licence.
Temperature and sampling
Temperature sharpens or flattens the odds before sampling
Low temperature for facts and code; higher for varied writing.
After the model scores the candidates, one must be drawn. Temperature reshapes the odds first. At a low temperature, the top token takes almost everything, so answers are repeatable. Use it for extraction and code. At one, you get the model’s own odds. At a high temperature, the odds flatten and unlikely tokens get picked: more variety, and more chance of nonsense.
Temperature and sampling · Going deeper · 1 of 2
Top-p, top-k, max tokens and stop sequences
Top-p keeps the likely tokens until they add up to p.
Temperature and sampling · Going deeper · 2 of 2
Text is scored by overlap, by meaning, or by a judge
ROUGE for summaries, BLEU for translation, BERTScore for meaning.
Generated text has no single right answer. Overlap metrics compare it with a reference written by a person. Here five of the six words match. ROUGE counts how much of the reference was covered, and is used for summaries. BLEU counts how much of the output appears in the reference, and is used for translation. Both miss that lay and sat mean nearly the same. BERTScore compares meanings instead. For open tasks, people, or another model following a rubric, grade the answers.
Diffusion models
Image models start from noise and remove it step by step
A diffusion model learns to predict and remove noise, guided by the prompt.
Images are generated differently. A diffusion model is trained by taking real pictures, adding noise little by little, and learning to predict the noise that was added. To create, run it backwards. Start from pure noise. Remove a little predicted noise. Again and again, steered by the prompt. After fifty steps a new image is there. Most image and video generators work this way.
Lesson complete
You finished “How large language models work”
Nicely done.