Tokens

A model reads and writes tokens, not words

"Tokenisation isn’t magic: it splits text into pieces."53 characters, 8 wordsTokenisation␣isn’t␣magic:␣it␣splits␣text␣into␣pieces.34048082453652911204254334157514951139986313each token is a number in a fixed vocabulary of about 100,000 to 200,000 entries; the model only ever sees these numbers12 tokens for 53 characters: 4.42 characters a token. Rule of thumb for English: ~4 characters, or 100 tokens ≈ 75 words.Common words are one token; rare words, code and many other languages split into more, so they cost more.A language model reads and writes tokens, not words or letters.A tokenizer cuts text into common fragments learned from data (byte-pair encoding), and maps each to an ID. Everything about a modelis counted in tokens: the price you pay, the speed it writes, and how much fits in its context. The split above is illustrative;every model family has its own tokenizer, so the same text is a different number of tokens on different models.

About 4 characters a token in English. Prices, speed and limits are all in tokens.

Byte-pair encoding · Going deeper

A vocabulary is built by merging the commonest pair, over and over

Training text: low × 5, lower × 2, newest × 6, widest × 3. Start: every word is letters plus an end mark ·MergeMost frequent pairSeenNew token"newest" is now1e + s9esn e w es t ·2es + t9estn e w est ·3est + ·9est·n e w est·4l + o7lon e w est·5lo + w7lown e w est·6n + e6nene w est·7ne + w6newnew est·8new + est·6newest·newest·Words it never sawlowestlowest·newernewer·Known pieces are reused; nothing is ever"out of vocabulary", only split smaller.A tokenizer's vocabulary is learned by merging the most frequent neighbouring pair, again and again.Byte-pair encoding starts from single characters (in practice, bytes) and adds one token per merge until the vocabulary reaches achosen size, tens of thousands of merges for a real model. Frequent words end up as one token, rare ones as several familiarpieces. Encoding new text replays the merges in the order learned. This is why token counts differ between model families.

Byte-pair encoding learns merges from text; new words split into known pieces.

Next-token prediction

An LLM only ever predicts the next token, again and again

"The capital of France is"modelProbability of each next tokenParis91.2%a4.1%the2.5%located1.7%Lyon0.5%… and about 100,000 other tokens, each with a tiny shareThen append the chosen token and run again:1. The capital of France is Paris2. The capital of France is Paris.3. The capital of France is Paris. ItA large language model does one thing: predict the next token, over and over.For every position it outputs a probability for every token in its vocabulary. One is chosen, appended to the text, and the whole thingruns again. An answer of 500 tokens is 500 runs. The model picks likely continuations, not checked facts, which is why it canwrite a fluent sentence that is false: a hallucination. Logits here are illustrative; probabilities are computed from them.

Score every token, pick one, append it, repeat. Fluent is not the same as true.

Embeddings

Embeddings place meanings as points, close when similar

catdogpuppycartruckbusParisLondonTokyothousands of dimensions, flattened to two for the drawingmanwomankingqueenking − man + woman ≈ queencosine similarity: king–queen 0.93, cat–car near 0 in a real modelAn embedding turns a token or a text into a point, placed near things with similar meaning.Each embedding is a list of numbers, often 768 to 3,072 of them. Distance in that space tracks meaning, and directions carryrelationships. Similarity is measured as the cosine of the angle between two vectors: 1 for the same direction. Embeddings powersearch by meaning, recommendations and RAG (lesson 10). Positions here are illustrative.

Similar meanings sit close. Measure closeness with cosine similarity.

Cosine similarity · Going deeper

Cosine by hand: direction counts, length does not

Question "refund window" = [0.9, 0.1, 0.3], length 0.954ChunkVectorDot productLengthCosineStraight-line distanceA "money back within 14 days"[0.8, 0.2, 0.4]0.8600.9170.9840.173B a long page on refunds[2.4, 0.5, 1.3]2.6002.7750.9821.847C "store opening hours"[0.1, 0.9, 0.2]0.2400.9270.2711.136A: dot = 0.9×0.8 + 0.1×0.2 + 0.3×0.4 = 0.86; cosine = 0.86 ÷ (0.954 × 0.917) = 0.984✓Cosine ranks A, B, C: both refund chunks score above 0.98, opening hours 0.27.✗Distance ranks A, C, B: the long page (1.85) looks further away than opening hours (1.14).Cosine compares direction only, so a long vector and a short one on the same topic still match.Cosine similarity is the dot product divided by both lengths: 1 for the same direction, 0 for unrelated. Raw distance alsopunishes a difference in length, which says nothing about the topic. Most embedding models return vectors of length 1, and thenall three measures agree: the dot product is the cosine, and distance² = 2 − 2 × cosine. Use the metric the model was trained with.

Cosine similarity = dot product ÷ (length × length). For unit vectors it is the dot product.

Attention

Attention lets each token draw on every other token

Theanimaldidn’tcrossthestreetbecauseitwastootired2%59%3%5%2%10%3%4%3%7%How much "it" attends to each other word (one attention head, illustrative scores, softmax computed)"it" draws 59% of its attention from "animal", so its meaning in this layer becomes the animal, not the street.Attention lets every token look at every other token and take in what it needs.Each token asks a question (a query), every token offers a key and a value, and the match between query and keys sets the weightsshown here. A transformer runs many such heads in parallel and stacks dozens of layers. This is what lets a model resolve "it",follow a long argument, or connect a question to a fact twenty pages back.

Attention weights decide which words shape each word’s meaning.

Attention · Going deeper

The transformer, and why long context is costly

Tokens inEmbeddingAttentionFeed-forward× N layersdozens to 100+OutputAttention compares every token with every other: the work grows with the square of the length.1,000 tokens1,000,000 pairs10,000 tokens100,000,000 pairs100,000 tokens10,000,000,000 pairs1,000,000 tokens1,000,000,000,000 pairslog scaleA transformer stacks the same block, attention then feed-forward, dozens of times.Introduced in 2017 ("Attention Is All You Need"), it processes all tokens of the input at once, so it trains in parallel on GPUs.The cost is that attention pairs every token with every other: ten times the context is a hundred times the pairs. Long-contextmodels rely on engineering (caching, sparse and efficient attention) to make that affordable.

Attention compares every pair of tokens: cost grows with the square of the length.

Context window

The context window is the model’s entire working memory

One request to a model with a 32,000-token context windowEarlier conversation: 9,000Retrieved documents: 14,000free 3,200System prompt: 1,500 · Question: 300 · Room for the answer: 4,000 (the narrow boxes)Typical windows in 2026: 128,000 tokens to over 1,000,000.200,000 tokens ≈ 150,000 words ≈ 500 pages of a book (at 300 words a page).Anything not in the window does not exist for the model. It keeps no memory between requests: each one sends the whole history again.The context window is the model’s entire working memory for one request.Instructions, the conversation so far, any documents and the answer it writes must all fit, counted in tokens. When a chat grows toolong, the oldest parts are dropped or summarised. Bigger windows cost more per request and run slower, and models attend lessreliably to the middle of a very long context, so put what matters most at the start or the end.

Instructions, history, documents and the answer must all fit in the context window.

Training stages

Pretraining teaches language; post-training teaches behaviour

Pretrainingmonths on thousands of GPUsDatapredict the next token ontrillions of tokens of web, booksand codeResulta model that continues text, butfollows no instructionsSupervised fine-tuningdaysDataexamples of instructions with goodanswers, written by peopleResulta model that answers instead oframblingPreference tuningdaysDatapeople (or models) rank pairs ofanswers: RLHF, DPOResultanswers that are helpful, honestand safeDeployed modelDataserved behind an API, sometimeswith tools and guardrailsResultwhat you call from your appPretraining teaches a model language and knowledge; post-training teaches it how to behave.Scale grew fast: GPT-3 (2020) learned from about 300 billion tokens; Llama 3 (2024) from over 15 trillion, fifty times more.Pretraining is self-supervised and costs the most by far. Fine-tuning and preference tuning use much smaller, curated data to turna text predictor into an assistant. You can repeat the later steps on your own data (lesson 12); almost nobody repeats the first.

Pretraining on trillions of tokens, then fine-tuning and RLHF for behaviour.

Training stages · Going deeper

Foundation models: one pretrained model, many tasks

Foundation model typeIn → outUsed forLarge language modeltext → textchat, summarise, write codeMultimodal modeltext, image, audio → textread a chart, describe a photoImage and video generatortext → image or videodiffusion modelsEmbedding modeltext or image → vectorsearch, RAG, clusteringSpeech modelaudio ↔ texttranscription, voiceSmall language modeltext → text, cheaplyon devices, high volumeA foundation model is pretrained once on broad data, then adapted to many tasks.Traditional ML trains one model per task on labelled data. A foundation model is large, general and adapted by prompting, RAG orfine-tuning. Some are proprietary and reached only through an API; open-weight models can be downloaded and hosted yourself.Bedrock, Foundry and Vertex AI offer catalogues of both. Choose by modality, quality, latency, cost, context size and licence.

Pick a foundation model by modality, quality, latency, cost and licence.

Temperature and sampling

Temperature sharpens or flattens the odds before sampling

temperature 0.3Paris100%a0%the0%located0%Lyon0%nearly always Paris: repeatabletemperature 1Paris91%a4%the2%located2%Lyon1%the model’s own oddstemperature 1.8Paris68%a12%the9%located7%Lyon4%flatter: varied, may driftTemperature sharpens or flattens the odds before a token is drawn.Each logit is divided by the temperature before softmax. Low temperature makes the top token dominate: use it for extraction,classification and code. Higher temperature spreads the odds: more varied writing, more risk of nonsense. Temperature 0 takes thetop token every time, though output can still vary slightly between runs on some services.

Low temperature for facts and code; higher for varied writing.

Temperature and sampling · Going deeper · 1 of 2

Top-p, top-k, max tokens and stop sequences

TokenP at temp. 1.8Running totalKept, top-p 0.9?Paris67.7%67.7%yesa12.1%79.7%yesthe9.2%88.9%yeslocated7.3%96.2%yesLyon3.8%100.0%noSettingWhat it doestop-kkeep only the k most likely tokenstop-pkeep the smallest set reaching pmax tokensstop after this many tokensstop sequencestop when this text appearsTop-p and top-k cut off the unlikely tail before sampling.Top-p (nucleus sampling) keeps the most likely tokens until their probabilities add up to p, here 0.9, and samples only among them.Top-k keeps a fixed number. Both stop a rare, odd token from derailing the text. Max tokens limits length and therefore cost; stopsequences end output at a marker. Change temperature or top-p, not usually both.

Top-p keeps the likely tokens until they add up to p.

Temperature and sampling · Going deeper · 2 of 2

Text is scored by overlap, by meaning, or by a judge

ReferencethecatsatonthematModel outputthecatlayonthematROUGE-1 recall = 5 ÷ 6 = 0.83BLEU-1 precision = 5 ÷ 6 = 0.83MethodMeasuresUse forROUGEword overlap with a reference (recall)summariesBLEUn-gram overlap with references (precision)translationBERTScoresimilarity of meaning, via embeddingsparaphrase-tolerant tasksPerplexityhow surprised a model is by real textlanguage model trainingHuman or LLM judgegraded against a rubricchat, tone, helpfulnessGenerated text has no single right answer, so it is scored by overlap, meaning or a judge."Lay" for "sat" costs a point in ROUGE and BLEU though the meaning barely changed; BERTScore would forgive it. Overlap metrics arecheap and repeatable but blind to meaning. A model as judge scales human-style grading, and needs its own checks. Bedrock, Foundryand Vertex AI each run these evaluations, including automatic and human-review jobs.

ROUGE for summaries, BLEU for translation, BERTScore for meaning.

Diffusion models

Image models start from noise and remove it step by step

Generating "a round shape": the model removes a little predicted noise at each steppure noisestep 10step 25step 40step 50: imageTraining runs the other way: add noise to real images, and teach the network to predict the noise that was added.A diffusion model starts from noise and removes it, step by step, guided by the prompt.It learned by watching real images dissolve into noise and predicting what was added. Run backwards, from random noise, itproduces a new image that fits the text. Image and video generators work this way (Stable Diffusion, Amazon Nova Canvas, Imagen).More steps give finer detail at more cost. Language models, by contrast, generate token by token.

A diffusion model learns to predict and remove noise, guided by the prompt.

Lesson complete

You finished “How large language models work”

Nicely done.

Open in a new tab