Ways to customise a model

Climb the customisation ladder only as far as you need

Promptinginstructions, examplesnoneminutesRAGyour documents, per querydocumentsdaysFine-tuningbehaviour, format, a skillhundreds to thousands oflabelled pairsdays, some GPUContinued pretrainingdomain languagelarge unlabelled textweeks, more GPUTrain from scratcheverythingtrillions of tokensmonths, millionsmore cost, data, time and skill; more change to the model itselfClimb only as far as you need: most problems are solved by the first two steps.Prompting and RAG change nothing in the model and can be undone in minutes. Fine-tuning changes behaviour; continuedpretraining teaches a domain’s language; training from scratch is for model builders.

Prompt, then RAG, then fine-tune; pretraining is rarely yours.

Fine-tuning and LoRA

LoRA fine-tunes a model by training a tiny add-on

W4,096 × 4,09616,777,216 weights, frozen+A: 4,096 × 8×B: 8 × 4,096Trained with LoRA2 × 4,096 × 8 = 65,53665,536 ÷ 16,777,216 = 0.39% of this layerAt use: W + B·A, or swap adaptersper task on one shared base model.Fine-tuning adjusts a pretrained model on your examples; LoRA trains a tiny add-on instead of every weight.Full fine-tuning updates billions of weights and needs the memory to match. LoRA freezes the model and learns two thinmatrices per layer whose product is the change: here 0.39% of the weights. It is cheaper, the adapter is a small file, andthe base is untouched. Any fine-tuning can cause catastrophic forgetting of general skills: keep an evaluation set for those too.

LoRA trains 0.39% of a layer’s weights; the base stays frozen.

Fine-tuning and LoRA · Going deeper

Kinds of fine-tuning

MethodDataResultInstruction fine-tuninglabelled prompt and response pairsfollow a format, tone or taskContinued pretrainingunlabelled domain textlearn the vocabulary of law, medicine, a codebasePreference tuning (RLHF, DPO)pairs ranked better or worsehelpful, harmless, preferred answersDistillationa large model’s answersa small model that copies a big one, cheaperTransfer learning (classic ML)a pretrained network, new headimage or text classifiers with little dataFine-tuning methods differ in the data they need and what they change.Most teams use instruction fine-tuning with a few hundred to a few thousand high-quality examples; quality beats quantity.Amazon Bedrock offers fine-tuning, continued pre-training and distillation for supported models; Foundry and Vertex AI offersupervised and preference tuning. A custom model on Bedrock is served through provisioned throughput or on-demand, by model.

Instruction tuning, continued pretraining, preference tuning, distillation.

Cost of inference

Tokens times price: the model you choose sets the bill

1,000 requests, each 2,000 tokens in and 500 tokens out. Prices per million tokens, illustrative.Small model$0.25 in, $1.25 out$1.13Mid model$3 in, $15 out$13.50Large model$15 in, $75 out$67.50Small: 2,000 × $0.25 + 500 × $1.25 = $0.001125 per request; × 1,000 = $1.13. Output tokens cost more.The model you pick sets the bill: here a 60× difference for the same traffic.Cost is tokens in × price + tokens out × price, per request. Use the smallest model that passes your test set, route only hardrequests to a big one, shorten prompts, cache repeated prompt prefixes, and use batch inference (often around half price) forwork that can wait. Larger models are also slower, so the same choice sets latency.

Cost = tokens in × price + tokens out × price. Use the smallest model that passes.

Resent history · Going deeper · 1 of 2

Every turn pays again for all the turns before it

Input tokens sent on each turn of one chat: system prompt 1,500, each question 100, each answer 30011,600232,4004563,6007894,8001011126,000turnOver 12 turnsnew text typed 1,200input tokens billed 45,600of which sent before 42,900share repeated 94%purple: the new question · pale: everything resentPrompt caching bills the repeated prefix at areduced rate, if the prefix is byte-for-byte the same.A chat or agent resends everything on every turn, so input cost grows with the square of its length.Turn 12 sends 6,000 tokens to ask a 100-token question. Across the chat, 94% of the input had been sent before. Three levers: keepthe stable part (system prompt, tool definitions, documents) at the front so a prompt cache can match it, never put a timestampor user name ahead of it, and summarise or trim old turns. Cache discounts and lifetimes differ by provider: check the price page.

Keep the stable prefix first so prompt caching can match it; trim old turns.

Latency · Going deeper · 2 of 2

Output length, not input length, sets how long a reply takes

Example model: first token after 0.5 s, then 60 tokens a secondA label or a JSON object: 30 output tokens1.0 sA short answer: 250 output tokens4.7 sA full page: 1,000 output tokens17.2 sgrey: waiting for the first token · purple: writing. Drawn to scale.Response time is the wait for the first token plus the output length divided by the writing speed.The model reads its input in one parallel pass but writes one token at a time, so output length dominates: the full page takes17 times as long as the label. To cut latency, ask for less (a cap on output tokens, a tighter format), stream so the user reads fromthe first token, and pick a faster model for simple steps. Measure time to first token and tokens a second separately. Example figures.

Latency = time to first token + output tokens ÷ tokens per second.

Quantisation

Fewer bits per weight shrinks a model to fit the hardware

A 70-billion-parameter model: memory for the weights aloneFP324 bytes per weight280 GBFP16 / BF162 bytes per weight140 GBINT81 byte per weight70 GBINT40.5 bytes per weight35 GBone 80 GB GPUQuantisation stores weights with fewer bits, so a model needs less memory and runs faster.70 billion weights × 2 bytes = 140 GB, too big for one 80 GB GPU; at 8 bits it is 70 GB and fits. Four bits halves it againwith a small, measurable loss of quality. Running needs more memory than the weights: the key-value cache for long contexts andmany users adds a lot. Smaller and quantised models are how you serve on fewer GPUs, or on a laptop or phone.

Quantisation: 140 GB at 16 bits, 70 GB at 8, 35 GB at 4.

Quantisation · Going deeper · 1 of 2

On-demand, batch or provisioned

OptionHowFitsOn-demandpay per token, shared capacityvariable or low traffic, getting startedBatchsubmit a file, results laterlarge jobs that can wait; lower priceProvisioned throughputreserved capacity, fixed hourly pricesteady high traffic; some custom modelsSelf-hostedyour GPUs or endpointopen-weight models, full controlPay per token when traffic varies; reserve capacity when it is steady and high.On-demand is the default for foundation models on Bedrock, Foundry and Vertex AI. Provisioned throughput (Bedrock) orprovisioned throughput units (Azure) guarantee capacity for a commitment period. Batch inference trades speed for price.Self-hosting on SageMaker AI, Azure Machine Learning or Vertex AI endpoints puts scaling and patching on you.

On-demand for variable traffic; provisioned for steady; batch when it can wait.

Quantisation by hand · Going deeper · 2 of 2

Quantising rounds each weight to a grid: fine at 8 bits, coarse at 4

Weight (float)8-bit integerread back4-bit integerread back0.03160.03000.000−0.214−43−0.212−2−0.1790.412840.41450.447−0.074−15−0.074−1−0.0890.128260.12810.089−0.626−127−0.626−7−0.6260.00920.01000.0000.305620.30630.268scale = largest |weight| ÷ top level; integer = round(weight ÷ scale); read back = integer × scale8 bitslevels −127 to 127, step 0.005mean error 0.0009 (0.4% of a typical weight)4 bitslevels −7 to 7, step 0.089mean error 0.0251 (11% of a typical weight)2 of 8 weights round to zero at 4 bits.Quantising snaps every weight to the nearest point on a coarse grid, and stores only the grid number.With 8 bits the grid has 255 points and the rounding error is far below the weights themselves. With 4 bits there are 15, andsmall weights collapse to zero. That is why 8-bit models lose almost nothing and 4-bit ones lose a little, and why real schemesset a scale per block or per channel, calibrate on sample data, or fine-tune after quantising. Example weights, uniform symmetric grid.

Integer = round(weight ÷ scale); fewer bits mean a coarser grid and more error.

Lesson complete

You finished “Customising and running models”

Nicely done.

Open in a new tab