Ways to customise a model
Climb the customisation ladder only as far as you need
Prompt, then RAG, then fine-tune; pretraining is rarely yours.
There are five ways to make a model fit your needs, each costing more than the last. Better prompts: minutes, no data. RAG, to give it your documents. Fine-tuning, to change how it behaves, with labelled examples. Continued pretraining on a mass of domain text. And training a new model from scratch, which few companies ever need. Start at the bottom and climb only when a test shows you must.
Fine-tuning and LoRA
LoRA fine-tunes a model by training a tiny add-on
LoRA trains 0.39% of a layer’s weights; the base stays frozen.
Fine-tuning continues training a pretrained model on your own examples. Updating every weight is expensive: this one layer alone has nearly seventeen million. LoRA freezes them. It adds a thin matrix A, and a thin matrix B. Their product is the change. That is sixty five thousand weights, 0.39 percent of the layer. The adapter is a small file you can swap per task.
Fine-tuning and LoRA · Going deeper
Kinds of fine-tuning
Instruction tuning, continued pretraining, preference tuning, distillation.
Cost of inference
Tokens times price: the model you choose sets the bill
Cost = tokens in × price + tokens out × price. Use the smallest model that passes.
Generative AI is billed per token. Take a thousand requests, each two thousand tokens in and five hundred out. A small model costs about $1.13. A mid-size model $13.50. A large one $67.50, 60 times more for the same traffic. So test whether the small model is good enough before paying for the large one.
Resent history · Going deeper · 1 of 2
Every turn pays again for all the turns before it
Keep the stable prefix first so prompt caching can match it; trim old turns.
Latency · Going deeper · 2 of 2
Output length, not input length, sets how long a reply takes
Latency = time to first token + output tokens ÷ tokens per second.
Quantisation
Fewer bits per weight shrinks a model to fit the hardware
Quantisation: 140 GB at 16 bits, 70 GB at 8, 35 GB at 4.
A seventy billion parameter model is seventy billion numbers. At four bytes each, two hundred eighty gigabytes. At two bytes, the usual for serving, one hundred forty: too big for one eighty gigabyte GPU. Quantise to one byte, and it is seventy, and fits. Four bits gives thirty five. Each step loses a little quality, so measure it on your test set.
Quantisation · Going deeper · 1 of 2
On-demand, batch or provisioned
On-demand for variable traffic; provisioned for steady; batch when it can wait.
Quantisation by hand · Going deeper · 2 of 2
Quantising rounds each weight to a grid: fine at 8 bits, coarse at 4
Integer = round(weight ÷ scale); fewer bits mean a coarser grid and more error.
Lesson complete
You finished “Customising and running models”
Nicely done.