Caching
A cache answers repeated reads from memory and spares the database
A cache keeps recent answers in memory. At 90% hits the average read drops from 10.5 ms to 1.5 ms.
Most systems read the same things over and over. Without a cache, ten thousand reads a second become ten thousand queries, each taking about ten milliseconds, and the database works hard on answers it gave a second ago. Put an in-memory cache beside the application. Nine reads in ten find their answer there in half a millisecond. Only a thousand a second reach the database. The average read takes about a millisecond and a half, and the database, the hardest part of the system to scale, does a tenth of the work.
Layers of caching · Going deeper
Caches sit at every layer, from the browser to the database
The nearer the user a request is answered, the less every layer behind it does: browser, CDN, distributed cache, database.
Hit ratio
The last few percent of hits cut database load the most
Database load is the miss ratio: 99% hits sends ten times fewer reads to the database than 90%.
How good a cache is comes down to one number, the hit ratio. As it rises, the average read time falls toward the cache’s own speed. At ninety percent, the database still sees one read in ten. At ninety-nine percent, one in a hundred: ten times less load, from what looks like a small improvement. More memory, longer lifetimes and simpler keys raise the ratio. And if most reads are unique, no cache will help.
Cache-aside
The application checks the cache, and fills it on a miss
In cache-aside (lazy loading) the app reads the cache, falls back to the database on a miss, and stores the result with a TTL.
Here is the most common pattern, step by step. The application asks the cache for product forty-two. It is not there: a miss. So the application queries the database. And writes the answer into the cache, with a lifetime of three hundred seconds. The next request finds it. When the product changes, the application updates the database and deletes the cached copy, so the next read loads the new value. Only data that is read gets cached, and if the cache disappears the application still works, just slower.
Caching patterns · Going deeper
Four patterns, and Redis or Memcached to hold them
Write-through keeps the cache current; write-behind is fastest but can lose data; Redis adds replicas and persistence.
TTL and invalidation
A TTL caps how stale data can get; deleting on write keeps it fresh
With only a TTL, a change can stay hidden for up to the whole TTL. Invalidation on write closes the gap.
Every cached value is a copy, and copies go stale. A price of twenty dollars is cached with a lifetime of three hundred seconds. Two minutes later the price changes to twenty-five in the database. The cache does not know. For the next three minutes, users see twenty. If the code that changes the price also deletes the cached key, the next read fetches twenty-five at once. Choose each lifetime from how wrong the data may be, and invalidate on write for data your own code changes.
Cache stampede · Going deeper
When a hot key expires, let one request rebuild it
A cache stampede sends every request to the database at once; a lock, early refresh or jittered TTLs prevent it.
Content delivery networks
A CDN answers users nearby, and the origin sees only the misses
A CDN caches at edge locations near users. At 95% hits the origin sends 7.5 TB a day instead of 150 TB.
The same idea, placed near the user. A user in Sydney asks for a photo. The nearest edge location already has it and answers in fifteen milliseconds. Another photo is not there yet. The edge fetches it from the origin in Virginia, about two hundred milliseconds away, returns it, and keeps a copy for the next person. With ninety-five percent of requests answered at the edge, the origin sends seven and a half terabytes a day instead of a hundred and fifty. Users get speed; you get a smaller bill and an origin that is hard to overwhelm.
HTTP caching
Version file names and cache them for a year; keep the page short-lived
Cache-Control sets who may keep a response and for how long; versioned file names make long caching safe.
Browsers and CDNs follow instructions in each response. A script whose file name includes a fingerprint of its contents can be cached for a year, because a new build produces a new name. The HTML page that names those files is cached briefly: a minute in the browser, five minutes at the CDN. A deploy changes the page, and the page points at the new files. Personal responses are marked private, never kept by a shared cache. And keep the cache key small: each header or cookie it includes splits the cache and lowers the hit ratio.
Lesson complete
You finished “Caching and content delivery”
Nicely done.