Caching

A cache answers repeated reads from memory and spares the database

Without a cache: every read is a queryapp10,000 queries/sdatabase10 ms per readWith a cache: 90% of reads are hitsapp9,000/sin memory1,000/sdatabase1.5 ms on averageExample figures: a cache read 0.5 ms, a database query 10 ms. Average = 0.9 × 0.5 + 0.1 × (0.5 + 10).A cache answers repeated reads from memory: reads get 7× faster and the database does 10% of the work.Most systems read the same data again and again: the same product page, the same profile, the same feed. Keeping recentanswers in memory turns a 10 ms query into a sub-millisecond lookup, and the database, the hardest part to scale, seesonly the misses. It is the cheapest scaling there is. The price: the cache can be out of date. The rest of this lesson.

A cache keeps recent answers in memory. At 90% hits the average read drops from 10.5 ms to 1.5 ms.

Layers of caching · Going deeper

Caches sit at every layer, from the browser to the database

LayerWhereRead costNotesBrowser cachethe user’s own device0 msCache-Control headersCDN edgea city near the user5 to 20 msCloudFront · Front Door · Cloud CDNRegional edge / shieldone layer behind the edges+10 to 30 msprotects the origin from repeatsIn-process memoryinside each app servermicrosecondssmall, per server, lost on restartDistributed cachea shared cluster in the zoneabout 0.5 msElastiCache · Azure Cache · MemorystoreDatabase cachepages kept in the DB’s memoryautomaticor DAX in front of DynamoDBCaches sit at every layer. The nearer the user a request is answered, the less every layer behind it does.A request answered in the browser never reaches the network. One answered at the CDN never reaches your region. Oneanswered by the distributed cache never reaches the database. Each layer is a separate decision about what may bestale and for how long. Personal or rapidly changing data is cached close to the data; public, static files close to the user.

The nearer the user a request is answered, the less every layer behind it does: browser, CDN, distributed cache, database.

Hit ratio

The last few percent of hits cut database load the most

0%50%90%100%0 ms5 ms10 mshit ratio: the share of reads answered by the cacheaverage read time90% hits: 1.50 ms;database sees 10% of reads99% hits: 0.60 ms;database sees 1% of readsThe last few percent of hits matter most: 99% sends the database ten times fewer reads than 90%.The database load is the miss ratio: 10% misses at 90% hits, 1% at 99%. So raising the hit ratio from 90 to 99 cutsdatabase reads by ten. Hit ratio rises with more memory (more keys fit), longer TTLs and fewer variations in the cachekey. It falls when the data is mostly unique, which is when a cache does not help at all.

Database load is the miss ratio: 99% hits sends ten times fewer reads to the database than 90%.

Cache-aside

The application checks the cache, and fills it on a miss

ApplicationCacheDatabase1 GET product:422 miss (nothing there)3 SELECT … WHERE id = 424 app writes it to the cache, TTL 300 sOn a writeupdate the database, thendelete product:42 from the cacheCache-aside: the application checks the cache, and fills it from the database on a miss.Also called lazy loading. Only data that is actually read gets cached, and if the cache fails the application stillworks, just slower. The first read of anything is a miss, and the copy can go stale, so every entry has a TTL and awrite deletes the key, letting the next read load the new value. It is the default pattern with Redis or Valkey.

In cache-aside (lazy loading) the app reads the cache, falls back to the database on a miss, and stores the result with a TTL.

Caching patterns · Going deeper

Four patterns, and Redis or Memcached to hold them

PatternHowGainsCostsCache-asideread cache; on a miss, read DB and fillcaches only what is readfirst read slow; may be staleWrite-througheach write updates cache and DBcache always currentcaches unread data; slower writesWrite-behindwrite to cache; DB later, in batchesfastest writesloses data if the cache failsRead-throughthe cache loads from the DB itselfsimple app codecache must know the DB (DAX)EngineData typesCopies and persistenceChoose it forValkey / Redis OSSstrings, hashes, lists, sorted sets, streamsreplicas, failover, snapshotssessions, leaderboards, pub/subMemcachedplain keys and valuesnone: a lost node loses its keyssimple, multi-threaded cachesFour ways to keep cache and database together, and two engines to keep them in.Combine patterns: cache-aside for reads plus write-through for data that must be fresh, such as a price. The twoengines answer different needs: Valkey and Redis OSS bring rich data types, replicas and failover (ElastiCache, AzureCache for Redis, Memorystore); Memcached is a plain, fast, disposable store. Exam clue: “replication” or “persistence” means Redis.

Write-through keeps the cache current; write-behind is fastest but can lose data; Redis adds replicas and persistence.

TTL and invalidation

A TTL caps how stale data can get; deleting on write keeps it fresh

0 s60 s120 s180 s240 s300 scached: price $20TTL 300 s: the entry lives until heredatabase: price $25stale for 180 s: users see $20Delete the key on write and the next read loads $25: stale for milliseconds, not minutes.A TTL caps how stale a cached value can get. Deleting the key on every write keeps it fresh.Pick each TTL from how wrong the data may be: seconds for stock levels, minutes for a product page, a day for acountry list. Expiry alone means users may see the old value for up to a whole TTL after a change. Invalidating onwrite closes that gap for data your own code changes; the TTL still protects against anything that slips past.Add a little random jitter to TTLs so that thousands of keys set together do not all expire in the same second.

With only a TTL, a change can stay hidden for up to the whole TTL. Invalidation on write closes the gap.

Cache stampede · Going deeper

When a hot key expires, let one request rebuild it

A popular key expireshomepage: expired2,000 identical queriesevery request misses at once and rebuilds the same entryFixes✓one rebuilds, the rest wait: a lock or request coalescing✓refresh before expiry, in the background✓jitter TTLs so keys expire at different times✓serve the stale value for a moment while refreshing✓CDN: Origin Shield collapses repeats from many edgesWhen a hot key expires, every request misses at once. Let one rebuild it while the others wait or reuse the old copy.This is a cache stampede (thundering herd). It arrives exactly when load is highest, because that is when the keyis popular, and it can take the database down just as the cache was meant to protect it. The fixes all make surethat one miss causes one database query, not thousands.

A cache stampede sends every request to the database at once; a lock, early refresh or jittered TTLs prevent it.

Content delivery networks

A CDN answers users nearby, and the origin sees only the misses

Originus-east-1S3 bucketload balancerSydneyedge: Sydneyhit: 15 msSydneyedge: missmiss: fetched from the origin, about 200 ms,then kept at the edge for the next userOrigin sends 7.5 TB a daynot 150 TB: 95% served at the edgeA CDN keeps copies at hundreds of edge locations, so most users are answered nearby and the origin sees only misses.A content delivery network caches responses in cities around the world. Static files (images, scripts, video) cachewell; dynamic pages can still use it for TLS near the user and a fast path to the origin. Edge data transfer is usuallycheaper than transfer from the region. CloudFront, Azure Front Door and Google Cloud CDN; keep the origin bucket private.

A CDN caches at edge locations near users. At 95% hits the origin sends 7.5 TB a day instead of 150 TB.

HTTP caching

Version file names and cache them for a year; keep the page short-lived

GET /assets/app.3f9a1c.jsCache-Control: public, max-age=31536000, immutablethe file name changes with every build,so a cached copy can live for a yearGET /index.htmlCache-Control: public, max-age=60, s-maxage=300the page names the current files; browsersrecheck after 60 s, the CDN after 300 sGET /api/meCache-Control: private, no-storepersonal data: never stored by a shared cachecache key = path + only the query strings, headersand cookies that change the answerName files by version and cache them for a year. Keep the page that lists them short-lived.Cache-Control tells browsers and CDNs what to keep and for how long: max-age for everyone, s-maxage for shared caches,private or no-store for personal responses. Versioned file names make deploys instant without invalidations, whichare slow and charged beyond a free allowance. Keep the cache key small: each extra header or cookie in it lowers the hit ratio.

Cache-Control sets who may keep a response and for how long; versioned file names make long caching safe.

Lesson complete

You finished “Caching and content delivery”

Nicely done.

Open in a new tab