The problem

Start from what users do and what the business promises

Upload a photostore it for ever, make thumbnailsView the feednewest photos from people you followLike a photoa small write, shown to the ownerRequirements, with numbersFeed loads in under 500 ms for 99 in 100 requestsAvailable 99.9% of the timeA photo, once uploaded, is never lostLaunch with 1,000 daily users; plan for a millionA small team: prefer managed, proven partsA design starts from what users do and what the business promises, written as numbers.A photo-sharing app does three things: upload, view a feed, like. The requirements put numbers on what matters: how fast the feed is,how often the app may be down, and that photos are never lost. It launches with a thousand daily users and hopes for a million. The rulefor this case: at each size, estimate the load, find the first thing that breaks, and change only that.

Upload, feed, like. Feed p99 under 500 ms, 99.9% available, photos never lost: requirements as numbers.

The estimate

Turn users into requests a second and bytes a year

Per daily user each day: 30 feed loads, 0.1 uploads of 0.5 MB, 10 likes. Peak is 3.5× the average.Daily usersFeed reads a second, peakLikes a second, peakPhotos a year1,0001018 GB100,000122411.8 TB1,000,0001,21540518 TBAt a million daily users: about 1,215 feed reads a second at peak, and 18 TB of new photos a year.30 feed loads a day per user, spread over 86,400 seconds and multiplied by 3.5 for the evening peak, gives the load that matters. Readsoutnumber writes: likes peak at about 405 a second and uploads at 4. Photos grow by 18 TB a year and must never be lost. Sothis is a read-heavy app with a large, growing pile of files. That one sentence will choose most of the design.

At a million daily users: about 1,215 feed reads a second at peak and 18 TB of photos a year.

The first design

One server is the right first design

1,000 usersOne serverone zoneAppDatabasePhoto files on its diskAt 1,000 users1.2 feed reads a second at peak18 GB of photos a yearCheap, simple, and fast enough.One server is the right first design. It is cheap, easy to change, and fast enough for a thousand users.At a thousand daily users the app serves about 1.2 feed reads a second at peak and stores 18 GB of photos a year: nothing a singlemachine notices. Building for a million on day one would cost more, slow the team down and guess at problems the app may neverhave. The skill is knowing which limit it will hit first, and roughly when.

App, database and files on one machine: cheap, simple and fast enough for a thousand users.

The limits

Limits arrive in an order you can predict

1k10k100k1MOne server, one zone, from day oneany failure takes the app down; 99.9% needs two of everythingDisk: 2 TB of photos, ~110,000 usersfills within a year; photos on one disk can be lostDatabase: ~300 feed queries a second, ~247,000 usersfeed slows past the 500 ms target at peakLimits arrive in an order you can predict. Write the order down before you hit the first one.The single server is a single point of failure from day one: one disk, one zone, so 99.9% is out of reach. At 0.5 MB a photo, 2 TB of diskfills within a year at about 110,000 daily users. And if one database answers about 300 feed queries a second, the feed slows at peak fromabout 247,000 users. Each limit has its own fix; none of them needs the others.

A single point of failure from day one, the disk at ~110,000 users, the database at ~247,000.

The fixes

Each limit gets its own fix

Photosobject storage + CDNnever lost, no diskApp serversstateless, 2 zones,behind a balanceradd more at peakFeedscache, 90% hit ratio1,215 → 122 DB reads/sDatabaseprimary + 2 readreplicasstandby in the 2nd zoneThumbnailsa queue and workersuploads return at onceDatabase feed reads a second at peak, to scale1,215 without a cache122 with a 90% hit ratio, shared by 2 replicasEach limit gets its own fix, and each fix is a part this course taught on its own.Photos move to object storage, which keeps copies across zones, with a CDN in front. App servers keep no state, so a balancer can spreadthem over two zones and add more at peak. A cache answers 90% of feed reads, cutting the database's load from about 1,215 to 122 asecond, which read replicas share. Thumbnails are made by workers from a queue, so an upload returns at once.

Object storage and a CDN, stateless servers in two zones, a feed cache, replicas and a queue. Database reads fall to 122 a second.

The trade-off

Every fix gives up a little; sharding gives up the most, so it waits

ChangeWhat it gives upWhy that is acceptableFeed cachea feed can be up to 30 s oldfine: nobody notices a photo arriving lateRead replicasreplicas lag the primary, usually under a secondread your own new photo from the primaryThumbnail queuea new photo shows a placeholder brieflyworkers scale with the queueShardingnot done: the primary handles writes409 writes a second at peakEvery fix gives up a little freshness or simplicity. Sharding gives up the most, so it waits.A cache and replicas make the feed slightly stale, which a photo app can live with; a user's own new photo is read from the primary. Thequeue makes thumbnails appear a moment late. Sharding the database would remove the last single writer, but it is the hardest change toundo, and at about 409 writes a second the primary is far from its limit. Postpone what you cannot reverse.

A feed up to 30 s stale, replica lag, late thumbnails. Sharding is postponed: writes peak at 409 a second.

The finished design

At a million users, every part answers one limit

1M dailyCDNphotos · L5Load balancer2 zones · L4App serversstateless · L3Cachefeeds · L5Database+ replicas · L6Queuethumbnails · L7Workersresize · L8Object storagephotos · L10Timeouts, retriesevery call · L9Encryptionkeys · L11Two zones for every part: 99.9% (L2)Review: cost, monitoring, the six pillars (L12)The design at a million users uses nearly every lesson in this course, each for one reason.Nothing here was added for its own sake: each part answers one limit found on the way up, and the requirement it serves can be named.Each box opens the lesson that teaches it (L is the lesson number). The next limits, writes on one primary and the cost of the cache, arewritten down with the user count at which they will matter, so the next change can be planned rather than rushed.

CDN, balancer, stateless servers, cache, database and replicas, queue, storage, timeouts and encryption. Each links to its lesson.

A real case

Instagram at 14 million users had the same parts

Instagram, 201114 million users in just over a yearrun by 3 engineersPrincipleskeep it very simple; don’t reinvent the wheel;use proven and solid technologiesLoad balancingElastic Load Balancer, 3 nginxApp servers25+ Django on EC2, GunicornDatabasePostgreSQL, sharded, 12 + 12 replicasFeeds, sessionsRedis; Memcached on 6 instancesPhotosS3, served through CloudFrontBackground workGearman queue, about 200 workersInstagram at 14 million users had the same parts as our design, run by three engineers.Instagram’s engineering post, a little over a year after launch, lists the stack: a load balancer, stateless Django servers, PostgreSQLwith replicas in another zone and already sharded, Redis and Memcached, photos in S3 behind a CDN, and a queue with workers for fan-outand notifications. Its rule was ours: keep it simple, use proven parts, and change what the numbers say is breaking.

Three engineers, 14 million users: a balancer, Django servers, sharded PostgreSQL, Redis, S3 and a CDN, and a queue.

Lesson complete

You finished “A photo app from a thousand users to a million”

Nicely done.

Open in a new tab