The problem
Start from what users do and what the business promises
Upload, feed, like. Feed p99 under 500 ms, 99.9% available, photos never lost: requirements as numbers.
A photo app does three things: upload a photo, view a feed, and like. The requirements put numbers on what matters: the feed loads in under half a second for ninety-nine requests in a hundred, the app is up 99.9 percent of the time, and a photo is never lost. It launches with a thousand daily users and hopes for a million.
The estimate
Turn users into requests a second and bytes a year
At a million daily users: about 1,215 feed reads a second at peak and 18 TB of photos a year.
At a thousand users, the load is tiny. At a hundred thousand, about a hundred and twenty feed reads a second at peak. At a million, about 1,215, with likes at about 405 a second and 18 terabytes of new photos a year. A read-heavy app with a growing pile of files: that sentence chooses most of the design.
The first design
One server is the right first design
App, database and files on one machine: cheap, simple and fast enough for a thousand users.
App, database and photo files on one server. At a thousand users it barely notices the load. Building for a million on day one would cost more and guess at problems the app may never have. The skill is knowing which limit comes first.
The limits
Limits arrive in an order you can predict
A single point of failure from day one, the disk at ~110,000 users, the database at ~247,000.
The single server fails the availability target from the start: one disk, one zone. At half a megabyte a photo, a two terabyte disk fills within a year at about a hundred and ten thousand users. And if one database answers about 300 feed queries a second, the feed slows at peak from about a quarter of a million users. Each limit has its own fix.
The fixes
Each limit gets its own fix
Object storage and a CDN, stateless servers in two zones, a feed cache, replicas and a queue. Database reads fall to 122 a second.
Photos move to object storage, which keeps copies across zones, with a CDN in front. App servers keep no state, so a balancer spreads them over two zones. A cache answers nine feed reads in ten, so the database's load falls from about 1,215 to 122 a second. Read replicas share what is left, with a standby in the second zone. And thumbnails are made from a queue, so uploads return at once.
The trade-off
Every fix gives up a little; sharding gives up the most, so it waits
A feed up to 30 s stale, replica lag, late thumbnails. Sharding is postponed: writes peak at 409 a second.
The cache makes a feed up to 30 seconds old. Nobody notices a photo arriving late. Replicas lag slightly, so a user's own new photo is read from the primary. Thumbnails appear a moment late. And sharding, the hardest change to undo, is postponed: writes peak at about 409 a second, far from the primary's limit.
The finished design
At a million users, every part answers one limit
CDN, balancer, stateless servers, cache, database and replicas, queue, storage, timeouts and encryption. Each links to its lesson.
Here is the design at a million users. A CDN for photos, a load balancer over two zones, stateless app servers, a cache for feeds, a database with replicas, a queue and workers for thumbnails, object storage, timeouts and retries on every call, and encryption throughout. Every part answers one limit found on the way up. Each box opens the lesson that teaches it.
A real case
Instagram at 14 million users had the same parts
Three engineers, 14 million users: a balancer, Django servers, sharded PostgreSQL, Redis, S3 and a CDN, and a queue.
A little over a year after launch, Instagram had fourteen million users and three engineers. Their principles: keep it very simple, do not reinvent the wheel, and use proven technology. Their post lists the stack: a load balancer, more than twenty-five Django servers, PostgreSQL with replicas in another zone and already sharded, Redis and Memcached, photos in S3 behind CloudFront, and a queue with about two hundred workers. The same parts as our design, chosen for the same reasons.
Lesson complete
You finished “A photo app from a thousand users to a million”
Nicely done.