The parts of a system

What is a system that serves millions made of?

UsersLoad balancershares out requestslesson 4App serversmany identical copieslesson 3Databasereplicas and partitionslesson 6CDNcopies near the userlesson 5Cacheanswers kept in memorylesson 5Queuework to do laterlesson 7Workerstake jobs from the queueAround every partAvailabilitylesson 2Application shapelesson 8Designing for failurelesson 9Disaster recoverylesson 10Securitylesson 11Cost and operationslesson 12Almost every large system is these few parts. This lesson is how you decide which of them you need.One server can hold all of it, and for a small system that is the right design. Each part is added only when a numberin the requirements says so: more requests than one server carries, more data than one disk holds, a promise aboutdowntime. So the course starts with writing requirements as numbers, then takes the parts one lesson at a time.

A large system is a few standard parts: load balancer, app servers, cache, database, queue and CDN. Requirements decide which ones you need.

Requirements

What it must do is easy; how well it must do it shapes the design

The brief: “A photo-sharing app. People upload photos, follow each other and scroll a feed.”Functional: what it must doUpload a photo with a captionFollow and unfollow other peopleSee a feed of photos from people followedLike and comment on a photoSearch for people by nameNon-functional: how well it must do itScale10 million daily users, doubling each yearSpeedfeed loads in under 300 ms for 99% of requestsAvailability99.9% each monthDurabilityan accepted upload is never lostCompliancepersonal data stays in the EUFunctional requirements decide what is built. Non-functional requirements decide how it is built.Any design that stores photos and lists them meets the left column; a laptop under a desk would. The rightcolumn is what rules that out: ten million users, a 300 ms promise, 99.9% and data kept in the EU decide theservers, the zones, the database and the region. Architecture is mostly the work of meeting the right column.

Functional requirements say what the system does; non-functional requirements say how well, and they decide the architecture.

Constraints · Going deeper

Constraints are fixed facts, and each one removes options

Own datacentreKuberneteson 3 cloudsVirtualmachines, USManagedcontainers, EUFunctions,EUOne largeserver, EUBudget: $20,000 a month✗✗✓✓✓✓Team of four, none knows Kubernetes✗✗✓✓✓✓Personal data stays in the EU✓✓✗✓✓✓Live in three months✗✗✓✓✓✓99.9% available (no single point of failure)✓✓✓✓✓✗Options leftruled outruled outruled outstill openstill openruled outConstraints are fixed facts you design within, and each one removes options.A requirement is a goal you may trade against others; a constraint is not negotiable: budget, deadline, the skillsof the team, the law, the systems you must connect to. Listing them first saves designing what cannot be built.Here five constraints leave 2 of 6 designs, and choosing between those two is the real decision.

A constraint is not negotiable: budget, deadline, team skills, law. Five constraints here leave two of six designs.

Quality attributes

A requirement without a number cannot be designed for or tested

As people say itAs an architect writes it: a number, a condition and how it is measured✗“It must be fast”✓the feed returns within 300 ms for 99% of requests at 10,000 a second✗“It must always be up”✓99.9% of minutes each month, measured at the load balancer✗“It must scale”✓handles three times today’s peak with no change to the design✗“It must be secure”✓encrypted in transit and at rest; staff sign in with MFA; access logged✗“Never lose a photo”✓an upload is acknowledged only once it is stored in two zonesA requirement you cannot measure cannot be designed for, or tested.“Fast” has no answer: fast at what load, for how many requests, measured where? Write each quality as a targetwith a number and a way to check it. These are the quality attributes (ISO 25010 lists them: performance,reliability, security, maintainability and more), and each one turns into a test, an alarm or a service level objective.

Each quality attribute becomes a target with a number, a condition and a way to measure it.

Back-of-envelope estimates

Daily users become requests per second with one division

Reads: opening the feed10 milliondaily users300 millionrequests a day3,472a second, on average10,417a second at the peak×30 each÷86,400 s×3 peakWrites: uploading a photo10 milliondaily users2 millionuploads a day23a second, on average69a second at the peak×0.2 each÷86,400 s×3 peakReads outnumber writes 150 to 1.A read-heavy system: reads get caches and copies; writes can be few and careful.Daily users become requests per second with one division, and that number sizes everything.A day has 86,400 seconds, so 300 million requests a day is about 3,472 a second. Traffic is not flat: the eveningpeak here is assumed at three times the average, and the design must carry the peak, not the average. Round freely:the goal is the order of magnitude, which tells you one server, ten, or a thousand.

300 million requests a day is about 3,472 a second. Design for the peak, here three times the average.

Back-of-envelope estimates · Going deeper

Six conversions, and the bandwidth the feed needs

Conversions to keep at hand1 day86,400 s, about 10⁵1 monthabout 2.6 million s1 million a dayabout 12 a second1 billion a monthabout 385 a second1 Gbit/s125 MB every second1 TB a dayabout 93 Mbit/s, constantlyBytes, not requests: the feed’s bandwidth300 million feedsa day× 10 thumbnailsof 50 KB each150 TBsent out a day13.9 Gbit/saverage; 42 at the peakMore than any one server’s network card: this is a job for a CDN.Six conversions turn any brief into rates, and bandwidth often matters more than requests.3,472 requests a second is modest, but each one pulls half a megabyte of images. 150 TB a day out of one regionis tens of gigabits a second and a large data transfer bill. Serving those bytes from edge caches near the users,lesson 5, is what makes the design affordable. Estimate requests, storage and bandwidth before choosing anything.

Thumbnails alone send 150 TB a day, about 14 gigabits a second: bandwidth, not requests, calls for a CDN.

Storage growth

Data only grows: size the store for year three, not launch day

2.6 PBend of year 17.9 PBend of year 218.4 PBend of year 339.4 PBend of year 481.5 PBend of year 53.6 MB per upload × 2 million a day = 7.2 TB a day; uploads double each yearsolid: added that year · light: all storedStorage only grows. Size it for year three, not for launch day.Year one stores 2.6 PB; by year five the total is 81 PB, because nothing is deleted and growth compounds.Two design choices follow at once: storage that grows without limits or migrations (object storage), and alifecycle that moves photos nobody opens any more to cheaper, colder tiers. Requests grow too; data never shrinks.

Year one stores 2.6 PB; with uploads doubling yearly, five years hold 81 PB.

Latency numbers

A network round trip costs thousands of memory reads

100 ns1 µs10 µs100 µs1 ms10 ms100 ms1 sRead from main memory100 nsRandom read from a local SSD20 µsRound trip inside one zone0.5 msRound trip between zones1 to 2 msHard disk seek5 msRound trip across the US65 msRound trip Europe to California150 msfeels instant to a person →A network round trip costs thousands of memory reads. Count the hops in every request.Each step on the scale is ten times the last; typical round figures, to show the order of magnitude. Inside a zonea call to another service costs about half a millisecond, so a page that makes twenty calls in a row has spent tenbefore any work is done. Crossing an ocean costs more than the whole 100 ms that feels instant to a person.

A round trip inside a zone takes about 0.5 ms, between zones 1 to 2 ms, across an ocean about 150 ms.

Latency budget · Going deeper

Split the target between the hops, and each part has a number

A latency budget: 300 ms at the 99th percentile, shared between every hop on the request’s pathDNS and TLS (reused): 5 msLoad balancer: 2 msWeb service: 25 msFeed service: 40 msCache read: 3 msDatabase (on a miss): 60 msMargin: 165 msSplit the target between the hops, and every team knows the number it must hit.The database is the largest share, and only on a cache miss. The margin absorbs slow moments, retries and growth;a design with no margin misses its target on the first bad day. If a hop cannot meet its share, the answer is acache, a copy nearer the user, or fewer hops in a row, not a faster server.

A latency budget divides the 300 ms target between every hop, and keeps a margin for bad days.

Percentiles

The average hides the slow requests; the 99th percentile shows them

0 ms200 ms400 ms600 ms800 msmedian (p50): 72 msaverage: 91 msp95: 182 msp99: 520 ms1 request in 100 is slower than this10,000 simulated requestsThe average hides the slow requests. Set targets at the 99th percentile.Half the requests finish within 72 ms and the average looks healthy at 91 ms, yet one in a hundred takes over 520 ms.Those are real people, and often the most active ones, who make the most requests. So requirements and alarmsname a percentile: p50 for the typical case, p99 for the tail. “Under 300 ms” always means “for 99% of requests”.

The median here is 72 ms and the average 91 ms, but the p99 is 520 ms: one request in a hundred is slower.

The tail at scale · Going deeper

Wait for many services, and the rare slow reply becomes common

02550751000%50%100%calls a page makes to services, each slow for 1 request in 1001 call: 1% of pages are slow10 calls: 10% of pages are slow100 calls: 63% of pages are slowWhen a page waits for many services, the rare slow request becomes the common case.A page that calls 100 services and waits for all of them is slowed by at least one p99 response 63% of the time.This is the “tail at scale”: the more parts a request fans out to, the more its speed is set by the slowest.Fixes: fewer calls on the critical path, timeouts, and sending a second copy of a slow request (hedging).

A page that waits on 100 services is slowed by a p99 reply 63% of the time.

Trade-offs

Every choice gives something up; write down what and why

Decision record 7: how the feed is builtstatus: acceptedContext150 reads for every upload; the feed must load in 300 ms at p99; people follow about 200 others.OptionsA. Build on readquery everyone followed when the app opens✓simple; uploads stay cheap✗200 lookups and a merge on every openB. Build on write (fan-out)on upload, add it to every follower’s feed list✓opening the feed is one key lookup✗400 million feed inserts a day (4,630 a second)DecisionB, except for accounts with over a million followers, whose posts are merged in at read time.Consequenceswrites need a queue and workers (lesson 7); feed lists use a key-value store; an unfollow must clean up.Every choice gives something up. Write down what, and why, while the reasons are fresh.A decision record is one short page per significant choice: the context, the options considered, the decisionand its consequences. It lets a reviewer check the reasoning, and lets the team change the decision later when thecontext changes, instead of guessing why it was made. The numbers from the estimates are what make it defensible.

A decision record keeps the context, the options, the decision and its consequences for each significant choice.

How the exam asks · Going deeper

An exam question is a requirement plus one deciding phrase

The phrase in the questionWhat it asks you to optimise“most cost-effective”the cheapest design that still meets every stated requirement“least operational overhead”managed or serverless services: nothing to patch, scale or replace yourself“highly available”survives the loss of one zone: at least two zones, no single point of failure“fault tolerant”no interruption at all when a part fails: full redundancy, already running“decouple the components”a queue or an event bus between the parts (lesson 7)“without changing the code”fixes outside the application: a proxy, a cache, a balancer, a bigger instance“lowest latency for global users”content at the edge: a CDN, or a global accelerator for non-HTTP trafficArchitect exams are a requirement plus one deciding constraint. Find the phrase, and it picks the answer.Usually two or three options would work. The qualifier at the end of the question, “most cost-effective”, “leastoperational overhead”, decides between them, exactly as a constraint does in a real design. Read the last line first,then rule out each option that breaks a stated requirement, then choose by the qualifier.

Find the qualifier, such as least operational overhead or most cost-effective, and it chooses between workable answers.

Lesson complete

You finished “Requirements and estimates”

Nicely done.

Open in a new tab