How far to split a system is a trade between coordination and isolation
From monolith to microservices to cells: each step buys team independence and contained failure, and costs a network.
Architecture begins with one question: how many separately deployed programs does the system run? A monolith is one. Every feature lives in one process with one database. Calls between features take nanoseconds. Its limit is human: with enough people on one codebase, teams block each other. A modular monolith keeps the single deploy but builds walls inside, so the seams are ready if you ever need to cut. Microservices cut along those seams. Each team ships on its own, and a bug stays in one service. But every call that was free now crosses a network. That is only worth it past roughly fifty to a hundred engineers. Cells go one step further: several complete copies of everything, each serving a slice of users. A bad release then hurts one slice, not everyone.
Monoliths and microservices · Going deeper · 1 of 2
The architecture spectrum
Conway’s law made literal: the shape of the system mirrors the shape of the teams. Microservices also require service discovery, tracing, versioned APIs and a platform team on call. Below about 50 – 100 engineers you pay that tax without earning the benefit.
Monoliths and microservices · Going deeper · 2 of 2
Cells: a full stack per shard
Each cell holds a load balancer, the services and the datastores for one shard of users, chosen at the edge by a hash of the user id. Re-sharding is mechanical: start a new cell, drain users into it, retire the old one. Slack adopted cells after a 2021 outage.
REST, gRPC and GraphQL
REST, gRPC and GraphQL answer the same request in three shapes
An API is the contract between programs: REST exposes resources, gRPC exposes functions, GraphQL lets the caller choose fields.
Once there are two services, they need an agreed way to talk. Take one need, a user’s name and their three latest post titles, and ask for it three ways. REST treats everything as a resource at a web address. It takes two requests here, and each returns every field whether you wanted it or not. But every tool on the internet already speaks it, and responses cache naturally. That is why public APIs use it. gRPC makes it a function call with a checked schema and a compact binary format. It is fast and strict, which suits calls between your own services, but a browser cannot use it directly. GraphQL lets the client write a query naming exactly the fields it wants, in one trip. That suits mobile apps on weak connections, though ordinary web caching no longer works.
REST, gRPC and GraphQL · Going deeper · 1 of 3
Three styles compared
REST’s cost is follow-up calls: a user, then their posts, then each post’s comments. gRPC runs on HTTP/2 and supports streaming in both directions. GraphQL moves the many-calls problem to the server, where one query can fan out to dozens of backend calls, and authorisation becomes field by field.
REST, gRPC and GraphQL · Going deeper · 2 of 3
Versioning without breaking callers
REST versions by a URL prefix such as /v1/ or by a header. gRPC relies on Protocol Buffers’ rules: never reuse a field number, never change a field’s type. GraphQL adds and deprecates fields with no version at all. All three reduce to: add, never break.
REST, gRPC and GraphQL · Going deeper · 3 of 3
Who is calling: OAuth with PKCE
The user signs in at the identity provider, so the app never sees the password. The app receives a one-time code and swaps it for a short-lived token. PKCE binds that code to the app that started: the app sends only a hash of a secret at first, and must present the secret itself to redeem the code, so a stolen code is useless.
Circuit breakers and rate limits
Guard every call: stop calling what is sick, ration what is greedy
A circuit breaker fails fast when a dependency is down; a rate limiter keeps one caller from taking everything.
Two small mechanisms keep one problem from becoming everyone’s problem. First, on the way out. A circuit breaker wraps each call to a dependency. Normally it is closed, and calls flow. If half of the last twenty calls fail, it opens. Now every call fails instantly, without even trying. That sounds worse, but it is far better than a thousand callers each waiting thirty seconds on a service that is not going to answer. After a cool-down it lets a single call through as a probe. If that works, it closes again. Second, on the way in. A token bucket holds ten tokens and refills five every second. Each request takes one. A short burst goes through. But a caller that keeps going empties the bucket and gets told: too many requests.
Circuit breakers and rate limits · Going deeper · 1 of 2
The circuit breaker’s three states
Pair the breaker with a fallback, such as a cached value, a default or a structured error, so the caller still gets something useful. Without a breaker, retries against a failing service fill your own thread pool with waiting calls and you go down with it.
Circuit breakers and rate limits · Going deeper · 2 of 2
The token bucket
Capacity sets how large an honest burst may be; the refill rate bounds sustained load. State is kept per key, by IP address, API key or user, usually in Redis so every gateway instance shares one count. Over the limit, reply 429 with a hint for when to retry.
Check yourself
A full bucket holds 10 tokens and refills at 5 a second. A client sends 25 requests in the same instant. How many succeed?
A burst can only spend what the bucket holds. The other 15 get “429 Too Many Requests” and may retry as tokens refill.
Event-driven design
Publish events instead of calling: downstream failures stay downstream
With a message broker, producers publish what happened and consumers react when they can. The cost is eventual consistency.
There are two ways for services to work together. The first is to call and wait. Order calls inventory, which calls payment, which calls email. If email is down, the whole order fails, for the sake of a message. The second is to publish. The order service writes one fact, order forty two was created, to a topic in a broker, and its job is done. The broker keeps events in order, for days. Inventory and payment each read from the topic at their own pace. Email is down. Nothing breaks. Its events wait in the topic, and it catches up when it returns. And months later a new analytics service is added. It replays the topic from the beginning, and the order service never has to change.
Event-driven design · Going deeper · 1 of 2
Topics, and events as the record
Event sourcing stores the events themselves as the system of record. An order’s state is what you get by replaying created, item added, paid, shipped. Auditing and time travel come free. The cost: history cannot be edited, so a bug is fixed with a correcting event, never a delete.
Event-driven design · Going deeper · 2 of 2
Separate models for writing and reading (CQRS)
Writes go through one path and produce events. Reads are served from views built by consuming those events, one view per kind of question. The sides scale independently, and a lost view is rebuilt from the log. The views lag by milliseconds to seconds.
Sagas
Across services nothing can be rolled back, only corrected
A saga is a chain of local transactions joined by events, with a compensating step for each one that may need undoing.
Placing an order touches three services, each with its own database. No single transaction can cover them. So the work becomes a chain. The order service saves order forty two as pending, and announces it. Inventory hears that, reserves one widget, and announces that. Payment hears that and charges the card. The card is declined. Now the chain runs in reverse. Payment announces the failure. Inventory hears it and releases the widget. The order service hears it and marks the order cancelled. Nothing was rolled back. For most of a second that widget really was unavailable to everyone else. Each step was corrected by a new step.
Sagas · Going deeper · 1 of 2
Forward path and compensations
In choreography each service reacts to events with no central brain: fewest parts, hardest to debug. In orchestration a coordinator such as Temporal or Step Functions drives the sequence and the retries: easier to observe, more code. Either way every consumer must ignore duplicates, because brokers redeliver.
Sagas · Going deeper · 2 of 2
The outbox pattern
Write the business row and an event row in one local transaction. A relay reads the outbox table, or tails the database’s change log, publishes each event and marks it sent. A crash before publishing loses nothing; a crash after publishing sends it twice. Delivery is at least once, which consumers handle with idempotency keys.
Caching
A cache answers in half a millisecond what the origin needs 300 for
A cache keeps a computed answer near the asker. Layers of them mean most requests never reach the origin.
Computing the same answer again and again is wasted work, so systems keep answers close to whoever asks. A request tries each place in turn. First the browser’s own copy. If it is still fresh, no request is made at all. Next a content delivery network: a copy in a city near the user, about twenty milliseconds away. Then the application’s own cache, holding hot results in memory. Half a millisecond. Then the database: a real query, about fifty milliseconds. And if nothing has the answer, the code builds it from scratch: three hundred. Look at the bars. The whole game is to be answered as high up this list as possible.
Caching · Going deeper
The cache hierarchy, and who writes to it
Cache-aside is the default: read the cache, on a miss query the database and store the result. Write-through updates cache and database together: consistent, slower writes. Write-behind updates the cache and flushes later: fast, and loses unflushed writes in a crash. Browser and CDN caching are governed by HTTP headers such as Cache-Control and ETag.
Check yourself
A popular cache entry expires. Rebuilding it takes 800 ms, and 1,000 requests arrive every second. With no protection, about how many rebuilds start?
Every request that misses starts its own rebuild until the first finishes: a stampede. Let one request rebuild while the others wait, and 800 becomes 1.
Cloud service models
The cloud rents compute, storage and network on a sliding scale
Virtual machines, containers and functions trade control for convenience. And data leaving the cloud is the bill nobody expects.
A cloud provider rents out a data centre as a set of services you call from code. Three kinds cover it. Compute is where your code runs, and it is a sliding scale. A virtual machine is a whole computer: you control everything and you patch everything. A container is a packaged program that starts in seconds. A function is just your code, started in milliseconds when something happens, and billed per call. Storage has three shapes: a disk attached to one machine, for databases; object storage, which keeps any number of blobs by key; and a shared folder. Network joins it up: a private network, load balancers, and delivery networks near users. And here is the surprise. Sending a gigabyte out to the internet costs several times more than keeping it for a month.
Cloud service models · Going deeper · 1 of 3
The three axes
On-demand capacity is paid by the hour. Reserved capacity is far cheaper for a one or three year commitment. Spot capacity is deeply discounted but can be reclaimed with two minutes’ notice, which suits batch jobs and stateless web tiers. A query returning a kilobyte per row is free in development and tens of thousands of dollars a month at 100,000 requests a second across regions.
Cloud service models · Going deeper · 2 of 3
Load balancing in layers
At global scale five layers each make one decision: DNS picks a region, anycast routing reaches the nearest entry point, a layer-7 balancer reads HTTP and picks a service or cell, a layer-4 balancer spreads connections, and a final rule picks one process. Each layer fails over on its own.
Cloud service models · Going deeper · 3 of 3
Service mesh
A small proxy beside every service instance handles encryption between services, retries, timeouts and telemetry, so application code does none of it. A control plane configures all the proxies, which is how a 1% canary becomes one configuration change. The cost: twice as many processes and a control plane to run.
Infrastructure as code
Infrastructure as code: declare what you want, and a loop makes it so
Infrastructure as code keeps the desired setup in files; a tool compares that with reality and applies the difference.
Clicking through a cloud console works for a demo. It fails for a real system, because nobody can review it, repeat it, or tell what changed. The alternative is a loop. You write down what you want in files, kept in Git: three web servers, in this region. The tool looks at what is actually running. Two servers. It works out the difference, and shows it to you as a plan: add one web server. You approve, and it makes the fewest changes needed to close the gap. Three things follow. Every change is reviewed like code. A lost region can be rebuilt with one command. And running it again changes nothing, while anything changed by hand shows up as a difference.
Infrastructure as code · Going deeper
The reconciliation loop
Terraform and its open fork OpenTofu keep a state file mapping declared resources to real ones. It must live in shared storage with locking and backups, and never be edited by hand: lose it, and the next run tries to recreate everything on top of what exists. Scheduled plans catch drift from manual changes.
Progressive delivery
Release to a few before releasing to everyone
Rolling, blue/green, canary and shadow releases differ in how many users a bad version can reach and how fast you can undo it.
Getting new code to users is its own design choice, and it comes down to risk. A rolling release replaces a few machines at a time. Simple, but slow to finish and slow to undo. Blue green builds a complete second fleet with the new version, tests it, then switches all traffic over at once. If it goes wrong, switch back. You pay for two fleets while it lasts. A canary sends one percent of real users to the new version, and watches errors and speed. Then ten percent, then half. If the numbers turn bad it stops on its own. A shadow release copies real traffic to the new version and throws its answers away. Real load, and no user ever sees it.
Progressive delivery · Going deeper
Four strategies
The strategy is a function of blast radius and confidence. Canary steps are gated on the same indicators that define the service’s reliability target, so a release that harms users halts before most of them are exposed.
Observability and error budgets
Three signals show what is wrong; an error budget decides what to do
Observability: metrics, traces and logs. An SLO turns them into an error budget the team agreed to in advance.
You cannot fix what you cannot see, so systems report on themselves in three ways. Metrics are numbers over time, like error rate. They are cheap, and they tell you whether the system is healthy right now. Here, something spiked. A trace follows one request through every service it touched. This one spent four hundred and eighty of its milliseconds in the auth service. Now you know where. Logs are the notes each step wrote down. This one says a key expired. Now you know why. But when is a problem bad enough to act on? Decide in advance. Promise that ninety nine point nine percent of requests succeed over thirty days. That leaves forty three minutes of failure. Two outages have used all of it, so risky releases stop until reliability is repaired.
Observability and error budgets · Going deeper · 1 of 2
Metrics, logs, traces
Metrics are counters, gauges and histograms, cheap to keep and aggregate. Logs are expensive at volume: keep errors, sample 1% of successes, expire after 30 days. Traces link spans by an id carried in request headers (W3C Trace Context); at 100,000 requests a second only a sample is traced.
Observability and error budgets · Going deeper · 2 of 2
SLO and error budget
An indicator is a measurable property such as success rate or p99 latency. An objective is a target on it over a rolling window. The budget is the complement: 0.1% of 30 days is 43 minutes. The hard part is human: if leaders ship anyway when the budget is gone, the objective is fiction.
Check yourself
A service promises 99.9% success over 30 days. This month it has had the two outages shown. What should happen to Friday’s risky release?
25 + 18 = 43: the budget is spent. The whole point of an error budget is that this decision was made in advance.
Summary
One request, and every decision in this lesson along its path
Architecture, APIs, events, cloud and observability are one design, not five. Next: data, the state all of this depends on.
Follow one request, and the whole lesson is on its path. It leaves you, and most of the time a cache answers it. Otherwise a load balancer picks a healthy cell and hands it to a service through an API. That service does its part and publishes events, which other services act on when they can. Around every one of those calls stand guards: timeouts, careful retries, circuit breakers and rate limits. Underneath is rented compute, storage and network, written down as code and kept in line by a loop. And across all of it, the signals that say whether it is working, with a budget that decides when to stop and repair. These parts have shared written standards too, from OpenAPI for describing interfaces to OpenTelemetry for the signals.
Lesson complete
You finished “Systems Design and Cloud”
Answer the questions along the way to see your score here.