Monoliths and microservices

How far to split a system is a trade between coordination and isolation

Monolith✓calls cost nanoseconds✗teams block each otherStack OverflowModular monolith✓one deploy, clear seams✗still one release trainShopifyMicroservices✓teams ship independently✗every call crosses a networkworth it past 50 – 100 engineersCellsa bad release: 25% of users✓a failure hits one shard✗the most to build and runAWS, Slacksimpler to runmore independent, failures containedSplitting a system trades easy coordination for independent teams and contained failures.The trap is the distributed monolith: services that must deploy together and call each other in a chain. Ten calls at 50 ms eachis 500 ms before any retry: the network tax is paid, and the coupling is still there.

From monolith to microservices to cells: each step buys team independence and contained failure, and costs a network.

Monoliths and microservices · Going deeper · 1 of 2

The architecture spectrum: monolith to modular monolith to microservices to cells Monolith Modular monolith Microservices Cell-based app one process orders billing users notify A B C D E F cell 1 cell 2 cell 3 cell 4 A·B·C A·B·C A·B·C A·B·C 1 process · 1 deploy 1 process · modules N processes · N deploys N services × M cells team-coordination cost → 1 team 3–5 teams 10+ teams platform org pick the smallest topology that solves your team-coordination problem

The architecture spectrum

Conway’s law made literal: the shape of the system mirrors the shape of the teams. Microservices also require service discovery, tracing, versioned APIs and a platform team on call. Below about 50 – 100 engineers you pay that tax without earning the benefit.

Monoliths and microservices · Going deeper · 2 of 2

Cell-based deployment: shard users across isolated stacks within a region Edge router · shard by user-id hash consistent hash · routes user → cell deterministically cell α · 25% cell β · 25% cell γ · DOWN cell δ · 25% LB api db cache LB api db cache LB api db cache LB api db cache users 0–25% users 25–50% users 50–75% users 75–100% cell γ failing affects only its 25% · the other three never noticed

Cells: a full stack per shard

Each cell holds a load balancer, the services and the datastores for one shard of users, chosen at the edge by a hash of the user id. Re-sharding is mechanical: start a new cell, drain users into it, retire the old one. Slack adopted cells after a 2021 outage.

REST, gRPC and GraphQL

REST, gRPC and GraphQL answer the same request in three shapes

The app needs one user’s name and the titles of their 3 latest posts.REST: resources at URLsGET /users/123GET /users/123/posts?limit=32 trips, every field of each✓cacheable, every tool speaks it✗extra trips and extra fields→public APIsgRPC: a typed function callGetUserSummary(  user_id: 123,  posts: 3)✓checked schema, small binary✗no browsers, cannot curl it→between your own servicesGraphQL: the client picks fields{ user(id: 123) {    name    posts(last: 3) { title }} }✓exactly what was asked for✗ordinary HTTP caching breaks→mobile and web front endsThree sweet spots, not three rivals. Many systems use all of them at once.Whatever you choose, the contract outlives the server: an app on someone’s phone may keep calling a three-year-old endpoint.So only ever add fields. Never remove one or change what it means.

An API is the contract between programs: REST exposes resources, gRPC exposes functions, GraphQL lets the caller choose fields.

REST, gRPC and GraphQL · Going deeper · 1 of 3

REST, gRPC, and GraphQL: three answers to the same question REST gRPC GraphQL resources + verbs function calls + streams client picks fields GET /users/123 → {id, name, ...} GET /users/123/ posts → [{id, title}] POST /users PUT /users/123 DELETE /users/123 service Users { rpc Get(Req) returns (User); rpc Stream(Req) returns (stream Event); } // proto3 wire query { user(id: 123) { name posts(last: 3) { title } } } + HTTP-cache friendly + universal toolchain + verbs as semantics − N+1 round trips − over-fetching − no built-in streams + binary, much smaller + schema-typed stubs + HTTP/2 streams − not browser-native − opaque to curl − proxy-unfriendly + exact-shape responses + one endpoint + schema + introspection − HTTP caching breaks − resolver fan-out − per-field auth public APIs · CRUD internal microservices mobile + aggregating UIs pick by caller · real platforms layer all three

Three styles compared

REST’s cost is follow-up calls: a user, then their posts, then each post’s comments. gRPC runs on HTTP/2 and supports streaming in both directions. GraphQL moves the many-calls problem to the server, where one query can fan out to dozens of backend calls, and authorisation becomes field by field.

REST, gRPC and GraphQL · Going deeper · 2 of 3

Three API versioning strategies and their coupling trade-offs URL versioning Header versioning Field evolution explicit · breaking content-negotiated additive only GET /v1/users/42 → {id, name} GET /v2/users/42 → {id, name, email} // v1 frozen // route by prefix // two codepaths GET /users/42 Accept: application/ vnd.ex.v1+json GET /users/42 Accept: application/ vnd.ex.v2+json // one URL · two shapes GET /users/42 → {id, name} // later, additive: → {id, name, email} // never remove // never retype // deprecate slow + obvious in logs + caches cleanly − forks the spec + same URL forever + negotiation built-in − proxies miss it + no version bumps + old clients keep working − discipline required URL is loud; header is clean but invisible; evolution is cheapest if discipline holds

Versioning without breaking callers

REST versions by a URL prefix such as /v1/ or by a header. gRPC relies on Protocol Buffers’ rules: never reuse a field number, never change a field’s type. GraphQL adds and deprecates fields with no version at all. All three reduce to: add, never break.

REST, gRPC and GraphQL · Going deeper · 3 of 3

OAuth 2.0 authorization code flow with PKCE Browser App Auth server Resource API 1. user clicks "log in" 2. app: verifier=rand · challenge=SHA256(verifier) 3. redirect to auth · code_challenge · redirect_uri 4. auth screen · user enters credentials 5. redirect back · auth_code 6. POST /token · code · code_verifier 7. auth checks SHA256(verifier) = challenge 8. access_token · short-lived bearer 9. resource returns user data PKCE binds the redeemed code to the original initiator

Who is calling: OAuth with PKCE

The user signs in at the identity provider, so the app never sees the password. The app receives a one-time code and swaps it for a short-lived token. PKCE binds that code to the app that started: the app sends only a hash of a secret at first, and must present the secret itself to redeem the code, so a stolen code is useless.

Circuit breakers and rate limits

Guard every call: stop calling what is sick, ration what is greedy

A sick dependency: stop calling itClosedcalls flowhalf of the last 20 calls failedOpenfail instantlyafter a cool-downHalf-openlet one call tryit worksif it fails, open againA greedy caller: ration itrefills 5 tokens a secondholds 10each request takes a tokena burst of 10 goes throughbucket empty:429 Too Many RequestsEvery service needs a guard on the way out and a guard on the way in.A circuit breaker turns slow failure into instant failure: 1,000 instant errors are harmless, 1,000 thirty-second timeoutsuse up your threads and take you down too. A token bucket allows honest bursts and caps the sustained rate.

A circuit breaker fails fast when a dependency is down; a rate limiter keeps one caller from taking everything.

Circuit breakers and rate limits · Going deeper · 1 of 2

Circuit breaker state machine: closed, open, half-open CLOSED calls flow count failures HALF-OPEN one probe call only OPEN fail fast no calls go out failure rate over threshold cooldown elapsed (probe) probe ok probe fails — re-open trip · 50% failures over last 20 calls (configurable) cooldown · 30 s typical · prevents stampede of retries probe · one in-flight call gates the recovery decision

The circuit breaker’s three states

Pair the breaker with a fallback, such as a cached value, a default or a structured error, so the caller still gets something useful. Without a breaker, retries against a failing service fill your own thread pool with waiting calls and you go down with it.

Circuit breakers and rate limits · Going deeper · 2 of 2

Token bucket: capacity B, refill rate R, request consumes one token Refill source Bucket (capacity B) Requests drip · R tok/s e.g. R=10/s capacity B · burst limit accept 429 reject Retry-After −1 token if empty behavior burst of B requests · accepted instantly · bucket drains sustained rate ≤ R · steady-state accepted sustained rate over R · excess rejected with 429 + Retry-After per-key state · one bucket per IP / API key / user

The token bucket

Capacity sets how large an honest burst may be; the refill rate bounds sustained load. State is kept per key, by IP address, API key or user, usually in Redis so every gateway instance shares one count. Over the limit, reply 429 with a hint for when to retry.

Check yourself

25 requests at oncerefills 5 a second

A full bucket holds 10 tokens and refills at 5 a second. A client sends 25 requests in the same instant. How many succeed?

Event-driven design

Publish events instead of calling: downstream failures stay downstream

Call and waitOrderInventoryPaymentEmailEmail is down, so the order fails.And every hop adds its own delay.Publish and subscribeOrdertopic: orders#40#41#42kept for days, in orderInventoryreserves stockPaymentcharges the cardEmaildown: its events wait,and it catches up laterAnalyticsadded later; replays the logPublish what happened and let whoever cares react: the producer no longer needs its consumers to be up.The price is eventual consistency: readers lag the log by milliseconds to seconds, so “I just ordered, where is it?” must be designed for.

With a message broker, producers publish what happened and consumers react when they can. The cost is eventual consistency.

Event-driven design · Going deeper · 1 of 2

Pub/sub topology and an event-sourced order timeline Producer Topic Consumer groups order-svc orders.events billing-svc email-svc analytics-svc OrderCreated · 42 ItemAdded · 41 Paid · 40 append-only log · partitioned retention 7 d · replayable Event-sourced timeline · order #42 OrderCreated ItemAdded Paid Shipped 10:00:01 10:00:08 10:01:14 10:42:00 derived state status: shipped items: 1 total: $42.00 events are the source of truth · current state is a fold over the timeline

Topics, and events as the record

Event sourcing stores the events themselves as the system of record. An order’s state is what you get by replaying created, item added, paid, shipped. Auditing and time travel come free. The cost: history cannot be edited, so a bug is fixed with a correcting event, never a delete.

Event-driven design · Going deeper · 2 of 2

CQRS: write side and read side share the event log but optimize for different shapes Command PlaceOrder Validator invariants Event store append-only Write store aggregate roots Query GetOrders projector A projector B projector C read store · order list read store · stats read store · search UI user dashboard write side · ↑ read side · ↓ one source of truth, many derived shapes

Separate models for writing and reading (CQRS)

Writes go through one path and produce events. Reads are served from views built by consuming those events, one view per kind of question. The sides scale independently, and a lost view is rebuilt from the log. The views lag by milliseconds to seconds.

Sagas

Across services nothing can be rolled back, only corrected

OrderInventoryPaymentorder #42: pendingOrderCreatedreserves 1 widgetInventoryReservedcard declinedPaymentFailedreleases the widgetorder #42: cancelledforward: one local transaction, then an eventcompensate, newest step firstNo undo buttonThe widget really wasunavailable for 800 ms.A compensation correctsforward. It does noterase what happened:a refund is not anun-charge.Across services there is no shared transaction: only a chain of local ones, each announcing what it did.Trap: “save the order, then publish the event” is two writes. Crash in between and the order waits forever.Fix (the outbox): save the row and its event in one local transaction; a relay publishes it afterwards. It may publish twice,so every consumer ignores repeats.

A saga is a chain of local transactions joined by events, with a compensating step for each one that may need undoing.

Sagas · Going deeper · 1 of 2

A saga: forward path with compensating events on failure Order Inventory Payment Shipping Done create order reserve items charge card · FAILS schedule ship commit OrderCreated ItemsReserved compensation path · roll backward cancel order release items payment declined PaymentFailed InventoryReleased choreography each service subscribes to events · no central brain · fewest moving parts orchestration a coordinator owns the workflow · easier to observe, more code to write

Forward path and compensations

In choreography each service reacts to events with no central brain: fewest parts, hardest to debug. In orchestration a coordinator such as Temporal or Step Functions drives the sequence and the retries: easier to observe, more code. Either way every consumer must ignore duplicates, because brokers redeliver.

Sagas · Going deeper · 2 of 2

The outbox pattern: business row and event row in one transaction, relay publishes after commit Service Database (1 transaction) Broker order-svc handle PlaceOrder orders table id=42 status=new total=42.00 outbox table id=99 type=OrderCreated payload=... sent=false INSERT INSERT COMMIT — both rows visible together relay poll · or CDC tail orders.events topic read unsent publish mark sent why this works one local transaction · no distributed commit crash before publish · relay retries from the outbox crash after publish · duplicate event · consumer dedups by id

The outbox pattern

Write the business row and an event row in one local transaction. A relay reads the outbox table, or tails the database’s change log, publishes each event and marks it sent. A crash before publishing loses nothing; a crash after publishing sends it twice. Delivery is at least once, which consumers handle with idempotency keys.

Caching

A cache answers in half a millisecond what the origin needs 300 for

0 ms100 ms200 ms300 msBrowser cachethe copy already on the deviceno request at allCDN edgea copy in a city near the user20 msApplication cachehot results in memory (Redis)0.5 msDatabase querythe real lookup50 msBuild it from scratchrun the code at the origin300 msa request tries each in turn, top to bottomKeep the answer near the asker: each layer catches what the one above missed, so most requests never reach the origin.A 95% hit rate on a 0.5 ms cache in front of a 300 ms origin averages about 15 ms: twenty times faster.The hard part is knowing when a stored answer has gone stale: let it expire after a set time, delete it on every change,or put a version in the key.

A cache keeps a computed answer near the asker. Layers of them mean most requests never reach the origin.

Caching · Going deeper

The cache hierarchy of a typical web request Browser CDN edge App cache DB cache DB HTTP cache Cloudflare etc. Redis · in-proc buffer pool origin ≈0 ms ≈20 ms ≈0.5 ms ≈5 ms ≈50 ms hit ratio hit ratio hit ratio hit ratio ≈30% ≈95% ≈80% ≈99% TTL hours TTL minutes TTL seconds page LRU write patterns cache-aside · app reads cache; on miss reads DB and writes cache (default) write-through · writes go to cache and DB together · consistent, slow write-behind · writes hit cache, async flush to DB · fast, can lose on crash

The cache hierarchy, and who writes to it

Cache-aside is the default: read the cache, on a miss query the database and store the result. Write-through updates cache and database together: consistent, slower writes. Write-behind updates the cache and flushes later: fast, and loses unflushed writes in a crash. Browser and CDN caching are governed by HTTP headers such as Cache-Control and ETag.

Check yourself

entry expires800 ms laterone rebuild takes this longrequests keep arriving, 1,000 a second

A popular cache entry expires. Rebuilding it takes 800 ms, and 1,000 requests arrive every second. With no protection, about how many rebuilds start?

Cloud service models

The cloud rents compute, storage and network on a sliding scale

you manage less, and control less →ComputeVirtual machinestarts in 30 – 90 secondsyou patch the operating systemContainerstarts in secondsyou ship a packaged imageFunctionstarts in millisecondsyou ship one function, pay per callStorageBlocka disk for one machinedatabasesObjectblobs by key, at any sizebackups, media, data lakes (S3)Filea folder many machines shareshared directoriesNetworkPrivate networkyour own address spaceLoad balancerspreads traffic over serversCDNcopies close to usersThe surprisekeep 1 GB for a month$0.02send 1 GB out to the internet$0.09The cloud rents three things, compute, storage and network, each on a scale from “you run it” to “they run it”.Moving data out costs several times more than keeping it, so on a busy service the transfer bill dwarfs the storage bill.The architecture review and the cost review are the same meeting.

Virtual machines, containers and functions trade control for convenience. And data leaving the cloud is the bill nobody expects.

Cloud service models · Going deeper · 1 of 3

The three cloud axes: compute, storage, network Compute Storage Network VM Container Function Block Object File VPC Load balancer NAT CDN whole OS · 30–90 s boot k8s pod · 1–10 s boot stateless · per-invocation one-VM disk · any filesystem flat key-value · multi-AZ NFS · multi-attach private IP space L4 / L7 fan-out private → public egress edge cache · many PoPs → less control, more managed → pick by access pattern → same primitives every cloud

The three axes

On-demand capacity is paid by the hour. Reserved capacity is far cheaper for a one or three year commitment. Spot capacity is deeply discounted but can be reclaimed with two minutes’ notice, which suits batch jobs and stateless web tiers. A query returning a kilobyte per row is free in development and tens of thousands of dollars a month at 100,000 requests a second across regions.

Cloud service models · Going deeper · 2 of 3

Multi-tier load balancing: DNS, anycast, L7, L4, instance DNS Anycast L7 LB L4 LB Instance geo / latency · region BGP · nearest PoP path · header · cell TCP · round-robin one pod · one PID picks picks picks picks instance why this works DNS TTL too low: lookup load · too high: slow failover anycast pulls users to a sick PoP unless health checks withdraw the route L7 is HTTP-aware (rewrites, auth, WAF) · L4 is dumb and fast consistent-hash at L4 keeps a user on the same backend for cache locality

Load balancing in layers

At global scale five layers each make one decision: DNS picks a region, anycast routing reaches the nearest entry point, a layer-7 balancer reads HTTP and picks a service or cell, a layer-4 balancer spreads connections, and a final rule picks one process. Each layer fails over on its own.

Cloud service models · Going deeper · 3 of 3

Service mesh: sidecar proxy per pod, control plane configures all sidecars Control plane routing config · cert authority pod A pod B pod C app sidecar app sidecar app sidecar localhost only localhost only localhost only push config mTLS mTLS what the sidecar gives you for free mTLS · retries · timeouts · circuit-break · traffic-split distributed traces · metrics · access logs · no app-code changes cost · 2× pod count · control plane to operate

Service mesh

A small proxy beside every service instance handles encryption between services, retries, timeouts and telemetry, so application code does none of it. A control plane configures all the proxies, which is how a 1% canary becomes one configuration change. The cost: twice as many processes and a control plane to run.

Infrastructure as code

Infrastructure as code: declare what you want, and a loop makes it so

Desired: files in Gitweb_servers = 3region = "eu-west-1"Actual: what is running2 serversPlan: the difference+ 1 web server2 exist, 3 are wantedapply: the fewest changes that close the gapReviewableevery change is a pull request,with an author and a historyRepeatablea lost region is rebuilt by onecommand, not 200 clicksSelf-correctingrun it twice and nothing changes;manual edits show up as driftDescribe what you want, and let a loop compare it with what exists and close the gap.Terraform runs this loop for cloud resources; Kubernetes runs the same loop for programs. The record that links code to realresources (the state file) is precious: lose it and the tool believes the cloud is empty.

Infrastructure as code keeps the desired setup in files; a tool compares that with reality and applies the difference.

Infrastructure as code · Going deeper

The reconciliation loop at the heart of Terraform Desired state Planner Diff Apply State file Cloud provider main.tf · HCL declarative parse + graph topo sort + create / − destroy ≈ update / replace API calls in order last-known actual remote backend actual state refresh drift detection apply desired ↔ actual · planner emits the minimal call sequence to converge them

The reconciliation loop

Terraform and its open fork OpenTofu keep a state file mapping declared resources to real ones. It must live in shared storage with locking and backups, and never be edited by hand: lose it, and the next run tries to recreate everything on top of what exists. Scheduled plans catch drift from manual changes.

Progressive delivery

Release to a few before releasing to everyone

new versionold versionRollinga few machines at a timesimple and slowfine for services that hold no stateBlue / greena second fleet, then a switcholdnew ← all trafficundo is one switch backcosts a second fleet while it runsCanarya small share of real users1% → 10% → 50% → 100%, watching errorslimits who can be hurthalts by itself if errors riseShadowa copy of the trafficanswers usersreplies discardedreal load, nobody exposedtests speed and crashes, safelyA release is the riskiest routine thing a team does, so choose by how much can break and how fast you can undo it.Most platforms combine them: a canary inside a blue / green switch, gated by automatic health checks.If you cannot state the rollback in one sentence, pick again.

Rolling, blue/green, canary and shadow releases differ in how many users a bad version can reach and how fast you can undo it.

Progressive delivery · Going deeper

Four deployment strategies: rolling, blue/green, canary, shadow Rolling Blue / green Canary Shadow t0 t1 t2 t3 replace n at a time · slow · simple blue 100% green 0% flip two fleets · 2× cost · instant rollback 1% 5% 25% 50% 100% step + watch SLOs · halt on regression live shadow discard copy real load · zero user exposure picking one stateless web → rolling DB schema → blue/green user-facing → canary rewrites → shadow first SLO-gated automated rollback on error budget burn

Four strategies

The strategy is a function of blast radius and confidence. Canary steps are gated on the same indicators that define the service’s reliability target, so a release that harms users halts before most of them are exposed.

Observability and error budgets

Three signals show what is wrong; an error budget decides what to do

Metricsnumbers over timeIs it healthy right now?Tracesone request’s pathgateway 12 msauth 480 msdatabase 30 msWhere did the time go?Logsnotes from each step{"req":"a41f","err":"key expired"}What exactlyhappened?From signals to a decisionThe promise: 99.9% of requestssucceed, measured over 30 days.Allowed failure: 43 minutesoutage: 25 min18 minBudget spent: no risky releasesuntil the window moves on.A target nobody enforces is decoration.A metric says something is wrong, a trace says where, and a log says why.An error budget turns those signals into a rule agreed in advance: spend it on releases and experiments, and when it is gone,stop shipping features and fix reliability.

Observability: metrics, traces and logs. An SLO turns them into an error budget the team agreed to in advance.

Observability and error budgets · Going deeper · 1 of 2

The three pillars of observability: metrics, logs, traces Metrics Logs Traces numbers over time timestamped events request journeys requests_total {svc="api"} 10:00:01 INFO user 42 login 10:00:02 WARN cache miss 10:00:03 ERROR db timeout 10:00:04 INFO retry success api auth db notify cache trace-id: 7f3a... span-id: a12 small · aggregable "is it healthy?" large · cardinal "what happened?" linked spans "where did it go?" scrape every 15 s indexed · sampled head-sampled at entry use metrics to alert, logs to inspect, traces to navigate

Metrics, logs, traces

Metrics are counters, gauges and histograms, cheap to keep and aggregate. Logs are expensive at volume: keep errors, sample 1% of successes, expire after 30 days. Traces link spans by an id carried in request headers (W3C Trace Context); at 100,000 requests a second only a sample is traced.

Observability and error budgets · Going deeper · 2 of 2

SLO and error budget over a 30-day window 100% 99.9% 99% day 0 day 30 SLO incident incident budget exhausted freeze releases SLI · success rate budget = 100% − SLO · burn-rate alerts fire when the 30-day spend pace puts you under target

SLO and error budget

An indicator is a measurable property such as success rate or p99 latency. An objective is a target on it over a rolling window. The budget is the complement: 0.1% of 30 days is 43 minutes. The hard part is human: if leaders ship anyway when the budget is gone, the objective is fiction.

Check yourself

error budget for the window: 43 minutesoutage: 25 minoutage: 18 min

A service promises 99.9% success over 30 days. This month it has had the two outages shown. What should happen to Friday’s risky release?

Summary

One request, and every decision in this lesson along its path

YouCDN + cachesanswer most requestsLoad balancerpicks a healthy cellServiceREST, gRPC, GraphQLBrokerevents, replayableOther servicesreact when they canGuards on every calltimeouts · retries with backoff · circuit breakers · rate limits · sagas for multi-step workCloud underneathcompute, storage and network, described as code and continuously reconciledObservability acrossmetrics, traces and logs, and an error budget that decides when to stop shippingEvery choice here is a hedge against partial failure, and each choice constrains the others.Choosing many small services, for instance, forces decisions about APIs, events, deployment and monitoring.

Architecture, APIs, events, cloud and observability are one design, not five. Next: data, the state all of this depends on.

Lesson complete

You finished “Systems Design and Cloud”

Answer the questions along the way to see your score here.

Open in a new tab