Timeouts

Without a timeout, one slow dependency holds every thread

100 requests/s; each holds one of 50 worker threads while it waits for a dependency that has hungNo timeout: each waits 30 sthreads needed = 100 × 30 s = 3,000all 50 taken in 0.5 s: every request now failsTimeout 300 ms: give up and move onthreads needed = 100 × 0.3 s = 3020 spare: other requests still servedWithout a timeout, one slow dependency holds every thread, and the failure spreads to everything else.Threads, connections and memory in use = arrival rate × how long each waits. A hung dependency makes the wait huge, sothe pool fills in half a second and the healthy parts of the service fail too. Every network call needs a timeout, setjust above the dependency’s normal p99, so that a slow dependency becomes a fast, contained error.

Resources in use = rate × wait. A hung call with no timeout needs 3,000 threads; a timeout of 300 ms needs 30.

Bulkheads · Going deeper

Give each dependency its own share, so one cannot sink the rest

One shared pool40 threads, sharedthe slow reviews service takes all 40;checkout cannot get a threadBulkheads: a pool per dependencyreviewscheckoutsearchreviews fills its own 10 and fails alone;checkout and search carry onBulkheads give each dependency its own share, so one failure cannot sink the rest.Named after the watertight walls in a ship’s hull. Separate thread pools, connection pools or even separatedeployments per dependency or per customer group keep a fault in one compartment. Cells and shuffle sharding applythe same idea at scale: each customer is served by a small slice of the fleet, so one bad slice hurts few.

Bulkheads split pools per dependency or customer, keeping a failure in one compartment.

Retries

Retries multiply through layers into a storm

Browser1 callWeb tiereach call tries 4 times (1 + 3 retries)4 callsOrders serviceeach call tries 4 times (1 + 3 retries)16 callsDatabaseeach call tries 4 times (1 + 3 retries)64 callsRetries multiply through layers: 4 tries at each of 3 layers turns one click into 64 database calls.A retry helps with a brief glitch. But when a dependency is overloaded, every caller retrying makes it worse, and retriesat several layers multiply. Retry at one layer only (usually the one nearest the failure), cap the attempts, retry onlyerrors that may pass (timeouts, throttling, 5xx), only safe (idempotent) calls, and use a retry budget across the service.

4 tries at each of 3 layers turns one click into 64 database calls. Retry at one layer, with a cap.

Backoff and jitter · Going deeper

Back off exponentially, and add randomness

Backoff without jitter: 200 clients retry at 1, 2, 4 and 8 s, togetherpeaks of 200 at onceWith full jitter: each waits a random time up to the backoff0 s4 s8 s12 s16 sat most 88 at onceBack off exponentially and add jitter, so retries spread out instead of arriving as a new wave.Doubling the wait between attempts (1, 2, 4, 8 s, with a cap) gives a struggling dependency room to recover. Withoutrandomness, every client that failed together retries together, recreating the spike on each round. Full jitter (waita random time between zero and the backoff) spreads them evenly. AWS SDKs do this by default.

Exponential backoff spaces retries out; jitter stops 200 clients retrying at the same instant.

Circuit breakers

Stop calling a dependency that keeps failing, and fail fast

Closedcalls pass through50% of calls failing in 10 sOpenfail fast, no callsafter 30 sHalf-openlet a few trial calls intrials succeed: closetrials fail: open againA circuit breaker stops calling a dependency that keeps failing, fails fast, and tests it again later.Like the switch in a fuse box. While the breaker is open, callers get an immediate error or a fallback instead ofwaiting on timeouts, and the failing service gets a pause in which to recover. After a cool-down a few trial callsdecide whether to close it again. Libraries (Resilience4j, Polly) and service meshes provide it.

A circuit breaker opens after repeated failures, fails fast while open, and goes half-open to test with a few calls.

Graceful degradation

When a part fails, serve a smaller answer instead of an error

Product pagePhoto and pricecatalogue: upBuy buttoncart and checkout: upReviewsreviews: slow, so hiddenRecommended for you → Popular this weekpersonalisation down: a cached list insteadFor each part of the page, decide:✓is it essential? (price, buy button)✓if not, what is the fallback?· a cached copy· a default value· leave it out✗never let an optional part take the page downWhen a part fails, serve a smaller answer instead of an error.Users can still buy with no reviews and generic recommendations; they cannot buy from an error page. Mark eachdependency essential or optional, give each optional one a timeout and a fallback (a cached copy, a default, ornothing), and test the fallbacks: a fallback that runs only during an outage is the code most likely to be broken.

Graceful degradation: essential parts must work; optional parts get a timeout and a fallback.

Load shedding · Going deeper

Under overload, reject the excess so the rest succeed

01,0002,0003,000requests offered a second1,000answered successfully a secondno shedding: everything slows, then failsshedding: excess gets a fast 429 or 503; the rest succeedUnder overload, reject the excess quickly. A service that tries to serve everyone ends up serving no one.Past capacity, queues grow, every request slows past its client’s timeout, and the work done is wasted (illustrative curves).Load shedding refuses requests beyond what can be served in time, early and cheaply, so the rest succeed. Throttlingdoes the same per client: API Gateway usage plans, WAF rate rules. Clients then back off with jitter.

Load shedding refuses requests beyond capacity quickly, so throughput stays up instead of collapsing.

Chaos engineering

Break things on purpose to find weak spots before users do

Steady statedefine normal: p99, errorsHypothesis“losing a zone does not change it”Injectstop instances, add latency, cut a zoneObservedid it hold? how long to recover?Fix and repeatthen automate it on a scheduleFaults worth injecting· an instance or container killed· a zone made unreachable· 200 ms added to a dependency· a dependency returning errors· the database failed over· credentials expiringAWS Fault Injection Service · Azure Chaos StudioBreak things on purpose, carefully and often, to find the weak spots before a real failure does.Chaos engineering is an experiment: state what normal looks like, predict that a fault will not change it, inject thefault with a limited blast radius and a stop condition, and watch. Each surprise is a fix made calmly instead ofduring an outage. Start in test, then game days, then small regular experiments in production.

Chaos engineering runs controlled experiments: a steady state, a hypothesis, an injected fault, and observation.

Safe deployments · Going deeper

Most outages start with a change: release in small steps

MethodHowGainsCostsRollingreplace servers a few at a timeno extra costtwo versions run at onceBlue/greenbuild a full new copy, switch trafficinstant rollback: switch backdouble capacity during the switchCanarysend 5% to the new version, watch, widena bad release hurts fewneeds good metrics to judgeImmutablenever change a server: replace itno drift; every server identicalslower than patching in placeMost outages begin with a change. Release in small steps, watch alarms, and roll back automatically.Every deployment is a planned fault injection. Release to one zone or a small percentage first, compare its errorsand latency with the rest, and let an alarm stop and reverse it. Immutable infrastructure (new servers from a new image,never edited in place) makes every server known and every rollback a simple switch. CodeDeploy, ECS and Lambda supportcanary and linear traffic shifting; load balancer weights and DNS weights do the same for anything else.

Canary, blue/green and rolling releases with automatic rollback; immutable infrastructure replaces instead of patching.

Lesson complete

You finished “Designing for failure”

Nicely done.

Open in a new tab