Timeouts
Without a timeout, one slow dependency holds every thread
Resources in use = rate × wait. A hung call with no timeout needs 3,000 threads; a timeout of 300 ms needs 30.
Failure in distributed systems is usually not a crash. It is something slow. A service handles a hundred requests a second with fifty threads. A dependency hangs, and with no timeout each request waits thirty seconds. That would need three thousand threads. The fifty are gone in half a second, and now every request fails, even the ones that never touch the slow dependency. With a timeout of three hundred milliseconds, each request gives up quickly. Only thirty threads are busy, and the rest of the service carries on. Every network call needs a timeout.
Bulkheads · Going deeper
Give each dependency its own share, so one cannot sink the rest
Bulkheads split pools per dependency or customer, keeping a failure in one compartment.
Retries
Retries multiply through layers into a storm
4 tries at each of 3 layers turns one click into 64 database calls. Retry at one layer, with a cap.
Retrying seems harmless. Watch it multiply. One click in the browser. The web tier calls the orders service, and on failure tries again, up to four times. The orders service does the same with the database. Sixteen. And the database layer retries too. Sixty-four calls for one click, all arriving at the moment the database is already struggling. Retry at one layer only, cap the attempts, and retry only calls that are safe to repeat.
Backoff and jitter · Going deeper
Back off exponentially, and add randomness
Exponential backoff spaces retries out; jitter stops 200 clients retrying at the same instant.
Circuit breakers
Stop calling a dependency that keeps failing, and fail fast
A circuit breaker opens after repeated failures, fails fast while open, and goes half-open to test with a few calls.
Timeouts make each failed call quick. A circuit breaker avoids making the calls at all. Normally it is closed, and calls pass through. When half the calls in ten seconds fail, it opens. Now callers get an immediate error, or a fallback, without waiting. The struggling service gets a rest. After thirty seconds it goes half-open and lets a few trial calls through. If they succeed, it closes and traffic resumes. If they fail, it opens again and waits another round.
Graceful degradation
When a part fails, serve a smaller answer instead of an error
Graceful degradation: essential parts must work; optional parts get a timeout and a fallback.
A page is made of parts, and not all are equal. The photo, the price and the buy button are essential. Reviews are not: when the reviews service is slow, the page simply hides them. Personal recommendations are down too, so the page shows a cached list of popular items instead. The customer can still buy. So for each part decide: is it essential? If not, what is the fallback: a cached copy, a default, or nothing at all. And test the fallbacks, because they only run when something else is already wrong.
Load shedding · Going deeper
Under overload, reject the excess so the rest succeed
Load shedding refuses requests beyond capacity quickly, so throughput stays up instead of collapsing.
Chaos engineering
Break things on purpose to find weak spots before users do
Chaos engineering runs controlled experiments: a steady state, a hypothesis, an injected fault, and observation.
How do you know any of this works? You test it, on purpose. Define normal: orders a minute, response times, error rate. Predict: losing a zone will not change it. Inject the fault, with a limited blast radius and a stop button: stop the instances in one zone. Watch. Did the numbers hold? How long did recovery take? Fix whatever surprised you, and repeat, until the experiment runs on a schedule. Services like Fault Injection Service and Chaos Studio make these experiments safe and repeatable.
Safe deployments · Going deeper
Most outages start with a change: release in small steps
Canary, blue/green and rolling releases with automatic rollback; immutable infrastructure replaces instead of patching.
Lesson complete
You finished “Designing for failure”
Nicely done.