Load balancers
One address in front, many servers behind, only healthy ones used
A load balancer spreads requests over a target group and sends nothing to targets that fail their health checks.
A fleet of servers needs a single front door. Clients connect to one name, and the load balancer behind it. The balancer itself runs in several zones, managed by the provider. It usually holds the TLS certificate, so encryption ends here and the servers are spared the work. Behind it sits a target group: the servers that do the work. Each healthy one receives a share. One server has stopped answering its health check. The balancer simply stops sending it traffic. Users never see it.
Cross-zone balancing · Going deeper
Uneven zones run hot unless the balancer may cross zones
Without cross-zone load balancing, 2 targets in one zone take 25% each while 8 in the other take 6.25%.
Layer 4 and layer 7
Layer 4 forwards connections; layer 7 reads every request
A layer 4 balancer sees addresses and ports; a layer 7 balancer reads HTTP: host, path, headers and cookies.
Load balancers come in two main kinds, named after the network layer they read. A layer four balancer sees only addresses and ports. It forwards whole connections without opening them. That makes it fast, able to carry any protocol including UDP, able to keep fixed IP addresses, and able to show the server the client’s real address. A layer seven balancer ends the encryption and reads each request: the method, the path, the host, the cookies. Now it can make decisions: send the API calls one way and the images another, redirect, or block an attack. For web traffic, that is usually what you want.
Balancing algorithms · Going deeper · 1 of 2
Round robin is fair to requests, not to servers
With uneven requests, round robin gives one server 80 units of work; least outstanding requests evens it to 38 at most.
ALB and NLB · Going deeper · 2 of 2
ALB or NLB: nine behaviours that decide
The ALB reads HTTP and adds headers; the NLB passes connections, keeps the client’s IP and has a static IP per zone.
Listener rules
Route by host and path, and one entry point fronts many services
Listener rules match a host, path or header and forward to a target group, in priority order.
A layer seven balancer is a router for requests. The first rule: if the host is api dot example dot com, forward to the API service running in containers. If the path starts with images, forward to a set of functions. If a test header is present, send the request to version two: this is how a canary release is tested. Everything else goes to the web front end. One certificate, one address, four services behind it. And because a rule can send a percentage of traffic to each group, the balancer becomes the switch for releasing new versions safely.
Health checks
Two missed checks take a server out; five passes bring it back
With defaults of 30 s, 2 and 5, a hung server serves errors for up to 60 s and returns after 150 s.
Health checks deserve numbers. The balancer asks each server for a health page every thirty seconds. A server hangs. The next check fails, then the one after. Two failures in a row and it is marked unhealthy. Until then, it was still receiving its share of requests, for up to a minute. Restarted, it must pass five checks in a row, two and a half minutes, before it is trusted again. These are the defaults, and they can be tuned. One warning: if the health page checks a shared database, one slow database marks every server unhealthy at once, and the whole site goes dark.
DNS routing
DNS chooses the region before the first packet is sent
Routing policies let one name answer by weight, latency, location or health; answers are cached for their TTL.
Before a load balancer sees anything, DNS has already decided where the user goes. A managed DNS service can answer the same name in different ways. Weighted: ninety percent to the current version, ten percent to the new one, then shift step by step. Latency: the service knows the latency from each user network to each region, and answers with the fastest. A user in Frankfurt is sent to Frankfurt. Failover: while the primary passes its health check, answer with it; when it fails, answer with the standby. One caution. Answers are cached for their time to live, so keep it short on names you may need to move.
Route 53 policies · Going deeper
Eight routing policies, and the exam clue for each
Simple, weighted, latency, failover, geolocation, geoproximity, multivalue and IP-based: each pairs with health checks.
Global routing
Anycast gives one address worldwide and fails over without DNS
Anycast announces one IP from every edge, so users enter the provider’s network nearby and failover does not wait for DNS caches.
DNS has one weakness: caching. After a failover, users whose resolver still holds the old answer keep going to the dead region until it expires, and some clients hold it much longer than they should. Anycast avoids this. One IP address is announced from every edge location in the world. A user in Tokyo and a user in Paris use the same address, and each enters the provider’s network at the nearest edge. From there the provider sends traffic to a healthy region, and can switch regions in seconds while the address never changes. It also crosses the ocean on the provider’s network instead of the public internet.
Load balancers on three clouds · Going deeper
The same four entry points, named on three clouds
Layer 7, layer 4, a global edge and DNS exist on every cloud: Front Door and Google’s global load balancer do in one product what AWS composes.
Lesson complete
You finished “Load balancing”
Nicely done.