Load balancers

One address in front, many servers behind, only healthy ones used

clientsshop.example.com :443Load balancerone address, in two zonesTLS ends herecertificate attachedTarget group: web serversport 8080healthy: receives 333 req/shealthy: receives 333 req/shealthy: receives 333 req/sfailed its health check:no traffic until it passes againOne address in front, many servers behind, and only the healthy ones receive traffic.A load balancer is the stable entry point of a scaled-out tier. It spreads requests across a group of targets, checkseach one every few seconds and stops sending to any that fail, so a dead server is invisible to users. It usuallyends TLS too, and is itself spread across zones by the provider, so it is not a single point of failure.

A load balancer spreads requests over a target group and sends nothing to targets that fail their health checks.

Cross-zone balancing · Going deeper

Uneven zones run hot unless the balancer may cross zones

Cross-zone off: each zone gets half the trafficzone A2 targets, 25% eachzone B8 targets, 6.25% eachCross-zone on: each target gets an equal sharezone A2 targets, 10% eachzone B8 targets, 10% eachWithout cross-zone balancing every zone gets an equal share, so the zone with fewer targets runs hot.DNS returns one balancer node per zone, and clients split evenly between them. Off: a node uses only its own zone, so eachof the 2 targets in zone A takes 25%, 4 times the load of a target in zone B (6.25%). On: every node uses all 10 targets.ALB: always on. NLB and Gateway Load Balancer: off by default, and on an NLB the traffic that crosses zones is billed.

Without cross-zone load balancing, 2 targets in one zone take 25% each while 8 in the other take 6.25%.

Layer 4 and layer 7

Layer 4 forwards connections; layer 7 reads every request

What a layer 4 balancer readsIP 203.0.113.9 → 198.51.100.4 TCP port 443encrypted bytes it does not open…Network Load Balancer · Azure Load Balancer · Google passthrough NLBTCP, UDP and TLS; keeps the client’s IP; static IP per zonemillions of requests a second at very low latencyWhat a layer 7 balancer readsIP and port, then it ends TLS and reads:GET /api/orders/42 HTTP/2Host: shop.example.comCookie: session=…Application Load Balancer · Application Gateway · Google Application LBroutes by host, path, header, method or queryHTTP/2, gRPC, WebSockets; WAF, sign-in and redirectsA layer 4 balancer forwards connections by address and port. A layer 7 balancer reads each request.Choose layer 7 for web traffic: it can send /api and /images to different services and block attacks on the way.Choose layer 4 for anything that is not HTTP, for static IP addresses, for extreme throughput, or to keep theclient’s own IP visible to the server. A third kind, the Gateway Load Balancer, passes traffic through firewall appliances.

A layer 4 balancer sees addresses and ports; a layer 7 balancer reads HTTP: host, path, headers and cookies.

Balancing algorithms · Going deeper · 1 of 2

Round robin is fair to requests, not to servers

30 requests arrive in a repeating pattern: cheap, cheap, expensive (1, 1, 8 units).Round robineach server in turnserver 110 units of workserver 210 units of workserver 380 units of workLeast outstanding requeststo the server with the least in progressserver 138 units of workserver 230 units of workserver 332 units of workRound robin is fair to requests, not to servers. When requests differ in cost, balance by work in progress.Round robin counts requests, so if every third one is expensive, one server gets all of them and runs hot while theothers idle. Least outstanding requests (ALB), least connections, or weighted choices follow the actual load. Layer 4balancers hash the connection’s addresses instead, so one long connection always lands on the same server.

With uneven requests, round robin gives one server 80 units of work; least outstanding requests evens it to 38 at most.

ALB and NLB · Going deeper · 2 of 2

ALB or NLB: nine behaviours that decide

Application Load Balancer (layer 7)Network Load Balancer (layer 4)ProtocolsHTTP, HTTPS, gRPC, WebSocketsTCP, UDP, TLSPicks a target byround robin, least outstanding requests, weighted randoma hash of the flow: a connection stays on one targetAddressa DNS name; the IP addresses behind it changeone static IP per zone; an Elastic IP if you wantClient’s IP addressin the X-Forwarded-For headerkept on the packet (client IP preservation)Cross-zone balancingalways on; can be turned off per target groupoff by defaultIdle timeout60 s by defaultTCP 350 s (60 to 6,000); UDP 120 s, fixedStickinessa cookie lasting 1 second to 7 daysby source IP addressTargetsinstances, IP addresses, a Lambda functioninstances, IP addresses, an ALBOnly hereAWS WAF, user sign-in, redirects, fixed responsesPrivateLink services; millions of requests a secondTake the ALB for HTTP. Take the NLB for anything else, for a fixed IP address, or for PrivateLink.Exam clues for the NLB: “static IP”, “UDP”, “millions of connections”, “the server must see the client’s address” or“offer the service to other accounts privately”. For the ALB: “path”, “host name”, “WAF”, “sign users in”, “Lambda”.Need both? Register the ALB as the NLB’s target: fixed addresses in front, request routing behind.

The ALB reads HTTP and adds headers; the NLB passes connections, keeps the client’s IP and has a static IP per zone.

Listener rules

Route by host and path, and one entry point fronts many services

One listenerHTTPS :443if Host is api.example.com1API servicecontainersif Path is /images/*2Image servicefunctionsif Header x-canary: yes3Version 2 (canary)5% of serversif Anything else (default)4Web front endserversrules are checked in priority orderA layer 7 balancer routes by host and path, so one entry point can front many services.Each rule has a priority, a condition and an action: forward to a target group, redirect (HTTP to HTTPS), return afixed response, or ask the user to sign in. Targets can be servers, containers, IP addresses or functions. Weightedforwarding sends a percentage to a new version, which makes the balancer the switch for blue/green and canary releases.

Listener rules match a host, path or header and forward to a target group, in priority order.

Health checks

Two missed checks take a server out; five passes bring it back

0 s60 s120 s180 s240 s300 sapp hangs✗✗2 failures in a row:out of serviceup to 60 s of errors for its share of usersrestarted5 passes: back in service5 × 30 s = 150 s before it receives traffic againAWS defaults: check every 30 s; 2 failures to leave; 5 passes to return; a check times out after 5 s. Tune them to the application.Health checks decide who gets traffic. Their interval and thresholds set how long users see a broken server.With the defaults a hung server keeps receiving requests for up to 60 seconds. A shorter interval finds it sooner but costsmore checks. Point the check at a page that tests what matters (can it reach its database?) but not at shareddependencies, or one slow database marks every server unhealthy at once. Removing a server drains it: up to 300 s by default.

With defaults of 30 s, 2 and 5, a hung server serves errors for up to 60 s and returns after 150 s.

DNS routing

DNS chooses the region before the first packet is sent

WeightedDNS90%version 110%version 2shift traffic step by stepLatencyFrankfurt12 mseu-central-195 msus-east-1answer with the fastest regionFailoverDNSprimarycheck failingsecondarynow answeredhealth check flips the answerDNS chooses where a user goes before the first packet is sent, by the policy on the name.A DNS service such as Route 53, Azure Traffic Manager or Cloud DNS can answer the same name differently: by weight,by measured latency, by the user’s location, or by health. Answers are cached for their TTL, so a change reaches usersonly as caches expire: keep TTLs short (60 seconds) on names you may need to move quickly.

Routing policies let one name answer by weight, latency, location or health; answers are cached for their TTL.

Route 53 policies · Going deeper

Eight routing policies, and the exam clue for each

Route 53 policyHow it answersUse it forSimpleone answer (or several, in random order)a single resourceWeightedanswers in set proportionscanary and blue/green releasesLatencythe region with the lowest measured latencyserving users from several regionsFailoverprimary while healthy, else secondaryactive-passive disaster recoveryGeolocationby the user’s country or continentlanguage, licensing, data residencyGeoproximityby distance, shifted by a biasmoving load between regionsMultivalue answerup to eight healthy answers, at randomsimple client-side spreadingIP-basedby the user’s network range (CIDR)routing known ISPs or officesEight DNS policies, and each can be combined with health checks so a failed endpoint stops being answered.Exam clues: “closest region” or “lowest latency” → latency; “by country” or “legal” → geolocation; “gradually” or “apercentage” → weighted; “standby site” → failover. Alias records point a name at an AWS load balancer, CloudFront or S3website, including the zone apex (example.com), which a CNAME cannot. Azure Traffic Manager has similar methods.

Simple, weighted, latency, failover, geolocation, geoproximity, multivalue and IP-based: each pairs with health checks.

Global routing

Anycast gives one address worldwide and fails over without DNS

DNS-based failoverresolver cache: TTL 60 sold region (down)users follow the cached answer until it expires;some clients ignore TTLs and hold it far longerAnycast: one address announced from every edgeTokyoParisedgeedge75.2.10.4the same IP for bothregion Aregion Btraffic joins the provider’s network at the nearest edge; failover is in secondsAnycast gives one address worldwide, enters the provider’s network near the user, and fails over without DNS.With DNS, the user is sent to a region’s address and keeps it until caches expire. With anycast the address neverchanges; the network routes each packet to the nearest edge, and the provider moves traffic between regions behindit. AWS Global Accelerator (two static IPs, any TCP or UDP), Azure Front Door and Google’s global load balancers.

Anycast announces one IP from every edge, so users enter the provider’s network nearby and failover does not wait for DNS caches.

Load balancers on three clouds · Going deeper

The same four entry points, named on three clouds

AWSAzureGoogle CloudHTTP, in one regionApplication Load BalancerApplication Gatewayregional Application Load BalancerTCP and UDPNetwork Load BalancerAzure Load Balancerpassthrough Network Load BalancerHTTP, worldwideCloudFront in front of regional ALBsFront Doorglobal external Application LBOne anycast addressGlobal Acceleratorcross-region Load Balancerglobal external ALB or proxy NLBDNS routingRoute 53 routing policiesTraffic ManagerCloud DNS routing policiesWeb firewallAWS WAFWeb Application FirewallCloud ArmorFirewall appliancesGateway Load BalancerGateway Load Balancerinternal passthrough NLB as next hopEvery cloud has the same entry points: layer 7, layer 4, a global edge and DNS.They differ in how the global part is built. Google’s global load balancer is one anycast address in front of back endsin many regions. Azure Front Door is the same idea with a CDN built in. AWS composes it: one ALB per region, thenRoute 53, Global Accelerator or CloudFront in front to choose the region. AZ-305 and the Google exam ask for these names.

Layer 7, layer 4, a global edge and DNS exist on every cloud: Front Door and Google’s global load balancer do in one product what AWS composes.

Lesson complete

You finished “Load balancing”

Nicely done.

Open in a new tab