09 — Load Balancers: Distributing Traffic Across Servers

"Just NGINX" was how I conflated load balancers, which hid the job they actually do. Writing it down gave the component a clear purpose: a load balancer distributes incoming client requests across computing resources — application servers, databases — and returns each response to the appropriate client. [1] Its value is threefold: it keeps requests off unhealthy servers, it prevents any one resource from being overloaded, and it helps eliminate the single point of failure that a lone server represents.

The framing that landed is that a load balancer is the component that turns N identical servers into one logical server from the client's perspective. Without it, scaling horizontally just gives you N islands the client has to know about. With it, the client talks to one address and the balancer worries about which backing server actually handles each request.

What it does, and what it costs

A load balancer can be hardware (expensive) or software (HAProxy, NGINX) [1]. Beyond the core distribution job, it usually bundles two useful side-features:

• SSL termination — it decrypts incoming requests and encrypts server responses, so the backend servers do not have to. This removes the need to install X.509 certificates on every server [1].
• Session persistence — it can issue cookies and route a specific client's requests to the same instance, useful when the web apps do not keep track of sessions themselves [1].

The honest costs the roadmap flags: a load balancer can become a performance bottleneck if it is under-resourced or misconfigured; it adds complexity; and a single load balancer is itself a single point of failure, so you often need multiple load balancers in a failover setup — which further increases complexity [1].

Layer 4 vs layer 7: where the decision is made

The distinction I had to nail down is _what information_ the balancer uses to route. That is the layer-4 vs layer-7 split [3][4]:

• Layer 4 looks at info at the transport layer — generally the source and destination IP addresses and ports in the header, but _not_ the contents of the packet. It forwards network packets to and from the upstream server, performing Network Address Translation (NAT) [4]. It is cheaper and faster because it does not inspect the message body.
• Layer 7 looks at the application layer to decide how to distribute requests — the contents of the header, message, and cookies [3]. It terminates the network traffic, reads the message, makes a routing decision, then opens a connection to the selected server. A layer-7 balancer can direct video traffic to video servers while sending sensitive billing traffic to security-hardened servers — routing by content, not just by address [3].

The performance gap between the two has narrowed on modern commodity hardware, so the choice is increasingly about _capability_ (do I need content-based routing?) rather than raw throughput [3].

The routing algorithms

Once I know which layer I'm at, the next question is the algorithm — the rules the balancer uses to pick a server [2]. Two families:

• Static algorithms distribute traffic without considering current server state. Round-robin (cycle through servers), random, and IP-hash (hash the client IP to a consistent server) are static.
• Dynamic algorithms account for the current state of each server — least-connections (send to the server with fewest active connections), least-response-time (send to the fastest-responding), and so on.

Dynamic costs more to compute (the balancer must track per-server state) but adapts when one server is slow or degraded. The roadmap points at Cloudflare's summary for the full taxonomy, and I keep that as a reference rather than memorizing every variant [2].

Load balancer vs reverse proxy

One more distinction worth making, because I conflated these for years. A load balancer distributes traffic across multiple servers; a reverse proxy can be useful even with a single server — it sits in front and provides benefits like SSL termination, caching, and request rewriting [5]. Solutions like NGINX and HAProxy do both layer-7 reverse proxying and load balancing [5]. The way of thinking: a reverse proxy is the _position_ (in front of servers); a load balancer is the _behavior_ (distributing across servers). Most production reverse proxies are also load balancers, but not every reverse proxy has more than one backing server to balance across.

Horizontal scaling: the load balancer's reason for existing

Load balancers also enable horizontal scaling — improving performance and availability by adding more commodity machines rather than buying a bigger single machine (vertical scaling) [6]. Scaling out with commodity hardware is more cost-efficient and yields higher availability than scaling up a single expensive server. The catch the roadmap emphasizes: servers behind a horizontal-scaled balancer should be stateless — they must not hold user-related data like sessions or profile pictures, because any request might land on any server [6]. Sessions move to a centralized store (a database, or a persistent cache like Redis/Memcached), so any server can serve any request.

How I use this

The habit is to put a load balancer in front of any tier with more than one instance, and to treat the backing servers as stateless from the start — sessions and uploads go to shared storage, never to local disk on one instance. When I need content-based routing (different traffic classes to different pools), I reach for layer 7; when I just need even distribution at high throughput, layer 4 is simpler and faster. And I never deploy a single load balancer in production without a failover pair, because the balancer that removes the single point of failure must not itself be one.

References

[1] cs.fyi, "Scalability — for dummies," 2021. [Online]. Available: https://cs.fyi/guide/scalability-for-dummies

[2] Cloudflare, "Types of load balancing algorithms," 2024. [Online]. Available: https://www.cloudflare.com/learning/performance/types-of-load-balancing-algorithms/

[3] NGINX, "Inside NGINX: how we designed for performance and scale," NGINX Blog. [Online]. Available: https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/

[4] F5, "Layer 4 load balancing," F5 Glossary. [Online]. Available: https://www.f5.com/glossary/layer-4-load-balancing

[5] NGINX, "Reverse proxy vs load balancer," NGINX Glossary. [Online]. Available: https://www.nginx.com/resources/glossary/reverse-proxy-vs-load-balancer/

[6] HAProxy, "HAProxy architecture guide." [Online]. Available: http://www.haproxy.org/download/1.2/doc/architecture.txt