--- title: "09 — Load Balancers: Distributing Traffic Across Servers" uid: load-balancers tags: ["l4-l7", "scaling", "load-balancing", "roadmap:system-design", "system-design", "reverse-proxy"] excerpt: "A load balancer spreads requests across servers to prevent overload and remove single points of failure — deciding by network packet (layer 4) or application message (layer 7)." date: 2026-08-13T03:27:33+0000 source: https://www.aveshina.my.id/en/blog/load-balancers --- "Just NGINX" was how I conflated load balancers, which hid the job they actually do. Writing it down gave the component a clear purpose: **a load balancer distributes incoming client requests across computing resources — application servers, databases — and returns each response to the appropriate client.** [1] Its value is threefold: it keeps requests off unhealthy servers, it prevents any one resource from being overloaded, and it helps eliminate the single point of failure that a lone server represents. The framing that landed is that a load balancer is the component that turns N identical servers into one logical server from the client's perspective. Without it, scaling horizontally just gives you N islands the client has to know about. With it, the client talks to one address and the balancer worries about which backing server actually handles each request. ## What it does, and what it costs A load balancer can be hardware (expensive) or software (HAProxy, NGINX) [1]. Beyond the core distribution job, it usually bundles two useful side-features: - **SSL termination** — it decrypts incoming requests and encrypts server responses, so the backend servers do not have to. This removes the need to install X.509 certificates on every server [1]. - **Session persistence** — it can issue cookies and route a specific client's requests to the same instance, useful when the web apps do not keep track of sessions themselves [1]. The honest costs the roadmap flags: a load balancer can become a performance bottleneck if it is under-resourced or misconfigured; it adds complexity; and a single load balancer is itself a single point of failure, so you often need multiple load balancers in a failover setup — which further increases complexity [1]. ## Layer 4 vs layer 7: where the decision is made The distinction I had to nail down is _what information_ the balancer uses to route. That is the layer-4 vs layer-7 split [3][4]: ```figure ``` - **Layer 4** looks at info at the transport layer — generally the source and destination IP addresses and ports in the header, but _not_ the contents of the packet. It forwards network packets to and from the upstream server, performing Network Address Translation (NAT) [4]. It is cheaper and faster because it does not inspect the message body. - **Layer 7** looks at the application layer to decide how to distribute requests — the contents of the header, message, and cookies [3]. It terminates the network traffic, reads the message, makes a routing decision, then opens a connection to the selected server. A layer-7 balancer can direct video traffic to video servers while sending sensitive billing traffic to security-hardened servers — routing by content, not just by address [3]. The performance gap between the two has narrowed on modern commodity hardware, so the choice is increasingly about _capability_ (do I need content-based routing?) rather than raw throughput [3]. ## The routing algorithms Once I know which layer I'm at, the next question is the algorithm — the rules the balancer uses to pick a server [2]. Two families: - **Static** algorithms distribute traffic without considering current server state. Round-robin (cycle through servers), random, and IP-hash (hash the client IP to a consistent server) are static. - **Dynamic** algorithms account for the current state of each server — least-connections (send to the server with fewest active connections), least-response-time (send to the fastest-responding), and so on. Dynamic costs more to compute (the balancer must track per-server state) but adapts when one server is slow or degraded. The roadmap points at Cloudflare's summary for the full taxonomy, and I keep that as a reference rather than memorizing every variant [2]. ## Load balancer vs reverse proxy One more distinction worth making, because I conflated these for years. A load balancer distributes traffic across multiple servers; a reverse proxy can be useful even with a single server — it sits in front and provides benefits like SSL termination, caching, and request rewriting [5]. Solutions like NGINX and HAProxy do both layer-7 reverse proxying and load balancing [5]. The way of thinking: a reverse proxy is the _position_ (in front of servers); a load balancer is the _behavior_ (distributing across servers). Most production reverse proxies are also load balancers, but not every reverse proxy has more than one backing server to balance across. ## Horizontal scaling: the load balancer's reason for existing Load balancers also enable **horizontal scaling** — improving performance and availability by adding more commodity machines rather than buying a bigger single machine (vertical scaling) [6]. Scaling out with commodity hardware is more cost-efficient and yields higher availability than scaling up a single expensive server. The catch the roadmap emphasizes: **servers behind a horizontal-scaled balancer should be stateless** — they must not hold user-related data like sessions or profile pictures, because any request might land on any server [6]. Sessions move to a centralized store (a database, or a persistent cache like Redis/Memcached), so any server can serve any request. ## How I use this The habit is to put a load balancer in front of any tier with more than one instance, and to treat the backing servers as stateless from the start — sessions and uploads go to shared storage, never to local disk on one instance. When I need content-based routing (different traffic classes to different pools), I reach for layer 7; when I just need even distribution at high throughput, layer 4 is simpler and faster. And I never deploy a single load balancer in production without a failover pair, because the balancer that removes the single point of failure must not itself be one. ## References [1] cs.fyi, "Scalability — for dummies," 2021. [Online]. Available: [https://cs.fyi/guide/scalability-for-dummies](https://cs.fyi/guide/scalability-for-dummies) [2] Cloudflare, "Types of load balancing algorithms," 2024. [Online]. Available: [https://www.cloudflare.com/learning/performance/types-of-load-balancing-algorithms/](https://www.cloudflare.com/learning/performance/types-of-load-balancing-algorithms/) [3] NGINX, "Inside NGINX: how we designed for performance and scale," NGINX Blog. [Online]. Available: [https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/](https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/) [4] F5, "Layer 4 load balancing," F5 Glossary. [Online]. Available: [https://www.f5.com/glossary/layer-4-load-balancing](https://www.f5.com/glossary/layer-4-load-balancing) [5] NGINX, "Reverse proxy vs load balancer," NGINX Glossary. [Online]. Available: [https://www.nginx.com/resources/glossary/reverse-proxy-vs-load-balancer/](https://www.nginx.com/resources/glossary/reverse-proxy-vs-load-balancer/) [6] HAProxy, "HAProxy architecture guide." [Online]. Available: [http://www.haproxy.org/download/1.2/doc/architecture.txt](http://www.haproxy.org/download/1.2/doc/architecture.txt) ```quiz Q: A layer 4 load balancer makes routing decisions based on… - the HTTP headers, cookies, and message body - source/destination IP addresses and ports (not the packet contents) correct: 1 explain: Layer 4 operates at the transport layer and inspects IP and port only. Layer 7 inspects the application message — headers, cookies, path. Q: Why must servers behind a horizontally-scaled load balancer be stateless? - because stateless servers are faster - because any request can land on any server, so user-specific state must live in shared storage correct: 1 explain: Horizontal scaling distributes requests arbitrarily. If a server held session data locally, the next request for that session might hit a different server that lacks it. State moves to a shared store. Q: The key difference between a load balancer and a reverse proxy is… - there is no difference; they are the same product - a reverse proxy is about position (in front of servers); a load balancer is about behavior (distributing across servers) correct: 1 explain: A reverse proxy can be useful with a single server; a load balancer distributes across multiple. Most modern products (NGINX, HAProxy) do both. Q: Dynamic load-balancing algorithms differ from static ones because they… - ignore server state entirely - factor in current server state (e.g. least-connections, least-response-time) correct: 1 explain: Dynamic algorithms track per-server state and adapt — useful when servers degrade. Static algorithms like round-robin distribute without considering state. ```