11 Jun 2026 · Rehurz
Load Balancer in System Architecture: L4 vs L7 Design
When you're asked to design a system that handles thousands or millions of requests, the first thing that breaks is a single server. A load balancer sits between your clients and your application servers, routing traffic to keep any one server from drowning. In a system design interview, showing you understand load balancers, what they do, where they sit, how they decide which server gets the next request, and what happens when they themselves fail, separates candidates who know the topology from those who have read about it.
Quick answer: A load balancer is a dedicated service that distributes incoming traffic across multiple backend servers to avoid overload on any single machine. It works at Layer 4 (transport, dealing with TCP/UDP ports) or Layer 7 (application, understanding HTTP and content). It can use algorithms like round robin (each server in turn), least connections (to the least busy), IP hash (same client always to same server), or consistent hashing (preserves assignments when servers are added or removed). Layer 4 is fast and simple; Layer 7 is slower but smarter, letting you route based on URL path, hostname, or request headers. Load balancers also check server health, terminate SSL connections, and use techniques like active-passive redundancy or DNS-based failover to avoid being a single point of failure themselves.
What Does a Load Balancer Do?
A load balancer receives a request from a client and decides which of several backend servers should handle it. That is the core job. Without it, you either have one server (which is slow and fails completely if it goes down) or you tell clients "pick a server yourself" (which is chaotic and clients get inconsistent experiences).
The load balancer sits in the network path. Requests come in, the balancer examines them, and forwards them to a chosen backend. Responses come back from the backend, and the balancer sends them to the client. From the client's perspective, they talk to one IP or hostname; they never know there are multiple backends.
This solves two problems. First, it spreads load, so no single server becomes a bottleneck. Second, it hides the backend complexity. If a server fails, the load balancer can stop sending it traffic and route around it.
A good load balancer is also stateless. It does not keep data between requests. It sees request N, makes a decision, and forgets about it. This means you can run multiple load balancers in parallel without them fighting over state.
Layer 4 vs Layer 7 Load Balancing
The OSI model has seven layers. Layer 4 is the transport layer (TCP, UDP). Layer 7 is the application layer (HTTP, HTTPS, TLS).
Layer 4 load balancing looks at the TCP or UDP port and IP address. It sees "this packet came from client IP 192.168.1.5 on port 12345, going to your service on port 80." It makes a routing decision and forwards the entire packet stream to a backend server. It is fast because it is not parsing the request body or headers. It is simple. But it knows nothing about the request itself. It cannot say "requests to /admin go to server 1, requests to /api go to server 2."
Layer 7 load balancing sits higher in the stack. It understands HTTP. It can read the request path, headers, hostname, and method. So it can route intelligently: requests with hostname api.example.com go to one pool, requests with hostname web.example.com go to another. Requests to /static/* go to a fast cache server. This is powerful but slower because the balancer must parse and understand the request.
In practice, many large systems use both. A Layer 4 load balancer (like HAProxy or a cloud network load balancer) sits at the edge and does fast, dumb distribution to multiple Layer 7 load balancers. Each Layer 7 load balancer (like nginx or envoy) then applies smart routing to a pool of application servers.
Load Balancing Algorithms
Once a load balancer decides it will handle a request, it must pick a backend. Which one?
Round robin is the simplest. Servers are in a list: server 1, 2, 3, 1, 2, 3, and so on. Each request goes to the next in line. It is fair on average but assumes all servers are equally fast. If server 2 is slow, it still gets the same share.
Weighted round robin fixes that. Server 1 gets weight 3, server 2 gets weight 1. So the pattern is 1, 1, 1, 2, 1, 1, 1, 2. You choose weights based on server capacity.
Least connections sends the next request to the server currently handling the fewest active connections. This adapts to real time. If server 1 suddenly has 100 long-lived connections and server 2 has 5, new requests go to server 2. It is better for real-world variability but requires the load balancer to track connection counts.
IP hash or source hash hashes the client IP address and always routes that client to the same backend. This is useful if you have in-memory session state on servers. Client 192.168.1.5 always hits server 2; client 192.168.1.6 always hits server 1. The downside: if a server fails, all its clients suddenly move to a different server, losing their session. Also, if servers vary in capacity, hashing may distribute poorly.
Consistent hashing is a more sophisticated approach. It arranges servers and requests on a ring, so that removing a server affects only a small fraction of requests, not a third of them. When you add or remove a server, most clients still hash to the same server. It is more complex but much better for distributed caches and state that you do not want to lose.
Health Checks and Failover
A load balancer does not just assume every server is alive. It periodically sends a health check: a simple request (often just a ping or a GET /health endpoint) to each backend. If the backend responds, it is marked healthy and gets traffic. If it does not respond within a timeout, it is marked unhealthy and traffic is paused. After some recovery time, it may be checked again.
This is critical. A server might crash or hang. Without health checks, the load balancer would send traffic to a dead machine and clients would timeout. With checks, a failed server is removed in seconds.
Health checks are not free. If you have 1000 backends and check each every 5 seconds, that is 200 requests per second just for health. So you must balance frequency (faster detection) against cost (polling load).
Sticky Sessions and Session Affinity
If your application servers keep state in memory (a user logs in, their session is stored in RAM on server 1), you need the same user to always hit server 1. Otherwise, on the next request, server 2 has no idea who they are.
Sticky sessions (also called session affinity) tell the load balancer: once you send a request from this client to server 1, send all future requests from this client to server 1. This is often done with a cookie. The load balancer sets a cookie with the server ID, and on the next request, reads it and routes accordingly.
The downside: sticky sessions break horizontal scaling. You cannot easily add or remove servers without disrupting clients. Also, if server 1 fails, the client is stuck and must re-authenticate on server 2.
A better pattern is stateless application servers. Store sessions in a shared cache (like Redis) or a database. Then any server can handle any user because the session is not on the server. This scales much better.
SSL Termination
When a client connects with HTTPS, it needs to establish a TLS handshake and encryption. That handshake is CPU-expensive. If your application servers each do TLS, you are wasting compute.
Instead, the load balancer terminates TLS. The client connects to the load balancer over HTTPS, negotiates encryption, and sends the request. The load balancer decrypts it, sees the HTTP request inside, and forwards it to the backend server in plaintext (or with fast internal TLS to the backend over a private network). The backend server responds in plaintext, the load balancer encrypts it, and sends it to the client.
This shifts the crypto work to one place (the load balancer) and keeps the backend servers focused on application logic. It also lets the load balancer see the request (it is no longer encrypted), so Layer 7 routing becomes possible.
Load Balancer Redundancy and High Availability
Here is a trap: what if the load balancer itself fails? All your backends are alive, but no traffic can reach them because the single load balancer is down.
The answer is redundancy. You run multiple load balancers. But how do clients know which one to use? Common patterns:
Active-passive: Two load balancers, one is active, one is a hot standby. A virtual IP (VIP) or DNS name points to the active one. If it fails, a monitor (using heartbeat or health checks) detects the failure and updates the VIP to point to the passive one. The passive becomes active. Tools like Keepalived automate this. Downside: the passive server is underutilized.
Active-active: Multiple load balancers are active at the same time. Clients are distributed among them (via DNS round robin or an upper-level load balancer). If one fails, clients using others are unaffected. Those using the failed one reconnect and are redirected to a live one. This is more efficient but requires careful design to avoid cascading failures.
DNS-based: Clients resolve a hostname to multiple IP addresses (one per load balancer). Their OS or library picks one, connects, and uses it. If a load balancer fails, future DNS lookups might return a dead IP, but many clients will keep using the old IP and timeout. Recovery is slow. This is common but not ideal for low-latency failover.
Anycast: All load balancers advertise the same IP address from multiple geographic locations or data centers. The network routes each client to the nearest one. If one fails, the network automatically reroutes. Requires BGP and carrier support but gives fast, transparent failover.
Where Load Balancers Sit in a Multi-Tier Architecture
A typical web architecture has multiple layers. Load balancers appear at multiple points.
Clients
|
v
[Internet]
|
v
[DNS] ---> IP of LB
|
v
[Load Balancer 1] (public, Layer 4 or 7)
|
+---> [App Server 1]
+---> [App Server 2]
+---> [App Server 3]
|
v
[Internal LB] (Layer 7, optional)
|
+---> [Microservice A]
+---> [Microservice B]
|
v
[Database] (with read replicas)
|
v
[Cache Layer] (Redis, Memcached)
The edge load balancer (the one clients see) handles SSL termination and coarse routing. It distributes traffic to multiple application servers.
If your application is microservices, you might have an internal load balancer (often an API gateway or service mesh like Istio) that routes requests within the backend. It understands the microservice topology and can do intelligent retries, circuit breaking, and fine-grained routing.
Behind that are databases. Databases usually do not use load balancers in the same way because writes must go to a single primary, and you need strong consistency. But reads can be load-balanced across read replicas.
Caches (Redis, Memcached) are often accessed directly from application servers using consistent hashing, not through a load balancer.
Scaling and Connection Limits
A load balancer has finite capacity. If you have one load balancer and it receives 100,000 requests per second, the balancer itself becomes the bottleneck.
Scaling a load balancer involves:
- Vertical scaling: More powerful hardware, more CPU cores. At some point, you hit physical limits.
- Horizontal scaling: Multiple load balancers, with DNS or an upper-layer balancer distributing clients among them.
- Stateless design: Ensure the load balancer itself does not keep per-client state, so requests from the same client can be handled by different load balancers.
Cloud providers often offer managed load balancers (AWS ELB, Google Cloud Load Balancing, Azure Load Balancer) that auto-scale. You define a policy (e.g. trigger scaling at 70% CPU), and the cloud handles adding capacity.
Algorithm Comparison
Algorithm | Use Case | Trade-offs
------------------+-------------------+-------------
Round Robin | Uniform loads | Ignores actual server
| | load; simple
Weighted RR | Mixed capacity | Manual weight tuning
| | required
Least Conn | Variable job | Requires connection
| durations | tracking; adaptive
IP Hash | Sticky sessions | Poor distribution on
| | server failure
Consistent Hash | Caches, state | Complex; minimal
| preservation | disruption on change
Frequently Asked Questions
Q: Is a load balancer the same as a reverse proxy? A: A load balancer is a type of reverse proxy. A reverse proxy sits between clients and servers, and may or may not distribute traffic (a single-server reverse proxy is still a reverse proxy). A load balancer specifically distributes traffic. All load balancers are reverse proxies, but not all reverse proxies are load balancers.
Q: Can I use DNS for load balancing instead of a dedicated load balancer? A: Yes, partially. DNS round robin returns multiple IPs for one hostname; clients pick one. It is cheap and works, but failover is slow (clients hold DNS caches) and you lose Layer 7 routing. For production, dedicated load balancers are more reliable.
Q: What if my load balancer becomes a bottleneck? A: Run multiple load balancers (active-active), scale vertically to more powerful hardware, or use cloud-managed load balancers that auto-scale. You can also shard by geography or use content delivery networks to move load distribution closer to clients.
Q: Should I use Layer 4 or Layer 7? A: Layer 4 is faster and simpler; use it when you do not need smart routing. Layer 7 is slower but lets you route by hostname, path, or headers; use it when you need that flexibility. Many systems use both: Layer 4 at the edge, Layer 7 inside.
Q: How do I handle session state with a load balancer? A: Either use sticky sessions (clients stick to one server) or move state out of the server (store in Redis, a database, or client-side). Sticky sessions are simpler but less scalable. Stateless design is better long-term.
Q: What is the difference between connection pooling and load balancing? A: Connection pooling reuses connections from an application to a database. Load balancing distributes traffic among servers. They are orthogonal; you can use both. You might load balance traffic across app servers, and each app server pools connections to a single database.
Practising System Design with Rehurz
System design interviews are spoken rounds. You talk through your architecture, make tradeoffs, and defend your choices under cross-questioning. Understanding load balancers is half the battle; articulating why you chose a specific algorithm, how you would handle failover, and what happens at scale separates confident candidates from those who know the theory but falter in the room.
Rehurz is a voice-based AI mock interview platform. In a system design session on Rehurz, you describe your design out loud, and the interviewer, tuned to your seniority and domain, asks sharp follow-ups: "Why not use consistent hashing here?" or "How does this scale to 10 million users?" You get to practice the speaking part, not just the whiteboarding. After the interview, you get a detailed scorecard with a hire signal and curated resources to close gaps. Try a live technical interview. Your first interview is free, no card required. If you want to deepen your system design chops before the real thing, start now.
Mastering load balancers is one pillar of system design. They are not theoretical, every production system at scale uses them. When you see a question about distributed systems, high availability, or scaling, you will find a load balancer in the answer. Study the algorithms, understand the tradeoffs, and be ready to explain your choices clearly.