How to Design a Highly Available Rate Limiter

When traffic spikes hit your API, your database can collapse in seconds if you're not careful. A rate limiter is one of the oldest and most critical tools in a backend engineer's toolkit, yet designing a highly available one that scales across distributed systems trips up even experienced candidates in system design interviews.

This post walks you through why rate limiting matters, where it sits in your architecture, the main algorithms, and how to implement it reliably at scale with Redis and handle the edge cases that interviewers love to ask about.

Quick answer: A rate limiter enforces a quota on how often a user or IP can call your API, returning HTTP 429 (Too Many Requests) when exceeded. Distributed rate limiters use Redis with atomic increment and expiry commands (INCR + EXPIRE) to track per-user or per-IP counters across multiple servers. Token bucket and leaky bucket algorithms handle bursts gracefully, while sliding window approaches offer accuracy. The key tradeoff is between precision (slow, memory-intensive) and speed (fast, approximate). Choose fail-open for availability and fail-closed for security.

Why Rate Limiting Exists

Rate limiting serves three distinct purposes, and conflating them during an interview signals misunderstanding.

First, it protects your infrastructure from overload. A single buggy client or a traffic spike can consume all database connections or exhaust your servers. Rate limiting acts as an early circuit breaker before damage happens. Without it, one misbehaving customer's retry loop can take down service for everyone.

Second, it enforces fair use and prevents abuse. If you offer a free tier, rate limits prevent one user from hogging all your resources. For public APIs, it stops scrapers and bots from draining your quota. For subscription tiers, it becomes part of your billing model: 1000 requests per month for the free plan, 10,000 for Pro.

Third, it mitigates denial-of-service attacks. A distributed rate limiter makes it harder for attackers to flood your system; they have to spread requests across many IPs or accounts, each hitting the limit individually instead of one coordinated spike.

Where Does a Rate Limiter Live?

Architecture placement is crucial and worth drawing during your interview.

A gateway-level rate limiter sits at the edge, before requests reach your application servers. It might live in an API gateway, a reverse proxy, or a dedicated middleware service. This is where you stop the bulk of malicious traffic, never even routing it to your backend. It's fast, centralized, and reduces wasted resources.

An application-level rate limiter lives inside your service code, often as middleware. You might use it to rate-limit per-user or per-account (finer granularity than gateway-level IP-based limiting). It can enforce business logic: "Pro users get 10x the quota."

Most mature systems use both. The gateway catches obvious abuse (single IP hammering you). Your app middleware enforces fair-use quotas per user or tenant.

Client --> [API Gateway Rate Limiter]
             (IP-based, coarse-grained)
             |
             v
         [App Servers]
         (user/account-level,
          fine-grained, business logic)
         |
         v
     [(Database / Cache)]

Rate Limiting Algorithms

Five main approaches exist. Each trades off memory, speed, accuracy, and burst handling. Here's the comparison:

Algorithm         | Accuracy | Memory  | Burst  | Impl Ease
------------------+----------+---------+--------+-----------
Fixed Window      | Low      | Minimal | Fair   | Easy
Sliding Window Log | High     | High    | Exact  | Medium
Sliding Window Cnt | Medium   | Low     | Good   | Medium
Token Bucket      | High     | Low     | Excellent | Hard
Leaky Bucket      | High     | Low     | Excellent | Hard

Fixed Window

The simplest approach. Divide time into fixed intervals (e.g., 1-minute windows). Count requests in the current window; reject if over the limit.

How it works: A user makes requests at 10:59:50 (9 requests) and 10:00:10 (8 requests). Each falls in a different 1-minute window, so both are allowed even though 17 requests in 70 seconds might violate the true intent.

Pros: trivial to implement, minimal memory, fast.

Cons: boundary spike problem. If a limit is "100 per minute" and you have two 1-minute windows back-to-back, a user can make 100 requests at the tail of window 1 and 100 at the head of window 2, totaling 200 in less than 2 minutes. Looks like a burst escape hatch.

When to use: backend jobs, non-critical APIs, or when the boundary spike is acceptable.

Sliding Window Log

Track exact timestamps of every request for a user. When a new request arrives, discard old entries outside the window and count remaining requests.

How it works: Limit is 100 per minute. At 10:00:45, a request arrives. You drop all requests before 09:59:45, then count remaining. If count < 100, allow.

Pros: accurate, handles bursts exactly, no boundary spike.

Cons: memory scales linearly with request rate. A heavy user with 1000 requests per minute needs to store 1000 log entries. For millions of users, this gets expensive.

When to use: small user bases, strict accuracy requirements, or when memory is cheap.

Sliding Window Counter (Hybrid)

A compromise. Instead of logging every timestamp, keep two counters: one for the current 1-minute window, one for the previous. Estimate requests in the sliding window by weighted sum.

Formula: current_requests + (requests_in_previous_window * (time_elapsed_in_current_window / total_window_duration))

How it works: At 10:00:45, you've seen 30 requests in the current window (00:30 to 00:45) and 100 in the previous window (09:59 to 10:00). Elapsed time is 45 seconds of a 60-second window. Estimate: 30 + (100 * 45/60) = 30 + 75 = 105. If limit is 100, reject.

Pros: memory-efficient, reasonably accurate, faster than sliding window log.

Cons: still not perfect if requests cluster. The weighted approximation can allow slight boundary spikes but usually acceptable.

When to use: most production systems. It's the sweet spot for scalability and accuracy.

Token Bucket

Imagine a bucket that fills at a fixed rate (e.g., 100 tokens per minute). Each request costs one token. If the bucket is empty, request is rejected. Unused tokens can accumulate (up to a maximum), enabling bursts.

How it works: The bucket holds a max of 100 tokens, refilling at a rate of 1.67 per second. A user with an empty bucket at 10:00:00 can burst 100 requests in the first second. Then requests trickle in at 1.67 per second. If the user goes idle for 10 seconds, tokens accumulate to 100 again, allowing another burst.

Pros: handles bursts gracefully, memory-efficient, predictable behavior, widely understood.

Cons: can be complex to implement correctly across distributed systems. You need to calculate available tokens based on last refill time and current time; floating-point math can introduce subtle bugs.

When to use: APIs that benefit from bursty traffic (webhooks, batch jobs, integrations). SaaS pricing tiers often use token bucket.

Leaky Bucket

Requests are added to a queue. They drain from the queue at a fixed rate. If the queue fills up, new requests are rejected.

How it works: Queue max is 100, drain rate is 10 per second. Requests arrive at 150 per second. The first 100 get queued, 50 are rejected. Then the queue drains at 10 per second, making room for new arrivals.

Pros: smooths traffic, guarantees steady output rate, prevents spikes downstream.

Cons: adds latency (requests wait in queue); hard to tune (what's the right drain rate?); less intuitive than token bucket.

When to use: systems where you want to protect downstream services from bursts (e.g., a print queue, batch processors).

For interviews, token bucket and sliding window counter are the most commonly expected answers.

Implementing Distributed Rate Limiting with Redis

Staying on a single server is a non-starter for production. Redis is the standard choice for distributed rate limiters because it's fast and supports atomic operations.

The core pattern uses Redis's INCR (increment) and EXPIRE (set timeout) commands. Here's pseudocode for fixed window:

key = f"rate_limit:{user_id}:{current_minute}"
current_count = INCR(key)
if current_count == 1:
    EXPIRE(key, 60)
if current_count > limit:
    return 429 Too Many Requests
return 200 OK

The trick: INCR and EXPIRE must be atomic. If they're separate commands, a race condition can occur. A process crashes after INCR but before EXPIRE, leaving the key without a TTL forever. Use a Redis Lua script to ensure atomicity:

if redis.call("incr", KEYS[1]) == 1 then
    redis.call("expire", KEYS[1], ARGV[1])
end
return redis.call("get", KEYS[1])

Or use Redis commands directly if your client library supports pipelining and you're willing to accept eventual consistency.

For token bucket in Redis, you compute available tokens based on elapsed time:

key = f"bucket:{user_id}"
last_refill = GET(f"{key}:last_refill")
now = current_time()
elapsed = now - last_refill
tokens_added = elapsed * rate_per_second
current_tokens = GET(key) or max_tokens
new_tokens = min(current_tokens + tokens_added, max_tokens)
if new_tokens >= cost:
    SET(key, new_tokens - cost)
    SET(f"{key}:last_refill", now)
    return 200 OK
return 429

Again, wrap this in Lua to prevent race conditions. Redis's single-threaded model guarantees Lua scripts run atomically, so the entire token deduction happens without interference.

Handling Race Conditions and Distributed Failures

Distributed rate limiting introduces failure modes.

Problem 1: Multiple servers updating the same counter. Two servers read current_count = 99 simultaneously, both INCR to 100, both allow the request. Limit is breached. Solution: Redis's INCR is atomic at the Redis node level, so if both servers use the same Redis instance, this is safe. If you shard Redis, ensure one user always routes to one shard (consistent hashing on user ID).

Problem 2: Redis goes down. Your rate limiter can't query counters. Options: (a) fail-open: allow all requests until Redis recovers (loose security but high availability); (b) fail-closed: deny all requests (high security but degrades UX). Most systems choose fail-open for user-facing APIs and fail-closed for internal rate limiting.

Problem 3: Clock skew. If server A thinks it's 10:00 but server B thinks it's 09:59, they might disagree on which window a request falls into. Solution: use server time, not client time. Better, use a centralized time source or let Redis (which is on a single machine or a clustered setup with clock sync) decide time.

Per-User vs Per-IP Keys

The key design determines what you're limiting.

Per-IP: rate_limit:1.2.3.4, tracks requests from a single IP address. Good for basic DDoS mitigation. Weak against distributed attacks or shared networks (office WiFi looks like one IP to you).

Per-user: rate_limit:user_12345, tracks authenticated users. Strong against abuse within your own system. Weak if user accounts are cheap (attacker creates many free accounts).

Per-tenant: rate_limit:org_567, for multi-tenant SaaS, tracks requests per customer organization. Enforces fair use across customers.

Hybrid: Apply limits at multiple levels. Gateway limits per IP (broad protection), app limits per user (fair use), and per-account quotas (billing tiers). Stack them:

if INCR(f"ip:{ip_address}") > 1000:
    return 429
if INCR(f"user:{user_id}") > 10000:
    return 429
if INCR(f"org:{org_id}") > 100000:
    return 429
return 200

HTTP Response and Retry Logic

When you reject a request, the HTTP response matters.

Status code: 429 Too Many Requests (RFC 6585). Some systems used 503 or 420; don't. Use 429.

Headers: Include Retry-After to tell clients when to retry. Either a number of seconds or an HTTP date:

Retry-After: 60
Retry-After: Wed, 21 Oct 2026 07:28:00 GMT

Also include RateLimit-* headers for transparency:

RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 1718876880

Clients use Retry-After to implement exponential backoff. Good clients obey it; bad ones hammer you. Your rate limiter has to protect against both.

Fail-Open vs Fail-Closed

When your rate limiter itself fails (Redis down, network partition), you choose your posture.

Fail-open: In the absence of a rate limit, allow the request. You protect availability. If an attacker exploits the downtime, at least your users can access the service. Most user-facing APIs do this.

Fail-closed: In the absence of a rate limit, deny the request. You protect security. An attacker can trigger a DoS by taking down Redis, but you prevent them from bypassing your limits. Choose this for internal APIs or payment processing.

Implement fail-open by catching Redis timeouts and logging them, then allowing the request to proceed. Implement fail-closed by catching errors and returning 429.

Frequently Asked Questions

Q: Should I rate-limit on request count or bandwidth (bytes)? A: Both, separately. A single 1 MB upload counts as one request but consumes more bandwidth than 100 small requests. Rate-limit requests per minute and bytes per minute independently.

Q: What if a user has multiple devices or accounts? A: Use per-user ID if you have auth. For unauthenticated APIs, you're stuck with IP-based limits. Accept some collateral damage (blocking innocent users on shared networks) or add CAPTCHA to distinguish humans from bots.

Q: Can I rate-limit on response time instead of request count? A: Yes, but it's harder. You're tracking whether requests are slow, not how many arrive. Combine with request-count limits; if responses are slow, lower the limit. Most interviews expect you to propose hybrid limits.

Q: How do I handle legitimate bursts from integrations? A: Whitelist trusted IPs or accounts, or offer a higher tier. Token bucket handles bursts naturally. You can also provision extra capacity for partners.

Q: What's the latency of a rate limit check? A: In-process memory (< 1 ms). Redis single-digit ms (network round trip). For Lua scripts, add parsing overhead. Expect 5 to 50 ms total in a geographically distributed system. If latency matters, cache the rate limit decision locally (best-effort).

Q: Can I rate-limit based on user reputation or machine learning? A: Yes, advanced systems do. Score users by historical behavior, flag anomalies, and adjust limits dynamically. This requires ML infrastructure and is interview gold if you propose it as an optimization. Start simple, then layer it on.

Practising This with Rehurz

System design interviews test your ability to think through tradeoffs under cross-questioning. A rate limiter is a canonical problem because it combines architecture (where does it sit?), algorithms (which one?), distributed systems (Redis, failure modes), and API design (HTTP status codes, retry headers).

On Rehurz, you practice this as a spoken technical round. You explain your design choice for algorithm and placement, then your interviewer probes: "What happens if Redis goes down? How would you handle a burst from a trusted partner? Can a user with multiple IPs bypass your limit?" You defend your tradeoffs in real time, adapt when challenged, and think out loud like you would in a real interview. Your feedback report shows where you missed edge cases and curates resources to close those gaps.

Try a live technical interview. Your first round is free.

---

Rate limiting is not glamorous, but it's unglamorous in the way infrastructure often is: foundational and everywhere. Master it, and system design interviews become less about luck and more about pattern recognition.

Your first interview is free, no card required. Start your first interview.