8 Jun 2026 · Rehurz
Scaling to 10 Million Users: An Architectural Roadmap
You've launched. Your app works. Traffic grows. Then one day, a request takes 10 seconds to respond. Your database is at 95% CPU. Queries queue. You add a server. Traffic keeps growing. A week later, the same crisis repeats.
This is the scaling problem. How do you architect a system that grows reliably from one thousand users to ten million, removing bottlenecks as they appear without overbuilding for problems you don't have yet?
Quick answer: Start monolithic on one server, then evolve through six stages: separate compute and data, load-balance stateless app servers, replicate reads, add caching layers, introduce async work via message queues, and finally shard the database when write volume becomes the limit. At each stage, measure first, identify the bottleneck, fix it, and accept the new bottleneck as the cost of the next stage.
Why Scaling Matters (and When It Doesn't)
Scaling is a word that gets misused. It doesn't mean "make the code run fast." It means "keep the system responding as the number of simultaneous users, the request volume, and the data size all grow together."
Most startups don't need to think about scaling to 10 million users. They need to reach 100 users first. But system design for scale teaches you architecture patterns that matter much earlier: why statelessness is non-negotiable, how to identify bottlenecks before they break you, and why premature optimization wastes time.
Scaling is not one problem. It is a sequence of bottlenecks. Each solution creates a new bottleneck downstream. Understanding what breaks at each stage teaches you to build intentionally.
Stage 1: The Monolith (1 to 100 Thousand Users)
You have one server. On it: your application code, your database, your cache, everything. No load balancer. No replicas. One IP address.
This works for a very long time. A modest cloud instance handles thousands of concurrent users if your code is reasonably efficient. Twitter ran on a monolith for years.
The bottleneck: CPU and memory on a single machine. When you hit the instance's ceiling (8 cores, 32 GB), you must choose: upgrade the machine (vertical scaling) or split the work.
When to stay: if you have less than a million users and your database fits on one machine, a monolith is faster to ship and cheaper to operate.
When to leave: when your instance type maxes out or when vertical scaling costs more than horizontal scaling.
Stage 2: Separate App and Database (100 Thousand to 1 Million Users)
Your first split: move the database to a different machine.
Your application server no longer competes with the database for CPU and memory. You can now scale them independently. Need a bigger database? Upgrade that server. Need more app throughput? Add another app server (though without a load balancer, you can't yet use it).
The bottleneck: now two bottlenecks coexist. The app server's CPU under request load. The database's CPU under query load. Whichever you neglect first will break you.
Why this matters: separating concerns is the foundation of all scaling. Monoliths can scale vertically up to a point, but distributed systems scale horizontally. This is your first step.
Stage 3: Stateless Apps and a Load Balancer (1 to 5 Million Users)
Now you have three app servers. But they are not connected to a load balancer, so users don't distribute across them. Most traffic lands on one.
You add a load balancer (a hardware box or cloud service that distributes traffic by round-robin or connection count). Behind it: three identical, stateless app servers. Stateless means each server contains no user session data. If a user's session is stored in the server's memory, they must always hit the same server. Instead, store sessions in a shared layer: a cache or database.
This is the hardest conceptual shift. Many junior engineers want to store session data in memory. This immediately prevents load balancing. The rule: app servers are cattle, not pets. They are interchangeable.
The bottleneck: the database now handles both application queries and session reads. Database CPU spikes. Queries start queuing.
Cost of this stage: a load balancer, session store setup (Redis or similar), and the discipline to keep application code stateless.
Stage 4: Read Replicas and Caching (5 to 20 Million Users)
Your database is now the bottleneck. Write queries (inserts, updates) still go to the primary. But read queries (selects) can scatter to read replicas, which are read-only copies of the primary.
This roughly triples your read throughput without increasing write capacity. But most production systems are read-heavy. Seventy to ninety percent of queries are reads. Read replicas buy you a lot.
Alongside this, add a cache layer (Redis or Memcached). Expensive queries (joins across 10 tables, sorting 100,000 rows) are computed once and cached for 5 to 60 seconds. Subsequent requests hit the cache, which responds in microseconds.
The bottleneck: write throughput. The primary database still serializes all writes. If your application needs to write 50,000 times per second, a single database instance cannot keep up.
Stage 5: Message Queues and Async Work (10 to 50 Million Users)
Not all writes need to be synchronous. If a user uploads a profile photo, you don't need to resize the thumbnail, optimize it, and store it before returning a 200 OK. Instead:
- Store the photo in durable storage.
- Add a job to a message queue: "resize photo #12345."
- Return 200 OK.
- A background worker (one or more servers) consumes the job and processes it asynchronously.
This decouples fast paths from slow paths. The API responds quickly. The expensive work happens in the background.
Message queues also let you smooth traffic spikes. If you receive 100,000 requests in one second but can only process 10,000 per second, the queue buffers the rest. Workers process them as capacity allows.
The bottleneck: database write throughput persists, but less acutely, because you have eliminated needless synchronous writes. Remaining bottleneck: the scale of your data and the complexity of your queries.
Stage 6: Database Sharding (20 to 100+ Million Users)
At 20 million users, your primary database might hold 50 terabytes of data. Backups take 12 hours. A single transaction takes 5 minutes because it must scan billions of rows. Replication lag grows.
Sharding partitions the data horizontally: instead of one database holding all users, you have 10 databases, each holding users 1 to 1 million, 1 million to 2 million, and so on. Shards are independent. User 500,000's data lives on Shard 5 only.
This increases your write capacity linearly with the number of shards. It also makes data smaller and queries faster on each shard.
The cost: sharding breaks transactions and joins across shards. A query that was one line of SQL becomes code that queries multiple shards, aggregates results, and handles partial failures.
The bottleneck: operational complexity and code architecture. Sharding is the last scaling lever. After this, you have hit the physical limits of distributed systems and must live with tradeoffs: consistency vs. availability, latency vs. durability.
Vertical vs. Horizontal Scaling: When Each Wins
Vertical scaling (buying a bigger machine) is simple and works until it doesn't. One machine has limits. The largest cloud instances are in the terabyte range. Beyond that, you buy a bigger database appliance, which costs exponentially more. Vertical scaling is also risky: one machine is one point of failure.
Horizontal scaling (adding more machines) is complex but resilient. You add load capacity almost linearly. A tenth server gives you 90 percent more capacity, not 10 percent. But it requires stateless design, distributed coordination, and operational discipline.
In practice, most systems blend both. You might vertically scale the database to 16 cores, then horizontally scale app servers across 50 instances.
Measuring Before Scaling
The worst scaling mistake is to optimize based on intuition. "Our database queries are slow" is not actionable. "Queries average 500ms but peak at 5 seconds, with 20 percent of time spent on a single join" is. Measure first.
Use these tools:
- Application profiling: identify which functions consume the most CPU.
- Database profiling: which queries are slowest? Which consume the most IO?
- Request tracing: which requests are slowest end-to-end?
- Load testing: generate expected traffic and see where the system breaks.
Measurement often reveals that your bottleneck is not where you think. Many teams have optimized a function that runs 0.1 percent of the time, while a function that runs 30 percent of the time goes untouched. Measurement prevents this waste.
Architectural Growth: A Visual Guide
Here is how the system evolves from Stage 1 to Stage 6:
Stage 1: Monolith
+-------------------+
| App + DB + Cache |
+-------------------+
Stage 2: Split Compute and Data
+----------+ +----------+
| App | | Database |
+----------+ +----------+
Stage 3: Load Balanced Apps
[Load Balancer]
/ | \
+---+ +---+ +---+
|App| |App| |App|
+---+ +---+ +---+
|
+--------+
|Database|
+--------+
Stage 4: Read Replicas and Cache
[Load Balancer]
/ | \
+---+ +---+ +---+
|App| |App| |App|
+---+ +---+ +---+
| [Cache]
+----+----+----+
|
+--+--+
| |
[Prim] [Read]
[Replicas]
Stage 5: Message Queue
[LB]
/ | \
+---+ +---+
|API| |API|
+---+ +---+
|
[Queue]
|
+---------+
|Workers |
+---------+
|
[Cache] ---- [DB Primary]
|
[Replicas]
Stage 6: Database Sharding
[LB]
/ | \
+--+--+
|API |
+--+--+
|
[Cache]
|
+--+--+--+
|Sh|Sh|Sh| ... (Shards 1-10)
+--+--+--+
Stage by Stage: Bottlenecks and Fixes
Stage | Bottleneck | Fix
------+----------------------+--------------------------------
1 | Machine resources | Vertical scale / split layers
2 | Database CPU+memory | Read replicas + caching
3 | Database reads | Caching + read optimization
4 | Sync write latency | Async work + message queues
5 | DB write throughput | Sharding + data partitioning
6 | Operational complex | Accept tradeoffs, scale org
Common Mistakes (and How to Avoid Them)
Premature optimization: Building for 100 million users when you have 1,000 wastes months. Build for 10x your current scale, not 100x.
Misidentifying the bottleneck: Optimize the wrong thing and nothing improves. Always measure.
Over-caching: A cache is only useful if queries miss the cache frequently enough to justify invalidation complexity. Over-caching wastes memory.
Underestimating operational complexity: Sharding is not a code problem; it is an operational one. You must monitor shard balance, rebalance when one shard grows too large, and handle shard failures. Factor in this cost before committing.
Ignoring failure modes: A distributed system fails partially: one shard is down, one read replica is slow, one load balancer is overloaded. Code must handle these gracefully. This is harder than it sounds.
Frequently Asked Questions
How do I know when to scale?
When your system is at 60 to 70 percent capacity, start designing for the next stage. By the time you hit 90 percent, you are in crisis mode. If you measure continuously, you have runway to plan.
Should I use a managed database or run my own?
Managed (RDS, Cloud SQL, Supabase) handles backups, failover, and patches. You pay a premium. Running your own gives control and saves cost but adds operational burden. For early-stage systems, managed is usually better. For scale past a million users, many teams run their own.
How do I handle sharding complexity?
Most teams use a sharding library or ORM abstraction (Vitess, CockroachDB, or Citus Postgres). These handle routing, rebalancing, and cross-shard queries. Building sharding from scratch is rarely worth it.
Is caching always good?
Caching is only useful when the cost of computing the value exceeds the cost of invalidating it. A cache hit must be faster and cheaper than a database read. In practice, this is true for most read-heavy systems, but not all.
Can I scale to a million users on a single database?
Yes, with proper indexing, query optimization, and read replicas. Many million-user systems use a single primary database. The constraint is operational burden, not technical feasibility. A single database is easier to reason about.
Practising System Design with Rehurz
System design is a core technical skill in engineering interviews. The systems you build in interviews mirror the scaling challenges you will face in production: from monolithic design to multi-tier architecture, from single points of failure to resilient distributed systems.
Rehurz provides a spoken system design round. You talk through your architecture, defend tradeoffs, and answer sharp cross-questioning from an interviewer who catches inconsistencies and probes edge cases. The feedback captures not just your final design but your reasoning under pressure, your ability to navigate tradeoffs, and how you react when challenged.
Get started with your first free interview.
Scaling is not magic; it is a sequence of architectural decisions driven by bottleneck identification and measurement. Master these patterns and you can architect systems for any scale.