- 01 Terminology Glossary
- 02 Real-Time Chat (WhatsApp)
- 03 Video Streaming (YouTube)
- 04 Low-Latency Systems
- 05 Ordered Message Queues
- 06 URL Shortener
- 07 Rate Limiter
- 08 Notification System
- 09 News Feed
- 10 Offline Messaging
- 11 Circuit Breaker
- 12 Self-Healing Infrastructure
- 13 Scaling to 5M Users
- 14 Handling 5M Requests
- 15 End-to-End Encryption
- 16 Real-Time React Dashboard
- 17 Fast-to-Slow File Transfer
- 18 Ride-Hailing (Uber, Ola)
- 19 Ticket Booking (BookMyShow)
- 20 Unique ID Generator
Blueprint 14 · 9 diagrams
Design an Elastic System That Scales up to 5 Million Requests Without Breaking
Elastic system that scales up and down to 5 million requests without breaking
1. Summary
The system must handle a load that changes all the time. At night, the load can be a few thousand requests. During a sale or a live event, the load can increase to 5 million requests. The system must not break at any load level.
"Not breaking" does not mean "serve every request at any cost." It means:
- The system scales up when the load increases.
- The system scales down when the load decreases. Then the cost stays low.
- When the load increases faster than the system can scale, the system stays alive. It serves the most important requests and rejects or delays the others in a controlled way.
- The system recovers automatically when the load becomes normal.
The design uses three principles:
| Principle | Meaning | Main tools |
|---|---|---|
| 1. Scale | Add capacity when the load increases. Remove capacity when the load decreases. | Autoscaling, cells, serverless, on-demand databases |
| 2. Absorb | Hold or serve extra load at cheap layers while the system scales. | CDN, cache, queues, request coalescing |
| 3. Protect | Keep the core system alive when the load is more than the capacity. | Rate limits, load shedding, backpressure, circuit breakers, degradation |
2. Clarify the load
In an interview, ask for the time unit. This document uses 5 million requests per second (RPS) as the maximum peak. The same design works for "5 million per minute" (approx. 83,000 RPS) with fewer cells.
| Requirement | Peak RPS | Design size |
|---|---|---|
| 5 million per day | approx. 175 | Small. Autoscaling of a few servers is sufficient. |
| 5 million per hour | approx. 4,000 | Medium. See 13-scaling-5-million-users.md. |
| 5 million per minute | approx. 83,000 to 250,000 | Large. One region, 2 to 3 cells. |
| 5 million per second | 5,000,000 | Very large. Many regions, many cells. |
Ask also:
- What is the minimum load? (This decides the minimum cost.)
- How fast can the load increase? (For example, from 500,000 to 5 million RPS in 1 minute.)
- Are the peaks known in advance (a sale, a match), or are they random (viral content)?
- Which requests must never fail (payment, login)? Which requests can wait or fail (recommendations, analytics)?
3. Requirements
Functional requirements
- Serve normal API traffic (90% reads, 10% writes).
Non-functional requirements
| Requirement | Target |
|---|---|
| Load range | From approx. 10,000 RPS to 5,000,000 RPS |
| Spike speed | 10 times the current load within 60 seconds |
| Availability | 99.99% for critical requests |
| Latency | p99 less than 200 ms during normal operation |
| Behavior at overload | No crash. No cascading failure. Critical requests continue. |
| Cost | Cost follows the load. Low cost at low load. |
4. Load levels and system behavior
The system has a defined behavior for each load level.
flowchart LR
L0["LOW: 10K to 100K RPS. Minimum capacity. Low cost."] --> L1["NORMAL: 100K to 1M RPS. Autoscaling active."]
L1 --> L2["HIGH: 1M to 5M RPS. Pre-scaled cells. All layers at high use."]
L2 --> L3["OVERLOAD: more than capacity. Shed low-priority load. Degrade features."]
L3 --> L2
L2 --> L1
L1 --> L0
| Level | Load | What the system does |
|---|---|---|
| Low | 10K to 100K RPS | Runs at minimum capacity. Uses committed (cheap) capacity. Some cells scale in. |
| Normal | 100K to 1M RPS | Reactive autoscaling adds and removes instances. |
| High | 1M to 5M RPS | All cells are active. Scheduled and predictive scaling add capacity before the peak. |
| Overload | More than current capacity | Admission control and load shedding protect the core. Non-critical features stop. Queues hold writes. |
5. High-level architecture
flowchart TB
U["Clients"] --> DNS["GeoDNS + Anycast"]
DNS --> EDGE["CDN edge + WAF + DDoS protection + edge rate limits"]
EDGE -- "cache hit (absorb)" --> U
EDGE --> GW["Regional L4 load balancer + L7 gateway (admission control)"]
GW --> CR["Cell router"]
CR --> C1["Cell 1"]
CR --> C2["Cell 2"]
CR --> CN["Cell N (added when load grows)"]
subgraph "One cell (scaling unit)"
SVC["Stateless services (autoscaling)"] --> L1C["In-process cache"]
SVC --> RC[("Redis cluster")]
SVC --> K["Kafka / queue (absorb writes)"]
SVC --> DB[("Database shards")]
K --> WK["Workers (autoscale on queue lag)"]
WK --> DB
end
CTRL["Scaling controller: reactive + scheduled + predictive"] --> GW
CTRL --> CR
MON["Metrics: RPS, latency, errors, saturation, queue lag"] --> CTRL
Request funnel at peak
Each layer removes load before the next layer. The database never gets the full client load.
flowchart TB
A["5,000,000 RPS from clients"] --> B["Edge / CDN serves approx. 60%"]
B --> C["approx. 2,000,000 RPS reach regions"]
C --> D["Gateway rejects abuse and over-limit traffic"]
D --> E["approx. 1,900,000 RPS reach cells"]
E --> F["Caches serve approx. 90% of reads"]
F --> G["approx. 170,000 reads/s reach databases"]
E --> H["approx. 190,000 writes/s go to queues"]
H --> I["Workers write to databases in batches"]
6. Principle 1: Scale
6.1 Each layer scales at a different speed
This is the most important point of the design. A spike can come in seconds. Some layers need minutes or hours to scale.
| Layer | Time to add capacity | Strategy |
|---|---|---|
| CDN / edge | Immediate (provider capacity) | Cache as much as possible. |
| Serverless functions | Seconds (with concurrency limits) | Use for spiky, short tasks. Request higher limits before events. |
| Containers on existing nodes | 10 to 60 seconds | Reactive autoscaling. |
| New nodes (VMs) | 1 to 5 minutes | Keep a warm pool and spare nodes. |
| Cache cluster | Minutes; resharding is slow | Size for peak. Keep headroom. |
| Database read replicas | 5 to 20 minutes | Scale before known peaks. |
| Database shards / write capacity | Hours to days | Plan for peak. Use on-demand modes where possible. |
| Cloud quotas and limits | Days (support request) | Request higher limits in advance. |
Rule: If a layer scales slowly, give it headroom (spare capacity) or put a buffer in front of it.
6.2 Cells: the unit of scale
A cell is a complete, independent copy of the stack. One cell serves a fixed group of users.
| Item | Value |
|---|---|
| Tested capacity of one cell | approx. 100,000 RPS |
| Cells at low load | 2 to 4 (minimum for availability) |
| Cells at peak | approx. 20 to 26 (2 million RPS after the edge + 30% spare) |
Why cells help elasticity
- Known capacity. You test one cell to its limit. Then capacity planning is simple: required RPS ÷ cell capacity = number of cells.
- Linear growth. To support more load, add more cells. You do not make one big system bigger.
- Small blast radius. A failure in one cell affects only approx. 5% of users.
- Cost control. At low load, the cells run with few instances. Their minimum size is small.
6.3 Autoscaling methods
Use three methods together.
flowchart LR
S1["Scheduled scaling: known events (sale at 12:00)"] --> CAP["Desired capacity"]
S2["Predictive scaling: forecast from past daily and weekly patterns"] --> CAP
S3["Reactive scaling: current CPU, RPS, queue lag, latency"] --> CAP
CAP --> MAX{"Take the HIGHEST value"}
MAX --> ACT["Scale instances, nodes, workers"]
| Method | When it works | Example tools |
|---|---|---|
| Scheduled | Known peaks (sales, matches, product launches, 9:00 AM traffic) | Kubernetes CronJobs, AWS scheduled scaling |
| Predictive | Regular daily or weekly patterns | AWS Predictive Scaling, custom forecast models |
| Reactive | Unknown or random changes | Kubernetes HPA, KEDA, Cluster Autoscaler, Karpenter |
Reactive scaling signals
| Component | Scale on | Reason |
|---|---|---|
| API services | RPS per instance and CPU | CPU alone reacts late for I/O-heavy services. |
| Workers | Queue lag (messages waiting) | Lag shows directly that workers are too slow. |
| WebSocket gateways | Open connections | Connections use memory, not CPU. |
| Database replicas | CPU and replica lag | Lag shows the replica cannot follow the primary. |
6.4 Scale up fast, scale down slowly
| Rule | Value | Reason |
|---|---|---|
| Scale-up step | Large (for example, +50% or +100%) | A spike needs capacity now. Small steps are too slow. |
| Scale-up delay | Short (15 to 30 seconds) | Act quickly. |
| Scale-down step | Small (for example, -10%) | The load can come back. |
| Scale-down delay | Long (5 to 10 minutes stabilization window) | Prevents "flapping" (up and down many times). |
| Scale-down action | Drain connections. Finish current requests. Then stop. | Prevents errors for users. |
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-service
spec:
scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: api-service }
minReplicas: 6 # Low load: small and cheap, but available in 3 zones
maxReplicas: 600 # Cost and safety limit
metrics:
- type: Pods
pods:
metric: { name: http_requests_per_second }
target: { type: AverageValue, averageValue: "800" }
behavior:
scaleUp:
stabilizationWindowSeconds: 15
policies:
- { type: Percent, value: 100, periodSeconds: 15 } # Double quickly
scaleDown:
stabilizationWindowSeconds: 600
policies:
- { type: Percent, value: 10, periodSeconds: 60 } # Decrease slowly
6.5 Make new capacity ready faster
- Small container images. A small image downloads faster on a new node.
- Fast startup. The service must be ready in seconds. Load configuration and warm connections at startup.
- Warm pools. Keep some stopped or idle nodes ready. They start faster than new nodes.
- Over-provisioning pods. Run low-priority "placeholder" pods. When real pods need space, Kubernetes removes the placeholders immediately. Then the cluster autoscaler adds nodes in the background.
- Cloud quotas. Check instance, IP address, and API limits. Request higher limits before peak events. An autoscaler cannot pass a quota.
6.6 Databases with variable load
| Need | Elastic option |
|---|---|
| Key-value with unpredictable load | DynamoDB on-demand mode, or provisioned mode with autoscaling |
| Relational with variable load | Aurora Serverless v2, Aurora replica autoscaling |
| Very large write scale | Pre-sharded Cassandra / ScyllaDB / Vitess, sized for peak |
| Read spikes | Caches + autoscaling read replicas |
Important rules:
- Use connection pooling (PgBouncer, RDS Proxy). When 500 new app instances start, they must not open 50,000 new database connections. A "connection storm" can stop the database.
- Create more shards than you need at the start (for example, 1,024 logical shards on 16 physical nodes). Later, move logical shards to new nodes. You do not need to split data.
- Plan write capacity for peak. Writes are the most difficult part to scale quickly. Put a queue in front of non-critical writes.
7. Principle 2: Absorb
Absorb layers give the system time to scale.
7.1 CDN and edge
- Cache static files and public API responses at the edge.
- Use short TTLs (1 to 10 seconds) for data that changes often. Even a 1-second TTL removes most of the load for very popular data.
- Use "stale-while-revalidate." The edge serves old data while it gets new data in the background.
- Use "stale-if-error." The edge serves old data if the origin fails.
7.2 Caches
flowchart LR
REQ["Request"] --> L1["L1: in-process cache"]
L1 -- miss --> L2["L2: Redis cluster"]
L2 -- miss --> CO{"Request coalescing: one request for each key"}
CO --> DB[("Database")]
- Request coalescing: If 10,000 requests ask for the same missing key, send one request to the database.
- Hot-key replication: Copy a very popular key to many cache nodes. Read from a random copy.
- Cache warming: Load popular data before a known peak.
7.3 Queues for writes
sequenceDiagram
participant C as Client
participant S as Service
participant Q as Kafka / queue
participant W as Workers
participant DB as Database
C->>S: Write request (idempotency key)
S->>Q: Store event (durable)
S-->>C: 202 Accepted
Note over Q: During a spike, the queue grows. Nothing breaks.
Q->>W: Events in batches
W->>DB: Batch writes at a safe rate
Note over W: Workers autoscale on queue lag
- The queue takes the spike. The database gets a steady rate.
- Workers scale on queue lag. When the spike ends, the queue becomes empty and workers scale down.
- Use idempotency keys. A retry must not write the same change two times.
- Keep critical writes (payments, stock changes) on a separate synchronous path with reserved capacity.
8. Principle 3: Protect
When the load is more than the capacity, the system must not try to serve everything. If it does, all requests become slow, timeouts occur, clients retry, and the system crashes. Protection keeps the system alive.
8.1 Protection layers
flowchart LR
R["Requests"] --> E["Edge: block bots, DDoS, per-IP limits"]
E --> G["Gateway: per-user and per-API-key rate limits"]
G --> AC["Admission control: is there capacity?"]
AC -- "yes" --> P{"Priority"}
AC -- "no, low priority" --> REJ["Fast reject: HTTP 429/503 + Retry-After"]
P -- "critical" --> RES["Reserved capacity pool"]
P -- "normal" --> SH["Shared capacity pool"]
RES --> SVC["Services with timeouts, bulkheads, circuit breakers"]
SH --> SVC
8.2 Controls
| Control | Function |
|---|---|
| Rate limiting | Limits each client, user, API key, and IP address. See 07-rate-limiter.md. |
| Admission control | Accepts a request only if the service has capacity (for example, concurrent requests less than the limit). |
| Load shedding | Rejects new low-priority requests quickly when the system is full. A fast rejection is better than a slow timeout. |
| Priority classes | Critical (login, checkout) > Normal (browse) > Low (recommendations, analytics, prefetch). |
| Bulkheads | Separate thread pools and connection pools for each dependency and priority. One slow part cannot use all resources. |
| Timeouts | Each call has a time limit. No request waits without limit. |
| Circuit breakers | Stop calls to a failing dependency. See 11-circuit-breaker.md. |
| Backpressure | When a queue is full, tell producers to slow down. |
| Retry budget | Retries are limited to approx. 10% of requests. This prevents a "retry storm." |
| Client jitter and backoff | Clients wait a random time before a retry. This prevents synchronized waves. |
8.3 Graceful degradation levels
Define the levels in advance. The system moves between levels automatically, based on load and error rate. Engineers can also change the level manually.
| Level | Trigger | Action |
|---|---|---|
| 0: Normal | Load less than 70% of capacity | All features on. |
| 1: Reduced | Load more than 80% | Turn off personal recommendations. Increase cache TTLs. Stop background prefetch. |
| 2: Essential | Load more than 95% or high error rate | Serve cached pages only for browsing. Queue all non-critical writes. Show simplified pages. |
| 3: Critical only | Severe overload | Allow only login, checkout, and payment. Show a "busy" page or a virtual waiting room for others. |
Use feature flags to turn features off quickly with no deployment.
8.4 Virtual waiting room
For extreme known events (ticket sales, limited product drops):
- Put users in a queue at the edge.
- Let users into the system at a rate the system can serve.
- Show each user their position and the estimated wait time.
This changes an uncontrolled spike into a controlled flow.
9. Spike timeline: from 500,000 to 5,000,000 RPS in 60 seconds
sequenceDiagram
participant T as Traffic
participant E as Edge / CDN
participant G as Gateway
participant A as Autoscaler
participant Q as Queues
participant S as Services
T->>E: Load increases 10 times
E->>E: 0 s: CDN serves cached responses (absorb)
E->>G: Remaining load
G->>S: 0 to 15 s: Services use spare headroom (30%)
S->>Q: Writes go to queues (absorb)
G->>G: 15 s: Admission control starts to shed low-priority requests (protect)
A->>S: 15 to 60 s: Pods scale up on existing and warm nodes (scale)
A->>A: 1 to 5 min: New nodes join. More pods start (scale)
G->>G: Degradation level goes to 1 or 2 if necessary (protect)
S->>S: 5 min: Capacity matches load. Shedding stops.
Q->>S: Workers process the queue backlog
G->>G: Degradation level returns to 0
Result: Some low-priority requests fail for a short time. Critical requests continue. The system does not crash. It returns to normal automatically.
10. Low load: keep the cost low
| Method | Description |
|---|---|
| Small minimum capacity | Keep only enough for availability (for example, 2 instances in each of 3 zones in each cell). |
| Committed pricing for the base | Use Savings Plans or Reserved Instances for the minimum load only. |
| Spot instances for the variable part | Stateless services and workers can use Spot. Keep a base of On-Demand capacity. |
| Scale cells in | At very low load, route users to fewer active cells. Keep the other cells at minimum size. |
| Serverless for spiky or rare work | Pay only when the code runs. |
| On-demand databases | DynamoDB on-demand or Aurora Serverless v2 for variable database load. |
| Turn off non-production at night | Development and test environments do not need to run 24 hours. |
flowchart TB
subgraph "Capacity mix"
SP["Spot: variable and peak load (cheapest, can be interrupted)"]
OD["On-Demand: flexible middle load"]
RI["Savings Plan / Reserved: always-on minimum"]
end
RI --> OD --> SP
11. Failures caused by scaling
Scaling itself can cause failures. Plan for these.
| Failure | Cause | Prevention |
|---|---|---|
| Connection storm | Hundreds of new instances connect to the database at the same time. | Connection pooler. Slow start with jitter. |
| Cold cache stampede | A cache restart or a new cell sends all reads to the database. | Cache warming. Request coalescing. Gradual traffic shift to new cells. |
| Quota limit | The cloud account reaches its instance or IP limit. | Request higher quotas in advance. Alert at 70% of quota. |
| Slow new instances | Large images or slow startup. | Small images. Warm pools. Fast readiness. |
| Flapping | The autoscaler adds and removes capacity many times. | Long scale-down stabilization window. |
| Hot shard | One key or shard gets most of the load. | Good shard key. Hot-key replication. Split hot partitions. |
| Retry storm after recovery | All clients retry at the same time. | Retry budgets. Exponential backoff with jitter. |
| Dependency limit | A third-party API cannot scale with you. | Rate limit calls to it. Cache its results. Queue requests. Use a circuit breaker. |
12. Capacity plan by load level
| Item | 100K RPS | 1M RPS | 5M RPS (peak) |
|---|---|---|---|
| Load after CDN (approx. 40%) | 40K | 400K | 2M |
| Active cells | 2 | 5 to 6 | 20 to 26 |
| API instances (approx. 8,000 RPS each) | 6 to 10 | 50 to 70 | 250 to 330 |
| Redis nodes | 6 | 20 to 30 | 80 to 120 |
| Database reads after cache | approx. 4K/s | approx. 35K/s | approx. 170K/s |
| Writes into queues | approx. 4K/s | approx. 40K/s | approx. 190K/s |
| Regions | 1 to 2 | 2 to 3 | 3+ |
These numbers are approximate. Real values come from load tests of one cell.
13. Testing
You must prove that the system does not break. Test each behavior.
| Test | Purpose |
|---|---|
| Load test | Find the real capacity of one instance and one cell. |
| Step test | Increase the load in steps. Find where latency starts to increase. |
| Spike test | Increase the load 10 times in 60 seconds. Check absorb, shed, and scale behavior. |
| Soak test | Run high load for several hours. Find memory leaks and slow degradation. |
| Scale-down test | Decrease the load. Check that instances drain with no user errors. |
| Chaos test | Stop instances, a zone, a cache node, or a dependency during load. |
| Game day | Practice a peak event with the full team before the real event. |
Tools: k6, Gatling, Locust, JMeter, AWS Distributed Load Testing, Chaos Mesh, AWS Fault Injection Service.
14. Monitoring
| Signal | Examples |
|---|---|
| Traffic | RPS by cell, region, endpoint, and priority class |
| Latency | p50, p99, p99.9 |
| Errors | Error rate, shed rate (requests rejected on purpose) |
| Saturation | CPU, memory, connections, cache hit rate, queue lag, replica lag |
| Scaling | Desired vs actual instances, time to scale, pending pods, quota use |
| Degradation | Current degradation level and active feature flags |
| Cost | Cost per million requests, cost per cell |
Alert on user symptoms (errors and latency). Show the shed rate separately from real errors. A controlled rejection is not a failure of the system.
15. Trade-offs
- Headroom vs cost: Spare capacity handles spikes immediately. But it costs money at low load.
- Asynchronous writes vs consistency: Queues absorb spikes. But the data becomes eventually consistent.
- Load shedding vs user experience: Rejecting some requests keeps the system alive. But some users get errors.
- Cells vs simplicity: Cells give known capacity and a small blast radius. But they add routing and operations work.
- Predictive scaling vs accuracy: Forecasts add capacity early. But a wrong forecast adds cost or misses a spike. Always combine it with reactive scaling.
- Spot vs reliability: Spot is cheap. But it can be interrupted. Use it only for stateless and retryable work.
16. Interview follow-up questions
- The load increases 10 times in 30 seconds, but autoscaling needs 3 minutes. What happens in those 3 minutes?
- Which parts of the system cannot scale quickly, and how do you protect them?
- How do you decide which requests to reject during overload?
- How do you scale down without errors for users?
- How do you prevent a database connection storm when 500 new instances start?
- How do you keep the cost low at night when the load is only 2% of the peak?
- How do you prove that the system can handle 5 million RPS before the real event?