System Design: Scaling WebSockets to 1 Million Concurrent Users

Building a chat app for 100 users is a weekend project. Building a real-time notification system for 1 million concurrent users is an engineering nightmare. Unlike HTTP requests which spin up, serve data, and die, WebSockets keep a persistent TCP connection open. This fundamentally changes how you design your infrastructure.

The C10K Problem is Dead, Meet C1M

A single modern Linux server can handle millions of connections if tuned correctly (bumping up file descriptor limits via ulimit and tweaking TCP keepalive settings). However, the bottleneck isn't usually the OS; it's the application memory footprint. If your Node.js WebSocket server allocates 50KB of memory per connection, 1 million users require 50GB of RAM just to sit idle.

The Load Balancer Dilemma

You can't just slap a standard HTTP load balancer in front of WebSockets. If User A connects to Server 1, and User B connects to Server 2, how do they chat? They are physically connected to different machines.

This is where the Pub/Sub (Publish-Subscribe) pattern is mandatory. We utilized Redis Pub/Sub as the central nervous system. When User A sends a message to a chat room, Server 1 publishes that message to a Redis channel. Server 2, which is subscribed to that channel, receives the message and pushes it down the WebSocket pipe to User B.

Handling Connection Storms

The scariest moment is a deployment or a brief network blip. If your servers restart, 1 million clients disconnect simultaneously. Two seconds later, 1 million clients attempt to reconnect simultaneously. This "Thundering Herd" will instantly crash your infrastructure.

We solved this by implementing "jitter" on the client side. If a connection drops, the client doesn't reconnect immediately. It waits a random amount of time between 1 and 30 seconds before retrying. This smooths out the reconnection spike, giving our servers breathing room to establish the TCP handshakes gracefully.