One busy chat room can melt a giant database

When Discord outgrew MongoDB, it moved messages to Cassandra, a database designed to spread data across many machines. The idea: pick a partition key, hash it, and let each server own a slice of the keyspace. Discord bucketed messages by channel (plus a time window), so every channel's history lived together — great for the common query of "show me this channel's recent messages."

The trap was the hot partition. Cassandra sends every read and write for a given key to the same replicas. A tiny, hyperactive channel — think a massive announcement server where everyone posts at once — funneled all its traffic onto one unlucky set of nodes. Those nodes got buried while the rest of the cluster sat nearly idle. This is data skew: the data was balanced by count, but not by access.

Adding more servers doesn't fix this, because the bottleneck is one key, not total capacity. Discord's answer was partly architectural (they later moved to ScyllaDB) and partly a clever intermediary: a request coalescing layer that collapsed many simultaneous reads of the same hot channel into a single database query, then fanned the result back out.

The lesson generalizes far beyond chat: in any sharded system, your slowest component is decided by your most skewed key, not your average load.