How Kafka turned data into a river you can replay

Most systems treat data as state — a row in a database that you overwrite when it changes. Apache Kafka flips this around: it treats data as an endless log of events, immutable facts like "order #42 placed" or "sensor read 30°C." This is called event streaming, and Kafka is the battle-tested platform that made it mainstream, originally built at LinkedIn to move firehoses of activity data.

The core trick is deceptively simple. Producers append events to the end of a named topic, and Kafka stores them durably on disk as an ordered, append-only log. Consumers read forward through that log at their own pace, each tracking its own position (offset). Because the log isn't deleted when it's read, ten different services — a fraud checker, a dashboard, a data warehouse loader — can all replay the same stream independently, even reprocessing old events after a bug fix.

Tap to zoom

Kafka runs as a distributed cluster of brokers, splitting each topic into partitions spread across machines for scale and replicated for fault tolerance. That combination — durable storage plus publish/subscribe plus stream processing — is why Kafka became the backbone of event-driven architectures, connecting databases, microservices, and analytics through one always-on data pipeline.