Apache Kafka

Apache Kafka's architecture is built around a distributed commit log, where records are organized into topics and partitioned across brokers for parallelism and scalability. Each partition is an ordered, immutable sequence of records, and producers append records while consumers read them at their own pace. This design allows Kafka to provide both pub/sub messaging and durable storage, with retention policies that can keep data for a configured period or size.

Kafka relies on ZooKeeper (or, in newer versions, KRaft mode) for cluster coordination, managing metadata such as broker membership and topic configurations. The system achieves fault tolerance through replication: each partition has a leader and multiple followers, and if a leader fails, a follower is elected to take over. This ensures high availability and durability even in the face of broker failures.¹

Kafka is often used as the central nervous system for event-driven architectures, connecting microservices, data lakes, and stream-processing engines. It powers use cases such as log aggregation, metrics collection, activity tracking, and change data capture (CDC). The Kafka Connect framework simplifies integration with external systems, while Kafka Streams provides a lightweight library for building stream-processing applications in Java and Scala.

Major companies, including LinkedIn, Netflix, Uber, and Airbnb, have adopted Kafka for real-time data pipelines. It is also a key component in many open-source projects, such as Apache Flink and Apache Spark, which can consume from and produce to Kafka topics. The platform's ability to handle millions of events per second with low latency makes it suitable for both online and offline data processing.²

Kafka's origins trace back to a team at LinkedIn led by Jay Kreps, Neha Narkhede, and Jun Rao, who named it after the writer Franz Kafka because the system was "optimized for writing" — a nod to the author's literary output. The project was open-sourced in 2011 and became a top-level Apache project in 2012.

One lesser-known feature is Kafka's support for exactly-once semantics (EOS) in stream processing, which was a major engineering effort and is still considered advanced. Another is the "tombstone" record, a special null-value message used in log compaction to mark a key for deletion. Kafka also has a built-in "mirror maker" tool for cross-datacenter replication, and it can be used as a distributed database of sorts, though it lacks query capabilities. The KRaft mode, introduced in 2.8, removes the ZooKeeper dependency, simplifying operations and improving scalability.

Running Kafka in production requires careful tuning of brokers, producers, and consumers to balance throughput, latency, and durability. Key parameters include replication factor, acks, and retention settings. Monitoring is critical, and tools like Burrow and Kafka Monitor help track consumer lag and broker health.

Kafka faces competition from other messaging systems such as RabbitMQ, Apache Pulsar, and Amazon Kinesis. Pulsar, for instance, offers a different architecture with separate storage and serving layers, while Kinesis is a fully managed cloud service. However, Kafka's maturity, ecosystem, and community support keep it a dominant choice. The Confluent company, founded by the original creators, offers a commercial platform and cloud service built on Kafka.³

Broker

A server that stores and serves Kafka partitions.

Topic

A named channel to which records are published.

Partition

A unit of parallelism within a topic, an ordered log.

Consumer group

A set of consumers that jointly consume a topic's partitions.

Log compaction

A retention mechanism that keeps only the latest value for each key.