# Apache Kafka Architecture

Apache Kafka is a distributed event streaming platform used to publish, store, process, and consume streams of records in real time. It supports the publish-subscribe messaging model while providing high throughput, scalability, fault tolerance, and durability.   
   
**Why Kafka?**

Traditional applications may fail when they receive a sudden spike in events because the application cannot process every request immediately. Kafka acts as a durable buffer between producers and consumers, allowing applications to ingest and process millions of messages per second while scaling horizontally across multiple servers.

### **Kafka vs Traditional Message Queues**

*   Kafka retains messages for a configurable period (or indefinitely).
    
*   Consumers maintain their own offsets and can replay messages.
    
*   Multiple consumer groups can independently consume the same data.
    
*   Kafka is optimized for high-throughput event streaming rather than simple task queues.
    

### **Core Components**

![](https://cdn.hashnode.com/uploads/covers/6725cc5ed900c71437ff6562/18047f5f-803b-4c50-aa8f-f1457ada0408.png align="center")

**Kafka Cluster:** A collection of brokers working together to provide scalability, fault tolerance, and high availability.

**Broker:** A Kafka server that stores partitions, serves producer/consumer requests, and replicates data.

**Producer:** Client application that publishes records to Kafka topics.

**Topic:** A logical stream of records to which producers write and consumers subscribe.

**Partition:** An append-only ordered log. FIFO ordering is guaranteed only within a partition. Each record receives a unique offset within that partition.

**Replication:** Each partition has one leader and zero or more follower replicas. Producers write only to the leader. Followers replicate the leader's log for fault tolerance.

**Consumer:** Application that polls records from topics.

**Consumer Group:** A set of consumers cooperating to process a topic. Within a consumer group, one partition is assigned to only one consumer instance at a time. Different consumer groups can read the same records independently.

### How Messages Are Distributed

Kafka uses a partitioner to determine the destination partition: • Key-based partitioning: hash(key) % number\_of\_partitions. • Sticky partitioning (default when no key is provided). • Explicit partition specified by the producer. • Custom partitioner implementing application-specific logic.

![](https://cdn.hashnode.com/uploads/covers/6725cc5ed900c71437ff6562/7c35b4bc-1d73-4a87-b2f7-c22cea1b65f1.png align="center")

**Message Lifecycle**

1.  Producer sends a record to the leader of the selected partition.
    
2.  The leader appends the record to the end of the partition and assigns the next offset.
    
3.  Follower replicas copy the new record from the leader.
    
4.  The producer receives an acknowledgement based on the configured acks setting (0, 1, or all).
    
5.  Consumers poll records from the leader.
    
6.  After processing, consumers commit their offsets. The message is not deleted.
    

![](https://cdn.hashnode.com/uploads/covers/6725cc5ed900c71437ff6562/78981fef-ecdd-4f11-9544-d01fbde1c655.png align="center")

### Offsets and Retention

An offset is the position of a record within a partition, not a global message ID. Offsets allow consumers to resume reading after restarts. Reading or committing an offset does not remove the record. Kafka deletes records only when the configured retention policy or log compaction rules apply.

### Common Use Cases

Data pipelines

Event-driven microservices

Real-time analytics

Stream processing

Streaming ETL

Log aggregation

IoT data ingestion

### References

1.  Apache Kafka Official Documentation
    
    1.  [What Is Apache Kafka®? Architecture, Use Cases, and How It Works](https://www.confluent.io/what-is-apache-kafka/#how-does-confluent-relate-to-kafka)
        
2.  GFG
    
    1.  [Kafka Architecture - GeeksforGeeks](https://www.geeksforgeeks.org/apache-kafka/kafka-architecture/)