Skip to content

Apache Kafka core components

Apache Kafka is a distributed event streaming platform that functions as a high-throughput, fault-tolerant publish-subscribe system.^[600-developer-big-data-kafka-kafka-01.md]

Core Concepts

The architecture of Kafka relies on several key abstractions and components that work together to manage data streams.

  • Producer: The component (or client) responsible for sending (publishing) data records to the Kafka cluster.^[600-developer-big-data-kafka-kafka-01.md]
  • Consumer: The component (or client) that subscribes to topics and reads (consuming) data records from the Kafka cluster.^[600-developer-big-data-kafka-kafka-01.md]
  • Consumer Group: A mechanism where multiple consumers join a group to share the load of consuming data from topics, enabling scalability and fault tolerance.^[600-developer-big-data-kafka-kafka-01.md]
  • Broker: An individual Kafka server instance that forms part of the cluster; it receives data from producers and serves it to consumers.^[600-developer-big-data-kafka-kafka-01.md]
  • Topic: A specific category or feed name to which records are published and stored.^[600-developer-big-data-kafka-kafka-01.md]
  • Partition: A division of a topic that allows a topic to be split across multiple brokers for parallelism and scalability.^[600-developer-big-data-kafka-kafka-01.md]
  • Offset: A unique identifier assigned to every record within a partition, which acts as a pointer to the specific location of that record.^[600-developer-big-data-kafka-kafka-01.md]

Sources

  • 600-developer-big-data-kafka-kafka-01.md