Skip to content

Apache Kafka

Apache Kafka is a distributed event streaming platform used for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.^[600-developer-kafka-kafka.md]

Architecture

The core architecture of Kafka is designed around the principle of committing logs to facilitate the transmission of information between consumers and producers.^[600-developer-kafka-kafka.md]

Key Components

The system relies on several core components to function:

  • Producer: The component responsible for publishing (writing) data to the Kafka topics.^[600-developer-kafka-kafka.md]
  • Consumer: The component responsible for subscribing to (reading) data from Kafka topics.^[600-developer-kafka-kafka.md]
  • Broker: The server nodes that form the Kafka cluster. Brokers handle the storage of data and serve requests from producers and consumers.^[600-developer-kafka-kafka.md]
  • Topic: A specific stream or category to which records are published.^[600-developer-kafka-kafka.md]
  • Partition: A division of a topic that allows for scalability and parallelism.^[600-developer-kafka-kafka.md]

Design Principles

Kafka's architecture is defined by the following mechanisms:

  • Committing Logs: The fundamental mechanism used to persist data and ensure reliability.^[600-developer-kafka-kafka.md]
  • Partitioning: Data is divided across partitions, which enables the system to handle large volumes of data by distributing the load.^[600-developer-kafka-kafka.md]
  • [[Message Broker]]
  • [[Event-Driven Architecture]]

Sources

^[600-developer-kafka-kafka.md]