Skip to content

Kafka command-line operations

Kafka command-line operations refers to the suite of shell scripts provided by Kafka to manage the cluster lifecycle, configure server properties, and handle data streams via the console^[600-developer-big-data-kafka-kafka-01.md].

Server Lifecycle

To start a Kafka broker, the kafka-server-start.sh script is executed with the server properties configuration file^[600-developer-big-data-kafka-kafka-01.md].

bin/kafka-server-start.sh config/server.properties

To manage the running process in a shell environment, standard job control commands can be used: * Ctrl + Z: Suspend the process^[600-developer-big-data-kafka-kafka-01.md]. * bg: Resume the suspended process in the background^[600-developer-big-data-kafka-kafka-01.md]. * jobs -l: List background jobs^[600-developer-big-data-kafka-kafka-01.md]. * fg %1: Bring a specific job (e.g., job 1) to the foreground^[600-developer-big-data-kafka-kafka-01.md].

To stop the broker, the provided stop script is used^[600-developer-big-data-kafka-kafka-01.md].

bin/kafka-server-stop.sh

Console Producer and Consumer

Kafka provides command-line tools to interact with topics directly from the terminal for testing and debugging purposes^[600-developer-big-data-kafka-kafka-01.md].

To create a producer and send messages, the kafka-console-producer.sh script is used, specifying the broker list and the target topic^[600-developer-big-data-kafka-kafka-01.md].

bin/kafka-console-producer.sh --broker-list hadoop101:9092 --topic first

To read messages from a topic, the kafka-console-consumer.sh script connects to the Zookeeper ensemble and specifies the topic to consume^[600-developer-big-data-kafka-kafka-01.md].

bin/kafka-console-consumer.sh --zookeeper hadoop101:2181 --topic first

Broker Configuration

The server.properties file defines the critical behavior of the Kafka broker^[600-developer-big-data-kafka-kafka-01.md].

Key configuration parameters include^[600-developer-big-data-kafka-kafka-01.md]:

  • broker.id: A unique integer identifier for each broker in the cluster^[600-developer-big-data-kafka-kafka-01.md].
  • log.dirs: The directory where Kafka stores log files (e.g., /opt/kafka/logs)^[600-developer-big-data-kafka-kafka-01.md].
  • num.partitions: The default number of log partitions per topic^[600-developer-big-data-kafka-kafka-01.md].
  • zookeeper.connect: A comma-separated list of Zookeeper host:port pairs^[600-developer-big-data-kafka-kafka-01.md].
  • delete.topic.enable: A flag (set to true) to enable topic deletion^[600-developer-big-data-kafka-kafka-01.md].

Sources

  • 600-developer-big-data-kafka-kafka-01.md