Skip to content

Kafka manager deployment

Kafka manager deployment refers to the process of setting up and running the Kafka Manager application, a tool used for managing Apache Kafka clusters. In containerized environments, this typically involves building a container image, defining Kubernetes resources, and configuring network access via Ingress.

Architecture and Dependencies

Kafka Manager is a web-based tool that requires connectivity to a Zookeeper ensemble to function, as it relies on Zookeeper to discover and manage Kafka brokers^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

The standard deployment architecture involves: * Zookeeper: The backend coordination service (e.g., zk1.od.com:2181). * Kafka Manager: The application container running the web interface.

Container Image Creation

The deployment begins by building a Docker image for the application. A custom Dockerfile is typically used to install the necessary dependencies (like Java/Scala) and the Kafka Manager software itself^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

The build process often involves: 1. Using a base image with Scala and SBT (e.g., hseeberger/scala-sbt). 2. Downloading the specific Kafka Manager release source code. 3. Compiling the application using SBT. 4. Packaging it into the final image format^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

Once built, the image is pushed to a container registry (e.g., Harbor) to be pulled by the Kubernetes cluster^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

Kubernetes Configuration

Deployment on Kubernetes is managed using standard resource manifests.

Deployment Manifest

The Deployment resource defines the application's desired state^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]: * Replicas: Typically set to 1. * Environment Variables: * ZK_HOSTS: Configured with the address of the Zookeeper service. * APPLICATION_SECRET: A secret key used for authentication and encryption within the application (often set to a default like letmein for initial setups). * Ports: The application listens on port 9000.

Service and Networking

A Service is created to expose the Kafka Manager pod within the cluster on port 9000^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

External access is configured using an Ingress resource^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md]: * It maps a specific hostname (e.g., km.od.com) to the Kafka Manager service. * DNS configuration (e.g., in BIND) is updated to resolve this hostname to the cluster's Ingress controller IP.

Initial Configuration

Once the service is running and accessible via the configured URL, the initial setup requires: 1. Cluster Name: Defining a name for the Kafka cluster being managed. 2. Zookeeper Hosts: Providing the connection string for the Zookeeper ensemble (e.g., zk1.od.com:2181). 3. Kafka Version: Specifying the version of the Kafka cluster (e.g., 0.10.0.0 or similar).

After saving these details, the Kafka Manager UI connects to the cluster, allowing for the monitoring of topics, brokers, and consumer offsets^[400-devops__06-Kubernetes__k8s-paas__07.Promtheus监控k8s企业级应用.md].

Sources