Skip to content

Kubernetes cluster deployment workflow

The Kubernetes cluster deployment workflow refers to the systematic process of installing and configuring a high-availability Kubernetes cluster from scratch, typically using binary deployments on Linux nodes (e.g., CentOS).^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md]

Architecture Overview

A standard enterprise deployment involves segregating responsibilities across multiple nodes to form a distributed cluster^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

Node Types

  • Master Nodes: Host the control plane components responsible for managing the cluster state^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  • Compute Nodes: Run the application containers and handle networking proxying^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  • 运维主机: A dedicated machine for hosting infrastructure services like [[DNS]], [[Docker]] registry, and certificate management^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

Key Components

  • etcd: A high-availability key-value store used for cluster data storage and service discovery^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  • API Server: The central management entity that handles REST operations and validates/configures data for API objects^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  • Controller Manager: Maintains cluster state (e.g., replica counts) and handles automated tasks like scaling and rolling updates^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  • Scheduler: Assigns [[Pods]] to specific nodes based on resource requirements and constraints^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  • Kubelet: The primary node-agent that registers the node with the API server and ensures containers described in Pod specs are running^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  • Kube-proxy: Manages network rules on each node, allowing network communication to your [[Pods]] from network sessions inside or outside the cluster^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

Prerequisites

Before beginning the deployment, specific infrastructure and operating system configurations are required^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

System Preparation

  • OS Configuration: Disable firewalld and SELinux (set to disabled) to prevent conflicts with container networking^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  • Kernel Version: Ensure the Linux kernel is version 3.8 or higher^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  • Network: Configure NAT settings in virtualized environments and ensure all nodes can communicate.
  • Hostnames: Assign unique hostnames (e.g., hdss7-21.host.com) to all nodes^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

Dependency Installation

Install common tools and utilities (e.g., wget, net-tools, epel-release) on all nodes to facilitate downloads and network diagnostics^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

Deployment Steps

1. Deploy DNS Infrastructure

Kubernetes relies heavily on DNS for service discovery and Ingress routing^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

  1. Install Bind9: Deploy on a dedicated node (e.g., hdss7-11).
  2. Configure Zones:
    • Create a "host domain" (e.g., host.com) for static node resolution^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
    • Create a "business domain" (e.g., od.com) for dynamic Kubernetes services and Ingress^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  3. Update Resolvers: Configure /etc/resolv.conf on all nodes and local workstations to point to the internal DNS server^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

2. Establish Certificate Authority (PKI)

Kubernetes components require TLS certificates to secure communication^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

  1. Install CFSSL: Download cfssl, cfssljson, and cfssl-certinfo on the运维主机^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  2. Generate CA: Create a Root Certificate Authority (ca.pem, ca-key.pem) with a long expiry (e.g., 175,200 hours)^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

3. Prepare Container Runtime

All compute nodes and the运维主机 must run Docker^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

  1. Install Docker: Use installation scripts (e.g., from get.docker.com with Aliyun mirrors).
  2. Configure Daemon: Edit /etc/docker/daemon.json:
    • Set graph to a data directory (e.g., /data/docker).
    • Define bip (Bridge IP) uniquely per node (e.g., 172.7.21.1/24) to prevent IP conflict^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
    • Configure registry-mirrors for faster image pulls.
    • Set exec-opts for native.cgroupdriver=systemd^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

4. Deploy Private Image Registry

To ensure stability and speed, deploy a local private registry^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

  1. Install Harbor: Download and extract Harbor on the运维主机^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  2. Configure Nginx Proxy: Set up Nginx to proxy traffic for harbor.od.com to the Harbor HTTP port (e.g., 180)^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  3. DNS Resolution: Add an A record for harbor.od.com pointing to the运维主机 IP in the DNS configuration^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  4. Push Images: Pull an image (e.g., nginx:1.7.9), tag it for the private registry, and push it to verify functionality^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

5. Deploy etcd Cluster

Deploy etcd on the master/compute nodes to form the distributed backing store^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

  1. Generate Peer Certs: Create etcd-peer certificates using CFSSL, including all node IPs in the hosts field^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  2. Install Binaries: Create a dedicated etcd user and extract the etcd binaries to /opt/etcd^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  3. Configure Service: Create a startup script (etcd-server-startup.sh) defining:
    • --data-dir
    • --listen-peer-urls and --listen-client-urls
    • --initial-cluster listing all peer members^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md]
    • Certificate paths for CA and peer certs.
  4. Supervise: Use supervisord to manage the etcd process, ensuring it restarts automatically on failure^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  5. Verify: Check cluster health using ./etcdctl cluster-health^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

6. Deploy API Server

The API Server runs on master nodes and requires specific client and server certificates^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

  1. Generate Certs:
    • Client Cert: For the API Server to communicate with etcd^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
    • Server Cert: For the API Server itself (apiserver-csr.json), including specific IP addresses like 10.4.7.10 (the VIP) and master node IPs^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  2. Configure: Create kube-apiserver.sh startup script with arguments:
    • --etcd-servers: List of etcd endpoints.
    • --service-cluster-ip-range: Virtual IP range for Services.
    • --authorization-mode: RBAC^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
    • --audit-log-path: For security auditing^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  3. Supervise: Manage the process using supervisord^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

7. Deploy L4 Reverse Proxy (HA)

Provide a stable entry point for the cluster using Nginx and Keepalived^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

  1. Install Nginx + Keepalived: On the reverse proxy nodes (e.g., hdss7-11, hdss7-12).
  2. Configure Nginx Stream: Set up a TCP stream block upstreaming the kube-apiserver pods on nodes 21 and 22^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  3. Configure Keepalived:
    • Assign state MASTER to the primary node and BACKUP to the secondary^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
    • Configure vrrp_script to check the health of the Nginx port (e.g., 7443)^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
    • Define a virtual_ipaddress (e.g., 10.4.7.10) shared between nodes^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

8. Deploy Controller Manager & Scheduler

These components run on the master nodes^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

  1. Controller Manager:
    • Create startup script pointing to --master=http://127.0.0.1:8080 (local proxy).
    • Enable --leader-elect for high availability^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  2. Scheduler:
    • Create startup script similar to Controller Manager.
    • Enable --leader-elect^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  3. Verify: Check component status via kubectl get cs^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

9. Deploy kubelet

The kubelet runs on all compute nodes^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

  1. Generate Certs: Create a kubelet certificate with a broad hosts field covering all possible node IPs^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  2. Create Kubeconfig: Generate kubelet.kubeconfig using kubectl config set-cluster, set-credentials, and set-context pointing to the VIP https://10.4.7.10:7443^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  3. RBAC Binding: Create a ClusterRoleBinding to authorize the k8s-node user with system:node permissions^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  4. Prepare Pause Image: Push a pause infrastructure image to the private registry^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  5. Configure: Create startup script (kubelet.sh) setting --hostname-override, --cluster-dns, and --pod-infra-container-image^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  6. Label Nodes: Use kubectl label node to mark nodes as master or worker roles^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

10. Deploy Kube-Proxy

  1. Generate Certs: Create client certificate for system:kube-proxy^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  2. Load IPVS Modules: Ensure kernel modules for IPVS are loaded on all nodes^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  3. Configure: Create startup script (kube-proxy.sh) with --proxy-mode=ipvs and --hostname-override^[400-devos__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].
  4. Verify: Use ipvsadm -Ln and kubectl get svc to check proxy rules and services^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md].

Validation

To confirm the cluster is operational, deploy a test application.^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md]

  1. Create a resource definition (e.g., nginx-ds.yaml) for a DaemonSet.
  2. Apply using kubectl create -f nginx-ds.yaml.
  3. Verify pods are running and distributed across nodes using kubectl get pods -o wide.

Sources

^[400-devops__06-Kubernetes__k8s-paas__02.企业部署实战_K8S.md]