Skip to content

Kubernetes Smooth Upgrade Procedure

A Kubernetes smooth upgrade (or rolling upgrade) is the process of updating the cluster version without service interruption or the need to shut down physical servers. This procedure is critical in production environments where downtime can result in significant financial loss, and is typically performed during periods of low traffic[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md][400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md].

Prerequisites

Before initiating the upgrade, it is necessary to verify the current status of the cluster nodes and system pods^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md].

Procedure

1. Network Isolation

To prevent external traffic from being routed to the node during maintenance, the node should be removed from the load balancing layer.^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md]

  • Stop the L7 (Layer 7) network traffic by removing the node's configuration from the Nginx reverse proxy configuration (/etc/nginx/nginx.conf and related conf files).^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md]
  • Reload the Nginx configuration to apply changes.^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md]

2. Node Eviction

Once the node is isolated from external traffic, it must be removed from the Kubernetes cluster management plane.^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md]

  • Execute the deletion command on the master node (e.g., kubectl delete node <node_name>).^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md]
  • Monitor the cluster using kubectl get nodes and kubectl get pods -n kube-system -o wide.^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md]
  • The Kubernetes scheduler will automatically migrate resources and balance the load to other available healthy nodes.^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md]

3. Binary Replacement

With the node safely drained, the Kubernetes binaries can be updated^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md].

  1. Extract: Download and extract the new version (e.g., kubernetes-server-linux-amd64-vX.Y.Z.tar.gz) to a temporary directory (e.g., /opt/123/).^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md]
  2. Prepare: Rename the directory to indicate the version (e.g., /opt/kubernetes-v1.15.4).^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md]
  3. Clean: Remove unnecessary files (source tar, tag files) and copy the required configurations (conf/), certificates (cert/), and scripts from the old installation directory to the new one.^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md]
  4. Switch: Update the symbolic link /opt/kubernetes to point to the new version directory.^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md]

4. Service Restoration

Restart the Kubernetes components using the process manager (e.g., supervisorctl restart all) and verify that the node rejoins the cluster successfully^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md].

Sources

^[400-devops__06-Kubernetes__k8s-paas__04.dashboard插件及k8s实战交付.md]