Kubernetes - Upgrade

What does "upgrade" mean in Kubernetes?

Actions during a Kubernetes cluster upgrade:

• Upgrade Container Runtime: e.g. containerd, cri-o, etc.
• Upgrade Kubernetes components: The life-cycle of these components are managed by kubeadm
  • kube-apiserver
  • kube-scheduler
  • kube-controller-manager
  • etcd
  • kubelet
• Upgrade Controlplane LB: haproxy and keepalived.
• Upgrade Addons: e.g. CNI, kube-proxy, core-dns, Dataplane LoadBalancer, kubevirt, etc.

kubeadm upgrade

Use kubeadm upgrade to upgrade. The upgrade procedure on control plane nodes and worker nodes should be executed one node at a time.

If kubeadm upgrade fails and does not roll back, for example because of an unexpected shutdown during execution, you can run kubeadm upgrade again. This command is idempotent and eventually makes sure that the actual state is the desired state you declare.

kubeadm manages the lifecycles of the components like kube-apiserver, kube-scheduler-controller,kube-controller-manager, etcd, kubelet.

Drain and undrain

Use kubectl drain to safely evict all of the pods from a node before you perform maintenance on the node (e.g. kernel upgrade, hardware maintenance, etc.). Alternatively can call eviction API.

To undrain:

$ kubectl uncordon $NODE_NAME

Cluster Upgrade

The cluster first upgrades the control plane node pool, followed by the load balancer node pool and finally worker node pools.

Option 1: Drain and Upgrade

Recreat every VM in the node pool by first draining and deleting the old VM, and then creating a new VM with a new instance template. All pods must be stopped during the upgrade which could cause workload disruption depending on the number of pod replicas.

Option 2: Surge Upgrades (with a Surge VM)

For every node to be upgraded, create a new "surge VM", register the new node, move Pods to the new node, and then delete the old node. This "create-before-delete" method provides less disruption.

Does not work for bare-metal clusters.