Kubernetes - Control Plane

Last Updated: 2024-02-06

When we talk about the "Control Plane", sometimes we are referring to the API Server, as it is the only visible part of the Control Plane.

How to access API Server / Control Plane

• REST API.
  • discovery endpoints:
    • /openapi/v2, /openapi/v3
    • /api, /apis
  • OpenAPI v3 spec: /openapi/v3/apis/<group>/<version>?hash=<hash>
  • APIs:
    • /api/v1
    • /apis/rbac.authorization.k8s.io/v1alpha1
• CLI: kubectl, kubeadm etc.
• Client Libraries: e.g. client-go (CLIs like kubectl, kubeadm also depend on client-go).

How to load balance multiple API Servers

If you have 3 control plane nodes, there will be 3 kube-apiservers running, how to decide which kube-apiserver to call?

On Cloud

On a public cloud (AWS, Google Cloud) you can use the load balancing service provided by the cloud to distribute traffic across the 3 API servers. E.g. Google Network Load Balancer.

On Prem

• If you deploy your own k8s, you can use a haproxy to load balance the apiservers.
• Then the haproxy will be the single point of failure. Use keepalived to have BACKUP haproxy standing by.
• haproxy and keepalived can be deployed and static pods (i.e. add yamls to /etc/kubernetes/manifests)

HAProxy and Keepalived

• haproxy: load balance HTTP requests.
• keepalived: keepalived makes haproxy highly available; an implementation of VRRP.

Virtual IP

ip will appear in ip a s if the node is alive, otherwise will disappear (can be simulated by systemctl stop haproxy) and the vip will appear in another configured node.

Keepalived

In the event that the master server (haproxy1) fails, keepalived assigns the virtual IP address (e.g. 10.0.0.10/24) to the eth0 interface on the backup server (haproxy2), which becomes the master server.

The active (master) router informs the backup routers of its active status using the Virtual Router Redundancy Protocol (VRRP), which requires the master router to send out advertisements at regular intervals. If the active router stops sending advertisements, a new master is elected.

Keepalived implements Virtual Router Redundancy Protocol, also known as VRRP. Usually, the VRRP protocol ensures that only one of the participating instances is running as MASTER. The other instances are in the BACKUP state listening for multicast packets from an instance with a higher priority.

If the BACKUP instance fails to receive VRRP advertisements for a period longer than the KEEPALIVED_ADVERT_INT, the BACKUP instance moves to the MASTER state and it takes the VIP for itself. In case there is more than one instance in BA

A VIP address migrates from one LVS router to the other during a failover, thus maintaining a presence at that IP address. A VIP is also known as a floating IP addresses.

Deployment

haproxy and keepalived can be containerized or non-containerized.

If non-containerized:

Configs can be found in /etc folder:

/etc/haproxy/haproxy.cfg
/etc/keepalived/keepalived.conf

They are managed by systemctl

$ systemctl restart haproxy
$ systemctl enable haproxy
$ systemctl restart keepalived
$ systemctl enable keepalived

If containerized:

Check keepalived config:

$ kubectl -n kube-system exec kube-keepalived-master -- cat /etc/keepalived/keepalived.conf