Kubernetes - kubelet

kubelet Config Files

The configuration file installed by the kubeadm package is written to /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and is used by systemd.

kubelet config files:

/var/lib/kubelet/config.yaml
/etc/kubernetes/kubelet.conf

kubelet config flags:

• if you start kubelet with --register-node=false, you need to manually create a Node object; if true it will create Node object on the api server.
• if set the kubelet --authorization-mode flag to Webhook, it will use the SubjectAccessReview API to determine authorization.
• --network-plugin=cni: enables CNI (Container Network Interface).

kubelet deployment

kubelet is usually deployed as a systemd service; check status:

$ systemctl status kubelet

Communication

kubelet <=> kube-apiserver

kubelet needs a kubeconfig to authenticate itself to the API server.

kubelet.conf has this

users:
  - name: default-auth
    user:
      client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
      client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

/var/lib/kubelet/pki/kubelet-client-current.pem is used when talking to the api server. The cert has Subject: O = system:nodes, CN = system:node:<node_name>.

kubelet <=> CRI (e.g. containerd)

kubelet talks to container runtime to create containers; if the CRI is containerd, kubelet talks to it through a unix socket: unix:///run/containerd/containerd.sock.

The kubelet works in terms of a PodSpec. A PodSpec is a YAML or JSON object that describes a pod.

kubelet uses the podSpec and invokes the RunPodSandbox RPC. When a sandbox is created it involves creating a pause container.

• The pause container hosts pod-level resources (Linux namespaces) that workload containers use.
• The pause container is a container which holds the network namespace for the pod.

Being in the same network namespace allows containers in the same pod to refer to one another using localhost. To set the networking for a pod kubelet uses CNI (Container Network Interface) plugin. CNI concerns itself only with network connectivity of containers and removing allocated resources when the container is deleted.

Once the sandbox is created and active, the kubelet starts creating a container for it via CRI. The container runtime starts downloading the corresponding docker image and completes the task.

Auth

The kubelet calls the TokenReview API on the configured API server to determine user information from bearer tokens.

/var/lib/kubelet

Kubernetes uses /var/lib/kubelet for local storage:

• ephemeral storage
• logs
• container runtime

"ephemeral" means if the node is gone, the data is gone; on the contrary, the "persistent" data should be stored in external storage.

Pods

/var/lib/kubelet/pods/POD_ID has all the pods, identified by a POD_ID, which should match the pod's .metadata.uid field. (This id does not match anything in crictl pods results).

Volumes

Check mounts:

$ mount | grep /var/lib/kubelet

Or list all the volumes:

$ ls -lad /var/lib/kubelet/pods/*/volumes/*

Containers

The files under /var/lib/kubelet/pods/*/containers/* are the termination log for each pod's container, and are created each time a container starts up.

$ ls -lad /var/lib/kubelet/pods/*/containers/*

The files are cleaned up when a pod is evicted, but they are not cleaned up on container restart.

How is Node registered?

Example kubelet service config (/etc/systemd/system/kubelet.service):

[Service]
ExecStart=/usr/local/bin/kubelet \
  --config=/var/lib/kubelet/kubelet-config.yaml \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --register-node=true \
  --v=2

It specifies --kubeconfig; the kubeconfig file should have the apiserver address; if --register-node=true, kubelet will register the Node to the apiserver.