logo

Kubernetes

Last Updated: 2023-02-16

Why Kubernetes?

Kubernetes is winning the container orchestration war.

Orchestration tools: handle containers running stateless applications. The applications may be terminated at any time, and / or restarted from a different machine. E.g. Kuberenetes, Mesos, Nomad, etc.

Kubernetes vs OpenStack

Openstack was launched in 2010. AWS was the only Cloud, GCP didn't exist, Docker was not a thing. The goal was to provide an open source and private alternative to AWS; building on top of VMs.

Kubernetees was launched in 2014. AWS, Azure, GCP became dominant players of Cloud computing, Docker became the synonym of container. The goal was to be a bridge among the big 3, and between public cloud and private data centers; building on top of containers.

OpenStack is on the downtrend.

Kubernetes vs Nomad

  • Kubernetes aims to provide all the features needed to run Linux container-based applications including cluster management, scheduling, service discovery, monitoring, secrets management and more.
  • Nomad only aims to focus on cluster management and scheduling and is designed with the Unix philosophy of having a small scope while composing with tools like Consul for service discovery / service mesh and Vault for secret management.

Kubernetes Applications

  • Stateless applications: trivial to scale, with no coordination. These can take advantage of Kubernetes deployments directly and work great behind Kubernetes Services or Ingress Services.
  • Stateful applications: postgres, mysql, etc, which generally exist as single processes and persist to disks. These systems generally should be pinned to a single machine and use a single Kubernetes persistent disk. These systems can be served by static configuration of pods, persistent disks, etc or utilize StatefulSets.
  • Static distributed applications: zookeeper, cassandra, etc which are hard to reconfigure at runtime but do replicate data around for data safety. These systems have configuration files that are hard to update consistently and are well-served by StatefulSets.
  • Clustered applications: etcd, redis, prometheus, vitess, rethinkdb, etc are built for dynamic reconfiguration and modern infrastructure where things are often changing. They have APIs to reconfigure members in the cluster and just need glue to be operated natively seemlessly on Kubernetes, and thus the Kubernetes Operator concept.

How does Kubernetes work?

K8s = Resources + Controllers

2 key concepts in Kubernetes: Resources + Controllers

  • A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind.
    • Built-in Resrouces: e.g. Pod, Service, Job, etc.
    • Custom Resources: extensions of the Kubernetes API.
      • Defined in CRDs (Customer Resources Definition). The Go data models (as Go structs) are made into Kubernetes manifests at build time that are applied to the Kubernetes clusters.
      • The CRDs then become available as APIs on the Kubernetes cluster.
  • Each resource has reconcilers monitoring it. The reconcilers are then packaged into a larger controller.
    • Controller pattern: Controllers typically read an object's .spec, possibly do things, and then update the object's .status.
    • Controllers are clients that call into the API server (i.e. API server does not know who or where the controllers are, they are NOT registered, unlike webhooks).
    • e.g.
      • Deployment controller watches Deployment resources.
      • Job controller, tells API server to create or remove Pods.
      • Other examples: replication controller, endpoints controller, namespace controller, and serviceaccounts controller.
  • Controllers (Go code) live in a controller-manager (a binary / container)
    • built-in controllers that run inside the kube-controller-manager.

Note: not all resources have controllers monitoring them, e.g. one exception is ConfigMap, it just stores stuff (ConfigMap does not spec and status field, but a data field)

Hierarchy

cluster -> namespace -> node -> pod -> container

(a Node = a kubelet running)

X as Code

In kubernetes, everything is a code. Git repository which should act as single source of truth.

Naming

The K8S group has a tradition of using Greek names

  • Kubernetes (κυβερνήτης): helmsman or pilot
  • Istio (ιστίο): sail
  • Anthos (ἄνθος): flower

Standards

  • Container Runtime Interface (CRI):the main protocol for the communication between the kubelet and Container Runtime.
  • Container Storage Interface (CSI)
  • Container Network Interface (CNI)

Scopes

  • Cluster level.
  • Namespace-based scoping is applicable only for namespaced objects (e.g. Deployments, Services, pods, services, replication controllers, etc) and not for cluster-wide objects (e.g. StorageClass, Nodes, PersistentVolumes, etc). namespace resources are not themselves in a namespace. To get all the namespaces: kubectl get namespace

Configuration Management

manifest = yaml

  • plain (e.g. CRDs, deployment/service with no variables but only hard coded values, configmaps)
    • kubectl apply (apply the manifests, create/update resources)
  • Helm: templates + values => yaml, good for yamls you fully own
    • helm install
  • kustomize: literal yaml + patches (does not use templates) good for yamls you do not own
    • kubectl apply -k

Helm: A package manager for Kubernetes that uses Charts, which are Go-based templates that ultimately generate YAML-based manifests for deployments. you can use a simple command like helm install prometheus prometheus-community/prometheus to deploy a functional monitoring agent (and all its requisite resources) on your existing cluster without writing a single line of YAML.

Kustomize: A configuration management tool for customizing Kubernetes objects. With Kustomize, you can take an existing manifest and apply overrides without touching the original YAML file.

APIs

API Conventions: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md

  • core group:
    • REST Path: /api/v1
    • apiVersion: "core" is skipped, i.e. apiVersion: v1.
  • exntions:
    • REST Path: /apis/$GROUP_NAME/$VERSION
    • apiVersion: $GROUP_NAME/$VERSION, e.g. apiVersion: batch/v1

The /api endpoint is already legacy and used only for core resources (pods, secrets, configmaps, etc.). A more modern and generic /apis/<group-name> endpoint is used for the rest of resources, including user-defined custom resources.

API Groups

  • core: v1
  • apps/v1, batch/v1, policy/v1, autoscaling/v2
  • k8s.io: app.k8s.io, metrics.k8s.io, networking.k8s.io, node.k8s.io, storage.k8s.io, rbac.authorization.k8s.io
  • x-k8s.io: cluster.x-k8s.io

Add-ons

  • add-on examples: CoreDNS, Dashboards (e.g. Grafana), etc.
  • add-ons can be in the form of Kubernetes Operators.
  • add-ons can be installed by Helm.

Multi-tenancy / Admin clusters

Benefits of using admin clusters:

  • provides a Kubernetes Resource Model (KRM) API for managing the lifecycle (bootstrap, upgrade, update configs / policies, deletion) of multiple user clusters. (Otherwise you need some non-standard way to manage the clusters, like manually and running scripts.)
  • provides a single location to store / cache common policies for multiple user clusters.

In ABM, admin clusters run on-premises. To support edge deployments with limited resource footprints, the admin cluster can run remotely in a different datacenter or region, or a public cloud.

Upgrade

kubeadm upgrade

Use kubeadm upgrade to upgrade. The upgrade procedure on control plane nodes and worker nodes should be executed one node at a time.

If kubeadm upgrade fails and does not roll back, for example because of an unexpected shutdown during execution, you can run kubeadm upgrade again. This command is idempotent and eventually makes sure that the actual state is the desired state you declare.

kubeadm manages the lifecycles of the components like kube-apiserver, kube-scheduler-controller,kube-controller-manager, etcd, kubelet.s

Drain and undrain

Use kubectl drain to safely evict all of the pods from a node before you perform maintenance on the node (e.g. kernel upgrade, hardware maintenance, etc.). Alternatively can call eviction API.

To undrain:

$ kubectl uncordon <node name>

Terminologies

  • "in-tree": meaning their code was part of the core Kubernetes code and shipped with the core Kubernetes binaries.
  • KRM: the Kubernetes Resource Model. The declarative format you use to talk to the Kubernetes API. Basically the yaml file you see and interact with (the yaml with apiVersion, kind, metadata, spec, and status)
  • K8s Native: Native = using KRM apis.

How to debug

With kubectl:

kubectl get pod
kubectl get event
kubectl logs

check admin node

  • /etc/containerd/config.toml for container configs
  • /etc/kubernetes/manifests for static pod manifests
  • crictl logs for static pod logs
  • journalctl -u kubelet for kubelet logs

Change verbosity level:

/var/lib/kubelet/config.yaml

logging.verbosity => 4

How to add logs?

klog:

Containerized-Data-Importer (CDI)

Containerized-Data-Importer (CDI) is a persistent storage management add-on for Kubernetes. It's primary goal is to provide a declarative way to build Virtual Machine Disks on PVCs for Kubevirt VM.

CDI provides the ability to populate PVCs with VM images or other data upon creation.

Dev Lifecycle

Code - (Build) -> Container Images -> (Push) -> Registry -> (Deploy) -> K8s Cluster

Code Structure Convention

  • pkg/apis: define types; using kubebuilder
  • pkg/controllers: define logic

Access Control

Authn: IdP

According to CloudFlare: An identity provider (IdP) is a service that stores and verifies user identity. IdPs are typically cloud-hosted services, and they often work with single sign-on (SSO) providers to authenticate users.

IdP is only Authn, does not include Authz. E.g. active directory, okta.

RedHat Single Sign-On (the commercial version of Keycloak).

Standards / Protocols:

Authz

  • Namespaces: segment pods by application or work group, support multi-tenancy.
  • RBAC: assign roles to users for specific namespaces.

Databases

postgres: https://github.com/zalando/spilo

Billing an Provisioning

The Account Tracking and Automation Tool (ATAT) is a cloud provisioning tool developed by the Hosting and Compute Center of the Defense Information Systems Agency (DISA). ATAT is responsible for two major areas of functionality:

  • Provisioning: provisions resources in a cloud environment.
  • Billing: reports historical and forecasted cost information.

Learning Resources

logo
PRIVACY POLICYABOUT中文