logo

Kubernetes - Extensions

Last Updated: 2022-07-25

2 Primary ways to extend the Kubernetes API:

  1. with CustomResourceDefinitions
  2. with Kubernetes API Aggregation Layer

For Option 2: run an extension API server in Pod(s) that run in your cluster. It can be used to integrate your apiserver with whatever other external systems (e.g. different storage APIs rather than etcd). Unlike Custom Resource Definitions (CRDs), the Aggregation API involves another server - your Extension apiserver - in addition to the standard Kubernetes apiserver. https://github.com/kubernetes-sigs/apiserver-builder-alpha is one way to build api server extensions.

This page primarily discuss Option 1. (The controller-runtime)

The Operator Patterns

Operator (A New API) = Custom Resource + Custom Controller

CRD

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: stable.example.com
  ...

Apply CRD so the new resource can be recognized by k8s:

$ kubectl apply -f crd.yaml

Then create custom objects also by kubectl apply:

$ kubectl apply -f foo.yaml

On their own, custom resources let you store and retrieve structured data. When you combine a custom resource with a custom controller, custom resources provide a true declarative API.

Controller Implementation

You can implement an a Controller using any language / runtime that can act as a client for the Kubernetes API. E.g. kubebuilder for golang.

controller-runtime is a spin-off from the kubebuilder project. kubebuilder provides some additional scalfolding toolings on top of controller-runtime to further make it easy for beginners. Advanced users can use controller-runtime directly without using kubebuilder.

controller-runtime is built on top of client-go which is the official Kubernetes golang client for interacting with the API server.

Architecture

https://book.kubebuilder.io/architecture.html

Hierarchy: main -> Manager -> Controller -> Reconciler.

  • main.go: the entry point, calls the controller manager, packaged into a container.
  • Manager:
    • sigs.k8s.io/controller-runtime/pkg/manager
  • Controller
    • sigs.k8s.io/controller-runtime/pkg/controller
    • one per Kind / CRD, calls a Reconciler each time it gets an event, after filtering by Predicates
    • implemented as worker queues: unlike http handlers, Controllers DO NOT handle events directly, but enqueue Requests to eventually reconcile the object.
      • requires Watches to be configured to enqueue reconcile.Requests in response to events.
      • requires a Reconciler to be provided to perform the work pulled from the work queue.
  • Reconciler:
    • sigs.k8s.io/controller-runtime/pkg/reconcile
    • Reconciler is a function provided to a Controller that may be called at anytime with the Name and Namespace of an object.
    • Reconcile() actually performs the reconciling for a single named object.
    • Reconciler contains all of the business logic of a Controller.
    • Reconciler typically works on a single object type.
    • Reconciler does not care about the event contents or event type responsible for triggering the reconcile.
    • Request: just has a name, but we can use the client to fetch that object from the cache.
    • Result: return an empty result and no error, to indicates to controller-runtime that we’ve successfully reconciled this object and don’t need to try again until there’s some changes.

Other noteable pkg:

  • Predicate:
    • sigs.k8s.io/controller-runtime/pkg/predicate
    • filters a stream of events, pass those require action to reconciler
  • Webhook:
    • sigs.k8s.io/controller-runtime/pkg/webhook

Scheme

Defined in k8s.io/apimachinery/pkg/runtime

Read more: API Machinery

Manager

  • controllers
  • webhook manager (mgr.GetWebhookServer())
  • client (mgr.GetClient())

Controller

Controller-Runtime controllers use a cache to subscribe to events from Kubernetes objects and to read those objects more efficiently by avoiding to call out to the API.

  • watch: what should we monitor?
  • reconcile: what to do if there's a difference?

Flags / Options

flags for controllers: previously use flags, now use component config (ctrl.Options).

options := ctrl.Options{Scheme: scheme}
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), options)

config defined in config/manager/controller_manager_config.yaml or go file.

Deployment

The Controller will normally run outside of the control plane, much as you would run any containerized application. For example, you can run the controller in your cluster as a Deployment.

CRD in the crds folder of a Helm Chart

Markers

//+kubebuilder:object:root=true is a marker used by https://github.com/kubernetes-sigs/controller-tools to generate CRM manifests from go code.

For vs Owns vs Watches

  • For: to declare the target resources that this reconciler is responsible for reconciling.
  • Owns: to declare the resources whose lifecycles are controlled by the target resources; these resources are usually created/updated in a reconciliation, and they will have an owner reference in their metadata pointing to the target resource. When these resources get changed, the owner will get requeued for a re-reconciliation. Example: A Deployment owns a ReplicaSet, and a ReplicaSet owns Pods.
  • Watches: to declare resources that are "inputs" to the target resources. Since there's no explicit metadata associating these input resources with target resources, you need to provide a function that has the business logic to locate the target resource to requeue from any given input resource.

Webhook

Webhooks

  • one per Kind that is reconciled.
  • provides methods to build and bootstrap a webhook server.
  • same as controllers, a Webhook Server is a Runable (https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager#Runnable) (i.e. both controller and webhook server implement Start(ctx context.Context)) which needs to be registered to a manager. Arbitrary number of Runables can be registered to a manager, so a webhook server can run with other controllers in the same manager. They will share the same dependencies provided by the manager. For example, shared cache, client, scheme, etc.
  • Admission Webhooks: the API server will send AdmissionRequests to Webhook, Admission Webhooks require Handler(s) to be provided to process the received AdmissionReview requests.
  • Conversion Webhooks: When API clients, like kubectl or your controller, request a particular version of your resource, the Kubernetes API server needs to return a result that’s of that version. However, that version might not match the version stored by the API server. In that case, the API server needs to know how to convert between the desired version and the stored version. Since the conversions aren’t built in for CRDs, the Kubernetes API server calls out to a webhook to do the conversion instead. For Kubebuilder, this webhook is implemented by controller-runtime, and performs the hub-and-spoke conversions.

Webhook Server

Webhook server will start a httpserver (sigs.k8s.io/controller-runtime/pkg/internal/httpserver) it is similar to a controller in that it also implements Runnable interface.

The business logic for a Webhook exists in a Handler. A Handler implements the admission.Handler interface, which contains a single Handle method.

hookServer.Register(): register a handler

logo
PRIVACY POLICYABOUT中文