Kubernetes - Controllers

Last Updated: 2023-07-19

https://github.com/kubernetes-sigs/controller-runtime

controller-runtime: a set of go libraries for building Controllers.

controllers are one kind of clients, they also talk to API Server, so controller-runtime wraps client-go and provide a Client to easily perform CRUD.

Unlike http handlers, Controllers DO NOT handle events directly, but enqueue Requests to eventually reconcile the object.

Controllers require a Reconciler to be provided to perform the work pulled from the work queue.

Controller will read and write to API server; webhooks are only called by API Server. controllers and webhooks may live in the same binary.

Create your own controllers

From a high level:

manifest -> image -> main() -> manager -> controller -> reconciler
                                     | -> webhook server -> webhooks

You will need the following pieces:

• Reconcilers and / or Webhooks: the logic.
  • Reconcilers implements reconcile.Reconciler, which has a method Reconcile(ctx context.Context, req ctrl.Request)
• Controller Manager: host the reconcilers and or webhooks.
• Binary: a main() func to start the controller manager.
  • main() calls Start() of the controller manager sigs.k8s.io/controller-runtime/pkg/manager/manager.go
• Image: build as an image.
• Chart / Manifest: some yaml files on how to deploy the new controller.

Code Example

Create a new manager:

import ctrl "sigs.k8s.io/controller-runtime"
manager, err := ctrl.NewManager(cfg.RESTConfig, ctrlOptions)

Register reconcilers:

ctrl.NewControllerManagedBy(manager).For(monitoredObj).Owns(generatedObj).complete(reconciler)

Register webhooks:

validator := v1.NewPodValidator(manager.GetClient())
manager.GetWebhookServer().Register("/validate-foo", &webhook.Admission{Handler: validator})

Start the manager:

manager.Start(ctx)

For vs Owns vs Watches

• For: to declare the target resources that this reconciler is responsible for reconciling.
• Owns: to declare the resources whose lifecycles are controlled by the target resources; these resources are usually created/updated in a reconciliation, and they will have an owner reference in their metadata pointing to the target resource. When these resources get changed, the owner will get requeued for a re-reconciliation. Example: A Deployment owns a ReplicaSet, and a ReplicaSet owns Pods.
• Watches: to declare resources that are "inputs" to the target resources. Since there's no explicit metadata associating these input resources with target resources, you need to provide a function that has the business logic to locate the target resource to requeue from any given input resource.

Flags / Options

flags for controllers: previously use flags, now use component config (ctrl.Options).

options := ctrl.Options{Scheme: scheme}
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), options)

Manager

• controllers
• webhook manager (mgr.GetWebhookServer())
• client (mgr.GetClient())

Controller

Controller-Runtime controllers use a cache to subscribe to events from Kubernetes objects and to read those objects more efficiently by avoiding to call out to the API.

• watch: what should we monitor?
• reconcile: what to do if there's a difference?

Controller Implementation

You can implement an a Controller using any language / runtime that can act as a client for the Kubernetes API. E.g. kubebuilder for golang.

controller-runtime is a spin-off from the kubebuilder project. kubebuilder provides some additional scalfolding toolings on top of controller-runtime to further make it easy for beginners. Advanced users can use controller-runtime directly without using kubebuilder.

controller-runtime is built on top of client-go which is the official Kubernetes golang client for interacting with the API server.

Architecture

https://book.kubebuilder.io/architecture.html

Hierarchy: main -> Manager -> Controller -> Reconciler.

• main.go: the entry point, calls the controller manager, packaged into a container.
• Manager:
  • sigs.k8s.io/controller-runtime/pkg/manager
• Controller
  • sigs.k8s.io/controller-runtime/pkg/controller
  • one per Kind / CRD, calls a Reconciler each time it gets an event, after filtering by Predicates
  • implemented as worker queues: unlike http handlers, Controllers DO NOT handle events directly, but enqueue Requests to eventually reconcile the object.
    • requires Watches to be configured to enqueue reconcile.Requests in response to events.
    • requires a Reconciler to be provided to perform the work pulled from the work queue.
• Reconciler:
  • sigs.k8s.io/controller-runtime/pkg/reconcile
  • Reconciler is a function provided to a Controller that may be called at anytime with the Name and Namespace of an object.
  • Reconcile() actually performs the reconciling for a single named object.
  • Reconciler contains all of the business logic of a Controller.
  • Reconciler typically works on a single object type.
  • Reconciler does not care about the event contents or event type responsible for triggering the reconcile.
  • Request: just has a name, but we can use the client to fetch that object from the cache.
  • Result: return an empty result and no error, to indicates to controller-runtime that we’ve successfully reconciled this object and don’t need to try again until there’s some changes.

Other noteable pkg:

• Predicate:
  • sigs.k8s.io/controller-runtime/pkg/predicate
  • filters a stream of events, pass those require action to reconciler
• Webhook:
  • sigs.k8s.io/controller-runtime/pkg/webhook

Deployment

The Controller will normally run outside of the control plane, much as you would run any containerized application. For example, you can run the controller in your cluster as a Deployment.

CRD in the crds folder of a Helm Chart.