Kubernetes - Controllers
https://github.com/kubernetes-sigs/controller-runtime
controller-runtime
: a set of go libraries for building Controllers.
controllers are one kind of clients, they also talk to API Server, so controller-runtime wraps client-go and provide a Client to easily perform CRUD.
Unlike http handlers, Controllers DO NOT handle events directly, but enqueue Requests to eventually reconcile the object.
Controllers require a Reconciler to be provided to perform the work pulled from the work queue.
Controller will read and write to API server; webhooks are only called by API Server. controllers and webhooks may live in the same binary.
Create your own controllers
From a high level:
manifest -> image -> main() -> manager -> controller -> reconciler
| -> webhook server -> webhooks
You will need the following pieces:
- Reconcilers and / or Webhooks: the logic.
- Reconcilers implements
reconcile.Reconciler
, which has a methodReconcile(ctx context.Context, req ctrl.Request)
- Reconcilers implements
- Controller Manager: host the reconcilers and or webhooks.
- Binary: a
main()
func to start the controller manager.main()
callsStart()
of the controller managersigs.k8s.io/controller-runtime/pkg/manager/manager.go
- Image: build as an image.
- Chart / Manifest: some yaml files on how to deploy the new controller.
Code Example
Create a new manager:
import ctrl "sigs.k8s.io/controller-runtime"
manager, err := ctrl.NewManager(cfg.RESTConfig, ctrlOptions)
Register reconcilers:
ctrl.NewControllerManagedBy(manager).For(monitoredObj).Owns(generatedObj).complete(reconciler)
Register webhooks:
validator := v1.NewPodValidator(manager.GetClient())
manager.GetWebhookServer().Register("/validate-foo", &webhook.Admission{Handler: validator})
Start the manager:
manager.Start(ctx)
For vs Owns vs Watches
For
: to declare the target resources that this reconciler is responsible for reconciling.Owns
: to declare the resources whose lifecycles are controlled by the target resources; these resources are usually created/updated in a reconciliation, and they will have an owner reference in their metadata pointing to the target resource. When these resources get changed, the owner will get requeued for a re-reconciliation. Example: A Deployment owns a ReplicaSet, and a ReplicaSet owns Pods.Watches
: to declare resources that are "inputs" to the target resources. Since there's no explicit metadata associating these input resources with target resources, you need to provide a function that has the business logic to locate the target resource to requeue from any given input resource.
Flags / Options
flags for controllers: previously use flags, now use component config (ctrl.Options
).
options := ctrl.Options{Scheme: scheme}
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), options)
Manager
- controllers
- webhook manager (
mgr.GetWebhookServer()
) - client (
mgr.GetClient()
)
Controller
Controller-Runtime controllers use a cache to subscribe to events from Kubernetes objects and to read those objects more efficiently by avoiding to call out to the API.
- watch: what should we monitor?
- reconcile: what to do if there's a difference?
Controller Implementation
You can implement an a Controller using any language / runtime that can act as a client for the Kubernetes API. E.g. kubebuilder
for golang.
controller-runtime
is a spin-off from the kubebuilder
project. kubebuilder
provides some additional scalfolding toolings on top of controller-runtime
to further make it easy for beginners. Advanced users can use controller-runtime
directly without using kubebuilder
.
controller-runtime
is built on top of client-go
which is the official Kubernetes golang client for interacting with the API server.
Architecture
https://book.kubebuilder.io/architecture.html
Hierarchy: main -> Manager -> Controller -> Reconciler.
main.go
: the entry point, calls the controller manager, packaged into a container.- Manager:
sigs.k8s.io/controller-runtime/pkg/manager
- Controller
sigs.k8s.io/controller-runtime/pkg/controller
- one per Kind / CRD, calls a Reconciler each time it gets an event, after filtering by Predicates
- implemented as worker queues: unlike http handlers, Controllers DO NOT handle events directly, but enqueue Requests to eventually reconcile the object.
- requires
Watches
to be configured to enqueuereconcile.Requests
in response to events. - requires a
Reconciler
to be provided to perform the work pulled from the work queue.
- requires
- Reconciler:
sigs.k8s.io/controller-runtime/pkg/reconcile
Reconciler
is a function provided to aController
that may be called at anytime with theName
andNamespace
of an object.Reconcile()
actually performs the reconciling for a single named object.- Reconciler contains all of the business logic of a Controller.
- Reconciler typically works on a single object type.
- Reconciler does not care about the event contents or event type responsible for triggering the reconcile.
Request
: just has a name, but we can use the client to fetch that object from the cache.Result
: return an empty result and no error, to indicates to controller-runtime that we’ve successfully reconciled this object and don’t need to try again until there’s some changes.
Other noteable pkg:
- Predicate:
sigs.k8s.io/controller-runtime/pkg/predicate
- filters a stream of events, pass those require action to reconciler
- Webhook:
sigs.k8s.io/controller-runtime/pkg/webhook
Deployment
The Controller will normally run outside of the control plane, much as you would run any containerized application. For example, you can run the controller in your cluster as a Deployment
.
CRD in the crds
folder of a Helm Chart.