Operator SDK – Go

Docs: Installation Guide

Mon, 01 Jan 0001 00:00:00 +0000

Install `operator-sdk`

Follow the steps in the installation guide to learn how to install the operator-sdk CLI tool.

Additional Prerequisites

git
go version 1.22
docker version 17.03+.
kubectl and access to a Kubernetes cluster of a compatible version.

Docs: Quickstart for Go-based Operators

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks through an example of building a simple memcached-operator using tools and libraries provided by the Operator SDK.

Prerequisites

Go through the installation guide.
Make sure your user is authorized with cluster-admin permissions.
An accessible image registry for various operator images (ex. hub.docker.com, quay.io) and be logged in to your command line environment.
- example.com is used as the registry Docker Hub namespace in these examples. Replace it with another value if using a different registry or namespace.
- Authentication and certificates if the registry is private or uses a custom CA.

Steps

Create a project directory for your project and initialize the project:

mkdir memcached-operator
cd memcached-operator
operator-sdk init --domain example.com --repo github.com/example/memcached-operator

Note If your local environment is Apple Silicon (darwin/arm64) use the go/v4 plugin which provides support for this platform by adding to the init subCommand the flag --plugins=go/v4

Create a simple Memcached API:

operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource --controller

Build and push your operator’s image:

make docker-build docker-push IMG="example.com/memcached-operator:v0.0.1"

OLM deployment

Install OLM:

operator-sdk olm install

Bundle your operator, then build and push the bundle image (defaults to example.com/memcached-operator-bundle:v0.0.1):

make bundle IMG="example.com/memcached-operator:v0.0.1"
make bundle-build bundle-push BUNDLE_IMG="example.com/memcached-operator-bundle:v0.0.1"

Run your bundle. If your bundle image is hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

operator-sdk run bundle <some-registry>/memcached-operator-bundle:v0.0.1

Create a sample Memcached custom resource:

$ kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com/memcached-sample created

Uninstall the operator:

operator-sdk cleanup memcached-operator

Direct deployment

Deploy your operator:

make deploy IMG="example.com/memcached-operator:v0.0.1"

Create a sample Memcached custom resource:

$ kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com/memcached-sample created

Uninstall the operator:

make undeploy

Run locally (outside the cluster)

This is recommended ONLY for development purposes

Run the operator:

make install run

In a new terminal tab/window, create a sample Memcached custom resource:

$ kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com/memcached-sample created

Stop the operator by pressing ctrl+c in the terminal tab or window the operator is running in

Next Steps

Read the full tutorial for an in-depth walkthrough of building a Go operator.

Docs: Go Operator Tutorial

Mon, 01 Jan 0001 00:00:00 +0000

NOTE: If your project was created with an operator-sdk version prior to v1.0.0 please migrate, or consult the legacy docs.

Prerequisites

Go through the installation guide.
Make sure your user is authorized with cluster-admin permissions.
An accessible image registry for various operator images (ex. hub.docker.com, quay.io) and be logged in to your command line environment.
- example.com is used as the registry Docker Hub namespace in these examples. Replace it with another value if using a different registry or namespace.
- Authentication and certificates if the registry is private or uses a custom CA.

Overview

We will create a sample project to let you know how it works and this sample will:

Create a Memcached Deployment if it doesn’t exist
Ensure that the Deployment size is the same as specified by the Memcached CR spec
Update the Memcached CR status using the status writer with the names of the CR’s pods

Create a new project

Use the CLI to create a new memcached-operator project:

mkdir -p $HOME/projects/memcached-operator
cd $HOME/projects/memcached-operator
# we'll use a domain of example.com
# so all API groups will be <group>.example.com
operator-sdk init --domain example.com --repo github.com/example/memcached-operator

--domain will be used as the prefix of the API group your custom resources will be created in. API groups are a mechanism to group portions of the Kubernetes API. You’re probably already familiar with some of the core Kubernetes API groups, such as apps or rbac.authorization.k8s.io. API groups are used internally to version your Kubernetes resources and are thus used for many things. Importantly, you should name your domain to group your resource types in meaningful group(s) for ease of understanding and because these groups determine how access can be controlled to your resource types using RBAC. For more information, see the core Kubernetes docs and the Kubebuilder docs.

Note If your local environment is Apple Silicon (darwin/arm64) use the go/v4 plugin which provides support for this platform by adding to the init subCommand the flag --plugins=go/v4

To learn about the project directory structure, see Kubebuilder project layout doc.

A note on dependency management

operator-sdk init generates a go.mod file to be used with Go modules. The --repo=<path> flag is required when creating a project outside of $GOPATH/src, as scaffolded files require a valid module path. Ensure you activate module support by running export GO111MODULE=on before using the SDK.

Manager

The main program for the operator main.go initializes and runs the Manager.

See the Kubebuilder entrypoint doc for more details on how the manager registers the Scheme for the custom resource API definitions, and sets up and runs controllers and webhooks.

The Manager can restrict the namespace that all controllers will watch for resources:

mgr, err := ctrl.NewManager(cfg, manager.Options{Namespace: namespace})

By default this will be empty string which means watch all namespaces:

mgr, err := ctrl.NewManager(cfg, manager.Options{Namespace: ""})

Read the operator scope documentation on how to run your operator as namespace-scoped vs cluster-scoped.

Create a new API and Controller

Create a new Custom Resource Definition (CRD) API with group cache version v1alpha1 and Kind Memcached. When prompted, enter yes y for creating both the resource and controller.

$ operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource --controller
Writing scaffold for you to edit...
api/v1alpha1/memcached_types.go
controllers/memcached_controller.go
...

This will scaffold the Memcached resource API at api/v1alpha1/memcached_types.go and the controller at controllers/memcached_controller.go.

Note: In this tutorial we will be providing all the steps to show you how to implement an operator project. However, as a follow up you might want to check the Deploy Image plugin with which it is possible to have the whole code generated to deploy and manage an Operand(image). To do so, you can use the command $ operator-sdk create api --group cache --version v1alpha1 --kind Memcached --plugins="deploy-image/v1-alpha" --image=memcached:1.4.36-alpine --image-container-command="memcached,-m=64,modern,-v" --run-as-user="1001"

Note: This guide will cover the default case of a single group API. If you would like to support Multi-Group APIs see the Single Group to Multi-Group doc.

Understanding Kubernetes APIs

For an in-depth explanation of Kubernetes APIs and the group-version-kind model, check out these kubebuilder docs.

In general, it’s recommended to have one controller responsible for managing each API created for the project to properly follow the design goals set by controller-runtime.

Define the API

To begin, we will represent our API by defining the Memcached type, which will have a MemcachedSpec.Size field to set the quantity of memcached instances (CRs) to be deployed, and a MemcachedStatus.Conditions field to store a CR’s Conditions.

Define the API for the Memcached Custom Resource(CR) by modifying the Go type definitions at api/v1alpha1/memcached_types.go to have the following spec and status:

// MemcachedSpec defines the desired state of Memcached
type MemcachedSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// The following markers will use OpenAPI v3 schema to validate the value
	// More info: https://book.kubebuilder.io/reference/markers/crd-validation.html
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=5
	// +kubebuilder:validation:ExclusiveMaximum=false

	// Size defines the number of Memcached instances
	// +operator-sdk:csv:customresourcedefinitions:type=spec
	Size int32 `json:"size,omitempty"`

	// Port defines the port that will be used to init the container with the image
	// +operator-sdk:csv:customresourcedefinitions:type=spec
	ContainerPort int32 `json:"containerPort,omitempty"`
}

// MemcachedStatus defines the observed state of Memcached
type MemcachedStatus struct {
	// Represents the observations of a Memcached's current state.
	// Memcached.status.conditions.type are: "Available", "Progressing", and "Degraded"
	// Memcached.status.conditions.status are one of True, False, Unknown.
	// Memcached.status.conditions.reason the value should be a CamelCase string and producers of specific
	// condition types may define expected values and meanings for this field, and whether the values
	// are considered a guaranteed API.
	// Memcached.status.conditions.Message is a human readable message indicating details about the transition.
	// For further information see: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties

	// Conditions store the status conditions of the Memcached instances
	// +operator-sdk:csv:customresourcedefinitions:type=status
	Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type" protobuf:"bytes,1,rep,name=conditions"`
}

Add the +kubebuilder:subresource:status marker to add a status subresource to the CRD manifest so that the controller can update the CR status without changing the rest of the CR object:

// Memcached is the Schema for the memcacheds API
//+kubebuilder:subresource:status
type Memcached struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   MemcachedSpec   `json:"spec,omitempty"`
	Status MemcachedStatus `json:"status,omitempty"`
}

After modifying the *_types.go file always run the following command to update the generated code for that resource type:

make generate

The above makefile target will invoke the controller-gen utility to update the api/v1alpha1/zz_generated.deepcopy.go file to ensure our API’s Go type definitions implement the runtime.Object interface that all Kind types must implement.

Generating CRD manifests

Once the API is defined with spec/status fields and CRD validation markers, the CRD manifests can be generated and updated with the following command:

make manifests

This makefile target will invoke controller-gen to generate the CRD manifests at config/crd/bases/cache.example.com_memcacheds.yaml.

OpenAPI validation

OpenAPI validation defined in a CRD ensures CRs are validated based on a set of declarative rules. All CRDs should have validation. See the OpenAPI validation doc for details.

Implement the Controller

For this example replace the generated controller file controllers/memcached_controller.go with the example memcached_controller.go implementation.

Note: If you used a value other than github.com/example/memcached-operator for repository (--repo flag) when running the operator-sdk init command, modify accordingly in the import block of the file.

Note: The next two subsections explain how the controller watches resources and how the reconcile loop is triggered. If you’d like to skip this section, head to the deploy section to see how to run the operator.

Setup a Recorder

First, add a recorder when you initialize the Memcached reconciler in main.go.

if err = (&controllers.MemcachedReconciler{
	Client:   mgr.GetClient(),
	Scheme:   mgr.GetScheme(),
	Recorder: mgr.GetEventRecorderFor("memcached-controller"),
}).SetupWithManager(mgr); err != nil {
	setupLog.Error(err, "unable to create controller", "controller", "Memcached")
	os.Exit(1)
}

This recorder will be used within the reconcile method of the controller to emit events.

Resources watched by the Controller

The SetupWithManager() function in controllers/memcached_controller.go specifies how the controller is built to watch a CR and other resources that are owned and managed by that controller.

import (
	...
	appsv1 "k8s.io/api/apps/v1"
	...
)

func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&cachev1alpha1.Memcached{}).
		Owns(&appsv1.Deployment{}).
		Complete(r)
}

The NewControllerManagedBy() provides a controller builder that allows various controller configurations.

For(&cachev1alpha1.Memcached{}) specifies the Memcached type as the primary resource to watch. For each Memcached type Add/Update/Delete event the reconcile loop will be sent a reconcile Request (a namespace/name key) for that Memcached object.

Owns(&appsv1.Deployment{}) specifies the Deployments type as the secondary resource to watch. For each Deployment type Add/Update/Delete event, the event handler will map each event to a reconcile Request for the owner of the Deployment. Which in this case is the Memcached object for which the Deployment was created.

The dependent objects, in this case the Deployments, need to have an Owner References field that references their owner object. This will be added by using the method ctrl.SetControllerReference. More info

Note: The K8s api will manage the resources according to theownerRef which will be properly set by using this method. Therefore, the K8s API will know that these resources, such as the Deployment to run the Memcached Operand image, depend on the custom resource for the Memcached Kind. This allows the K8s API to delete all dependent resources when/if the custom resource is deleted. More info

Controller Configurations

There are a number of other useful configurations that can be made when initializing a controller. For more details on these configurations consult the upstream builder and controller godocs.

Set the max number of concurrent Reconciles for the controller via the MaxConcurrentReconciles option. Defaults to 1.

func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
  return ctrl.NewControllerManagedBy(mgr).
    For(&cachev1alpha1.Memcached{}).
    Owns(&appsv1.Deployment{}).
    WithOptions(controller.Options{MaxConcurrentReconciles: 2}).
    Complete(r)
}

Filter watch events using predicates
Choose the type of EventHandler to change how a watch event will translate to reconcile requests for the reconcile loop. For operator relationships that are more complex than primary and secondary resources, the EnqueueRequestsFromMapFunc handler can be used to transform a watch event into an arbitrary set of reconcile requests.

Reconcile loop

The reconcile function is responsible for enforcing the desired CR state on the actual state of the system. It runs each time an event occurs on a watched CR or resource, and will return some value depending on whether those states match or not.

In this way, every Controller has a Reconciler object with a Reconcile() method that implements the reconcile loop. The reconcile loop is passed the Request argument which is a Namespace/Name key used to lookup the primary resource object, Memcached, from the cache:

import (
	ctrl "sigs.k8s.io/controller-runtime"

	cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
	...
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  // Lookup the Memcached instance for this reconcile request
  memcached := &cachev1alpha1.Memcached{}
  err := r.Get(ctx, req.NamespacedName, memcached)
  ...
}

For a guide on Reconcilers, Clients, and interacting with resource Events, see the Client API doc.

The following are a few possible return options for a Reconciler:

With the error:
```
return ctrl.Result{}, err
```
Without an error:
```
return ctrl.Result{Requeue: true}, nil
```
Therefore, to stop the Reconcile, use:
```
return ctrl.Result{}, nil
```

Reconcile again after X time:

 return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil

For more details, check the Reconcile and its Reconcile godoc.

Specify permissions and generate RBAC manifests

The controller needs certain RBAC permissions to interact with the resources it manages. These are specified via RBAC markers like the following:

//+kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/finalizers,verbs=update
//+kubebuilder:rbac:groups=core,resources=events,verbs=create;patch
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  ...
}

The ClusterRole manifest at config/rbac/role.yaml is generated from the above markers via controller-gen with the following command:

make manifests

NOTE: If you receive an error, please run the specified command in the error and re-run make manifests.

Configure the operator’s image registry

All that remains is to build and push the operator image to the desired image registry.

Before building the operator image, ensure the generated Dockerfile references the base image you want. You can change the default “runner” image gcr.io/distroless/static:nonroot by replacing its tag with another, for example alpine:latest, and removing the USER 65532:65532 directive.

Your Makefile composes image tags either from values written at project initialization or from the CLI. In particular, IMAGE_TAG_BASE lets you define a common image registry, namespace, and partial name for all your image tags. Update this to another registry and/or namespace if the current value is incorrect. Afterwards you can update the IMG variable definition like so:

-IMG ?= controller:latest
+IMG ?= $(IMAGE_TAG_BASE):$(VERSION)

Once done, you do not have to set IMG or any other image variable in the CLI. The following command will build and push an operator image tagged as example.com/memcached-operator:v0.0.1 to Docker Hub:

make docker-build docker-push

Run the Operator

There are three ways to run the operator:

As a Go program outside a cluster
As a Deployment inside a Kubernetes cluster
Managed by the Operator Lifecycle Manager (OLM) in bundle format

1. Run locally outside the cluster

The following steps will show how to deploy the operator on the cluster. However, to run locally for development purposes and outside of a cluster use the target make install run.

Note that by using this plugin the Operand image informed will be stored via an environment variable in the config/manager/manager.yaml manifest.

Therefore, before running make install run you need to export any environment variable that you might have. Example:

export MEMCACHED_IMAGE="memcached:1.4.36-alpine"

2. Run as a Deployment inside the cluster

By default, a new namespace is created with name <project-name>-system, ex. memcached-operator-system, and will be used for the deployment.

Run the following to deploy the operator. This will also install the RBAC manifests from config/rbac.

make deploy

Verify that the memcached-operator is up and running:

$ kubectl get deployment -n memcached-operator-system
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
memcached-operator-controller-manager   1/1     1            1           8m

3. Deploy your Operator with OLM

First, install OLM:

operator-sdk olm install

Bundle your operator, then build and push the bundle image. The bundle target generates a bundle in the bundle directory containing manifests and metadata defining your operator. bundle-build and bundle-push build and push a bundle image defined by bundle.Dockerfile.

make bundle bundle-build bundle-push

Finally, run your bundle. If your bundle image is hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

operator-sdk run bundle <some-registry>/memcached-operator-bundle:v0.0.1

Check out the docs for a deep dive into operator-sdk's OLM integration.

Create a Memcached CR

Update the sample Memcached CR manifest at config/samples/cache_v1alpha1_memcached.yaml and define the spec as the following:

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  size: 3
  containerPort: 11211

Create the CR:

kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml

Ensure that the memcached operator creates the deployment for the sample CR with the correct size:

$ kubectl get deployment
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
memcached-sample                        3/3     3            3           1m

Check the pods and CR status to confirm the status is updated with the memcached pod names:

$ kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          1m
memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          1m
memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          1m

$ kubectl get memcached/memcached-sample -o yaml
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  clusterName: ""
  creationTimestamp: 2018-03-31T22:51:08Z
  generation: 0
  name: memcached-sample
  namespace: default
  resourceVersion: "245453"
  selfLink: /apis/cache.example.com/v1alpha1/namespaces/default/memcacheds/memcached-sample
  uid: 0026cc97-3536-11e8-bd83-0800274106a1
spec:
  size: 3
status:
  nodes:
  - memcached-sample-6fd7c98d8-7dqdr
  - memcached-sample-6fd7c98d8-g5k7v
  - memcached-sample-6fd7c98d8-m7vn7

Update the size

Update config/samples/cache_v1alpha1_memcached.yaml to change the spec.size field in the Memcached CR from 3 to 5:

kubectl patch memcached memcached-sample -p '{"spec":{"size": 5}}' --type=merge

Confirm that the operator changes the deployment size:

$ kubectl get deployment
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
memcached-sample                        5/5     5            5           3m

Cleanup

Run the following to delete all deployed resources:

kubectl delete -f config/samples/cache_v1alpha1_memcached.yaml
make undeploy

Next steps

Next, check out the following:

Validating and mutating admission webhooks.
Operator packaging and distribution with OLM.
The advanced topics doc for more use cases and under-the-hood details.

Docs: Admission Webhooks

Mon, 01 Jan 0001 00:00:00 +0000

Create a validating or mutating Admission Webhook

Admission webhooks are HTTP callbacks that receive admission requests and do something with them. It is registered with Kubernetes, and will be called by Kubernetes to validate or mutate a resource before being stored. There are two types of admission webhooks.

1. Validating admission webhook

Validating webhooks can be used to perform validations that go beyond the capabilities of OpenAPI schema validation, such as ensuring a field is immutable after creation or higher level permissions checks based on the user that is making the request to the API server. It can reject the request, but it cannot modify the object that they are receiving in the request.

2. Mutating admission webhook

Mutating webhooks are most frequently used for defaulting, by adding default values for unset fields in the resource on creation. They can modify objects by creating a patch that will be sent back in the admission response.

For more background on Admission webhooks, refer to the Kubebuilder documentation or the official Kubernetes documentation. You can also refer to the Kubebuilder webhook walkthrough, which is similar in content to this guide.

Create Validation Webhook

As an example, let’s walk through the scaffolding of a validation webhook for the sample memcached operator.

$ operator-sdk create webhook --group cache --version v1alpha1 --kind Memcached --defaulting --programmatic-validation

After, create webhook command, the following message will appear on the terminal. It scaffolds out api/<version>/<kind>_webhook.go file. In this example, it would be api/v1alpha1/memcached_webhook.go.

Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1alpha1/memcached_webhook.go

The --defaulting flag will scaffold the resources required for a mutating webhook, and the --programmatic-validation flag will scaffold the resources required for a validating webhook. In this case we have scaffolded both.

After running the create webhook command the file structure would be:

├── Dockerfile
├── Makefile
├── PROJECT
├── api
│   └── v1alpha1
│       ├── memcached_webhook.go
│       ├── webhook_suite_test.go
├── config
│   ├── certmanager
│   │   ├── certificate.yaml
│   │   ├── kustomization.yaml
│   │   └── kustomizeconfig.yaml
│   ├── default
│   │   ├── manager_webhook_patch.yaml
│   │   └── webhookcainjection_patch.yaml
│   └── webhook
│       ├── kustomization.yaml
│       ├── kustomizeconfig.yaml
│       └── service.yaml
├── go.mod
├── go.sum
└── main.go

The scaffolded file api/v1alpha1/memcached_webhook.go has method signatures which need to be implemented for the validation webhook.

Following this, there are a few steps which need to be done in your operator project to enable webhooks. This will involve:

Implementing the required methods for Validating or Mutating webhook in <kind>_webhook.go. An example of such implementation is provided here.
Uncommenting sections in config/default/kustomization.yaml to enable webhook and cert-manager configuration through kustomize. Cert-manager (or any third party solution) can be used to provision certificates for webhook server. This is explained in detail here.

Note If OLM is being used to deploy the operator, then the section prefixed with [CERT-MANAGER] need not be uncommented. This is because, OLM currently handles the cert generation and rotation for webhook deployment using self-signed certs. It also does not allow users to specify the name or mount location for the certs. More documentation on this issue can be found here.

Generate webhook manifests and enable webhook deployment

Once your webhooks are implemented, all that’s left is to create the WebhookConfiguration manifests required to register your webhooks with Kubernetes:

$ make manifests

Run your operator and webhooks

There are two ways to test your operator project with webhooks.

Run locally

Technically, the webhooks can be run locally, but for it to work you need to generate certificates for the webhook server and store them at /tmp/k8s-webhook-server/serving-certs/tls.{crt,key}. For more details about running webhook locally, refer here.

Run as a Deployment inside the cluster

Adding webhooks does not alter deploying your operator. For instructions on deploying your operator into a cluster, refer to the tutorial.

Docs: Operators Scope

Mon, 01 Jan 0001 00:00:00 +0000

Overview

A namespace-scoped operator watches and manages resources in a single Namespace, whereas a cluster-scoped operator watches and manages resources cluster-wide.

An operator should be cluster-scoped if it watches resources that can be created in any Namespace. An operator should be namespace-scoped if it is intended to be flexibly deployed. This scope permits decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions.

By default, operator-sdk init scaffolds a cluster-scoped operator. This document details conversion of default operator projects to namespaced-scoped operators. Before proceeding, be aware that your operator may be better suited as cluster-scoped. For example, the cert-manager operator is often deployed with cluster-scoped permissions and watches so that it can manage and issue certificates for an entire cluster.

IMPORTANT: When a Manager instance is created in the main.go file, the Namespaces are set via Cache Config as described below. These Namespaces should be watched and cached for the Client which is provided by the Manager. Only clients provided by cluster-scoped Managers are able to manage cluster-scoped CRD’s. For further information see: CRD scope doc.

Manager watching options

Watching resources in all Namespaces (default)

A Manager is initialized with no Cache option specified, or with a Cache.DefaultNamespaces of Namespace: "" will watch all Namespaces:

...
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    Scheme:             scheme,
    MetricsBindAddress: metricsAddr,
    Port:               9443,
    LeaderElection:     enableLeaderElection,
    LeaderElectionID:   "f1c5ece8.example.com",
})
...

Watching resources in specific Namespaces

To restrict the scope of the Manager’s cache to a specific Namespace, set Cache.DefaultNamespaces field in Options:

...
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    Scheme:             scheme,
    MetricsBindAddress: metricsAddr,
    Port:               9443,
    LeaderElection:     enableLeaderElection,
    LeaderElectionID:   "f1c5ece8.example.com",
    Cache: cache.Options{
      DefaultNamespaces: map[string]cache.Config{"operator-namespace": cache.Config{}},
    },
})
...

Watching resources in a set of Namespaces

It is also possible to use DefaultNamespaces to watch and manage resources in a set of Namespaces:

...
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    Scheme:             scheme,
    MetricsBindAddress: metricsAddr,
    Port:               9443,
    LeaderElection:     enableLeaderElection,
    LeaderElectionID:   "f1c5ece8.example.com",
    Cache: cache.Options{
      DefaultNamespaces: map[string]cache.Config{
        "operator-namespace1": cache.Config{},
        "operator-namespace2": cache.Config{},
      },
    },
})
...

In the above example, a CR created in a Namespace not in the set passed to Cache.DefaultNamespaces will not be reconciled by its controller because the Manager does not manage that Namespace. Further restrictions and qualifications can created on a per-namespace basis by setting fields in the cache.Config object, for further information see the controller runtime docs

IMPORTANT: Note that this is not intended to be used for excluding Namespaces, this is better done via a Predicate.

Restricting Roles and permissions

An operator’s scope defines its Manager’s cache’s scope but not the permissions to access the resources. After updating the Manager’s scope to be Namespaced, Role-Based Access Control (RBAC) permissions applied to the operator’s service account should be restricted accordingly.

These permissions are found in the directory config/rbac/. The ClusterRole in role.yaml and ClusterRoleBinding in role_binding.yaml are used to grant the operator permissions to access and manage its resources.

NOTE For changing the operator’s scope only the role.yaml and role_binding.yaml manifests need to be updated. For the purposes of this doc, the other RBAC manifests <kind>_editor_role.yaml, <kind>_viewer_role.yaml, and auth_proxy_*.yaml are not relevant to changing the operator’s resource permissions.

Changing the permissions to Namespaced

To change the scope of the RBAC permissions from cluster-wide to a specific namespace, you will need to:

Use Roles instead of ClusterRoles.

RBAC markers defined in the controller (e.g controllers/memcached_controller.go) are used to generate the operator’s RBAC ClusterRole (e.g config/rbac/role.yaml). The default markers don’t specify a namespace property and will result in a ClusterRole.

Update the RBAC markers in <kind>_controller.go with namespace=<namespace> where the Role is to be applied, such as:

//+kubebuilder:rbac:groups=cache.example.com,namespace=memcached-operator-system,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=cache.example.com,namespace=memcached-operator-system,resources=memcacheds/status,verbs=get;update;patch

Then run make manifests to update config/rbac/role.yaml. In our example it would look like:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: null
  name: manager-role
  namespace: memcached-operator-system

Use RoleBindings instead of ClusterRoleBindings. The config/rbac/role_binding.yaml needs to be manually updated:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: manager-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: manager-role
subjects:
- kind: ServiceAccount
  name: controller-manager
  namespace: system

Configuring watch namespaces dynamically

Instead of having any Namespaces hard-coded in the main.go file a good practice is to use an environment variable to allow the restrictive configurations. The one suggested here is WATCH_NAMESPACE, a comma-separated list of namespaces passed to the manager at deploy time.

Configuring Namespace scoped operators

Add a helper function in the main.go file:

// getWatchNamespace returns the Namespace the operator should be watching for changes
func getWatchNamespace() (string, error) {
    // WatchNamespaceEnvVar is the constant for env variable WATCH_NAMESPACE
    // which specifies the Namespace to watch.
    // An empty value means the operator is running with cluster scope.
    var watchNamespaceEnvVar = "WATCH_NAMESPACE"

    ns, found := os.LookupEnv(watchNamespaceEnvVar)
    if !found {
        return "", fmt.Errorf("%s must be set", watchNamespaceEnvVar)
    }
    return ns, nil
}

Use the environment variable value:

...
watchNamespace, err := getWatchNamespace()
if err != nil {
    setupLog.Error(err, "unable to get WatchNamespace, " +
       "the manager will watch and manage resources in all namespaces")
}

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    Scheme:             scheme,
    MetricsBindAddress: metricsAddr,
    Port:               9443,
    LeaderElection:     enableLeaderElection,
    LeaderElectionID:   "f1c5ece8.example.com",
    Cache: cache.Options{
      DefaultNamespaces: map[string]cache.Config{watchNamespace: cache.Config{}},
    },
})
...

Define the environment variable in the config/manager/manager.yaml:

spec:
  containers:
  - command:
    - /manager
    args:
    - --leader-elect
    image: controller:latest
    name: manager
    resources:
      limits:
        cpu: 100m
        memory: 30Mi
      requests:
        cpu: 100m
        memory: 20Mi
    env:
      - name: WATCH_NAMESPACE
        valueFrom:
          fieldRef:
            fieldPath: metadata.namespace
  terminationGracePeriodSeconds: 10

NOTE WATCH_NAMESPACE here will always be set as the namespace where the operator is deployed.

Configuring cluster-scoped operators with MultiNamespacedCacheBuilder

Add a helper function to get the environment variable value in the main.go file as done in the previous example (e.g getWatchNamespace())
Use the environment variable value and check if it is a multi-namespace scenario:

    ...
watchNamespace, err := getWatchNamespace()
if err != nil {
    setupLog.Error(err, "unable to get WatchNamespace, " +
        "the manager will watch and manage resources in all Namespaces")
}

options := ctrl.Options{
    Scheme:             scheme,
    MetricsBindAddress: metricsAddr,
    Port:               9443,
    LeaderElection:     enableLeaderElection,
    LeaderElectionID:   "f1c5ece8.example.com",
    Cache: cache.Options{
      DefaultNamespaces: map[string]cache.Config{watchNamespace: cache.Config{}},
    },
}

// Add support for MultiNamespace set in WATCH_NAMESPACE (e.g ns1,ns2)
if strings.Contains(watchNamespace, ",") {
    setupLog.Info("manager set up with multiple namespaces", "namespaces", watchNamespace)
    // configure cluster-scoped with MultiNamespacedCacheBuilder
    options.Namespace = ""
    options.NewCache = cache.MultiNamespacedCacheBuilder(strings.Split(watchNamespace, ","))
}
...

Define the environment variable in the config/manager/manager.yaml:

...
    env:
      - name: WATCH_NAMESPACE
        value: "ns1,ns2"
  terminationGracePeriodSeconds: 10
...

Updating your CSV’s installModes

If your operator is integrated with OLM, you will want to update your CSV base’s spec.installModes list to support the desired namespacing requirements. Support for multiple types of namespacing is allowed, so supporting multiple install modes in a CSV is permitted. After doing so, update your bundle or package manifests by following the linked guides.

Watching resources in all Namespaces (default)

Only the AllNamespaces install mode is supported: true by default, so no changes are required.

Watching resources in a single Namespace

If the operator can watch its own namespace, set the following in your spec.installModes list:

  - type: OwnNamespace
    supported: true

If the operator can watch a single namespace that is not its own, set the following in your spec.installModes list:

  - type: SingleNamespace
    supported: true

Watching resources in multiple Namespaces

If the operator can watch multiple namespaces, set the following in your spec.installModes list:

  - type: MultiNamespace
    supported: true

Docs: CRD Scope

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This page details the various methods to control the scope of a CRD. See the operator scope doc for information on configuring operator scope, such as which namespaces to watch.

Custom Resource Definitions (CRDs) contain a scope field that determines whether the resulting Custom Resource (CR) is cluster or namespace scoped. An operator author might use a namespaced-scoped CRD to restrict access to a CR to certain namespaces, or to have different versions of CRs accessible in different namespaces. Alternatively, an operator author might want a cluster-scoped CRD so all namespaces have visibility and access to CRs.

The CRD manifests are generated by the operator-sdk create api command in config/crd/bases. A CRD’s spec.scope field controls API scope; valid values are Cluster and Namespaced. For an Operator-sdk Go project, this value is determined by the operator-sdk create api --namespaced boolean flag, which edits the types.go file for the resource. For other operator types, the command edits spec.scope in the CRD’s YAML manifest directly.

Set `create api` –namespaced flag

When creating a new API, the --namespaced flag controls whether the resulting CRD will be cluster or namespace scoped. By default, --namespaced is set to true which sets the scope to Namespaced. An example command to create a cluster-scoped API would be:

$ operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource=true --controller=true --namespaced=false

Set Scope Marker in types.go

You can also manually set the scope in the Go types.go file by adding or changing the kubebuilder scope marker to your resource. This file is usually located in api/<version>/<kind>_types.go or apis/<group>/<version>/<kind>_types.go if you are using the multigroup layout. Once this marker is set, the CRD files will be generated with the approriate scope. Here is an example API type with the marker set to cluster scope:

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:resource:scope=Cluster

// Memcached is the Schema for the memcacheds API
type Memcached struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   MemcachedSpec   `json:"spec,omitempty"`
	Status MemcachedStatus `json:"status,omitempty"`
}

To set the scope to namespaced, the marker would be set to //+kubebuilder:resource:scope=Namespaced instead.

Set scope in CRD YAML file

The scope can be manually set directly in the CRD’s Kind YAML file, normally located in config/crd/bases/<group>.<domain>_<kind>.yaml. An example YAML file for a namespace-scoped CRD is shown below:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.2.5
  creationTimestamp: null
  name: memcacheds.cache.example.com
spec:
  group: cache.example.com
  names:
    kind: Memcached
    listKind: MemcachedList
    plural: memcacheds
    singular: memcached
  scope: Namespaced
  subresources:
    status: {}
...

Docs: Testing your Operator project

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The Operator SDK project recommends using controller-runtime’s envtest to write tests for your Operators projects. Envtest has a more active contributor community, it is more mature than Operator SDK’s test framework, and it does not require an actual cluster to run tests which can be a huge benefit in CI scenarios.

Using EnvTest

You will see that controllers/suite_test.go is created when a controller is scaffolded by the tool. This file contains boilerplate for executing integration tests using envtest with ginkgo and gomega. Setup instructions, including those for disconnected environments, are found here.

These tests are runnable as native Go tests:

go test controllers/ -v -ginkgo.v

The projects generated by using the SDK tool have a Makefile which contains the target tests which executes when you run make test. Note that this target will also execute when you run make docker-build IMG=<some-registry>/<project-name>:<tag>.

Operator SDK adopted this stack to write tests for its operators. It might be useful to check writing controller tests documentation and examples to learn how to better write tests for your operator. See, for example, that controller-runtime is covered by tests using the same stack as well.

e2e Integration tests

For Golang-based operators: you can create the e2e tests using Go. See the test directory for the Memcached sample under the testdata/go/v3/memcached-operator to see an example of e2e tests.
For Ansible-based operators: you can use Molecule, an Ansible testing framework. For further information see Testing with Molecule.
For Helm-based operators: you can also use Chart tests.

Alternatively, you can achieve the same goal using shell scripts. The following are a few examples of shell scripts used for testing projects built with SDK 1.0.0:

Other Options

Also, you can write tests for your operator in a declarative format using kuttl. Via kuttl, you can define YAML manifests that specify the expected before and after states of a cluster when your operator is used. For more info see Writing Kuttl Scorecard Tests.

An alternative and more modern solution to kuttl is chainsaw. Chainsaw offers more flexibility, a rich assertion model, and is actively maintained. Tests from kuttl can be automatically converted to chainsaw, see Migration from KUTTL.

To implement application-specific tests, the SDK’s test harness, scorecard, provides the ability to ship custom code in container images as well, which can be referenced in the test suite. Because this test suite definition metadata travels with the Operator Bundle, it allows for functional testing of the Operator without the source code or the project layout being available. See Writing Custom Scorecard Tests.

Docs: Advanced Topics

Mon, 01 Jan 0001 00:00:00 +0000

Manage CR status conditions

An often-used pattern is to include Conditions in the status of custom resources. A Condition represents the latest available observations of an object’s state (see the Kubernetes API conventions documentation for more information).

The Conditions field added to the MemcachedStatus struct simplifies the management of your CR’s conditions. It:

Enables callers to add and remove conditions.
Ensures that there are no duplicates.
Sorts the conditions deterministically to avoid unnecessary repeated reconciliations.
Automatically handles the each condition’s LastTransitionTime.
Provides helper methods to make it easy to determine the state of a condition.

To use conditions in your custom resource, add a Conditions field to the Status struct in _types.go:

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

type MyAppStatus struct {
    // Conditions represent the latest available observations of an object's state
    Conditions []metav1.Condition `json:"conditions"`
}

Then, in your controller, you can use Conditions methods to make it easier to set and remove conditions or check their current values.

Adding 3rd Party Resources To Your Operator

The operator’s Manager supports the core Kubernetes resource types as found in the client-go scheme package and will also register the schemes of all custom resource types defined in your project.

import (
    cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1
    ...
)

func init() {

    // Setup Scheme for all resources
    utilruntime.Must(cachev1alpha1.AddToScheme(scheme))
    //+kubebuilder:scaffold:scheme
}

To add a 3rd party resource to an operator, you must add it to the Manager’s scheme. By creating an AddToScheme() method or reusing one you can easily add a resource to your scheme. An example shows that you define a function and then use the runtime package to create a SchemeBuilder.

Register with the Manager’s scheme

Call the AddToScheme() function for your 3rd party resource and pass it the Manager’s scheme via mgr.GetScheme() or scheme in main.go. Example:

import (
    routev1 "github.com/openshift/api/route/v1"
)

func init() {
    ...

    // Adding the routev1
    utilruntime.Must(clientgoscheme.AddToScheme(scheme))

    utilruntime.Must(routev1.AddToScheme(scheme))
    //+kubebuilder:scaffold:scheme

    ...
}

If 3rd party resource does not have `AddToScheme()` function

Use the SchemeBuilder package from controller-runtime to initialize a new scheme builder that can be used to register the 3rd party resource with the manager’s scheme.

Example of registering DNSEndpoints 3rd party resource from external-dns:

import (
    ...
    "k8s.io/apimachinery/pkg/runtime/schema"
    "sigs.k8s.io/controller-runtime/pkg/scheme"
    ...
    // DNSEndpoints
    externaldns "github.com/kubernetes-incubator/external-dns/endpoint"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )

func init() {
    ...

    log.Info("Registering Components.")

    schemeBuilder := &scheme.Builder{GroupVersion: schema.GroupVersion{Group: "externaldns.k8s.io", Version: "v1alpha1"}}
    schemeBuilder.Register(&externaldns.DNSEndpoint{}, &externaldns.DNSEndpointList{})
    if err := schemeBuilder.AddToScheme(mgr.GetScheme()); err != nil {
        log.Error(err, "")
        os.Exit(1)
    }

    ...
}

NOTES:

After adding new import paths to your operator project, run go mod vendor if a vendor/ directory is present in the root of your project directory to fulfill these dependencies.
Your 3rd party resource needs to be added before add the controller in "Setup all Controllers".

Monitoring and Observability

This section covers how to create custom metrics, alerts and recording rules for your operator. It focuses on the technical aspects, and demonstrates the implementation by updating the sample memcached-operator.

For more information regarding monitoring best practices, take a look at our docs on observability-best-practices.

Prerequisites

The following steps are required in order to inspect the operator’s custom metrics, alerts and recording rules:

Install Prometheus and Prometheus Operator. We recommend using kube-prometheus in production if you don’t have your own monitoring system. If you are just experimenting, you can only install Prometheus and Prometheus Operator.
Make sure Prometheus has access to the operator’s namespace, by setting the corresponding RBAC rules.

Example: prometheus_role.yaml and prometheus_role_binding.yaml

Publishing Custom Metrics

If you wish to publish custom metrics for your operator, this can be easily achieved by using the global registry from controller-runtime/pkg/metrics. One way to achieve this is to declare your collectors as global variables, register them using RegisterMetrics() and call it in the controller’s init() function.

Example custom metric: MemcachedDeploymentSizeUndesiredCountTotal

package monitoring

import (
	"github.com/prometheus/client_golang/prometheus"
	"sigs.k8s.io/controller-runtime/pkg/metrics"
)

var (
	MemcachedDeploymentSizeUndesiredCountTotal = prometheus.NewCounter(
		prometheus.CounterOpts{
			Name: "memcached_deployment_size_undesired_count_total",
			Help: "Total number of times the deployment size was not as desired.",
		},
	)
)

// RegisterMetrics will register metrics with the global prometheus registry
func RegisterMetrics() {
	metrics.Registry.MustRegister(MemcachedDeploymentSizeUndesiredCountTotal)
}

The above example creates a new Counter metric. For other metrics’ types, see Prometheus Documentation.
For more information regarding operators metrics best-practices, please follow observability-best-practices.

init() function example:

package main


import (
   ...
   "github.com/example/memcached-operator/monitoring"
)

func init() {
   ...
   monitoring.RegisterMetrics()
   ...
}

The next step would be to set the controller’s logic according to which we update the metric’s value. In this case, the new metric type is Counter, thus a valid update operation would be to increment its value.

Metric update example:

...
size := memcached.Spec.Size
if *found.Spec.Replicas != size {
    // Increment MemcachedDeploymentSizeUndesiredCountTotal metric by 1
    monitoring.MemcachedDeploymentSizeUndesiredCountTotal.Inc()
}
...

Different metrics types have different valid operations. For more information, please follow Prometheus Golang client.

Publishing Alerts and Recording Rules

In order to add alerts and recording rules, which are unique to the operator’s needs, we’ll create a dedicated PrometheusRule object, by using prometheus-operator API.

PrometheusRule example:

package monitoring

import (
	monitoringv1 "github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/util/intstr"
)

// NewPrometheusRule creates new PrometheusRule(CR) for the operator to have alerts and recording rules
func NewPrometheusRule(namespace string) *monitoringv1.PrometheusRule {
	return &monitoringv1.PrometheusRule{
		TypeMeta: metav1.TypeMeta{
			APIVersion: monitoringv1.SchemeGroupVersion.String(),
			Kind:       "PrometheusRule",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      "memcached-operator-rules",
			Namespace: "memcached-operator-system",
		},
		Spec: *NewPrometheusRuleSpec(),
	}
}

// NewPrometheusRuleSpec creates PrometheusRuleSpec for alerts and recording rules
func NewPrometheusRuleSpec() *monitoringv1.PrometheusRuleSpec {
	return &monitoringv1.PrometheusRuleSpec{
		Groups: []monitoringv1.RuleGroup{{
			Name: "memcached.rules",
			Rules: []monitoringv1.Rule{
				createOperatorUpTotalRecordingRule(), 
				createOperatorDownAlertRule()
			},
		}},
	}
}

// createOperatorUpTotalRecordingRule creates memcached_operator_up_total recording rule
func createOperatorUpTotalRecordingRule() monitoringv1.Rule {
   return monitoringv1.Rule{
      Record: "memcached_operator_up_total",
      Expr:   intstr.FromString("sum(up{pod=~'memcached-operator-controller-manager-.*'} or vector(0))"),
   }
}

// createOperatorDownAlertRule creates MemcachedOperatorDown alert rule
func createOperatorDownAlertRule() monitoringv1.Rule {
	return monitoringv1.Rule{
		Alert: "MemcachedOperatorDown",
		Expr:  intstr.FromString("memcached_operator_up_total == 0"),
		Annotations: map[string]string{
			"description": "No running memcached-operator pods were detected in the last 5 min.",
		},
		For: "5m",
		Labels: map[string]string{
			"severity":    "critical",
		},
	}
}

Then, we may want to ensure that the new PrometheusRule is being created and reconciled. One way to achieve this is by expanding the existing Reconcile() function logic.

PrometheusRule reconciliation example:

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...
    ...
    // Check if prometheus rule already exists, if not create a new one
    foundRule := &monitoringv1.PrometheusRule{}
    err := r.Get(ctx, types.NamespacedName{Name: ruleName, Namespace: namespace}, foundRule)
    if err != nil && apierrors.IsNotFound(err) {
        // Define a new prometheus rule
        prometheusRule := monitoring.NewPrometheusRule(namespace)
        if err := r.Create(ctx, prometheusRule); err != nil {
            log.Error(err, "Failed to create prometheus rule")
            return ctrl.Result{}, nil
        }
    }
    
    if err == nil {
    // Check if prometheus rule spec was changed, if so set as desired
    desiredRuleSpec := monitoring.NewPrometheusRuleSpec()
    if !reflect.DeepEqual(foundRule.Spec.DeepCopy(), desiredRuleSpec) {
        desiredRuleSpec.DeepCopyInto(&foundRule.Spec)
        if r.Update(ctx, foundRule); err != nil {
            log.Error(err, "Failed to update prometheus rule")
            return ctrl.Result{}, nil
        }
    }
    ...
    ...
}

Please review the observability-best-practices for additional important information regarding alerts and recording rules.

Alerts Unit Testing

It is highly recommended implementing unit tests for prometheus rules. For more information, please follow the Prometheus unit testing documentation. For examples of unit testing in a Golang operator, see the sample memcached-operator alerts unit tests.

Inspecting the metrics, alerts and recording rules with Prometheus UI

Finally, in order to inspect the exposed metrics and alerts, we need to forward the corresponding port where metrics are published by Prometheus (usually 9090, which is the default value). This can be done with the following command:

$ kubectl -n monitoring port-forward svc/prometheus-k8s 9090

where we assume that the prometheus service is available in the monitoring namespace.

Now you can access Prometheus UI using http://localhost:9090. For more details on exposing prometheus metrics, please refer kube-prometheus docs.

Handle Cleanup on Deletion

Operators may create objects as part of their operational duty. Object accumulation can consume unnecessary resources, slow down the API and clutter the user interface. As such it is important for operators to keep good hygiene and to clean up resources when they are not needed. Here are a few common scenarios.

Internal Resources

A typical example of correct resource cleanup is the Jobs implementation. When a Job is created, one or multiple Pods are created as child resources. When a Job is deleted, the associated Pods are deleted as well. This is a very common pattern easily achieved by setting an owner reference from the parent (Job) to the child (Pod) object. Here is a code snippet for doing so, where “r” is the reconcilier and “ctrl” the controller-runtime library:

ctrl.SetControllerReference(job, pod, r.Scheme)

Note that the default behavior for cascading deletion is background propagation, meaning deletion requests for child objects occur after the request to delete the parent object. This Kubernetes doc provides alternative deletion types.

External Resources

Sometimes external resources or resources that are not owned by a custom resource, those across namespaces for example, need to be cleaned up when the parent resource is deleted. In that case Finalizers can be leveraged. A deletion request for an object with a finalizer becomes an update during which a deletion timestamp is set; the object is not deleted while the finalizer is present. The reconciliation loop of the custom resource’s controller will then need to check whether a the deletion timestamp is set, perform the external cleanup operation(s), then remove the finalizer to allow garbage collection of the object. Multiple finalizers may be present on an object, each with a key that should indicate what external resources require deletion by the controller.

The following is a snippet from a theoretical controller file controllers/memcached_controller.go that implements a finalizer handler:

import (
    ...
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
)

const memcachedFinalizer = "cache.example.com/finalizer"

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    reqLogger := r.log.WithValues("memcached", req.NamespacedName)
    reqLogger.Info("Reconciling Memcached")

    // Fetch the Memcached instance
    memcached := &cachev1alpha1.Memcached{}
    err := r.Get(ctx, req.NamespacedName, memcached)
    if err != nil {
        if errors.IsNotFound(err) {
            // Request object not found, could have been deleted after reconcile request.
            // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
            // Return and don't requeue
            reqLogger.Info("Memcached resource not found. Ignoring since object must be deleted.")
            return ctrl.Result{}, nil
        }
        // Error reading the object - requeue the request.
        reqLogger.Error(err, "Failed to get Memcached.")
        return ctrl.Result{}, err
    }

    ...

    // Check if the Memcached instance is marked to be deleted, which is
    // indicated by the deletion timestamp being set.
    isMemcachedMarkedToBeDeleted := memcached.GetDeletionTimestamp() != nil
    if isMemcachedMarkedToBeDeleted {
        if controllerutil.ContainsFinalizer(memcached, memcachedFinalizer) {
            // Run finalization logic for memcachedFinalizer. If the
            // finalization logic fails, don't remove the finalizer so
            // that we can retry during the next reconciliation.
            if err := r.finalizeMemcached(reqLogger, memcached); err != nil {
                return ctrl.Result{}, err
            }

            // Remove memcachedFinalizer. Once all finalizers have been
            // removed, the object will be deleted.
            controllerutil.RemoveFinalizer(memcached, memcachedFinalizer)
            err := r.Update(ctx, memcached)
            if err != nil {
                return ctrl.Result{}, err
            }
        }
        return ctrl.Result{}, nil
    }

    // Add finalizer for this CR
    if !controllerutil.ContainsFinalizer(memcached, memcachedFinalizer) {
        controllerutil.AddFinalizer(memcached, memcachedFinalizer)
        err = r.Update(ctx, memcached)
        if err != nil {
            return ctrl.Result{}, err
        }
    }

    ...

    return ctrl.Result{}, nil
}

func (r *MemcachedReconciler) finalizeMemcached(reqLogger logr.Logger, m *cachev1alpha1.Memcached) error {
    // TODO(user): Add the cleanup steps that the operator
    // needs to do before the CR can be deleted. Examples
    // of finalizers include performing backups and deleting
    // resources that are not owned by this CR, like a PVC.
    reqLogger.Info("Successfully finalized memcached")
    return nil
}

Complex cleanup logic

Similar to the previous scenario, finalizers can be used for implementing complex cleanup logic. Take CronJobs as an example: the controller maintains limited-size lists of jobs that have been created by the CronJob controller to check for deletion. These list sizes are configured by the CronJob fields .spec.successfulJobsHistoryLimit and .spec.failedJobsHistoryLimit, which specify how many completed and failed jobs should be kept. Check out the Kubebuilder CronJob tutorial for full implementation details.

Sensitive resources

Sensitive resources need to be protected against unintended deletion. An intuitive example of protecting resources is the PersistentVolume (PV) / PersistentVolumeClaim (PVC) relationship. A PV is first created, after which users can request access to that PV’s storage by creating a PVC, which gets bound to the PV. If a user tries to delete a PV currently bound by a PVC, the PV is not removed immediately. Instead, PV removal is postponed until the PV is not bound to any PVC. Finalizers again can be leveraged to achieve a similar behaviour for your own PV-like custom resources: by setting a finalizer on an object, your controller can make sure there are no remaining objects bound to it before removing the finalizer and deleting the object. Additionally, the user who created the PVC can specify what happens to the underlying storage allocated in a PV when the PVC is deleted through the reclaim policy. There are several options available, each of which defines a behavior that is achieved again through the use of finalizers. The key concept to take away is that your operator can give a user the power to decide how their resources are cleaned up via finalizers, which may be dangerous yet useful depending on your workloads.

Leader election

During the lifecycle of an operator it’s possible that there may be more than 1 instance running at any given time e.g when rolling out an upgrade for the operator. In such a scenario it is necessary to avoid contention between multiple operator instances via leader election so that only one leader instance handles the reconciliation while the other instances are inactive but ready to take over when the leader steps down.

There are two different leader election implementations to choose from, each with its own tradeoff.

Leader-with-lease: The leader pod periodically renews the leader lease and gives up leadership when it can’t renew the lease. This implementation allows for a faster transition to a new leader when the existing leader is isolated, but there is a possibility of split brain in certain situations.
Leader-for-life: The leader pod only gives up leadership (via garbage collection) when it is deleted. This implementation precludes the possibility of 2 instances mistakenly running as leaders (split brain). However, this method can be subject to a delay in electing a new leader. For instance when the leader pod is on an unresponsive or partitioned node, the pod-eviction-timeout dictates how long it takes for the leader pod to be deleted from the node and step down (default 5m).

By default the SDK enables the leader-with-lease implementation. However you should consult the docs above for both approaches to consider the tradeoffs that make sense for your use case.

The following examples illustrate how to use the two options:

Leader for life

A call to leader.Become() will block the operator as it retries until it can become the leader by creating the configmap named memcached-operator-lock.

import (
    ...
    "github.com/operator-framework/operator-lib/leader"
)

func main() {
    ...
    err = leader.Become(context.TODO(), "memcached-operator-lock")
    if err != nil {
        log.Error(err, "Failed to retry for leader lock")
        os.Exit(1)
    }
    ...
}

If the operator is not running inside a cluster leader.Become() will simply return without error to skip the leader election since it can’t detect the operator’s namespace.

Leader with lease

The leader-with-lease approach can be enabled via the Manager Options for leader election.

import (
    ...
    "sigs.k8s.io/controller-runtime/pkg/manager"
)

func main() {
    ...
    opts := manager.Options{
        ...
        LeaderElection: true,
        LeaderElectionID: "memcached-operator-lock"
    }
    mgr, err := manager.New(cfg, opts)
    ...
}

When the operator is not running in a cluster, the Manager will return an error on starting since it can’t detect the operator’s namespace in order to create the configmap for leader election. You can override this namespace by setting the Manager’s LeaderElectionNamespace option.

Multiple architectures

Authors may decide to distribute their bundles for various architectures: x86_64, aarch64, ppc64le, s390x, etc, to accommodate the diversity of Kubernetes clusters and reach a larger number of potential users. Each architecture requires however compatible binaries. Considerations on the topic are available in the Multiple Architectures page.

Docs: Go-based Operator Reference

Mon, 01 Jan 0001 00:00:00 +0000

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The motivations for the new layout are related to bringing more flexibility to users and part of the process to Integrating Kubebuilder and Operator SDK. Because of this integration you may be referred to the Kubebuilder documentation https://book.kubebuilder.io/ for more information about certain topics. When using this document just remember to replace $ kubebuilder <command> with $ operator-sdk <command>.

Note: It is recommended that you have your project upgraded to the latest SDK v1.y release version before following the steps in this guide to migrate to the new layout. However, the steps might work from previous versions as well. In this case, if you find an issue which is not covered here then check the previous Migration Guides which might help out.

What was changed

The deploy directory was replaced with the config directory including a new layout of Kubernetes manifests files:
- CRD manifests in deploy/crds/ are now in config/crd/bases
- CR manifests in deploy/crds/ are now in config/samples
- Controller manifest deploy/operator.yaml is now in config/manager/manager.yaml
- RBAC manifests in deploy are now in config/rbac/
build/Dockerfile is moved to Dockerfile in the project root directory
pkg/apis and pkg/controllers are now in the root directory.
cmd/manager/main.go is now in the root directory.

What is new

Scaffolded projects now use:

kustomize to manage Kubernetes resources needed to deploy your operator
A Makefile with helpful targets to build, test, deploy and tailor things based on your project needs
Helpers and options to work with webhooks. For further information see What is a Webhook?
Updated metrics configuration using kube-auth-proxy, a --metrics-addr flag, and kustomize-based deployment of a Kubernetes Service and prometheus operator ServiceMonitor
Scaffolded tests that use the envtest test framework
Preliminary support for CLI plugins. For more info see the plugins design document
A PROJECT configuration file to store information about GVKs, plugins, and help the CLI make decisions
A new option to create projects using ComponentConfig. For more info, see the enhancement proposal and the [Component config tutorial][component-config-tutorial]
Go version 1.15 (previously it was 1.13)

Generated files with the default API versions:

apiextensions/v1 for generated CRDs (apiextensions/v1beta1 was deprecated in Kubernetes 1.16 and will be removed in 1.22)
admissionregistration.k8s.io/v1 for webhooks (admissionregistration.k8s.io/v1beta1 was deprecated in Kubernetes 1.16 and will be removed in 1.22 )
cert-manager.io/v1 for the certificate manager when webhooks are used (cert-manager.io/v1alpha2 was deprecated in Cert-Manager 0.14. More info: CertManager v1.0 docs)

Note: You can still use the deprecated APIs which are only needed to support Kubernetes 1.15 and earlier.

How to migrate

The easy migration path is to initialize a new project, re-recreate APIs, then copy pre-v1.0.0 configuration files into the new project.

Prerequisites

Go through the installation guide.
Make sure your user is authorized with cluster-admin permissions.
An accessible image registry for various operator images (ex. hub.docker.com, quay.io) and be logged in to your command line environment.
- example.com is used as the registry Docker Hub namespace in these examples. Replace it with another value if using a different registry or namespace.
- Authentication and certificates if the registry is private or uses a custom CA.

Create a new project

In Kubebuilder-style projects, CRD groups are defined using two different flags (--group and --domain).

When we initialize a new project, we need to specify the domain that all APIs in our project will share, so before creating the new project, we need to determine which domain we’re using for the APIs in our existing project.

To determine the domain, look at the spec.group field in your CRDs in the deploy/crds directory.

The domain is everything after the first DNS segment. Using cache.example.com as an example, the --domain would be example.com.

So let’s create a new project with the same domain (example.com):

mkdir memcached-operator
cd memcached-operator
operator-sdk init --domain example.com --repo github.com/example/memcached-operator

Note:: operator-sdk attempts to automatically discover the Go module path of your project by looking for a go.mod file, or if in $GOPATH, by using the directory path. Use the --repo flag to explicitly set the module path.

Check if your project is multi-group

Before we start creating the APIs, check if your project has more than one group such as: foo.example.com/v1 and crew.example.com/v1. If you intend to work with multiple groups in your project, then run the command operator-sdk edit --multigroup=true to change the project’s layout to support multi-group.

Note: In multi-group projects, APIs are defined in apis/<group>/<version> and controllers are defined in controllers/<group>. For further information see Single Group to Multi-Group.

Migrate APIs and Controllers

Now that we have our new project initialized, we need to re-create each of our APIs. Using our API example from earlier (cache.example.com), we’ll use cache for the --group, v1alpha1 for the --version and Memcached for --kind flag.

For each API in the existing project, run:

operator-sdk create api \
    --group=cache \
    --version=<version> \
    --kind=<Kind> \
    --resource \
    --controller

How to keep `apiextensions.k8s.io/v1beta1` for CRDs?

From now on, the CRDs that will be created by controller-gen will be using Kubernetes API version apiextensions.k8s.io/v1 by default, instead of apiextensions.k8s.io/v1beta1.

The apiextensions.k8s.io/v1beta1 was deprecated in Kubernetes 1.16 and will be removed in Kubernetes 1.22.

If you would like to keep using the previous version, use the flag --crd-version=v1beta1 in the above command. This is only needed if you want your operator to support Kubernetes 1.15 and earlier.

APIs

Now let’s copy the API definition from pkg/apis/<group>/<version>/<kind>_types.go to api/<version>/<kind>_types.go. For our example, it is only required to copy the code from the Spec and Status fields.

This file is quite similar to the old one. Once you copy over your API definitions and generate manifests, you should end up with an identical API for your custom resource type. However, pay close attention to these kubebuilder Markers:

The +k8s:deepcopy-gen:interfaces=... marker was replaced with +kubebuilder:object:root=true.
If you are not using openapi-gen to generate OpenAPI Go code, then // +k8s:openapi-gen=true and other related openapi markers can be removed.

Note:: The operator-sdk generate openapi command was deprecated in 0.13.0 and was removed in 0.17 SDK release. Hence, it is recommended to use openapi-gen directly for OpenAPI code generation.

Our Memcached API types will look like:

// MemcachedSpec defines the desired state of Memcached
type MemcachedSpec struct {
	// Size is the size of the memcached deployment
	Size int32 `json:"size"`
}

// MemcachedStatus defines the observed state of Memcached
type MemcachedStatus struct {
	// Nodes are the names of the memcached pods
	Nodes []string `json:"nodes"`
}

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status

// Memcached is the Schema for the memcacheds API
type Memcached struct {...}

//+kubebuilder:object:root=true

// MemcachedList contains a list of Memcached
type MemcachedList struct {...}

Webhooks

SDK version 1.0.0 and later has support for webhooks by the CLI. If your project doesn’t require any webhooks, you can skip this section. However, if you have been using it via customizations in your project, you should use the tool to re-scaffold the webhooks.

A webhook can only be scaffolded for a pre-existent API in your project. Then, for each case you will run the command operator-sdk create webhook providing the --group, --kind and version of the API based on the flags that need to be used.

The valid flags for its types are: --defaulting, --programmatic-validation and --conversion. Use the same type that was used for scaffolding. To create defaulting and validating webhook:

operator-sdk create webhook \
    --group=cache \
    --version=<version> \
    --kind=<Kind> \
    --defaulting \
    --programmatic-validation

To create conversion webhook:

operator-sdk create webhook \
    --group=cache \
    --version=<version> \
    --kind=<Kind> \
    --conversion

After the webhook is generated, you will need to copy the webhook definition and content from your old project to the new one. You can find the respective file in api/v1/<kind>_webhook.go.

How to keep using `apiextensions.k8s.io/v1beta1` for Webhooks?

Hereafter, the webhooks that are created by SDK will use Kubernetes API version admissionregistration.k8s.io/v1 by default instead of admissionregistration.k8s.io/v1beta1 and cert-manager.io/v1 instead of cert-manager.io/v1alpha2.

Note that apiextensions/v1beta1 and admissionregistration.k8s.io/v1beta1 were deprecated in Kubernetes 1.16 and will be removed in Kubernetes 1.22. If you use apiextensions/v1 and admissionregistration.k8s.io/v1, then you need to use cert-manager.io/v1 which will be the default API adopted by the SDK CLI.

Note: If you are using the API cert-manager.io/v1alpha2, it is not compatible with the latest Kubernetes API version. (cert-manager.io/v1alpha2 was deprecated in Cert-Manager 0.14. For more info, refer to CertManager v1.0 docs)

If you would like to use the previous version, use the flag --webhook-version=v1beta1 in the above command which is only required if you want your operator to support Kubernetes 1.15 and earlier.

Controllers

Now let’s migrate the controller code from pkg/controller/<kind>/<kind>_controller.go to controllers/<kind>_controller.go following these steps:

Copy over any struct fields from the existing project into the new <Kind>Reconciler struct. Note: The Reconciler struct has been renamed from Reconcile<Kind> to <Kind>Reconciler. In our example, we would see ReconcileMemcached instead of MemcachedReconciler.
Replace the // your logic here in the new layout with your reconcile logic.
Copy the code under func add(mgr manager.Manager, r reconcile.Reconciler) to func SetupWithManager:

func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&cachev1alpha1.Memcached{}).
		Owns(&appsv1.Deployment{}).
		Complete(r)
}

In our example, the Watch implemented for the Deployment will be replaced with Owns(&appsv1.Deployment{}). Setting up controller Watches is simplified in more recent versions of controller-runtime, which has controller Builder helpers to handle more of the details.

Set RBAC permissions

The RBAC permissions are now configured via RBAC markers, which are used to generate and update the manifest files present in config/rbac/. These markers can be found (and should be defined) on the Reconcile() method of each controller.

In the Memcached example, they look like the following:

//+kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=pods,verbs=get;list

To update config/rbac/role.yaml after changing the markers, run make manifests.

By default, new projects are cluster-scoped (i.e. they have cluster-scoped permissions and watch all namespaces). Read the operator scope documentation for more information about changing the scope of your operator.

See the complete migrated memcached_controller.go code here.

Note: The version of controller-runtime used in the projects scaffolded by SDK 0.19.x+ was v0.6.0. Please check sigs.k8s.io/controller-runtime release docs from 0.7.0+ version for breaking changes.

Updating your ServiceAccount

New Go projects come with a ServiceAccount controller-manager in config/rbac/service_account.yaml. Your project’s RoleBinding and ClusterRoleBinding subjects, and Deployments spec.template.spec.serviceAccountName that reference a ServiceAccount already refer to this new name. When you run make deploy, your project’s name will be prepended to controller-manager, making it unique within a namespace, much like your old deploy/service_account.yaml. If you wish to use the old ServiceAccount, make sure to update all RBAC bindings and your manager Deployment.

Migrate `main.go`

By checking our new main.go we will find that:

The SDK leader.Become was replaced by the controller-runtime’s leader with lease mechanism. However, you can still use leader.Become if you wish:

func main() {
...
	ctx := context.TODO()
	// Become the leader before proceeding
	err = leader.Become(ctx, "memcached-operator-lock")
	if err != nil {
    	log.Error(err, "")
    	os.Exit(1)
	}
...
}

In order to use the previous one ensure that you have the operator-lib as a dependency of your project.

The default port used by the metric endpoint binds to :8080 from the previous :8383. To continue using port 8383, specify --metrics-bind-address=:8383 when you start the operator.
OPERATOR_NAME and POD_NAME environment variables are no longer used. OPERATOR_NAME was used to define the name for a leader election config map. Operator authors should use the LeaderElectionID attribute from the Manager Options which is hardcoded in main.go:

func main() {
...
	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		Scheme:                 scheme,
		MetricsBindAddress:     metricsAddr,
		Port:                   9443,
		HealthProbeBindAddress: probeAddr,
		LeaderElection:         enableLeaderElection,
		LeaderElectionID:       "86f835c3.example.com",
	})
...
}

Ensure that you copy all customizations made in cmd/manager/main.go to main.go. You’ll also need to ensure that all needed schemes have been registered, if you have been using third-party APIs (i.e Route Api from OpenShift).

Migrate your tests

For the new layout, you will see that controllers/suite_test.go is created when a controller is scaffolded by the tool. This file contains boilerplate for executing integration tests using envtest with ginkgo and gomega.

Operator SDK 1.0.0+ removes support for the legacy test framework and no longer supports the operator-sdk test subcommand. All affected tests should be migrated to use envtest.

The Operator SDK project recommends controller-runtime’s envtest because it has a more active contributor community, it is more mature than Operator SDK’s test framework, and it does not require an actual cluster to run tests, which can be a huge benefit in CI scenarios.

To learn more about how you can test your controllers, see the documentation about writing controller tests.

Migrate your Custom Resources

Custom resource samples are stored in ./config/samples using the new project structure. Copy the examples from your existing project into this directory. In existing projects, CR files have the format ./deploy/crds/<group>.<domain>_<version>_<kind>_cr.yaml.

In our example, we’ll copy the specs from deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml to config/samples/cache_v1alpha1_memcached.yaml

Configure your Operator

In case your project has customizations in the deploy/operator.yaml, it needs to be added to config/manager/manager.yaml. Note that OPERATOR_NAME and POD_NAME env vars are no longer used. For further information, check out the section Migrate main.go .

Export Metrics

If you are using metrics and would like to keep them exported, see that the func addMetrics() is no longer generated in the main.go and it is now configurable via kustomize.

Configure Prometheus metrics

Ensure that you have Prometheus installed in the cluster: To check if you have the required API resource to create the ServiceMonitor run:

kubectl api-resources | grep servicemonitors

If not, you can install Prometheus via kube-prometheus:

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.33/bundle.yaml

Now uncomment the line - ../prometheus in the config/default/kustomization.yaml file. It creates the ServiceMonitor resource which enables exporting the metrics:

# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
- ../prometheus

Use Handler from `operator-lib`

By using the InstrumentedEnqueueRequestForObject you will be able to export metrics from your Custom Resources. In our example, it would look like:

import (
    ...
	"github.com/operator-framework/operator-lib/handler"
    ...
)

func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
	// Create a new controller
	c, err := controller.New("memcached-controller", mgr, controller.Options{Reconciler: r})
	if err != nil {
		return err
	}
    ...
	err = c.Watch(&source.Kind{Type: &cachev1alpha1.Memcached{}}, &handler.InstrumentedEnqueueRequestForObject{})
	if err != nil {
		return err
	}
	...
	return nil
}

Note: Ensure that you have the operator-lib added to your go.mod.

In this way, the following metric with the resource info will be exported:

resource_created_at_seconds{"name", "namespace", "group", "version", "kind"}

Note: To check it you can create a pod to curl the metrics/ endpoint but note that it is now protected by the kube-auth-proxy which means that you will need to create a ClusterRoleBinding and obtain the token from the ServiceAccount’s secret which will be used in the requests. Otherwise, to test you can disable the kube-auth-proxy as well.

For more info see the metrics.

Operator image

The Dockerfile image also changes and now it is multi-stage, distroless and still rootless. However, users can change it to work as they want.

You might need to port some customizations made in your old Dockerfile as well. Also, if you wish to still use the previous UBI image replace:

# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
FROM gcr.io/distroless/static:nonroot

With:

FROM registry.access.redhat.com/ubi8/ubi-minimal:latest

Generate Manifests and Build the operator

Note that:

operator-sdk generate crds is replaced with make manifests, which generates CRDs and RBAC rules.
operator-sdk build is replaced with make docker-build IMG=<some-registry>/<project-name>:<tag>.

In this way, run:

make manifests docker-build docker-push IMG=example.com/memcached-operator:v0.0.1

Verify the migration

The project can now be deployed on cluster by running the command:

make deploy IMG=example.com/memcached-operator:v0.0.1

You can troubleshoot your deployment by checking container logs:

kubectl logs deployment.apps/memcached-operator-controller-manager -n memcached-operator-system -c manager

For further steps regarding the deployment of the operator, creation of custom resources, and cleaning up of resources, see the tutorial.

Operator SDK – Go

Docs: Installation Guide

Install operator-sdk

Additional Prerequisites

Docs: Quickstart for Go-based Operators

Prerequisites

Steps

OLM deployment

Direct deployment

Run locally (outside the cluster)

Next Steps

Docs: Go Operator Tutorial

Prerequisites

Overview

Create a new project

A note on dependency management

Manager

Create a new API and Controller

Understanding Kubernetes APIs

Define the API

Generating CRD manifests

OpenAPI validation

Implement the Controller

Setup a Recorder

Resources watched by the Controller

Controller Configurations

Reconcile loop

Specify permissions and generate RBAC manifests

Configure the operator’s image registry

Run the Operator

1. Run locally outside the cluster

2. Run as a Deployment inside the cluster

3. Deploy your Operator with OLM

Create a Memcached CR

Update the size

Cleanup

Next steps

Docs: Admission Webhooks

Create a validating or mutating Admission Webhook

1. Validating admission webhook

2. Mutating admission webhook

Create Validation Webhook

Generate webhook manifests and enable webhook deployment

Run your operator and webhooks

Run locally

Run as a Deployment inside the cluster

Docs: Operators Scope

Overview

Manager watching options

Watching resources in all Namespaces (default)

Watching resources in specific Namespaces

Watching resources in a set of Namespaces

Restricting Roles and permissions

Changing the permissions to Namespaced

Configuring watch namespaces dynamically

Configuring Namespace scoped operators

Configuring cluster-scoped operators with MultiNamespacedCacheBuilder

Updating your CSV’s installModes

Watching resources in all Namespaces (default)

Watching resources in a single Namespace

Watching resources in multiple Namespaces

Docs: CRD Scope

Overview

Set create api –namespaced flag

Set Scope Marker in types.go

Set scope in CRD YAML file

Docs: Testing your Operator project

Overview

Using EnvTest

e2e Integration tests

Other Options

Docs: Advanced Topics

Manage CR status conditions

Adding 3rd Party Resources To Your Operator

Register with the Manager’s scheme

If 3rd party resource does not have AddToScheme() function

Monitoring and Observability

Prerequisites

Publishing Custom Metrics

Publishing Alerts and Recording Rules

Install `operator-sdk`

Set `create api` –namespaced flag

If 3rd party resource does not have `AddToScheme()` function

How to keep `apiextensions.k8s.io/v1beta1` for CRDs?

How to keep using `apiextensions.k8s.io/v1beta1` for Webhooks?

Migrate `main.go`

Use Handler from `operator-lib`