Operator SDK – Go-based Operator Reference

Docs: Controller Runtime Client API

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The controller-runtime library provides various abstractions to watch and reconcile resources in a Kubernetes cluster via CRUD (Create, Update, Delete, as well as Get and List in this case) operations. Operators use at least one controller to perform a coherent set of tasks within a cluster, usually through a combination of CRUD operations. The Operator SDK uses controller-runtime’s Client interface, which provides the interface for these operations.

controller-runtime defines several interfaces used for cluster interaction:

client.Client: implementers perform CRUD operations on a Kubernetes cluster.
manager.Manager: manages shared dependencies, such as Caches and Clients.
reconcile.Reconciler: compares provided state with actual cluster state and updates the cluster on finding state differences using a Client.

Clients are the focus of this document. A separate document will discuss Managers.

Note: this document uses parts of the sample memcached-operator for example code. Import paths may be different for brevity.

Client Usage

Default Client

The SDK relies on a manager.Manager to create a client.Client interface that performs Create, Update, Delete, Get, and List operations within a reconcile.Reconciler's Reconcile function. The SDK will generate code to create a Manager, which holds a Cache and a Client to be used in CRUD operations and communicate with the API server. By default a Controller’s Reconciler will be populated with the Manager’s Client which is a split-client.

The following code, found in controllers/memcached_controller.go, demonstrates how the Manager’s client is passed to a reconciler.

import (
	appsv1 "k8s.io/api/apps/v1"
	ctrl "sigs.k8s.io/controller-runtime"

	cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
)

func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr). // mgr's Client is passed to r.
		For(&cachev1alpha1.Memcached{}).
		Owns(&appsv1.Deployment{}).
		Complete(r)
}

type MemcachedReconciler struct {
    client.Client // Populated above from a manager.Manager.

    Log    logr.Logger
    Scheme *runtime.Scheme
}

A split client reads (Get and List) from the Cache and writes (Create, Update, Delete) to the API server. Reading from the Cache significantly reduces request load on the API server; as long as the Cache is updated by the API server, read operations are eventually consistent.

Non-default Client

An operator developer may wish to create their own Client that serves read requests(Get List) from the API server instead of the cache, for example. controller-runtime provides a constructor for Clients:

// New returns a new Client using the provided config and Options.
func New(config *rest.Config, options client.Options) (client.Client, error)

client.Options allow the caller to specify how the new Client should communicate with the API server.

// Options are creation options for a Client
type Options struct {
    // Scheme, if provided, will be used to map go structs to GroupVersionKinds
    Scheme *runtime.Scheme

    // Mapper, if provided, will be used to map GroupVersionKinds to Resources
    Mapper meta.RESTMapper
}

Example:

import (
    "sigs.k8s.io/controller-runtime/pkg/client/config"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

cfg, err := config.GetConfig()
...
c, err := client.New(cfg, client.Options{})
...

Note: defaults are set by client.New when Options are empty. The default scheme will have the core Kubernetes resource types registered. The caller must set a scheme that has custom operator types registered for the new Client to recognize these types.

Creating a new Client is not usually necessary nor advised, as the default Client is sufficient for most use cases.

Reconcile and the Client API

A Reconciler implements the reconcile.Reconciler interface, which exposes the Reconcile method. Reconcilers are added to a corresponding Controller for a Kind; Reconcile is called in response to cluster or external Events, with a reconcile.Request object argument, to read and write cluster state by the Controller, and returns a ctrl.Result. SDK Reconcilers have access to a Client in order to make Kubernetes API calls.

// MemcachedReconciler reconciles a Memcached object
type MemcachedReconciler struct {
    // client, initialized using mgr.Client() above, is a split client
    // that reads objects from the cache and writes to the apiserver
    client.Client

    Log    logr.Logger

    // scheme defines methods for serializing and deserializing API objects,
    // a type registry for converting group, version, and kind information
    // to and from Go schemas, and mappings between Go schemas of different
    // versions. A scheme is the foundation for a versioned API and versioned
    // configuration over time.
    Scheme *runtime.Scheme
}

// Reconcile watches for Events and reconciles cluster state with desired
// state defined in the method body.
// The Controller will requeue the Request to be processed again if an error
// is non-nil or Result.Requeue is true, otherwise upon completion it will
// remove the work from the queue.
func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile is where Controller business logic lives, i.e. where Client API calls are made via MemcachedReconciler.client. A client.Client implementer performs the following operations:

Get

// Get retrieves an API object for a given object key from the Kubernetes cluster
// and stores it in obj.
func (c Client) Get(ctx context.Context, key client.ObjectKey, obj client.Object) error

Note: A client.ObjectKey is simply an alias for types.NamespacedName.

Example:

import (
    "context"

    ctrl "sigs.k8s.io/controller-runtime"

    cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    memcached := &cachev1alpha1.Memcached{}
    err := r.Get(ctx, request.NamespacedName, memcached)

    ...
}

List

// List retrieves a list of objects for a given namespace and list options
// and stores the list in obj.
func (c Client) List(ctx context.Context, list client.Object, opts ...client.ListOption) error

A client.ListOption is an interface that sets client.ListOptions fields. A client.ListOption is created by using one of the provided implementations: MatchingLabels, MatchingFields, InNamespace.

Example:

import (
    "context"
    "fmt"

    "k8s.io/api/core/v1"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    // Return all pods in the request namespace with a label of `instance=<name>`
    // and phase `Running`.
    podList := &v1.PodList{}
    opts := []client.ListOption{
        client.InNamespace(request.NamespacedName.Namespace),
        client.MatchingLabels{"instance": request.NamespacedName.Name},
        client.MatchingFields{"status.phase": "Running"},
    }
    err := r.List(ctx, podList, opts...)

    ...
}

Create

// Create saves the object obj in the Kubernetes cluster.
// Returns an error
func (c Client) Create(ctx context.Context, obj client.Object, opts ...client.CreateOption) error

A client.CreateOption is an interface that sets client.CreateOptions fields. A client.CreateOption is created by using one of the provided implementations: DryRunAll, ForceOwnership. Generally these options are not needed.

Example:

import (
    "context"

    "k8s.io/api/apps/v1"
    ctrl "sigs.k8s.io/controller-runtime"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    dep := &v1.Deployment{ // Any cluster object you want to create.
        ...
    }
    err := r.Create(ctx, dep)

    ...
}

Update

// Update updates the given obj in the Kubernetes cluster. obj must be a
// struct pointer so that obj can be updated with the content returned
// by the API server. Update does *not* update the resource's status
// subresource
func (c Client) Update(ctx context.Context, obj client.Object, opts ...client.UpdateOption) error

A client.UpdateOption is an interface that sets client.UpdateOptions fields. A client.UpdateOption is created by using one of the provided implementations: DryRunAll, ForceOwnership. Generally these options are not needed.

Example:

import (
    "context"

    "k8s.io/api/apps/v1"
    ctrl "sigs.k8s.io/controller-runtime"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    dep := &v1.Deployment{}
    err := r.Get(ctx, request.NamespacedName, dep)

    ...

    dep.Spec.Selector.MatchLabels["is_running"] = "true"
    err := r.Update(ctx, dep)

    ...
}

Patch

// Patch patches the given obj in the Kubernetes cluster. obj must be a
// struct pointer so that obj can be updated with the content returned by the Server.
func (c Client) Patch(ctx context.Context, obj client.Object, patch client.Patch, opts ...client.PatchOption) error

A client.PatchOption is an interface that sets client.PatchOptions fields. A client.PatchOption is created by using one of the provided implementations: DryRunAll, ForceOwnership. Generally these options are not needed.

Example:

import (
    "context"

    "k8s.io/api/apps/v1"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    dep := &v1.Deployment{}
    err := r.Get(ctx, request.NamespacedName, dep)

    ...

    // A merge patch will preserve other fields modified at runtime.
    patch := client.MergeFrom(dep.DeepCopy())
    dep.Spec.Selector.MatchLabels["is_running"] = "true"
    err := r.Patch(ctx, dep, patch)

    ...
}

Updating Status Subresource

When updating the status subresource from the client, the StatusWriter must be used. The status subresource is retrieved with Status() and updated with Update() or patched with Patch().

Update() takes variadic client.UpdateOption's, and Patch() takes variadic client.PatchOption's. See Client.Update() and Client.Patch() for more details. Generally these options are not needed.

Status

// Status() returns a StatusWriter object that can be used to update the
// object's status subresource
func (c Client) Status() (client.StatusWriter, error)

Example:

import (
    "context"

    ctrl "sigs.k8s.io/controller-runtime"

    cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    mem := &cachev1alpha1.Memcached{}
    err := r.Get(ctx, request.NamespacedName, mem)

    ...

    // Update
    mem.Status.Nodes = []string{"pod1", "pod2"}
    err := r.Status().Update(ctx, mem)

    ...

    // Patch
    patch := client.MergeFrom(mem.DeepCopy())
    mem.Status.Nodes = []string{"pod1", "pod2", "pod3"}
    err := r.Status().Patch(ctx, mem, patch)

    ...
}

Delete

// Delete deletes the given obj from Kubernetes cluster.
func (c Client) Delete(ctx context.Context, obj client.Object, opts ...client.DeleteOption) error

A client.DeleteOption is an interface that sets client.DeleteOptions fields. A client.DeleteOption is created by using one of the provided implementations: GracePeriodSeconds, Preconditions, PropagationPolicy.

Example:

import (
    "context"

    "k8s.io/api/core/v1"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    pod := &v1.Pod{}
    err := r.Get(ctx, request.NamespacedName, pod)

    ...

    if pod.Status.Phase == v1.PodUnknown {
        // Delete the pod after 5 seconds.
        err := r.Delete(ctx, pod, client.GracePeriodSeconds(5))
        ...
    }

    ...
}

DeleteAllOf

// DeleteAllOf deletes all objects of the given type matching the given options.
func (c Client) DeleteAllOf(ctx context.Context, obj client.Object, opts ...client.DeleteAllOfOption) error

A client.DeleteAllOfOption is an interface that sets client.DeleteAllOfOptions fields. A client.DeleteAllOfOption wraps a client.ListOption and client.DeleteOption.

Example:

import (
    "context"
    "fmt"

    "k8s.io/api/core/v1"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    // Delete all pods in the request namespace with a label of `instance=<name>`
    // and phase `Failed`.
    pod := &v1.Pod{}
    opts := []client.DeleteAllOfOption{
        client.InNamespace(request.NamespacedName.Namespace),
        client.MatchingLabels{"instance", request.NamespacedName.Name},
        client.MatchingFields{"status.phase": "Failed"},
        client.GracePeriodSeconds(5),
    }
    err := r.DeleteAllOf(ctx, pod, opts...)

    ...
}

Example usage

import (
    "context"
    "reflect"

    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/labels"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/types"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"

    cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
)

type MemcachedReconciler struct {
    client.Client
    Log    logr.Logger
    Scheme *runtime.Scheme
}

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {

    // Fetch the Memcached instance.
    memcached := &cachev1alpha1.Memcached{}
    err := r.Get(ctx, request.NamespacedName, memcached)
    if err != nil {
        if errors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }

    // Check if the deployment already exists, if not create a new deployment.
    found := &appsv1.Deployment{}
    err = r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found)
    if err != nil {
         if errors.IsNotFound(err) {
            // Define and create a new deployment.
            dep := r.deploymentForMemcached(memcached)
            if err = r.Create(ctx, dep); err != nil {
                return ctrl.Result{}, err
            }
            return ctrl.Result{Requeue: true}, nil
        } else {
            return ctrl.Result{}, err
        }
    }

    // Ensure the deployment size is the same as the spec.
    size := memcached.Spec.Size
    if *found.Spec.Replicas != size {
        found.Spec.Replicas = &size
        if err = r.Update(ctx, found); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{Requeue: true}, nil
    }

    // Update the Memcached status with the pod names.
    // List the pods for this CR's deployment.
    podList := &corev1.PodList{}
    listOpts := []client.ListOption{
        client.InNamespace(memcached.Namespace),
        client.MatchingLabels(labelsForApp(memcached.Name)),
    }
    if err = r.List(ctx, podList, listOpts...); err != nil {
        return ctrl.Result{}, err
    }

    // Update status.Nodes if needed.
    podNames := getPodNames(podList.Items)
    if !reflect.DeepEqual(podNames, memcached.Status.Nodes) {
        memcached.Status.Nodes = podNames
        if err := r.Status().Update(ctx, memcached); err != nil {
            return ctrl.Result{}, err
        }
    }

    return ctrl.Request{}, nil
}

// deploymentForMemcached returns a Deployment object for data from m.
func (r *MemcachedReconciler) deploymentForMemcached(m *cachev1alpha1.Memcached) *appsv1.Deployment {
    lbls := labelsForApp(m.Name)
    replicas := m.Spec.Size

    dep := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      m.Name,
            Namespace: m.Namespace,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: lbls,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: lbls,
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Image:   "memcached:alpine",
                        Name:    "memcached",
                        Command: []string{"memcached", "-a=64", "-b"},
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: 10000,
                            Name:          "memcached",
                        }},
                    }},
                },
            },
        },
    }

    // Set Memcached instance as the owner and controller.memcac
    // NOTE: calling SetControllerReference, and setting owner references in
    // general, is important as it allows deleted objects to be garbage collected.
    controllerutil.SetControllerReference(m, dep, r.Scheme)
    return dep
}

// labelsForApp creates a simple set of labels for Memcached.
func labelsForApp(name string) map[string]string {
    return map[string]string{"cr_name": name}
}

Docs: Logging

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Operator SDK-generated operators use the logr interface to log. This log interface has several backends such as zap, which the SDK uses in generated code by default. logr.Logger exposes structured logging methods that help create machine-readable logs and adding a wealth of information to log records.

Default zap logger

Operator SDK uses a zap-based logr backend when scaffolding new projects. To assist with configuring and using this logger, the SDK includes several helper functions.

In the simple example below, we add the zap flagset to the operator’s command line flags with BindFlags(), and then set the controller-runtime logger with zap.Options{}.

By default, zap.Options{} will return a logger that is ready for production use. It uses a JSON encoder, logs starting at the info level. To customize the default behavior, users can use the zap flagset and specify flags on the command line. The zap flagset includes the following flags that can be used to configure the logger:

--zap-devel: Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn) Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
--zap-encoder: Zap log encoding (‘json’ or ‘console’)
--zap-log-level: Zap Level to configure the verbosity of logging. Can be one of ‘debug’, ‘info’, ‘error’, or any integer value > 0 which corresponds to custom debug levels of increasing verbosity”)
--zap-stacktrace-level: Zap Level at and above which stacktraces are captured (one of ‘info’ or ‘error’)

Consult the controller-runtime godocs for more detailed flag information.

A simple example

Operators set the logger for all operator logging in main.go. To illustrate how this works, try out this simple example:

package main

import (
	"sigs.k8s.io/controller-runtime/pkg/log/zap"
	logf "sigs.k8s.io/controller-runtime/pkg/log"
)

var globalLog = logf.Log.WithName("global")
func main() {
	// Add the zap logger flag set to the CLI. The flag set must
	// be added before calling flag.Parse().
	opts := zap.Options{}
	opts.BindFlags(flag.CommandLine)
	flag.Parse()

	logger := zap.New(zap.UseFlagOptions(&opts))
	logf.SetLogger(logger)

	scopedLog := logf.Log.WithName("scoped")

	globalLog.Info("Printing at INFO level")
	globalLog.V(1).Info("Printing at DEBUG level")
	scopedLog.Info("Printing at INFO level")
	scopedLog.V(1).Info("Printing at DEBUG level")
}

Output using the defaults

$ go run main.go
INFO[0000] Running the operator locally in namespace default.
{"level":"info","ts":1587741740.407766,"logger":"global","msg":"Printing at INFO level"}
{"level":"info","ts":1587741740.407855,"logger":"scoped","msg":"Printing at INFO level"}

Output overriding the log level to 1 (debug)

$ go run main.go --zap-log-level=debug
INFO[0000] Running the operator locally in namespace default.
{"level":"info","ts":1587741837.602911,"logger":"global","msg":"Printing at INFO level"}
{"level":"debug","ts":1587741837.602964,"logger":"global","msg":"Printing at DEBUG level"}
{"level":"info","ts":1587741837.6029708,"logger":"scoped","msg":"Printing at INFO level"}
{"level":"debug","ts":1587741837.602973,"logger":"scoped","msg":"Printing at DEBUG level"}

Custom zap logger

In order to use a custom zap logger, zap from controller-runtime can be utilized to wrap it in a logr implementation.

Below is an example illustrating the use of zap-logfmt in logging.

Example

In your main.go file, replace the current implementation for logs inside the main function:

...
// Add the zap logger flag set to the CLI. The flag set must
// be added before calling flag.Parse().
	opts := zap.Options{}
	opts.BindFlags(flag.CommandLine)
	flag.Parse()

	logger := zap.New(zap.UseFlagOptions(&opts))
	logf.SetLogger(logger)
...

With:

	import(
	...
	zaplogfmt "github.com/sykesm/zap-logfmt"
	uzap "go.uber.org/zap"
	"go.uber.org/zap/zapcore"
	logf "sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/log/zap"
	...
)
	configLog := uzap.NewProductionEncoderConfig()
	configLog.EncodeTime = func(ts time.Time, encoder zapcore.PrimitiveArrayEncoder) {
		encoder.AppendString(ts.UTC().Format(time.RFC3339Nano))
	}
	logfmtEncoder := zaplogfmt.NewEncoder(configLog)

	// Construct a new logr.logger.
	logger := zap.New(zap.UseDevMode(true), zap.WriteTo(os.Stdout), zap.Encoder(logfmtEncoder))
	logf.SetLogger(logger)

NOTE: For this example, you will need to add the module "github.com/sykesm/zap-logfmt" to your project. Run go get -u github.com/sykesm/zap-logfmt.

Output using custom zap logger

$ go run main.go
ts=2020-04-30T20:35:59.551268Z level=info logger=global msg="Printing at INFO level"
ts=2020-04-30T20:35:59.551314Z level=debug logger=global msg="Printing at DEBUG level"
ts=2020-04-30T20:35:59.551318Z level=info logger=scoped msg="Printing at INFO level"
ts=2020-04-30T20:35:59.55132Z level=debug logger=scoped msg="Printing at DEBUG level"

By using sigs.k8s.io/controller-runtime/pkg/log, your logger is propagated through controller-runtime. Any logs produced by controller-runtime code will be through your logger, and therefore have the same formatting and destination.

Setting flags when running locally

When running locally with make run ENABLE_WEBHOOKS=false, you can use the ARGS var to pass additional flags to your operator, including the zap flags. For example:

$ make run ARGS="--zap-encoder=console" ENABLE_WEBHOOKS=false

Make sure to have your run target to take ARGS as shown below in Makefile.

# Run against the configured Kubernetes cluster in ~/.kube/config
run: manifests generate fmt vet
	go run ./main.go $(ARGS)

Setting flags when deploying to a cluster

When deploying your operator to a cluster you can set additional flags using an args array in your operator’s container spec in the file config/default/manager_metrics_patch.yaml For example:

- op: add
  path: /spec/template/spec/containers/0/args/0
  value: --zap-log-level=debug 
- op: add
  path: /spec/template/spec/containers/0/args/0
  value: --zap-encoder=console

Creating a structured log statement

There are two ways to create structured logs with logr. You can create new loggers using log.WithValues(keyValues) that include keyValues, a list of key-value pair interface{}'s, in each log record. Alternatively you can include keyValues directly in a log statement, as all logr log statements take some message and keyValues. The signature of logr.Error() has an error-type parameter, which can be nil.

An example from memcached_controller.go:

package memcached

import (
  ctrllog "sigs.k8s.io/controller-runtime/pkg/log"
)


// MemcachedReconciler reconciles a Memcached object
type MemcachedReconciler struct {
	client.Client
	Log    logr.Logger
	Scheme *runtime.Scheme
}

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := ctrllog.FromContext(ctx)

	// Fetch the Memcached instance
	memcached := &cachev1alpha1.Memcached{}
	err := r.Get(ctx, req.NamespacedName, memcached)
	if err != nil {
		if errors.IsNotFound(err) {
			// Request object not found, could have been deleted after reconcile request.
			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
			// Return and don't requeue
			log.Info("Memcached resource not found. Ignoring since object must be deleted")
			return ctrl.Result{}, nil
		}
		// Error reading the object - requeue the request.
		log.Error(err, "Failed to get Memcached")
		return ctrl.Result{}, err
	}

	// Check if the deployment already exists, if not create a new one
	found := &appsv1.Deployment{}
	err = r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found)
	if err != nil && errors.IsNotFound(err) {
		// Define a new deployment
		dep := r.deploymentForMemcached(memcached)
		log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
		err = r.Create(ctx, dep)
		if err != nil {
			log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
			return ctrl.Result{}, err
		}
		// Deployment created successfully - return and requeue
		return ctrl.Result{Requeue: true}, nil
	} else if err != nil {
		log.Error(err, "Failed to get Deployment")
		return ctrl.Result{}, err
	}

	...
}

Log records will look like the following (from log.Error() above):

2020-04-27T09:14:15.939-0400	ERROR	controllers.Memcached	Failed to create new Deployment	{"memcached": "default/memcached-sample", "Deployment.Namespace": "default", "Deployment.Name": "memcached-sample"}

Non-default logging

If you do not want to use logr as your logging tool, you can remove logr-specific statements without issue from your operator’s code, including the logr setup code in main.go, and add your own. Note that removing logr setup code will prevent controller-runtime from logging.

Docs: Proxy Friendly Operators

Mon, 01 Jan 0001 00:00:00 +0000

Proxy-friendly Operators should inspect their environment for the standard proxy variables (HTTPS_PROXY, HTTP_PROXY, and NO_PROXY) and pass the values to Operands.

Operator-lib provides a helper function proxy.ReadProxyVarsFromEnv that does this inspection, all you need to do is append the results to the Operand environments.

Using the memcached tutorial as an example, add the following to the Reconcile loop in controllers/memcached_controller.go:

import (
  ...
   "github.com/operator-framework/operator-lib/proxy"
)


for i, container := range dep.Spec.Template.Spec.Containers {
		dep.Spec.Template.Spec.Containers[i].Env = append(container.Env, proxy.ReadProxyVarsFromEnv()...)
}

You can set the environment variable on the Operator deployment. Using the memcached tutorial, edit config/manager/manager.yaml:

containers:
 - args:
   - --leader-elect
   - --leader-election-id=go-proxy-demo
   image: controller:latest
   name: manager
   env:
     - name: "HTTP_PROXY"
       value: "http_proxy_test"

Docs: Using Predicates for Event Filtering with Operator SDK

Mon, 01 Jan 0001 00:00:00 +0000

Events are produced by Sources assigned to resources a controller is watching. These events are transformed into Requests by EventHandlers and passed to Reconcile(). Predicates allow controllers to filter events before they are provided to EventHandlers. Filtering is useful because your controller may only want to handle specific types of events. Filtering also helps reduce chattiness with the API server, as Reconcile() is only called for events transformed by EventHandlers.

Predicate types

A Predicate implements the following methods that take an event of a particular type and return true if the event should be processed by Reconcile():

// Predicate filters events before enqueuing the keys.
type Predicate interface {
  Create(event.CreateEvent) bool
  Delete(event.DeleteEvent) bool
  Update(event.UpdateEvent) bool
  Generic(event.GenericEvent) bool
}

// Funcs implements Predicate.
type Funcs struct {
  CreateFunc func(event.CreateEvent) bool
  DeleteFunc func(event.DeleteEvent) bool
  UpdateFunc func(event.UpdateEvent) bool
  GenericFunc func(event.GenericEvent) bool
}

For example, all Create events for any watched resource will be passed to Funcs.Create() and filtered out if the method evaluates to false. If you do not register a Predicate method for a particular type, events of that type will not be filtered.

All event types contain Kubernetes metadata about the object that triggered the event, and the object itself. Predicate logic uses these data to make decisions about what should be filtered. Some event types include other fields pertaining to the semantics of that event. For example, event.UpdateEvent includes both old and new metadata and objects:

type UpdateEvent struct {
  // ObjectOld is the object from the event.
  ObjectOld runtime.Object

  // ObjectNew is the object from the event.
  ObjectNew runtime.Object
}

You can find all type definitions in the event package documentation.

Using Predicates

Any number of Predicates can be set for a controller via the builder method WithEventFilter(), which will filter an event if any of those Predicates evaluates to false. This first example is an implementation of a memcached-operator controller that simply filters Delete events on Pods that have been confirmed deleted; the controller receives all Delete events that occur, and we may only care about resources that have not been completely deleted:

import (
	"context"

	"github.com/go-logr/logr"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/event"
	"sigs.k8s.io/controller-runtime/pkg/predicate"

	cachev1alpha1 "github.com/example/app-operator/api/v1alpha1"
)

...

func ignoreDeletionPredicate() predicate.Predicate {
	return predicate.Funcs{
		UpdateFunc: func(e event.UpdateEvent) bool {
			// Ignore updates to CR status in which case metadata.Generation does not change
			return e.ObjectOld.GetGeneration() != e.ObjectNew.GetGeneration()
		},
		DeleteFunc: func(e event.DeleteEvent) bool {
			// Evaluates to false if the object has been confirmed deleted.
			return !e.DeleteStateUnknown
		},
	}
}

func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&cachev1alpha1.Memcached{}).
		Owns(&corev1.Pod{}).
		WithEventFilter(ignoreDeletionPredicate()).
		Complete(r)
}
  ...
}

Use cases

Predicates are not necessary for many operators, although filtering reduces the amount of chatter to the API server from Reconcile(). They are particularly useful for controllers that watch resources cluster-wide, i.e. without a namespace.

Docs: API Markers

Mon, 01 Jan 0001 00:00:00 +0000

This document describes code markers supported by the SDK.

ClusterServiceVersion markers

This section details ClusterServiceVersion (CSV) code markers and lists available markers.

Note: CSV markers can only be used in Go Operator projects. Annotations for Ansible and Helm Operator projects will be added in the future.

Usage

All CSV markers have the prefix +operator-sdk:csv.

`+operator-sdk:csv:customresourcedefinitions`

These markers populate owned customresourcedefinitions in your CSV.

Possible type-level markers:

+operator-sdk:csv:customresourcedefinitions:displayName="some display name"
- Configures the kind’s display name.
+operator-sdk:csv:customresourcedefinitions:resources={{Kind1,v1alpha1,dns-name-1},{Kind2,v1,"dns-name-2"},...}
- Configures the kind’s resources.
+operator-sdk:csv:customresourcedefinitions:order=1
- Configures the order of this type in the list. Markers with order omitted have the highest order, i.e. are at the end of the list. If more than one marker has the same order, the corresponding descriptions will be sorted alphabetically and placed above others with higher orders. Pre-existing list elements in the CSV will be appended to the set of other elements in the order corresponding to its index.

Possible field-level markers, all of which must contain the type=[spec,status] key-value pair:

+operator-sdk:csv:customresourcedefinitions:type=[spec,status],displayName="some field display name"
- Configures the field’s display name.
+operator-sdk:csv:customresourcedefinitions:type=[spec,status],xDescriptors={"urn:alm:descriptor:com.tectonic.ui:podCount","urn:alm:descriptor:io.kubernetes:custom"}
- Configures the field’s x-descriptors.
+operator-sdk:csv:customresourcedefinitions:type=[spec,status],order=1
- Configures the order of this type in the list. Markers with order omitted have the highest order, i.e. are at the end of the list. If more than one marker has the same order, the corresponding descriptions will be sorted alphabetically and placed above others with higher orders.

Top-level kind, name, and version fields are parsed from API code. All description fields are parsed from type declaration and struct type field comments. All path fields are parsed from a field’s JSON tag and merged with parent field path’s in dot-hierarchy notation.

x-descriptors

Check out the descriptor reference for available x-descriptors paths.

Examples

These examples assume Memcached, MemcachedSpec, and MemcachedStatus are the example projects’ kind, spec, and status.

Set a displayName and resources for a customresourcedefinitions kind entry:

//+operator-sdk:csv:customresourcedefinitions:displayName="Memcached App",resources={{Pod,v1,memcached-runner},{Deployment,v1,memcached-deployment}}
type Memcached struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   MemcachedSpec   `json:"spec,omitempty"`
    Status MemcachedStatus `json:"status,omitempty"`
}

Set displayName, path, xDescriptors, and description on a field for a customresourcedefinitions.specDescriptors entry:

type MemcachedSpec struct {
    // Size is the size of the memcached deployment. <-- This will become Size's specDescriptors.description.
    //+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Number of pods",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:podCount","urn:alm:descriptor:io.kubernetes:custom"}
    Size int32 `json:"size"` // <-- Size's specDescriptors.path is inferred from this JSON tag.
}

Let the SDK infer all unmarked paths on a field for a customresourcedefinitions.specDescriptors entry:
```
type MemcachedSpec struct {
    // Size is the size of the memcached deployment.
    //+operator-sdk:csv:customresourcedefinitions:type=spec
    Size int32 `json:"size"`
}
```
``

The SDK uses the Size fields’ json tag name as path, Size as displayName, and field comments as description.

A comprehensive example:

Infer path, description, displayName, and x-descriptors for specDescriptors and statusDescriptors entries.
Create three resources entries each with kind, version, and name values.

// Represents a cluster of Memcached apps
//+operator-sdk:csv:customresourcedefinitions:displayName="Memcached App",resources={{Pod,v1,memcached-runner},{Deployment,v1,memcached-deployment}}
type Memcached struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   MemcachedSpec   `json:"spec,omitempty"`
    Status MemcachedStatus `json:"status,omitempty"`
}

type MemcachedSpec struct {
    Pods MemcachedPods `json:"pods"`
}

type MemcachedStatus struct {
    Pods 		 MemcachedPods `json:"podStatuses"`
    //+operator-sdk:csv:customresourcedefinitions:type=status,displayName="Pod Count",xDescriptors="urn:alm:descriptor:com.tectonic.ui:podCount"
    PodCount int 					 `json:"podCount"`
}

type MemcachedPods struct {
    // Size is the size of the memcached deployment.
    //+operator-sdk:csv:customresourcedefinitions:type=spec
    //+operator-sdk:csv:customresourcedefinitions:type=status
    Size int32 `json:"size"`
}

The generated customresourcedefinitions will look like:

customresourcedefinitions:
  owned:
  - description: Represents a cluster of Memcached apps
    displayName: Memcached App
    kind: Memcached
    name: memcacheds.cache.example.com
    version: v1alpha1
    resources:
    - kind: Deployment
      name: memcached-deployment
      version: v1
    - kind: Pod
      name: memcached-runner
      version: v1
    specDescriptors:
    - description: The desired number of member Pods for the deployment.
      displayName: Size
      path: pods.size
    statusDescriptors:
    - description: The desired number of member Pods for the deployment.
      displayName: Size
      path: podStatuses.size
    - displayName: Size
      path: podCount
      x-descriptors:
      - 'urn:alm:descriptor:com.tectonic.ui:podCount'

Deprecated markers

Markers supported by operator-sdk prior to v1.0.0 are deprecated. You can migrate to the new marker system by running the following script:

$ curl -sSLo migrate-markers.sh https://raw.githubusercontent.com/operator-framework/operator-sdk/master/hack/generate/migrate-markers.sh
$ chmod +x ./migrate-markers.sh
$ ./migrate-markers.sh path/to/*_types.go

Docs: OpenAPI validation

Mon, 01 Jan 0001 00:00:00 +0000

OpenAPIv3 schemas are added to CRD manifests in the spec.validation block when the manifests are generated. This validation block allows Kubernetes to validate the properties in a Memcached Custom Resource when it is created or updated.

Markers (annotations) are available to configure validations for your API. These markers will always have a +kubebuilder:validation prefix.

Usage of markers in API code is discussed in the kubebuilder CRD generation and marker documentation. A full list of OpenAPIv3 validation markers can be found here.

To learn more about OpenAPI v3.0 validation schemas in CRDs, refer to the Kubernetes Documentation.