Operator SDK – Best Practices

Docs: Operator Best Practices

Mon, 01 Jan 0001 00:00:00 +0000

Development

Considerations for Operator developers:

An Operator should manage a single type of application, essentially following the UNIX principle: do one thing and do it well.
If an application consists of multiple tiers or components, multiple Operators should be written one for each of them. For example, if the application consists of Redis, AMQ, and MySQL, there should be 3 Operators, not one.
If there is significant orchestration and sequencing involved, an Operator should be written that represents the entire stack, in turn delegating to other Operators for orchestrating their part of it.
Operators should own a CRD and only one Operator should control a CRD on a cluster. Two Operators managing the same CRD is not a recommended best practice. An API that exists with multiple implementations is a typical example of a no-op Operator. The no-op Operator doesn’t have any deployment or reconciliation loop to define the shared API and other Operators depend on this Operator to provide one implementation of the API, e.g. similar to PVCs or Ingress.
Inside an Operator, multiple controllers should be used if multiple CRDs are managed. This helps in separation of concerns and code readability. Note that this doesn’t necessarily mean that we need to have one container image per controller, but rather one reconciliation loop (which could be running as part of the same Operator binary) per CRD.
An Operator shouldn’t deploy or manage other operators (such patterns are known as meta or super operators or include CRDs in its Operands). It’s the Operator Lifecycle Manager’s job to manage the deployment and lifecycle of operators. For further information check Dependency Resolution.
If multiple operators are packaged and shipped as a single entity by the same CSV for example, then it is recommended to add all owned and required CRDs, as well as all deployments for operators that manage the owned CRDs, to the same CSV.
Writing an Operator involves using the Kubernetes API, which in most scenarios will be built using same boilerplate code. Use a framework like the Operator SDK to save yourself time with this and to also get a suite of tooling to ease development and testing.
Operators shouldn’t make any assumptions about the namespace they are deployed in and they should not use hard-coded names of resources that they expect to already exist.
Operators shouldn’t hard code the namespaces they are watching. This should be configurable - having no namespace supplied is interpreted as watching all namespaces
Semantic versioning (aka semver) should be used to version an Operator. Operators are long-running workloads on the cluster and its APIs are potentially in need of support over a longer period of time. Use the semver.org guidelines to help determine when and how to bump versions when there are breaking or non-breaking changes.
Kubernetes API versioning guidelines should be used to version Operator CRDs. Use the Kubernetes sig-architecture guidelines to get best practices on when to bump versions and when breaking changes are acceptable.
When defining CRDs, you should use OpenAPI spec to create a structural schema for your CRDs.
Operators are instrumented to provide useful, actionable metrics to external systems (e.g. monitoring/alerting platforms). Minimally, metrics should represent the software’s health and key performance indicators, as well as support the creation of service levels indicators such as throughput, latency, availability, errors, capacity, etc.
Operators may create objects as part of their operational duty. Object accumulation can consume unnecessary resources, slow down the API and clutter the user interface. As such it is important for operators to keep good hygiene and to clean up resources when they are not needed. Here are instructions on how to handle cleanup on deletion.

Summary

One Operator per managed application
Multiple operators should be used for complex, multi-tier application stacks
CRD can only be owned by a single Operator, shared CRDs should be owned by a separate Operator
One controller per custom resource definition
Use a tool like Operator SDK
Do not hard-code namespaces or resources names
Make watch namespace configurable
Use semver / observe Kubernetes guidelines on versioning APIs
Use OpenAPI spec with structural schema on CRDs
Operators expose metrics to external systems
Operators cleanup resources on deletion

Running On-Cluster

Considerations for on-cluster behavior

Like all containers on Kubernetes, Operators need not run as root unless absolutely necessary. Operators should come with their own ServiceAccount and not rely on the default.
Operators should not self-register their CRDs. These are global resources and careful consideration needs to be taken when setting those up. Also this requires the Operator to have global privileges which is potentially dangerous compared to that little extra convenience.
Operators use CRs as the primary interface to the cluster user. As such, at all times, meaningful status information should be written to those objects unless they are solely used to store data in a structured schema.
Operators should be updated frequently according to server versioning.
Operators need to support updating managed applications (Operands) that were set up by an older version of the Operator. There are multiple models for this:

Model	Description
Operator fan-out	where the Operator allows the user to specify the version in the custom resource
single version	where the Operator is tied to the version of the operand.
hybrid approach	where the Operator is tied to a range of versions, and the user can select some level of the version.

An Operator should not deploy another Operator - an additional component on cluster should take care of this (OLM).
When Operators change their APIs, CRD conversion (webhooks) should be used to deal with potentially older instances of them using the previous API version.
Operators should make it easy for users to use their APIs - validating and rejecting malformed requests via extensive Open API validation schema on CRDs or via an admission webhook is good practice.
The Operator itself should be really modest in its requirements - it should always be able to deploy by deploying its controllers, no user input should be required to start up the Operator.
If user input is required to change the configuration of the Operator itself, a Configuration CRD should be used. Init-containers as part of the Operator deployments can be used to create a default instance of those CRs and then the Operator manages their lifecycle.

Summary:

On the cluster, an Operator…

Does not run as root
Does not self-register CRDs
Does not install other Operators
Does rely on dependencies via package manager (OLM)
Writes meaningful status information on Custom Resources objects unless pure data structure
Should be capable of updating from a previous version of the Operator
Should be capable of managing an Operand from an older Operator version
Uses CRD conversion (webhooks) if API/CRDs change
Uses OpenAPI validation / Admission Webhooks to reject invalid CRs
Should always be able to deploy and come up without user input
Offers (pre)configuration via a “Configuration CR” instantiated by InitContainers

Docs: Common recommendations and suggestions

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Any recommendations or best practices suggested by the Kubernetes community, such as how to develop Operator pattern solutions or how to use controller-runtime are good recommendations for those who are looking to build operator projects with operator-sdk. Also, see Operator Best Practices. However, here are some common recommendations.

Common Recommendations

Develop idempotent reconciliation solutions

When developing operators, it is essential for the controller’s reconciliation loop to be idempotent. By following the Operator pattern you will create Controllers which provide a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster. Breaking this recommendation goes against the design principles of controller-runtime and may lead to unforeseen consequences such as resources becoming stuck and requiring manual intervention.

Understanding Kubernetes APIs

Building your own operator commonly involves extending the Kubernetes API itself. It is helpful to understand exactly how Custom Resource Definitions interact with the Kubernetes API. Also, the Kubebuilder documentation on Groups and Versions and Kinds may be helpful to better understand these concepts as they relate to operators.

Avoid a design solution where more than one Kind is reconciled by the same controller

Having many Kinds (such as CRDs) which are all managed by the same controller usually goes against the design proposed by controller-runtime. Furthermore this might hurt concepts such as encapsulation, the Single Responsibility Principle, and Cohesion. Damaging these concepts may cause unexpected side effects, and increase the difficulty of extending, reusing, or maintaining the operator.

Ideally Operators does not manage other Operators

From best practices:

“Operators should own a CRD and only one Operator should control a CRD on a cluster. Two Operators managing the same CRD is not a recommended best practice. In the case where an API exists but with multiple implementations, this is typically an example of a no-op Operator because it doesn’t have any deployment or reconciliation loop to define the shared API and other Operators depend on this Operator to provide one implementation of the API, e.g. similar to PVCs or Ingress."
“An Operator shouldn’t deploy or manage other operators (such patterns are known as meta or super operators or include CRDs in its Operands). It’s the Operator Lifecycle Manager’s job to manage the deployment and lifecycle of operators. For further information check Dependency Resolution."

What does it mainly mean:

If you want to define that your Operator depends on APIs which are owned by another Operator or on another whole Operator itself you should use Operator Lifecycle Manager’s Dependency Resolution
If you want to reconcile core APIs (defined by Kubernetes) or External APIs (defined from other operators) you should not re-define the API as owned by your project. Therefore, you can create the controller in this cases by using the flag --resource=false. (i.e. $ operator-sdk create api --group ship --version v1beta1 --kind External --resource=false --controller=true). Attention: If you are using Golang-based language Operator then, you will need to update the markers and imports manually until it become officially supported by the tool. For further information check the issue #1999.

WARNING: if you create CRD’s via the reconciliations or via the Operands then, OLM cannot handle CRDs migration and update, validation.

NOTE: By not following this guidance you might probably to be hurting concepts like as single responsibility principle and damaging these concepts could cause unexpected side effects, such as; difficulty extending, reuse, or maintenance, only to mention a few.

Other common suggestions

Provide the images and tags used by the operator solution via environment variables in the config/manager/manager.yaml:

...
spec:
  ...
    spec:
      ...
      containers:
      - command:
        - /manager
        ...
        env:
        - name: MY_IMAGE
          value: "quay.io/example.com/image:0.0.1"

Manage your solutions using Status Conditionals
Use finalizers when/if required
Cover the project with tests/CI to ensure its quality:
- For any language-based operator, you can use Scorecard to implement functional tests
- For Go-based operators, you can also use envtest to cover the controllers. For further information see Testing with EnvTest. Also, see the test directory for the Memcached sample under the testdata/go/v3/memcached-operator to know how can you build e2e tests.
- For Ansible-based operators, you can also use Molecule, an Ansible testing framework. For further information see Testing with Molecule
- For Helm-based operators, you can also use Chart tests
Ensure that you checked the Can I customize the projects initialized with operator-sdk? and understand the Project Layout before starting to do your customizations as please you on top.
Optimize manager resource values in config/manager/manager.yaml according to project requirements. It is recommended to define resources limits in order to follow good practices and for security reasons. More info: Managing Resources for Containers and Docker Security Cheat Sheet.
Look for TODO(user) in the source code generated by the CLI to ensure that you follow all suggested customizations.
If you wish to integrate your project with OLM, you can also check its Best Practices section.

Docs: Resource pruning

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Operators can create Jobs or Pods as part of their normal operation, and when those Jobs or Pods complete, they can remain on the Kubernetes cluster if not specifically removed. These resources can consume valuable cluster resources like disk storage (e.g. etcd). These resources are not tied to a Custom Resource using an ownerReference.

Operator authors have traditionally had two pruning options:

leave the resource cleanup to a system admin to perform
implement some form of pruning within their operator solution

For our purposes when we say, prune, we mean to remove a resource (e.g. kubectl delete) from a Kubernetes cluster for a given namespace.

This documentation describes the pattern and library useful for implementing a solution within an operator.

operator-lib prune library

A simple pruning implementation can be found in the operator-lib prune package. This package is written in Go and is meant to be used within Go-based operators. This package was developed to include common pruning strategies as found in common operators. The package also allow for customization of hooks and strategies.

Pruning Configuration

Users can configure the pruning library by creating code similar to this example:

cfg = Config{
	Log:           logf.Log.WithName("prune"),
	DryRun:        false,
	Clientset:     client,
	LabelSelector: "app=churro",
	Resources: []schema.GroupVersionKind{
		{Group: "", Version: "", Kind: PodKind},
	},
	Namespaces: []string{"default"},
	Strategy: StrategyConfig{
		Mode:            MaxCountStrategy,
		MaxCountSetting: 1,
	},
	PreDeleteHook: myhook,
}

Config Field	Description
Log	a logr.Logger. It is optional if a logger is provided through the context to the Execute method, which is the case with the context of the Reconcile function of operator-sdk and controller-runtime
DryRun	a boolean determines whether to actually remove resources; `true` means to execute but not to remove resources
Clientset	a client-go Kubernetes ClientSet that will be used for Kube API calls by the library
LabelSelector	Kubernetes label selector expression used to find resources to prune
Resources	Kube resource Kinds, currently PodKind and JobKind are supported by the library
Namespaces	a list of Kube Namespaces to search for resources
Strategy	specifies the pruning strategy to execute
Strategy.Mode	currently MaxCountStrategy, MaxAgeStrategy, or CustomStrategy are supported
Strategy.MaxCountSetting	integer value for maxcount strategy, specifies how many resources should remain after pruning executes
Strategy.MaxAgeSetting	golang time.Duration string value (e.g. 48h), specifies age of resources to prune
Strategy.CustomSettings	a golang map of values that can be passed into a Custom strategy function
PreDeleteHook	optionally specifies a golang function to call before pruning a resource
CustomStrategy	optionally specifies a golang function that implements a custom pruning strategy

Pruning Execution

Users can invoke the pruning by running the Execute function on the pruning configuration as follows:

err := cfg.Execute(ctx)

Users might want to implement pruning execution by means of a cron package or simply call the prune library based on some other triggering event.

If a logger has been configured in the Config structure it takes precedence on the one provided through ctx. Adding a logger.Logger to the context can be done with logr.NewContext.

Pruning Strategies

maxcount Strategy

A strategy of leaving a finite set of resources is implemented called maxcount. This strategy seeks to leave a specific number of resources, sorted by latest, on your cluster. For example, if you have 10 resources that would be pruned, and you specified a maxcount value of 4, then 6 resources would be pruned (removed) from your cluster starting with the oldest resources.

maxage Strategy

A strategy of removing resources greater than a specific time is called maxage. This strategy seeks to remove resources older than a specified maxage duration. For example, a library user might specify a value of 48h to indicate that any resource older than 48 hours would be pruned. Durations are specified using golang’s time.Duration formatting (e.g. 48h).

Pruning Customization

preDelete Hook

Users can provide a preDelete hook when using the operator-lib prune package.
This hook function will be called by the library before removing a resource. This provides a means to examine the resource logs for example, extracting any valued content, before the resource is removed from the cluster.

Here is an example of a preDelete hook:

func myhook(ctx context.Context, cfg Config, res ResourceInfo) error {
        log := prune.Logger(ctx, cfg)
        log.V(4).Info("pre-deletion", "GVK", res.GVK, "namespace", res.Namespace, "name", res.Name)
       	if res.GVK.Kind == PodKind {
                req := cfg.Clientset.CoreV1().Pods(res.Namespace).GetLogs(res.Name, &v1.PodLogOptions{})
                podLogs, err := req.Stream(context.Background())
                if err != nil {
                        return err
                }
                defer podLogs.Close()

                buf := new(bytes.Buffer)
                _, err = io.Copy(buf, podLogs)
                if err != nil {
                        return err
                }

                log.V(4).Info("pod log before removing is", "log", buf.String())
        }
	return nil
}

Note if your custom hook returns an error, then the resource will not be removed by the prune library.

Custom Strategy

Library users can also write their own custom pruning strategy function to support advanced cases. Custom strategy functions are passed in the prune configuration and a list of resources selected by the library. The custom strategy builds up a list of resources to be removed, returning the list to the prune library which performs the actual resource removal. Here is an example custom strategy:

func myStrategy(ctx context.Context, cfg Config, resources []ResourceInfo) (resourcesToRemove []ResourceInfo, err error) {
        log := Logger(ctx, cfg)
        log.V(4).Info("myStrategy called", "resources", resources, "config", cfg)
	if len(resources) != 3 {
		return resourcesToRemove, fmt.Errorf("count of resources did not equal our expectation")
	}
	return resourcesToRemove, nil
}

To have your custom strategy invoked, you will specify your function within the prune configuration as follows:

cfg.Strategy.Mode = CustomStrategy
cfg.Strategy.CustomSettings = make(map[string]interface{})
cfg.CustomStrategy = myStrategy

Notice that you can optionally pass in settings to your custom function as a map using the cfg.Strategy.CustomSettings field.

Docs: Multi-Tenancy

Mon, 01 Jan 0001 00:00:00 +0000

NetworkPolicy

If your Operator creates or manages NetworkPolicy configurations ensure that your solution:

applies fine-grained network policies to the extent that is required for your managed application to function properly
applies fine-grained network policies to enable your managed application internal components to communicate among each other
allows users to configure your operator so it does not create or manage NetworkPolicy instances
does not create allow traffic from everywhere in the cluster type policies

NetworkPolicies are popular in multi-tenant cluster to provide an extra layer of segregation among tenants within SDN solutions. Users typically customize this extensively with goal of disallowing network traffic among unrelated tenants. Operators that deploy an accept all traffic from anywhere in the cluster style policy, are creating an obstacle in pursuing this goal, especially in instances where these policies cannot be disabled. In security-conscious environments policies like these are not allowed in production. In such cases your operator should minimally have the option to prevent NetworkPolicy objects from being created and leave this responsibility to the user. In more advanced cases, your operator should create NetworkPolicy configurations that follow the least-privilege principle, i.e. denying access to everything and from everywhere by default and only allowing access to specific authorized resources from specific authorized components.

Traffic sharding

The goal is to split or to isolated ingress traffic from certain environments, e.g. production and development environments, ending up on different routers and in this way, being managed by a different Ingress controller. This is a popular configuration option for heavily populated multi-tenant clusters, with several IngressController deployed.

If your Operator creates ingress resources the recommendation is to allow the users to customize them, through the use of a CRD. The required IngressClass needs then to be propagated to the ingress resources created so that they get picked up by the desired IngressController. Annotations are deprecated in favour of IngressClass.

Route resources

To run on the OpenShift distribution of Kubernetes you probably will use the Route API. When sharding these routes may be configured with a label selector. Based on this label selector they will amend their configuration when a route (having the label) is created or not (if the route does not have the label). The label is applied at the Route level and there is no pre-defined convention here, so users set these custom labels in accordance to how they configured their IngressController from operator.openshift.io/v1 instances.

In this way, your operator should allow the user to specify custom labels for any Route that it manages. Check the doc and this blog for an end-to-end examples and further information.

Docs: Designing Lean Operators

Mon, 01 Jan 0001 00:00:00 +0000

Overview

One of the pitfalls that many operators are failing into is that they watch resources with high cardinality like secrets possibly in all namespaces. This has a massive impact on the memory used by the controller on big clusters. Such resources can be filtered by label or fields. The original doc design for Filter cache ListWatch using selectors can be accessed from here

IMPORTANT NOTE Requests to a client backed by a filtered cache for objects that do not match the filter will never return anything. In other words, filtered caches make the filtered-out objects invisible to the client.

How is this done ?

When creating the manager, you can add an Options struct to configure the Cache
Each client.Object can be filtered with labels and fields

Examples

In this scenario, the user will configure the Cache to filter the secret object by it’s label. This will return a filtered cache for objects that match the filter.

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
  Cache: cache.Options{
    ByObject: map[client.Object]cache.ByObject{
      &corev1.Secret{}: cache.ByObject{
	Label: labels.SelectorFromSet(labels.Set{"app": "app-name"}),
      },
    },
  },
})

In this scenario, the user will configure the Cache to filter the node object by it’s field name. This will return a filtered cache for objects that match the filter.

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
  Cache: cache.Options{
    ByObject: map[client.Object]cache.ByObject{
      &corev1.Node{}: cache.ByObject{
	Fields: labels.SelectorFromSet(fields.Set{"metadata.name": "node01"}),
      },
    },
  },
})

Docs: Managing Resources

Mon, 01 Jan 0001 00:00:00 +0000

Managing Resources

Note that pretty much all controllers consume:

CPU: largely based on the number of reconciliations they perform, which are generally related to event activity for resources they’re watching.
Memory: largely based on the number of primary resources that exist (multiplied by some factor based on the number of operand resources they need to watch as a result) via informer caches.

And then there is a concern that one Pod or Container could monopolize all available resources and Cluster admins must consider the effects that one Pod or Container may have on other components.

In an effort to prevent a container from consuming all the resources on a cluster or affecting other workloads from being scheduled, many production clusters will define ResourceQuota configurations.

The ResourceQuota configuration also applies to tenant workloads that are managed by your Operator. Cluster administrators will typically set a ResourceQuota for each tenant’s namespace as part of the onboarding. If a LimitRange with default values has not been created in each namespace and your Operator creates Containers inside the tenant namespace without specifying at least resource requests for CPU and Memory of its Pods then, the system or quota may reject Pod creation. Check the following statements obtained from K8s docs:

“If a LimitRange is activated in a namespace for computing resources like CPU and memory, users must specify requests or limits for those values. Otherwise, the system may reject Pod creation." (Reference).
“If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation." (Reference).

In an effort to support clusters with the above configuration, to ensure safe operations and avoid negatively impacting other workloads: Operators should always include reasonable memory and CPU resource requests for their own deployment as well as for operands they deploy.

HINT Cluster admins might also able to avoid the above scenario by setting default values when they are not specified for each Pod and/or Container in a namespace.

Therefore, your Operator should always apply at least resource requests for CPU and memory for Pods/Deployments that it creates as part of the reconciliation logic. Ideally your Operator also applies memory limits to those Pods/Deployments. You may also consider CPU limits.

Resource requests and limits for the Operator Deployment can be defined by modifying theconfig/manager/manager.yaml as shown below:

  ...
  # TODO(user): Configure the resources accordingly based on the project requirements.
  # More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
  resources:
    requests:
      cpu: 10m
      memory: 64Mi
    ...

How to compute default values

IMPORTANT: A single configuration that fits all scenarios is not possible. In this way, Operators authors MUST to ensure that Cluster Admins and its users can change the resource requests/limits of the Operator/manager, and of its Operands.

However, you are able to benchmark your Operator by Monitoring the resource usage to ensure good and reasonable values for the general cases. Kubebuilder and SDK provide some metrics which can help you with.

NOTE Also, be aware that if the project was generated by Kubebuilder or SDK scaffold then some values for the Operator/manager(See config/manager/manager.yaml) are populated by default to get you started, however, you ought to optimize them based on your own tests and the specific needs of your operator.

How to change the Operator/manager resources values when under OLM management

If your operator is managed by OLM, administrators or users can configure your operator’s resource requests and limits via the subscription.

General guidelines

Following are some general recommendations to manage the resources:

MUST declare resource requests for both, CPU and Memory, for the Operator/manager and any Pod/Deployment managed by it
OPTIONALLY setting the resources limit for CPU and Memory for the Operator Pod and any Pod/Deployment managed by it.
SHOULD provide the mechanisms for Monitoring compute & memory resource usage so that, Cluster Admins can use these metrics to monitor and resize the Operator and its Operands.CAVEAT: If the Operator is integrated with OLM and the bundle has a PodMonitor or a ServiceMonitor the complete InstallPlan will fail on a cluster, which does not have these CRD/the Prometheus operator installed. In this case, you might want to ensure the dependency requirement with OLM dependency or make clear its requirement for the Operator consumers.
SHOULD allow admins to customize the requests/limits amounts defined for the Pod/Deployment created by the Operator and not hardcode these values.
SHOULD document how your Operator consumer can customize/rightsize the resources requests and limits for the Operator and Operands Pod/Deployments or describe how the solution could be configured to automatically adjust these values based on the environment. the Operator automatically adjust the values to the environment rather than asking its consumers to amend them. You might also consider leveraging the Vertical Pod Autoscaler to have the resources requested by the Operator automatically adjusted to the cluster where it is deployed. You might also look at allowing horizontal pod autoscaling for the other Pod/Deployments created by your Operator. In this case, be aware that resource requests are also required for horizontal pod autoscaling to work.CAVEAT: If you are using OpenShift as your Kubernetes distribution you might want to check the doc Automatically adjust pod resource levels with the vertical pod autoscaler. Also, if the VPA CRD is not available in the cluster where the operator gets deployed the InstallPlan will fail. Please, see how to work with OLM dependency if your project integrating with it.

Why should you set these?

Resource Requests

What happens when the resource requests are not set?

configurations made by the cluster administrators such as ResourceQuota might not work without LimitRanges. The LimitRanger admission controller can provide default values for resource requests and limits when they have not been defined.
the Operators consumers might face resource shortages on a node when resource usage increases, for example, during a daily peak in request rate.
the Operator’s consumers might be unable to successfully deploy the Operator because it does not have the minimal resources available.
the scheduler cannot make an informed placement choice when it picks the nodes the operator pods will be running on.
when there is memory contention on the node the pod is likely to get either evicted or OOM killed.
when there is CPU contention on the node the pod is likely to get starved of CPU cycles making the operator unresponsive.

Resource Limits

What happens when the resource limits are not set?

Wrong configurations or code implementations can consume all resources available, affecting other components on cluster. Also, it might leave the containers more vulnerable such as to Dos Attacks. More info. See that when there is memory contention on a node, pods will start getting evicted and possibly killed according to their OOM score. The node will be flagged with a MemoryPressure condition and eventually made unschedulable. However, when there is CPU contention on the node, neighbour pods may get slowed down to the CPU cores they requested.

However, a popular practice by cluster administrators is to leverage ResourceQuota to limit the total amount of resources that can be requested or allowed in a single namespace. This may protect against over consumption of resources by operands of a faulty or wrongly configured operator. On the other hand it also means that the operator may not be able to create additional pods, limiting its functionality when the limit has been reached.

Also, see might want to check in the K8s docs the following sections:

What happens when:

Limits reached

What happens when the resource limits have been reached?

For Memory: the container might be terminated with the reason OOM Killed. If it is restartable, the kubelet will restart it, as with any other type of runtime failure.
For CPU: the container might or might not be allowed to exceed its CPU limit for extended periods of time. However, it will not be killed for excessive CPU usage. The CPU is considered a “compressible” resource, and if the Pod starts hitting the CPU limits, Kubernetes uses kernel to starts throttling the container. That means the CPU will be artificially restricted, giving a potentially worse performance only.

You might want to check the Troubleshooting section in the Kubernetes documents to better understand how to debug these scenarios.

Limits are specified but not requests

What happens when the resource limits are defined but not the requests?

If you specify a CPU or Memory limit for a Container but do not specify a request, Kubernetes automatically assigns a CPU or Memory request that matches the limit. In this way, you will be requesting always the limit and will be allocating more resources than required. (NOT RECOMMENDED)

Values are too big

Memory and CPU requests and limits are associated with Containers, but be aware that the Memory and CPU requests and limits of a Pod are the sum of its specific computing types for all the containers in the Pod.

If you define that your Pods should have Memory or CPU request too big then, you might not only be allocating and blocking the usage of more than you ought unnecessarily. Also, your Operator consumers might be able to install your Operator via OLM, for example, but will be unable to check the Pods/Deployment running successfully when the amount defined exceeds the capacity available. In these scenarios, the Operator consumers will check that Pod(s) failed to schedule with event errors like Insufficient cpu and/or Insufficient memory.

Docs: Pod Security Standards

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The PodSecurityPolicy API is deprecated and will be removed from Kubernetes in version 1.25. This API is replaced by a new built-in admission controller (KEP-2579: Pod Security Admission Control) which allows cluster admins to enforce Pod Security Standards Labels.

What does that mean?

Namespace and Pod/Container can be defined with three different policies which are; Privileged, Baseline and Restricted. (More info). Therefore, Pod(s)/Container(s) that are not configured according to the enforced security standards defined globally or on the namespace level will not be admitted and it will not be possible to run them.

As a best practice, you must ensure that workloads (Operators and Operands) are defined to run under restricted permissions unless they need further privileges. For the cases where Pod/Container(s) requires escalating permissions, the recommendation is to use the label as described below

How should I configure my Operators and Operands to comply with the criteria?

For common cases that do not require escalating privileges: configure all containers to comply with the restricted policy as shown in the following the examples:

IMPORTANT NOTE The seccompProfile field to define that a container is restricted was introduced with K8s 1.19 and might not be supported on some vendors by default. Please, do not use this field if you are looking to build Operators that work on K8s versions < 1.19 or on vendors that do not support this field. Having this field when it is not supported can result in your Pods/Containers not being allowed to run (i.e. On Openshift versions < 4.11 with its default configuration the deployments will fail with errors like Forbidden: seccomp.)

In Kubernetes manifests:

    spec:
      securityContext:
        # WARNING: Ensure that the image used defines an UserID in the Dockerfile
        # otherwise the Pod will not run and will fail with `container has runAsNonRoot and image has non-numeric user`.
        # If you want your workloads admitted in namespaces enforced with the restricted mode in OpenShift/OKD vendors
        # then, you MUST ensure that the Dockerfile defines a User ID OR you MUST leave the `RunAsNonRoot` and
        # RunAsUser fields empty.
        runAsNonRoot: true
        # Please ensure that you can use SeccompProfile and do not use
        # if your project must work on old Kubernetes
        # versions < 1.19 or on vendors versions which
        # do NOT support this field by default (i.e. Openshift < 4.11 )
        seccompProfile:
          type: RuntimeDefault
      ...
      containers:
      - name: controller-manager
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
        ...

On Reconciliations, such as code implementation in Go:

Note: if you are setting the RunAsNonRoot value to true in the SecurityContext you will need to verify that the Pod or Container(s) are running with a numeric user that is not 0 (root). If the Pod or Container(s) do not use a non-zero numeric user, you can use the RunAsUser value to set the user to a non-zero numeric user. In this example, the memcached container does not use a non-zero numeric user and therefore the RunAsUser value is set to use a uid of 1000.

dep:= &appsv1.Deployment{
  ObjectMeta: metav1.ObjectMeta{
  ….
  },
  Spec: appsv1.DeploymentSpec{
    …
     Template: corev1.PodTemplateSpec{
       ….
        Spec: corev1.PodSpec{
           // Ensure restricted context for the Pod    
           SecurityContext: &corev1.PodSecurityContext{
              RunAsNonRoot: &[]bool{true}[0],
			  // Please ensure that you can use SeccompProfile and do NOT use
			  // this filed if your project must work on old Kubernetes
			  // versions < 1.19 or on vendors versions which 
			  // do NOT support this field by default (i.e. Openshift < 4.11)
              SeccompProfile: &corev1.SeccompProfile{
                 Type: corev1.SeccompProfileTypeRuntimeDefault,
              },
           },
           Containers: []corev1.Container{{
              Image:   "memcached:1.4.36-alpine",
              Name:    "memcached",
              // Ensure restricted context for the container  
              SecurityContext: &corev1.SecurityContext{
				 // WARNING: Ensure that the image used defines an UserID in the Dockerfile
				 // otherwise the Pod will not run and will fail with `container has runAsNonRoot and image has non-numeric user`.
				 // If you want your workloads admitted in namespaces enforced with the restricted mode in OpenShift/OKD vendors 
				 // then, you MUST ensure that the Dockerfile defines a User ID OR you MUST leave the `RunAsNonRoot` and
				 // RunAsUser fields empty. 
                 RunAsNonRoot:  &[]bool{true}[0],
                 AllowPrivilegeEscalation:  &[]bool{false}[0],
                 Capabilities: &corev1.Capabilities{
                    Drop: []corev1.Capability{
                       "ALL",
                    },
                 },
              },
           }},
        },
     },
  },
}

For Ansible and Helm language based Operators: Ansible playbooks or Helm charts MUST create manifests that comply with the requirements in the same way. You can find some examples by looking at the samples under the testdata directory.

For workloads that need elevated permissions: it is recommended that you ensure the namespace containing your solution is labeled accordingly. You can either update your operator to manage the namespace labels or include the namespace labeling as part of the manual install instructions.

It is recommended that you provide a description to help cluster admins understand why elevated permissions are required. You can add this information and the prerequisites to the description of your Operator Bundle (CSV).

Following you will find a detailed description of how to configure and test your solutions. The most straightforward way to ensure if your workloads will work in a restricted namespace is verifying if your solution can run in namespaces enforced as restricted.

NOTE: It is recommended that you test the desired behavior as part of an e2e test suite. Examples of an e2e test for this will be shown in a later section.

How the Operator bundle (CSV) must be configured to apply the standards to the Pod/Containers which are installed by OLM (Operator itself)?

For Operators integrated with OLM, there is an Operator bundle with a CSV where the spec.install.spec.deployments has a Deployment which defines the Pod/Container(s) that will be installed by OLM to get your Operator running on the cluster. In order for the security standards to be followed you will need to ensure the configurations are set correctly.

Note: Ensure the configuration is carried to the Pod/Containers on the bundle CSV after running make bundle. See that the Operator bundle generated with the target is built from the manifests under the config directory. To know more about the layout of your operator built with Operator-SDK see Project Layout.

To check an example of a CSV which complies with the restricted policy, see the Golang sample under the testdata/go/v4/memcached-operator/bundle/manifests/memcached-operator.clusterserviceversion.yaml

How can I verify my manifest?

Using Kind

To verify the policy of your Pod/Container(s) you might to use a Kind cluster as described in the K8s documentation.

Example

First lets create a namespace where we will test the files:

kubectl create ns mytest

Now, let’s label the namespace so that we can check our manifest

 kubectl label --overwrite ns mytest \
   pod-security.kubernetes.io/enforce=restricted \
   pod-security.kubernetes.io/enforce-version=v1.24 \
   pod-security.kubernetes.io/audit=restricted

Now, apply the following Pod which is not restricted on the namespace:

apiVersion: v1
kind: Pod
metadata:
 name: example
 namespace: mytest
spec:
 containers:
  - name: test
    securityContext:
     # see that we are allowing privilege escalation  
     allowPrivilegeEscalation: true 
    image: 'busybox:1.28'
    ports:
     - containerPort: 8080

Then, when we try to apply the Pod manifest above we should see an error:

$ kubectl apply -f mypodtest.yaml 
Error from server (Forbidden): error when creating "mypodtest.yaml": pods "example" is forbidden: violates PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

How do I check what policy (Privileged, Baseline and Restricted) is configured for my Operator and Operand(s)?

An easy way might be using the tool: psachecker. This tool is only able to be used to check locally the Deployments/Pods manifests and not the CSV.

Alternatively, you could test the policy enforcement by labeling the namespaces and running your operator or applying the manifests. More info.

How do I install psachecker?

The following steps will install psachecker in a Golang environment.

git clone git@github.com:stlaz/psachecker.git $GOPATH/src/github.com/stlaz/psachecker
cd $GOPATH/src/github.com/stlaz/psachecker
make build
cp kubectl-psachecker $GOPATH/bin/

How do I use psachecker to verify the policy of a manifests?

The following steps will demonstrate how we can use psachecker to verify that our Operator and deployments are properly configured in the CSV.

Create a new test.yaml file
Add Deployment schema to the file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-manifest
  namespace: test
spec:
...

Now, add the CSV deployment to this test (spec.install.spec.deployments)
Then, you can run the tool and check if its result will be restricted as expected (i.e.):

$ kubectl-psachecker inspect-workloads -f test.yaml
system: restricted

Can I use the metrics to check if my Pod/Containers are violating the PodSecurity policies?

Yes, you can. You need to label the namespaces with pod-security.kubernetes.io/audit: restricted (i.e. kubectl label --overwrite ns --all pod-security.kubernetes.io/enforce-version=v1.24 pod-security.kubernetes.io/audit=restricted). It is important to note that the results may include metrics that do not come from your Operator and Operand(s). If you are looking to use the metrics to do the checks, ensure that you check the results before and after performing the tests, for example:

kubectl get --raw /metrics | prom2json | jq '[.[] | select(.name=="pod_security_evaluations_total") ]'

[
  {
    "name": "pod_security_evaluations_total",
    "help": "[ALPHA] Number of policy evaluations that occurred, not counting ignored or exempt requests.",
    "type": "COUNTER",
    "metrics": [
      {
        "labels": {
          "decision": "allow",
          "mode": "enforce",
          "policy_level": "privileged",
          "policy_version": "latest",
          "request_operation": "create",
          "resource": "pod",
          "subresource": ""
        },
        "value": "29"
      },
      {
        "labels": {
          "decision": "allow",
          "mode": "enforce",
          "policy_level": "privileged",
          "policy_version": "latest",
          "request_operation": "update",
          "resource": "pod",
          "subresource": ""
        },
        "value": "0"
      },
      {
        "labels": {
          "decision": "deny",
          "mode": "audit",
          "policy_level": "restricted",
          "policy_version": "latest",
          "request_operation": "create",
          "resource": "controller",
          "subresource": ""
        },
        "value": "6"
      },
      {
        "labels": {
          "decision": "deny",
          "mode": "audit",
          "policy_level": "restricted",
          "policy_version": "latest",
          "request_operation": "create",
          "resource": "pod",
          "subresource": ""
        },
        "value": "7"
      },
      {
        "labels": {
          "decision": "deny",
          "mode": "warn",
          "policy_level": "restricted",
          "policy_version": "latest",
          "request_operation": "create",
          "resource": "controller",
          "subresource": ""
        },
        "value": "6"
      },
      {
        "labels": {
          "decision": "deny",
          "mode": "warn",
          "policy_level": "restricted",
          "policy_version": "latest",
          "request_operation": "create",
          "resource": "pod",
          "subresource": ""
        },
        "value": "7"
      }
    ]
  }
]

Also, you might be able to create Prometheus alerts such as:

sum (increase(pod_security_evaluations_total{decision="deny",mode="audit"}[1h])) by (policy_level)

(Recommended) How can I automate this check using e2e testing to ensure that my solutions can run under the policies?

In the tests, you can create and label a namespace with the desired policy (i.e. restricted). This would allow you to verify if any warnings/errors occur and if the Pod(s)/Container(s) are in the Running state. It can be something like:

NOTE See the test directory for the Memcached sample under the testdata/go/v3/memcached-operator to see a full example.

// namespace store the ns where the Operator and Operand(s) will be executed
const namespace = "my-operator-system"

var _ = Describe("my operator test", func() {
    BeforeEach(func() {
        ...
        
        // Where we will create the namespace for to install the Operator and Operands
        By("creating namespace")
        cmd := exec.Command("kubectl", "create", "ns", namespace)
        _, _ = utils.Run(cmd)
        
        // We will label as follows all namespaces so that we can check if warnings will be raised when 
		// a manifest be applied. (Not that some namespaces might not acced the label, i.e. if it
		// has a container running with a less restrictive policy )
        By("labeling all namespaces to warn against what can violate the restricted policy")
        cmd = exec.Command("kubectl", "label", "--overwrite", "ns", "--all",
            "pod-security.kubernetes.io/audit=restricted",
            "pod-security.kubernetes.io/enforce-version=v1.24",
            "pod-security.kubernetes.io/warn=restricted")
        _, err := utils.Run(cmd)
        ExpectWithOffset(1, err).NotTo(HaveOccurred())
        
		// We will enforce the restricted policy so that if our Operator
		// or Operand be unable to run as restricted we will be able to check
		// it by validating their status
        By("enforcing restricted to the ns where the Operator/Operand(s) will be checked")
        cmd = exec.Command("kubectl", "label", "--overwrite", "ns", namespace,
            "pod-security.kubernetes.io/audit=restricted",
            "pod-security.kubernetes.io/enforce-version=v1.24",
            "pod-security.kubernetes.io/enforce=restricted")
        _, err = utils.Run(cmd)
        Expect(err).To(Not(HaveOccurred()))
        })
    })
    
    AfterEach(func() {
        ...
    })

    It("should successfully run the Operator and Operand(s)", func() {
        // Then, here we build the operator and deploy the
        // manager as the operand in the namespaces were 
        // the policy restricted was enforced.
		
        // Therefore, we can check the Operator and Operand
        // status to ensure that all is Running.
		
        // Note that we can also verify if warns like with
        // the message Warning: would violate PodSecurity were 
        // returned when the manifest were applied on the cluster
    })
})

After following the recommendations to be restricted my workload is not running (CreateContainerConfigError). What should I do?

If you are encountering errors similar to Error: container has runAsNonRoot and image has non-numeric user or container has runAsNonRoot and image will run as root that means that the image used does not have a non-zero numeric user defined, i.e.:

USER 65532:65532 
OR
USER 1001

Due to the RunAsNonRoot field being set to true, we need to force the user in the container to a non-zero numeric user. It is recommended that the images used by your operator have a non-zero numeric user set in the image itself (similar to the example above). For further information check the note Consider an explicit UID/GID in the Dockerfile best practices guide. If your Operator will be distributed and used in vanilla Kubernetes clusters you can also fix the issue by defining the user via the security context configuration). (i.e. RunAsUser: &[]int64{1000}[0],).

NOTE If your Operator should work with specific vendors please ensure that you check if they have specific rules for Pod Security Admission. For example, we know that if you use RunAsUser on OpenShift it will disqualify the Pod from their restricted-v2 SCC. Therefore, if you want your workloads running in namespaces labeled to enforce restricted you must leave RunAsUser and RunAsNonRoot fields empty or if you want set RunAsNonRoot then, you MUST ensure that the image itself properly defines the UserID.

Docs: Operator Observability Best Practices

Mon, 01 Jan 0001 00:00:00 +0000

Operator Observability Best Practices

In this document, we provide best practices and examples for creating metrics, recording rules and alerts. It is based on the general guidelines in Operator Capability Levels.

Note: For technical documentation of how to add metrics to your operator, please read the Metrics section of the Kubebuilder documentation.

Operator Observability Recommended Components

Health and Performance metrics - for all of the operator components
1.1. Metrics should be implemented based on the guidelines below.
1.2. Metrics Documentation - All metrics should have documentation.
1.3. Metrics Tests - Metrics should include tests that verify that they exist and that their value is correct.
Alerts for when things are not working as expected for each of the operator’s components
2.1 Alerts should be implemented based on the guidelines below.
2.2. Alerts Runbooks - Each alert can include a runbook_url annotation and an alert runbook that describes it. See additional details below.
2.3. Alerts Tests - Alerts should include E2E Testing and unit tests.
Events - Custom Resources should emit custom events for the operations taking place.

Additional components would be Dashboards, Logs and Traces, which are not addressed in this document at this point.

Operators Observability General Guidelines

Important: It is highly recommended to separate your monitoring code from your core operator code.
We recommend to create a dedicated /monitoring subfolder that will include all the code of the Operator Observability Recommended Components, that are outlined above. For example, in the memcached-operator.

In your core operator code only call the functions that will update the metrics value from your desired location. For example, in the memcached-operator.

All operators start small. This separation will help you, as a developer, with easier maintenance of both your operator core code and the monitoring code and for other stakeholders to understand your monitoring code better.

Metrics Guidelines

Metrics Naming

Kubernetes components emit metrics in Prometheus format. This format is structured plain text, designed so that people and machines can both read it.

Your operator users should get the same experience when searching for a metric across Kubernetes operators, resources and custom resources.

Check if a similar Kubernetes metric, for node, container or pod, exists and try to align to it.
The metrics search list, in the Prometheus, Grafana UI and even in the /metrics end point, is sorted in alphabetical order. When searching for a metric, it should be easy to identify metrics that are related to a specific operator. That is why we recommend that your operator metrics name will follow this format: operator name prefix + the sub-operator name or entity + metric name based on the Prometheus naming conventions. For example, in the memcached-operator.

Note: In Prometheus Node Exporter metrics are separated like this:

node_network_**receive**_packets_total
node_network_**transmit**_packets_total

In this example, based on receive and transmit.

Please follow the same principle and don’t put similar metrics details as labels, so the user experience would be fluent.
Example for this in an operator:

kubevirt_vmi_network_**receive**_errors_total
kubevirt_vmi_network_**transmit**_bytes_total
kubevirt_migrate_vmi_**data_processed**_bytes
kubevirt_migrate_vmi_**data_remaining**_bytes

Your metric suffix should indicate the metric unit. For better compatibility, Prometheus base units should be used.
Prometheus supports four metric types. Gauge,Counter,Histogram and Summary. You can read more about the different types here, Understanding metrics types.
The most common types are:

Counter - Value can only increase or reset.
Gauge Value can be increased and decreased as needed.

_total suffix should be used for accumulating count. If your metrics has labels with high cardinality, like pod/container it usually means that you can aggregate it more, thus it will not require _total suffix.

Prometheus Labels

Prometheus labels are used to differentiate the characteristics of the thing that is being measured.

Important - Be cautious when adding labels to metrics. Labels can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values. Note: There are still cases when we will still need to have a high cardinality label like the pod name, but try to keep this to the minimum.
When creating a new metric, recording rule or alert, that reports a resource like a pod or a container, make sure that the namespace is included, in order to be able to uniquely identify it.

Metrics `Help` message

Your operator metrics help message should include the following details:

What does this metric measure?
What does the output mean?
What important labels does the metric use? (Optional. If applicable).

The Help message can be used to create auto-generated documentation, like it’s done in KubeVirt and generated by the KubeVirt metrics doc generator.

We recommend to auto-generated metrics documentation and save it in your operator repository, to a location like /docs/monitoring/, so that the users can find the information about your operator metrics easily.

See Alerts, Metrics and Recording Rules Tests section for metrics testing recommendations.

Prometheus Recording Rules Naming

As per Prometheus documentation, Recording rules allow you to pre-compute frequently needed or computationally expensive expressions and save their result as a new set of time series.

Note: The Prometheus recording rules appear in Prometheus UI as metrics. Recording rule names should follow the level:metric:operations format as specified in the Prometheus recording rules best practices. This naming convention makes it clear that the metric is a recording rule and helps consumers understand they need to examine the underlying query to fully understand what the metric provides.

level: represents the aggregation level and labels of the rule output
metric: is the metric name
operations: is a list of operations that were applied to the metric, newest operation first

For example: job:up:avg_over_time or instance:node_cpu_utilisation:rate5m

In addition to this format, your operator recording rules should also follow the same naming guidelines as metrics for consistency within your operator’s observability stack.

See Alerts, Metrics and Recording Rules Tests section for recording rules testing recommendations.

Prometheus Alerts Guidelines

Clear and actionable alerts are a key component of a smooth operational experience and will result in a better experience for the end users.

The following guidances aim to align alert naming, severities, labels, etc., in order to avoid alerts fatigue for administrators.

Alert Ownership

Individual operator authors are responsible for writing and maintaining alerting rules for their components, i.e. their operators and operands.

Operator authors should also take into consideration how their components interact with existing monitoring and alerting.

As an example, if your operator deploys a service which creates one or more PersistentVolume resources, and these volumes are expected to be mostly full as part of normal operation, it’s likely that this will cause unnecessary KubePersistentVolumeFillingUp alerts to fire.

You should work to find a solution to avoid triggering these alerts if they are not actionable.

Alerts Style Guide

Alert names MUST be CamelCase, e.g.: PrometheusRuleFailures
Alert names SHOULD be prefixed with a component, e.g.: AlertmanagerFailedReload
- There may be exceptions for some broadly scoped alerts, e.g.: TargetDown
Alerts MUST include a severity label indicating the alert’s urgency.
- Valid severities are: critical, warning, or info — see below for guidelines on writing alerts of each severity.
Alerts MUST include summary and description annotations.
- Think of summary as the first line of a commit message, or an email subject line. It should be brief but informative. The description is the longer, more detailed explanation of the alert.
Alerts SHOULD include a namespace label indicating the source of the alert.
- Many alerts will include this by virtue of the fact that their PromQL expressions result in a namespace label. Others may require a static namespace label — see for example, the KubeCPUOvercommit alert.

Optional Alerts Labels and Annotations

priority label indicating the alert’s level of importance and the order in which it should be fixed.
- Valid priorities are: high, medium, or low. Higher the priority the sooner the alert should be resolved.
- If the alert doesn’t include a priority label, we can assume it is a medium priority alert. This label will usually be used for alerts with warning severity, to indicate the order in which the alert should be addressed by, even though it doesn’t require immediate action.
runbook_url annotation is a link to an alert runbook which is intended to guide a cluster owner and/or operator through the steps of fixing problems on clusters, which are surfaced by alerts.
- An example for Runbook style documentation.
- Your operator alert runbooks can be saved at your operator repository, to /docs/monitoring/runbooks/ for example, at OpenShift Runbooks if your operator is shipped with OpenShift or another location that fits your operator.
- If you are using Github, you can use Github Pages for a better view of the runbooks.
kubernetes_operator_part_of label indicating the operator name. Label name is based on the Kubernetes Recommended Labels.

Alerts Severity

Critical Alerts

For alerting current and impending disaster situations. These alerts page an SRE. The situation should warrant waking someone in the middle of the night.

Timeline: ~5 minutes.

Reserve critical level alerts only for reporting conditions that may lead to loss of data or inability to deliver service for the cluster as a whole.
Failures of most individual components should not trigger critical level alerts, unless they would result in either of those conditions.
Configure critical level alerts so they fire before the situation becomes irrecoverable.
Expect users to be notified of a critical alert within a short period of time after it fires so they can respond with corrective action quickly.

Example critical alert: KubeAPIDown

- alert: KubeAPIDown
  annotations:
    summary: Target disappeared from Prometheus target discovery.
    description: KubeAPI has disappeared from Prometheus target discovery.
    runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeAPIDown.md
  expr: |
    absent(up{job="apiserver"} == 1)
  for: 15m
  labels:
    severity: critical

This alert fires if no Kubernetes API server instance has reported metrics successfully in the last 15 minutes.
This is a clear example of a critical control-plane issue that represents a threat to the operability of the cluster as a whole, and likely warrants paging someone.
The alert has clear summary and description annotations, and it links to a runbook with information on investigating and resolving the issue.

The group of critical alerts should be small, very well defined, highly documented, polished and with a high bar set for entry.

Warning Alerts

The vast majority of alerts should use this severity.
Issues at the warning level should be addressed in a timely manner, but don’t pose an immediate threat to the operation of the cluster as a whole.

Timeline: ~60 minutes

If your alert does not meet the criteria in “Critical Alerts” above, it belongs to the warning level or lower.

Use warning level alerts for reporting conditions that may lead to inability to deliver individual features of the cluster, but not service for the cluster as a whole. Most alerts are likely to be warnings.
Configure warning level alerts so that they do not fire until components have sufficient time to try to recover from the interruption automatically.
Expect users to be notified of a warning, but for them not to respond with corrective action immediately.

Example warning alert: ClusterNotUpgradeable

- alert: ClusterNotUpgradeable
  annotations:
    summary: One or more cluster operators have been blocking minor version cluster upgrades for at least an hour.
    description: In most cases, you will still be able to apply patch releases.
      Reason {{ "{{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}}"}}
      For more information refer to 'oc adm upgrade'{{ "{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}" }}.
    expr: |
      max by (name, condition, endpoint) (cluster_operator_conditions{name="version", condition="Upgradeable", endpoint="metrics"} == 0)
    for: 60m
    labels:
      severity: warning

This alert fires if one or more operators have not reported their Upgradeable condition as true in more than an hour.
The alert has a clear name and informative summary and description annotations.
The timeline is appropriate for allowing the operator a chance to resolve the issue automatically, avoiding the need to alert an administrator.

Info Alerts

Info level alerts represent situations an administrator should be aware of, but they don’t necessarily require any action.
Use these sparingly, and consider instead reporting this information via Kubernetes events.

Example info alert: MultipleContainersOOMKilled

- alert: MultipleContainersOOMKilled
  annotations:
    description: Multiple containers were out of memory killed within the past
      15 minutes. There are many potential causes of OOM errors, however issues
      on a specific node or containers breaching their limits is common.
      summary: Containers are being killed due to OOM
  expr: sum(max by(namespace, container, pod) (increase(kube_pod_container_status_restarts_total[12m]))
    and max by(namespace, container, pod) (kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}) == 1) > 5
  for: 15m
  labels:
    namespace: kube-system
    severity: info

This alert fires if multiple containers have been terminated due to out of memory conditions in the last 15 minutes.
This is something the administrator should be aware of, but may not require immediate action.

Alerts, Metrics and Recording Rules Tests

Add tests for alerts that validate that:
- Each alert includes all mandatory fields.
- Each runbook_url link is valid.
- Each alert that includes a pod or a container also includes the namespace.
Add e2e tests that inspect the alerts during upgrade and make sure that the alerts don’t fire when they shouldn’t (Zero noise).
Add tests for metrics and recording rules that validate that:
- Metric / Recording rule exists.
- Metric / Recording rule value is as expected.
- Metric / Recording rule name follows the best practices guideline.

Operator SDK – Best Practices

Docs: Operator Best Practices

Development

Summary

Running On-Cluster

Summary:

Docs: Common recommendations and suggestions

Overview

Common Recommendations

Develop idempotent reconciliation solutions

Understanding Kubernetes APIs

Avoid a design solution where more than one Kind is reconciled by the same controller

Ideally Operators does not manage other Operators

What does it mainly mean:

Other common suggestions

Docs: Resource pruning

Overview

operator-lib prune library

Pruning Configuration

Pruning Execution

Pruning Strategies

maxcount Strategy

maxage Strategy

Pruning Customization

preDelete Hook

Custom Strategy

Docs: Multi-Tenancy

NetworkPolicy

Traffic sharding

Route resources

Docs: Designing Lean Operators

Overview

How is this done ?

Examples

Docs: Managing Resources

Managing Resources

How to compute default values

How to change the Operator/manager resources values when under OLM management

General guidelines

Why should you set these?

Resource Requests

Resource Limits

What happens when:

Limits reached

Limits are specified but not requests

Values are too big

Docs: Pod Security Standards

Overview

What does that mean?

How should I configure my Operators and Operands to comply with the criteria?

How the Operator bundle (CSV) must be configured to apply the standards to the Pod/Containers which are installed by OLM (Operator itself)?

How can I verify my manifest?

Using Kind

How do I check what policy (Privileged, Baseline and Restricted) is configured for my Operator and Operand(s)?

How do I install psachecker?

How do I use psachecker to verify the policy of a manifests?

Can I use the metrics to check if my Pod/Containers are violating the PodSecurity policies?

(Recommended) How can I automate this check using e2e testing to ensure that my solutions can run under the policies?

After following the recommendations to be restricted my workload is not running (CreateContainerConfigError). What should I do?

Docs: Operator Observability Best Practices

Operator Observability Best Practices

Operator Observability Recommended Components

Operators Observability General Guidelines

Metrics Guidelines

Metrics Naming

Prometheus Labels

Metrics Help message

Prometheus Recording Rules Naming

Prometheus Alerts Guidelines

Recommended Reading

Alert Ownership

Alerts Style Guide

Alerts Severity

Critical Alerts

Warning Alerts

Info Alerts

Alerts, Metrics and Recording Rules Tests

Metrics `Help` message