Operator SDK – Operator SDK Documentation

Docs: Operator Best Practices

Mon, 01 Jan 0001 00:00:00 +0000

Development

Considerations for Operator developers:

An Operator should manage a single type of application, essentially following the UNIX principle: do one thing and do it well.
If an application consists of multiple tiers or components, multiple Operators should be written one for each of them. For example, if the application consists of Redis, AMQ, and MySQL, there should be 3 Operators, not one.
If there is significant orchestration and sequencing involved, an Operator should be written that represents the entire stack, in turn delegating to other Operators for orchestrating their part of it.
Operators should own a CRD and only one Operator should control a CRD on a cluster. Two Operators managing the same CRD is not a recommended best practice. An API that exists with multiple implementations is a typical example of a no-op Operator. The no-op Operator doesn’t have any deployment or reconciliation loop to define the shared API and other Operators depend on this Operator to provide one implementation of the API, e.g. similar to PVCs or Ingress.
Inside an Operator, multiple controllers should be used if multiple CRDs are managed. This helps in separation of concerns and code readability. Note that this doesn’t necessarily mean that we need to have one container image per controller, but rather one reconciliation loop (which could be running as part of the same Operator binary) per CRD.
An Operator shouldn’t deploy or manage other operators (such patterns are known as meta or super operators or include CRDs in its Operands). It’s the Operator Lifecycle Manager’s job to manage the deployment and lifecycle of operators. For further information check Dependency Resolution.
If multiple operators are packaged and shipped as a single entity by the same CSV for example, then it is recommended to add all owned and required CRDs, as well as all deployments for operators that manage the owned CRDs, to the same CSV.
Writing an Operator involves using the Kubernetes API, which in most scenarios will be built using same boilerplate code. Use a framework like the Operator SDK to save yourself time with this and to also get a suite of tooling to ease development and testing.
Operators shouldn’t make any assumptions about the namespace they are deployed in and they should not use hard-coded names of resources that they expect to already exist.
Operators shouldn’t hard code the namespaces they are watching. This should be configurable - having no namespace supplied is interpreted as watching all namespaces
Semantic versioning (aka semver) should be used to version an Operator. Operators are long-running workloads on the cluster and its APIs are potentially in need of support over a longer period of time. Use the semver.org guidelines to help determine when and how to bump versions when there are breaking or non-breaking changes.
Kubernetes API versioning guidelines should be used to version Operator CRDs. Use the Kubernetes sig-architecture guidelines to get best practices on when to bump versions and when breaking changes are acceptable.
When defining CRDs, you should use OpenAPI spec to create a structural schema for your CRDs.
Operators are instrumented to provide useful, actionable metrics to external systems (e.g. monitoring/alerting platforms). Minimally, metrics should represent the software’s health and key performance indicators, as well as support the creation of service levels indicators such as throughput, latency, availability, errors, capacity, etc.
Operators may create objects as part of their operational duty. Object accumulation can consume unnecessary resources, slow down the API and clutter the user interface. As such it is important for operators to keep good hygiene and to clean up resources when they are not needed. Here are instructions on how to handle cleanup on deletion.

Summary

One Operator per managed application
Multiple operators should be used for complex, multi-tier application stacks
CRD can only be owned by a single Operator, shared CRDs should be owned by a separate Operator
One controller per custom resource definition
Use a tool like Operator SDK
Do not hard-code namespaces or resources names
Make watch namespace configurable
Use semver / observe Kubernetes guidelines on versioning APIs
Use OpenAPI spec with structural schema on CRDs
Operators expose metrics to external systems
Operators cleanup resources on deletion

Running On-Cluster

Considerations for on-cluster behavior

Like all containers on Kubernetes, Operators need not run as root unless absolutely necessary. Operators should come with their own ServiceAccount and not rely on the default.
Operators should not self-register their CRDs. These are global resources and careful consideration needs to be taken when setting those up. Also this requires the Operator to have global privileges which is potentially dangerous compared to that little extra convenience.
Operators use CRs as the primary interface to the cluster user. As such, at all times, meaningful status information should be written to those objects unless they are solely used to store data in a structured schema.
Operators should be updated frequently according to server versioning.
Operators need to support updating managed applications (Operands) that were set up by an older version of the Operator. There are multiple models for this:

Model	Description
Operator fan-out	where the Operator allows the user to specify the version in the custom resource
single version	where the Operator is tied to the version of the operand.
hybrid approach	where the Operator is tied to a range of versions, and the user can select some level of the version.

An Operator should not deploy another Operator - an additional component on cluster should take care of this (OLM).
When Operators change their APIs, CRD conversion (webhooks) should be used to deal with potentially older instances of them using the previous API version.
Operators should make it easy for users to use their APIs - validating and rejecting malformed requests via extensive Open API validation schema on CRDs or via an admission webhook is good practice.
The Operator itself should be really modest in its requirements - it should always be able to deploy by deploying its controllers, no user input should be required to start up the Operator.
If user input is required to change the configuration of the Operator itself, a Configuration CRD should be used. Init-containers as part of the Operator deployments can be used to create a default instance of those CRs and then the Operator manages their lifecycle.

Summary:

On the cluster, an Operator…

Does not run as root
Does not self-register CRDs
Does not install other Operators
Does rely on dependencies via package manager (OLM)
Writes meaningful status information on Custom Resources objects unless pure data structure
Should be capable of updating from a previous version of the Operator
Should be capable of managing an Operand from an older Operator version
Uses CRD conversion (webhooks) if API/CRDs change
Uses OpenAPI validation / Admission Webhooks to reject invalid CRs
Should always be able to deploy and come up without user input
Offers (pre)configuration via a “Configuration CR” instantiated by InitContainers

Docs: OLM Integration Bundle Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

The Operator Lifecycle Manager (OLM) is a set of cluster resources that manage the lifecycle of an Operator. The Operator SDK supports both creating manifests for OLM deployment, and testing your Operator on an OLM-enabled Kubernetes cluster.

This document is intended to quickly walk through the steps to generate an OLM bundle. For further explanation, or if you’re using package manifests, see the Bundle Tutorial.

Important: this guide assumes your project was scaffolded with operator-sdk init --project-version=3. These features are unavailable to projects of version 2 or less; this information can be found by inspecting your PROJECT file’s version value.

Prerequisites

Have a working operator that you have uploaded to a container registry. This guide assumes the simple Golang Memcached operator from the building operators section at version 0.0.1.
Make sure your user is authorized with cluster-admin permissions.
Have OLM installed on your cluster. The command operator-sdk olm install will attempt to install a basic OLM deployment on your cluster.

Steps

Export environment variables

$ export USERNAME=<container-registry-username>
$ export VERSION=0.0.1
$ export IMG=docker.io/$USERNAME/memcached-operator:v$VERSION // location where your operator image is hosted
$ export BUNDLE_IMG=docker.io/$USERNAME/memcached-operator-bundle:v$VERSION // location where your bundle will be hosted

Create a bundle from the root directory of your project

$ make bundle

This will prompt you to enter basic information about your operator.

Build and push the bundle image

$ make bundle-build bundle-push

Validate the bundle

$ operator-sdk bundle validate $BUNDLE_IMG

Install the bundle with OLM

$ operator-sdk run bundle $BUNDLE_IMG

Next Steps

Read the full tutorial for a more in-depth look at creating and using a bundle.

Docs: OLM Integration Bundle Tutorial

Mon, 01 Jan 0001 00:00:00 +0000

This document succinctly walks through getting an Operator OLM-ready with bundles, and glosses over explanations of certain steps for brevity. The following documents contain more detail on these steps:

All operator-framework manifest commands supported by the SDK: CLI overview.
Generating operator-framework manifests: generation overview.

If you are working with package manifests, see the package manifests tutorial once you have completed the Setup section below.

Setup

Let’s first walk through creating an Operator for memcached, a distributed key-value store.

Follow one of the user guides to develop the memcached-operator in either Go, Ansible, or Helm, depending on which Operator type you are interested in. This guide assumes memcached-operator is on version 0.0.1, which is set in the Makefile variable VERSION.

Enabling OLM

Ensure OLM is enabled on your cluster before following this guide. operator-sdk olm has several subcommands that can install, uninstall, and check the status of particular OLM versions in a cluster.

Note: Certain cluster types may already have OLM enabled, but under a non-default ("olm") namespace, which can be configured by setting --olm-namespace=[non-default-olm-namespace] for operator-sdk olm status|uninstall subcommands.

You can check if OLM is already installed by running the following command, which will detect the installed OLM version automatically (0.15.1 in this example):

$ operator-sdk olm status
INFO[0000] Fetching CRDs for version "0.15.1"
INFO[0002] Fetching resources for version "0.15.1"
INFO[0002] Successfully got OLM status for version "0.15.1"

NAME                                            NAMESPACE    KIND                        STATUS
olm                                                          Namespace                   Installed
operatorgroups.operators.coreos.com                          CustomResourceDefinition    Installed
catalogsources.operators.coreos.com                          CustomResourceDefinition    Installed
subscriptions.operators.coreos.com                           CustomResourceDefinition    Installed
...

All resources listed should have status Installed.

Note: The operator-sdk olm status command is geared to detect the status of OLM that was installed by installation methods like operator-sdk olm install or by applying OLM manifests directly on the cluster. This command retrieves the resources that were compiled into SDK at the time of installation from the OLM manifests. However, if OLM was installed in a cluster in a custom fashion (such as in OpenShift clusters), it is possible that some resources will show a Not Found status when the operator-sdk olm status command is issued.

To check the true status of such resources in OCP clusters, run:

oc get <resource-name> -n <resource-namespace>

If OLM is not already installed, go ahead and install the latest version:

$ operator-sdk olm install
INFO[0000] Fetching CRDs for version "latest"
INFO[0001] Fetching resources for version "latest"
INFO[0007] Creating CRDs and resources
INFO[0007]   Creating CustomResourceDefinition "clusterserviceversions.operators.coreos.com"
INFO[0007]   Creating CustomResourceDefinition "installplans.operators.coreos.com"
INFO[0007]   Creating CustomResourceDefinition "subscriptions.operators.coreos.com"
...
NAME                                            NAMESPACE    KIND                        STATUS
clusterserviceversions.operators.coreos.com                  CustomResourceDefinition    Installed
installplans.operators.coreos.com                            CustomResourceDefinition    Installed
subscriptions.operators.coreos.com                           CustomResourceDefinition    Installed
catalogsources.operators.coreos.com                          CustomResourceDefinition    Installed
...

Note: By default, olm status and olm uninstall auto-detect the OLM version installed in your cluster. This can fail if the installation is broken in some way, so the version of OLM can be overridden using the --version flag provided with these commands.

Creating a bundle

If working with package manifests, see the package manifests tutorial.

We will now create bundle manifests by running make bundle in the root of the memcached-operator project.

$ make bundle
/home/user/go/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
operator-sdk generate kustomize manifests -q
kustomize build config/manifests | operator-sdk generate bundle -q --overwrite --version 0.0.1
INFO[0000] Building annotations.yaml
INFO[0000] Writing annotations.yaml in /home/user/go/src/github.com/test-org/memcached-operator/bundle/metadata
INFO[0000] Building Dockerfile
INFO[0000] Writing bundle.Dockerfile in /home/user/go/src/github.com/test-org/memcached-operator
operator-sdk bundle validate ./bundle
INFO[0000] Found annotations file                        bundle-dir=bundle container-tool=docker
INFO[0000] Could not find optional dependencies file     bundle-dir=bundle container-tool=docker
INFO[0000] All validation tests have completed successfully

The above command will have created the following bundle artifacts: a manifests directory (bundle/manifests) containing a CSV and all CRDs from config/crds, metadata directory (bundle/metadata), and bundle.Dockerfile have been created in the Operator project. These files have been statically validated by operator-sdk bundle validate to ensure the on-disk bundle representation is correct.

Deploying an Operator with OLM

At this point in development we’ve generated all files necessary to build the memcached-operator bundle. Now we’re ready to test and deploy the Operator with OLM.

Note: If testing a bundle whose image will be hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

Testing bundles

Before proceeding, make sure you’ve Installed OLM onto your cluster.

First, we need to build our bundle. To build a memcached-operator bundle, run:

$ make bundle-build bundle-push BUNDLE_IMG=<some-registry>/memcached-operator-bundle:v0.0.1

Now that the bundle image is present in a registry, operator-sdk run bundle can create a pod to serve that bundle to OLM via a Subscription, along with other OLM objects, ephemerally.

$ operator-sdk run bundle <some-registry>/memcached-operator-bundle:v0.0.1
INFO[0008] Successfully created registry pod: <some-registry>-memcached-operator-bundle-0-0-1
INFO[0008] Created CatalogSource: memcached-operator-catalog
INFO[0008] OperatorGroup "operator-sdk-og" created
INFO[0008] Created Subscription: memcached-operator-v0-0-1-sub
INFO[0019] Approved InstallPlan install-krv7q for the Subscription: memcached-operator-v0-0-1-sub
INFO[0019] Waiting for ClusterServiceVersion "default/memcached-operator.v0.0.1" to reach 'Succeeded' phase
INFO[0019]   Waiting for ClusterServiceVersion "default/memcached-operator.v0.0.1" to appear
INFO[0031]   Found ClusterServiceVersion "default/memcached-operator.v0.0.1" phase: Pending
INFO[0032]   Found ClusterServiceVersion "default/memcached-operator.v0.0.1" phase: Installing
INFO[0040]   Found ClusterServiceVersion "default/memcached-operator.v0.0.1" phase: Succeeded
INFO[0040] OLM has successfully installed "memcached-operator.v0.0.1"

Note: If the bundle that is being installed has dependencies, the --index-image flag allows adding a bundle to a catalog that contains that bundle’s dependencies.

Note: Version v1.22.0 and later of the operator-sdk use the new file-based catalog (FBC) bundle format by default. Earlier releases use the deprecated SQLite bundle format. If you use an earlier version of the Operator SDK, you must update to a newer version or specify the index image by adding the --index-image=quay.io/operator-framework/opm:v1.23.0 flag. For more information about this known issue, see the FAQ.

Upgrading a bundle to a newer version

We can use the operator-sdk run bundle-upgrade command with a newer version of bundle image to upgrade an existing operator bundle deployed on cluster. The command automates the manual orchestration typically required to upgrade an operator from one version to another. It extracts the package name from bundle, finds the existing subscription, updates the catalog source, deletes the existing registry pod and creates a new registry pod with the version of bundle image provided in the command.

Let’s upgrade the previously deployed memcached-operator bundle from version 0.0.1 to 0.0.2.

$ operator-sdk run bundle-upgrade <some-registry>/memcached-operator-bundle:v0.0.2
INFO[0002] Found existing subscription with name memcached-operator-bundle-0-0-1-sub and namespace default
INFO[0002] Found existing catalog source with name memcached-operator-catalog and namespace default
INFO[0007] Successfully created registry pod: <some-registry>-memcached-operator-bundle-0-0-2
INFO[0007] Updated catalog source memcached-operator-catalog with address and annotations
INFO[0008] Deleted previous registry pod with name "<some-registry>-memcached-operator-bundle-0-0-1"
INFO[0050] Approved InstallPlan install-c8fkh for the Subscription: memcached-operator-bundle-0-0-1-sub
INFO[0050] Waiting for ClusterServiceVersion "default/memcached-operator.v0.0.2" to reach 'Succeeded' phase
INFO[0050]   Waiting for ClusterServiceVersion "default/memcached-operator.v0.0.2" to appear
INFO[0052]   Found ClusterServiceVersion "default/memcached-operator.v0.0.2" phase: Pending
INFO[0057]   Found ClusterServiceVersion "default/memcached-operator.v0.0.2" phase: InstallReady
INFO[0058]   Found ClusterServiceVersion "default/memcached-operator.v0.0.2" phase: Installing
INFO[0095]   Found ClusterServiceVersion "default/memcached-operator.v0.0.2" phase: Succeeded
INFO[0095] Successfully upgraded to "memcached-operator.v0.0.2"

Note: If a bundle was installed using operator-sdk run bundle with a SQLite index image, the replaces field must be present and populated in the upgraded CSV’s spec.

Upgrading a bundle that was installed traditionally using OLM

An operator bundle can be upgraded even if it was originally deployed using OLM without using the run bundle command.

Let’s see how to deploy an operator bundle traditionally using OLM and then upgrade the operator bundle to a newer version.

First, create a CatalogSource by building the CatalogSource from a catalog.

$ oc create -f catalogsource.yaml

# catalogsource.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: etcdoperator
  namespace: default
spec:
  displayName: Etcd Operators
  image: <some-registry>/etcd-catalog:latest
  sourceType: grpc

Next, install the operator bundle by creating a subscription.

$ oc create -f subscription.yaml

# subscription.yaml
apiVersion: v1
items:
- apiVersion: operators.coreos.com/v1alpha1
  kind: Subscription
  metadata:
    name: etcd
    namespace: default
  spec:
    channel: "stable"
    installPlanApproval: Manual
    name: etcd
    source: etcdoperator
    sourceNamespace: default
    startingCSV: etcdoperator.v0.0.1

Once the Operator bundle is deployed, you can use the run bundle-upgrade command by specifying the new bundle image that you want to upgrade to.

$ operator-sdk run bundle-upgrade <some-registry>/etcd-bundle:v0.0.2
INFO[0000] Found existing subscription with name etcd and namespace default
INFO[0000] Found existing catalog source with name etcdoperator and namespace default
INFO[0005] Successfully created registry pod: <some-registry>-etcd-bundle-0-0-2
INFO[0005] Updated catalog source etcdoperator with address and annotations
INFO[0005] Deleted previous registry pod with name "<some-registry>-etcd-bundle-0-0-1"
INFO[0005] Approved InstallPlan install-6vrzh for the Subscription: etcd
INFO[0005] Waiting for ClusterServiceVersion "default/etcdoperator.v0.0.2" to reach 'Succeeded' phase
INFO[0005]   Waiting for ClusterServiceVersion "default/etcdoperator.v0.0.2" to appear
INFO[0007]   Found ClusterServiceVersion "default/etcdoperator.v0.0.2" phase: Pending
INFO[0008]   Found ClusterServiceVersion "default/etcdoperator.v0.0.2" phase: Installing
INFO[0018]   Found ClusterServiceVersion "default/etcdoperator.v0.0.2" phase: Succeeded
INFO[0018] Successfully upgraded to "etcdoperator.v0.0.2"

Deploying bundles in production

OLM and Operator Registry consumes Operator bundles via a catalog of Operators, implemented as an index image, which are composed of one or more bundles. To build and push a memcached-operator bundle image for version v0.0.1, run:

$ make bundle-build bundle-push BUNDLE_IMG=<some-registry>/memcached-operator-bundle:v0.0.1

Now you can build and push the catalog by running catalog-* Makfile targets, which use the Operator package manager tool opm to build the catalog:

$ make catalog-build catalog-push CATALOG_IMG=<some-registry>/memcached-operator-catalog:v0.0.1

Assuming IMAGE_TAG_BASE = <some-registry>/memcached-operator has the desired tag base, you can inline the above two commands to:

$ make bundle-build bundle-push catalog-build catalog-push

Which will build and push both <some-registry>/memcached-operator-bundle:v0.0.1 and <some-registry>/memcached-operator-catalog:v0.0.1.

Docs: Development

Mon, 01 Jan 0001 00:00:00 +0000

Installation

Prerequisites

git
go version 1.23

Download Operator SDK

Go to the operator-sdk repo and follow the fork guide to fork and set up a local repository.

Build the Operator SDK CLI

Build the Operator SDK CLI operator-sdk binary:

make install

Testing

See the testing and documentation guides for more information.

Releasing

See the release guide for more information.

Continuous Integration (CI)

The operator-sdk repo uses Github Actions to test each pull request and build images for both master commits and release tags. You can alter these processes by modifying the appropriate Action config.

Adding new architectures

The operator-sdk project builds binaries for several os’s/architectures. If you wish to add support for a new one, please create a feature request issue before implementing support for that platform and submitting a PR.

If you’d like to implement support yourself, you can test a new architecture by enabling Actions in your repository, add a platform pair to the deploy workflow’s build and push step, and push to your main branch. Once the updated Action passes, submit a PR linking the passing Action run.

Docs: Installation Guide

Mon, 01 Jan 0001 00:00:00 +0000

Install `operator-sdk`

Follow the steps in the installation guide to learn how to install the operator-sdk CLI tool.

Additional Prerequisites

docker version 17.03+
python version 3.8.6+
ansible version v2.9.0+
ansible-runner version 2.0.2+
ansible-runner-http version v1.0.0+
openshift version v0.12.0+
kubectl and access to a Kubernetes cluster of a compatible version.

Docs: Installation Guide

Mon, 01 Jan 0001 00:00:00 +0000

Install `operator-sdk`

Follow the steps in the installation guide to learn how to install the operator-sdk CLI tool.

Additional Prerequisites

git
go version 1.22
docker version 17.03+.
kubectl and access to a Kubernetes cluster of a compatible version.

Docs: Installation Guide

Mon, 01 Jan 0001 00:00:00 +0000

Install `operator-sdk`

Follow the steps in the installation guide to learn how to install the operator-sdk CLI tool.

Additional Prerequisites

docker version 17.03+.
kubectl and access to a Kubernetes cluster of a compatible version.

Docs: Backport Policy

Mon, 01 Jan 0001 00:00:00 +0000

Mainly critical issue fixes are backported to the most recent minor release. Special backport requests can be discussed during the weekly Triage meeting; this does not guarantee an exceptional backport will be created. Occasionally non-critical issue fixes will be backported, either at an approver’s discretion or by request as noted above. For information on contacting maintainers and attending meetings, check the community repository.

Process

Typically an issue will be fixed in the master branch, which will then be cherry-picked to the most recent release’s branch. Those with approver permissions and above can create a cherry-pick PR, assuming no conflicts, by commenting /cherry-pick <release branch> in the PR fixing the issue in master. Fixes that are only relevant to a specific release branch can be made against that branch directly.

Docs: Common recommendations and suggestions

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Any recommendations or best practices suggested by the Kubernetes community, such as how to develop Operator pattern solutions or how to use controller-runtime are good recommendations for those who are looking to build operator projects with operator-sdk. Also, see Operator Best Practices. However, here are some common recommendations.

Common Recommendations

Develop idempotent reconciliation solutions

When developing operators, it is essential for the controller’s reconciliation loop to be idempotent. By following the Operator pattern you will create Controllers which provide a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster. Breaking this recommendation goes against the design principles of controller-runtime and may lead to unforeseen consequences such as resources becoming stuck and requiring manual intervention.

Understanding Kubernetes APIs

Building your own operator commonly involves extending the Kubernetes API itself. It is helpful to understand exactly how Custom Resource Definitions interact with the Kubernetes API. Also, the Kubebuilder documentation on Groups and Versions and Kinds may be helpful to better understand these concepts as they relate to operators.

Avoid a design solution where more than one Kind is reconciled by the same controller

Having many Kinds (such as CRDs) which are all managed by the same controller usually goes against the design proposed by controller-runtime. Furthermore this might hurt concepts such as encapsulation, the Single Responsibility Principle, and Cohesion. Damaging these concepts may cause unexpected side effects, and increase the difficulty of extending, reusing, or maintaining the operator.

Ideally Operators does not manage other Operators

From best practices:

“Operators should own a CRD and only one Operator should control a CRD on a cluster. Two Operators managing the same CRD is not a recommended best practice. In the case where an API exists but with multiple implementations, this is typically an example of a no-op Operator because it doesn’t have any deployment or reconciliation loop to define the shared API and other Operators depend on this Operator to provide one implementation of the API, e.g. similar to PVCs or Ingress."
“An Operator shouldn’t deploy or manage other operators (such patterns are known as meta or super operators or include CRDs in its Operands). It’s the Operator Lifecycle Manager’s job to manage the deployment and lifecycle of operators. For further information check Dependency Resolution."

What does it mainly mean:

If you want to define that your Operator depends on APIs which are owned by another Operator or on another whole Operator itself you should use Operator Lifecycle Manager’s Dependency Resolution
If you want to reconcile core APIs (defined by Kubernetes) or External APIs (defined from other operators) you should not re-define the API as owned by your project. Therefore, you can create the controller in this cases by using the flag --resource=false. (i.e. $ operator-sdk create api --group ship --version v1beta1 --kind External --resource=false --controller=true). Attention: If you are using Golang-based language Operator then, you will need to update the markers and imports manually until it become officially supported by the tool. For further information check the issue #1999.

WARNING: if you create CRD’s via the reconciliations or via the Operands then, OLM cannot handle CRDs migration and update, validation.

NOTE: By not following this guidance you might probably to be hurting concepts like as single responsibility principle and damaging these concepts could cause unexpected side effects, such as; difficulty extending, reuse, or maintenance, only to mention a few.

Other common suggestions

Provide the images and tags used by the operator solution via environment variables in the config/manager/manager.yaml:

...
spec:
  ...
    spec:
      ...
      containers:
      - command:
        - /manager
        ...
        env:
        - name: MY_IMAGE
          value: "quay.io/example.com/image:0.0.1"

Manage your solutions using Status Conditionals
Use finalizers when/if required
Cover the project with tests/CI to ensure its quality:
- For any language-based operator, you can use Scorecard to implement functional tests
- For Go-based operators, you can also use envtest to cover the controllers. For further information see Testing with EnvTest. Also, see the test directory for the Memcached sample under the testdata/go/v3/memcached-operator to know how can you build e2e tests.
- For Ansible-based operators, you can also use Molecule, an Ansible testing framework. For further information see Testing with Molecule
- For Helm-based operators, you can also use Chart tests
Ensure that you checked the Can I customize the projects initialized with operator-sdk? and understand the Project Layout before starting to do your customizations as please you on top.
Optimize manager resource values in config/manager/manager.yaml according to project requirements. It is recommended to define resources limits in order to follow good practices and for security reasons. More info: Managing Resources for Containers and Docker Security Cheat Sheet.
Look for TODO(user) in the source code generated by the CLI to ensure that you follow all suggested customizations.
If you wish to integrate your project with OLM, you can also check its Best Practices section.

Docs: OLM Integration Package Manifests Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

Note As operator framework has moved to using bundle format by default, the package manifest commands have been deprecated and will be removed soon. It is suggested that you follow the bundle quickstart to package your operator.

This guide assumes you have followed the introduction and Setup section of the bundle quickstart, and have added the packagemanifests target to your Makefile as described here.

Creating package manifests

We will now create a package manifests format by running make packagemanifests in the root of the memcached-operator project:

$ make packagemanifests
/home/user/go/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
operator-sdk generate kustomize manifests -q
kustomize build config/manifests | operator-sdk generate packagemanifests -q --version 0.0.1

A versioned manifests directory packagemanifests/0.0.1 containing a CSV and all CRDs in config/crds and a package manifest YAML file packagemanifests/<project-name>.package.yaml have been created in the Operator project.

Deploying an Operator with OLM

At this point in development we’ve generated all files necessary to build a memcached-operator registry. Now we’re ready to test the Operator with OLM.

Testing package manifests

operator-sdk run packagemanifests will create an Operator registry from manifests and metadata in the memcached-operator project, and inform OLM that memcached-operator v0.0.1 is ready to be deployed. This process effectively replicates production deployment in a constrained manner to make sure OLM can deploy our Operator successfully before attempting real production deployment.

run packagemanifests performs some optionally configurable setup under the hood, but for most use cases the following invocation is all we need:

$ operator-sdk run packagemanifests --version 0.0.1
INFO[0000] Running operator from directory packagemanifests
INFO[0000] Creating memcached-operator registry         
INFO[0000]   Creating ConfigMap "olm/memcached-operator-registry-manifests-package"
INFO[0000]   Creating ConfigMap "olm/memcached-operator-registry-manifests-0-0-1"
INFO[0000]   Creating Deployment "olm/memcached-operator-registry-server"
INFO[0000]   Creating Service "olm/memcached-operator-registry-server"
INFO[0000] Waiting for Deployment "olm/memcached-operator-registry-server" rollout to complete
INFO[0000]   Waiting for Deployment "olm/memcached-operator-registry-server" to rollout: 0 of 1 updated replicas are available
INFO[0066]   Deployment "olm/memcached-operator-registry-server" successfully rolled out
INFO[0066] Creating resources                           
INFO[0066]   Creating CatalogSource "default/memcached-operator-ocs"
INFO[0066]   Creating Subscription "default/memcached-operator-v0-0-1-sub"
INFO[0066]   Creating OperatorGroup "default/operator-sdk-og"
INFO[0066] Waiting for ClusterServiceVersion "default/memcached-operator.v0.0.1" to reach 'Succeeded' phase
INFO[0066]   Waiting for ClusterServiceVersion "default/memcached-operator.v0.0.1" to appear
INFO[0073]   Found ClusterServiceVersion "default/memcached-operator.v0.0.1" phase: Pending
INFO[0077]   Found ClusterServiceVersion "default/memcached-operator.v0.0.1" phase: InstallReady
INFO[0078]   Found ClusterServiceVersion "default/memcached-operator.v0.0.1" phase: Installing
INFO[0036]   Found ClusterServiceVersion "default/memcached-operator.v0.0.1" phase: Succeeded
INFO[0037] Successfully installed "memcached-operator.v0.0.1" on OLM version "0.15.1"

NAME                            NAMESPACE    KIND                        STATUS
memcacheds.cache.example.com    default      CustomResourceDefinition    Installed
memcached-operator.v0.0.1       default      ClusterServiceVersion       Installed

As long as both the ClusterServiceVersion and all CustomResourceDefinition's return an Installed status, the memcached-operator has been deployed successfully.

Now that we’re done testing the memcached-operator, we should probably clean up the Operator’s resources. operator-sdk cleanup will do this for you:

$ operator-sdk cleanup memcached-operator
INFO[0000] subscription "memcached-operator-v0-0-1-sub" deleted
INFO[0000] customresourcedefinition "memcacheds.cache.example.com" deleted
INFO[0000] clusterserviceversion "memcached-operator.v0.0.1" deleted
INFO[0000] clusterrole "memcached-operator-metrics-reader" deleted
INFO[0000] serviceaccount "default" deleted
INFO[0000] role "memcached-operator.v0.0.1-jhjk7" deleted
INFO[0000] rolebinding "memcached-operator.v0.0.1-jhjk7-default-mxv6m" deleted
INFO[0000] catalogsource "memcached-operator-ocs" deleted
INFO[0000] operatorgroup "operator-sdk-og" deleted
INFO[0001] operator "memcached-operator" uninstalled

Migrating packagemanifests to bundles

In order to migrate packagemanifests to bundles, operator-sdk pkgman-to-bundle command can be used.

As an example, consider the packagemanifests directory to have the following structure:

packagemanifests
└── etcd
    ├── 0.0.1
    │   ├── etcdcluster.crd.yaml
    │   └── etcdoperator.clusterserviceversion.yaml
    ├── 0.0.2
    │   ├── etcdbackup.crd.yaml
    │   ├── etcdcluster.crd.yaml
    │   ├── etcdoperator.v0.0.2.clusterserviceversion.yaml
    │   └── etcdrestore.crd.yaml
    └── etcd.package.yaml

Here, we have manifests for two versions of the etcd operator. The following command will generate bundles for each of these versions.

$ operator-sdk pkgman-to-bundle packagemanifests --output-dir etcd-bundle/
INFO[0000] Packagemanifests will be migrated to bundles in bundle directory
INFO[0000] Creating etcd-bundle/bundle-0.0.1/bundle.Dockerfile
INFO[0000] Creating etcd-bundle/bundle-0.0.1/metadata/annotations.yaml
...

This will create output bundles in the directory etcd-bundle. The output directory will look like:

etcd-bundle/
├── bundle-0.0.1
│   ├── bundle
│   │   ├── manifests
│   │   │   ├── etcdcluster.crd.yaml
│   │   │   ├── etcdoperator.clusterserviceversion.yaml
│   │   ├── metadata
│   │   │   └── annotations.yaml
│   │   └── tests
│   │       └── scorecard
│   │           └── config.yaml
│   └── bundle.Dockerfile
└── bundle-0.0.2
    ├── bundle
    │   ├── manifests
    │   │   ├── etcdbackup.crd.yaml
    │   │   ├── etcdcluster.crd.yaml
    │   │   ├── etcdoperator.v0.0.2.clusterserviceversion.yaml
    │   │   ├── etcdrestore.crd.yaml
    │   └── metadata
    │       └── annotations.yaml
    └── bundle.Dockerfile

To build images for the bundles, the base container image name can be provided using --image-tag-base flag. This name should be provided without the tag (: and characters following), as the command will tag each bundle image with its packagemanifests directory name, i.e. <image-tag-base>:<dir-name>. For example, the following command for the above packagemnifests directory would build the bundles quay.io/example/etcd-bundle:0.0.1 and quay.io/example/etcd-bundle:0.0.2.

operator-sdk pkgman-to-bundle packagemanifests --image-tag-base quay.io/example/etcd-bundle

A custom command can also be specified to build images, using the --build-cmd flag. The default command is docker build. For example:

$ operator-sdk pkgman-to-bundle packagemanifests --output-dir etcd-bundle/ --image-tag-base quay.io/example/etcd --build-cmd "podman build -f bundle.Dockerfile . -t"

However, if using a custom command, it needs to be made sure that the command is in the PATH or a fully qualified path name is provided as input to the flag.

Once the command has finished building your bundle images and they have been added to a catalog image, delete all bundle directories except for the latest one. This directory will contain manifests for your operator’s head bundle, and should be versioned with version control system like git. Move this directory and its bundle.Dockerfile to your project’s root:

$ cp -r ./etcd-bundle/bundle-0.0.2/* .
$ rm -rf ./etcd-bundle

Try building then running your bundle on a live cluster to make sure it works as expected:

$ make bundle bundle-build bundle-push
$ operator-sdk run bundle quay.io/example/etcd-bundle:0.0.2

Docs: Project Layout

Mon, 01 Jan 0001 00:00:00 +0000

Operator SDK Project Layout

All projects initialized with operator-sdk init have a common base structure which builds on kubebuilder’s project layout. Each project type is customized further with code of that type’s language.

Common Base

The common structure contains the following items:

File/Directory	Description
`Dockerfile`	The Dockerfile of your operator project, used to build the image with `make docker-build`.
`Makefile`	Build file with helper targets to help you work with your project.
`PROJECT`	This file represents the project’s configuration and is used to track useful information for the CLI and plugins.
`bin/`	This directory contains useful binaries such as the `manager` which is used to run your project locally and the `kustomize` utility used for the project configuration. For other language types, it might have other binaries useful for developing your operator.
`bundle/`	This directory contains all the files used to integrate your project with OLM with the bundle format. It is built from the Makefile target `make bundle`.
`bundle/manifests/`	This directory has the OLM manifests of your bundle.
`bundle/metadata/`	This directory has the OLM metadata of your bundle e.g the index image annotations.
`bundle/tests/`	This directory has the Scorecard tests shipped with your operator bundle.
`config/`	Contains configuration files to launch your project on a cluster. Plugins might use it to provide functionality. For example, for the CLI to help create your operator bundle it will look for the CRD’s and CR’s which are scaffolded in this directory. You will also find all Kustomize YAML definitions as well.
`config/crd/`	Contains the Custom Resources Definitions.
`config/default/`	Contains a Kustomize base for launching the controller in a standard configuration.
`config/manager/`	Contains the manifests to launch your operator project as pods on the cluster.
`config/manifests/`	Contains the base to generate your OLM manifests in the bundle directory.
`config/prometheus/`	Contains the manifests required to enable project to serve metrics to Prometheus such as the `ServiceMonitor` resource.
`config/scorecard/`	Contains the manifests required to allow you test your project with Scorecard.
`config/rbac/`	Contains the RBAC permissions required to run your project.
`config/samples/`	Contains the Custom Resources.
`bundle.Dockerfile`	The Dockerfile to build the bundle image. Used to build the operator bundle image with `make bundle-build. \|

Ansible

Now, let’s look at the files and directories specific to Ansible-based operators.

File/Directory	Description
`config/testing/`	Manifest files to help you test your project. For example, to change the image policy for your Molecule tests or to enable debug level in the Ansible logs.
`molecule/`	Contain the manifests for your Molecule tests.
`molecule/default`	Contains the default Molecule task.
`molecule/kind`	Contains the Molecule task to be executed on the cluster.
`playbooks/`	Contains the Ansible playbooks.
`roles/`	Contains the Ansible role files for each Kind scaffold.
`requirements.yml`	This file specifies Ansible dependencies that need to be installed for your operator to function.
`watches.yaml`	Contains Group, Version, Kind, and the playbooks and rules location. Used to configure the Ansible watches.

Golang

Now, let’s look at the files and directories specific to Go-based operators.

File/Directory	Description
`api/`	Contains the api definition
`config/certmanager`	Contains the Kustomize manifests which configure the cert-manager by the Webhooks.
`config/webhook`	Contains the Kustomize manifests to configure the Webhook.
`controllers`	Contains the controllers.
`main.go`	Implements the project initialization.
`hack/`	Contains utility files, e.g. the file used to scaffold the license header for your project files.

Helm

Now, let’s look at the files and directories specific to Helm-based operators.

File/Directory	Description
`helm-charts`	Contains the Helm charts for each Kind scaffold which can be initialized with `operator-sdk init --plugins=helm [options]` or `operator-sdk create api [options]` .
`watches.yaml`	Contains Group, Version, Kind, and Helm chart location. Used to configure the Helm watches.

Docs: Quickstart for Ansible-based Operators

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks through an example of building a simple memcached-operator powered by Ansible using tools and libraries provided by the Operator SDK.

Prerequisites

Go through the installation guide.
Make sure your user is authorized with cluster-admin permissions.
Have an accessible image registry for various operator images (ex. hub.docker.com, quay.io) and be logged in to your command line environment.
- example.com is used as the registry Docker Hub namespace in these examples. Replace it with another value if using a different registry or namespace.
- Authentication and certificates if the registry is private or uses a custom CA.

Steps

Create a project directory for your project and initialize the project:

mkdir memcached-operator
cd memcached-operator
operator-sdk init --domain example.com --plugins ansible

Create a simple Memcached API:

operator-sdk create api --group cache --version v1alpha1 --kind Memcached --generate-role

Build and push your operator’s image:

make docker-build docker-push IMG="example.com/memcached-operator:v0.0.1"

OLM deployment

Install OLM:

operator-sdk olm install

Bundle your operator, then build and push the bundle image (defaults to example.com/memcached-operator-bundle:v0.0.1):

make bundle IMG="example.com/memcached-operator:v0.0.1"
make bundle-build bundle-push

Run your bundle. If your bundle image is hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

operator-sdk run bundle example.com/memcached-operator-bundle:v0.0.1

Create a sample Memcached custom resource:

$ kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com/memcached-sample created

Uninstall the operator:

operator-sdk cleanup memcached-operator

Direct deployment

Deploy your operator:

make deploy IMG="example.com/memcached-operator:v0.0.1"

Create a sample Memcached custom resource:

$ kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com/memcached-sample created

Uninstall the operator:

make undeploy

Next Steps

Read the full tutorial for an in-depth walkthrough of building an Ansible operator.

Docs: Cheat Sheet

Mon, 01 Jan 0001 00:00:00 +0000

Below you will find a cheat sheet with options and helpers for projects, which are built with the SDK and are respecting its proposed layout.

Common commands and options

Command	Description
`operator-sdk init`	To initialize an operator project in the current directory.
`operator-sdk init --plugins=<plugin-key>`	To initialize an operator project in the current directory using a specific plugin. To check the available plugins you can run `operator-sdk --help`. E.g (`operator-sdk init --plugins=helm`).
`operator-sdk create api [flags]`	Lets you create your own APIs with its GKV by Extending the Kubernetes API with CustomResourceDefinitions, or lets you use external/core-types. Also generates their respective controllers.
`operator-sdk create webhook [flags]`	To scaffold Webhooks for the APIs declared in the project. Currently, only the Go-based project supports this option.
`make docker-build IMG=<some-registry>/<project-name>:<tag>`	Build the operator image.
`make docker-build docker-push IMG=<some-registry>/<project-name>:<tag>`	Build and push the operator image for your registry.
`make install`	Install the CRDs into the cluster.
`make uninstall`	Uninstall the CRDs into the cluster.
`make run`	Run your controller locally and outside of the cluster. Note that this will run in the foreground, so switch to a new terminal if you want to leave it running.
`make deploy`	Deploy your project on the cluster.
`make undeploy`	Undeploy your project on the cluster.

To create bundles, catalogs, and develop for OLM

For further information check Operator SDK Integration with Operator Lifecycle Manager.

Command	Description
`make bundle`	Create/update the bundle based on the project manifests in the `bundle/` directory. For more info see Create a bundle.
`operator-sdk bundle validate ./bundle`	To validate your bundle spec definition.
`operator-sdk bundle validate ./bundle --select-optional suite=operatorframework`	Validate your bundle against OperatorHub.io criteria. For further information use the flag `--help`.
`operator-sdk olm install`	To install OLM on your cluster for development purposes.
`operator-sdk olm uninstall`	To uninstall OLM from your cluster.
`make bundle-build BUNDLE_IMG=<some-registry>/<project-name-bundle>:<tag>`	To build your bundle operator image.
`make bundle-build bundle-push BUNDLE_IMG=<some-registry>/<project-name-bundle>:<tag>`	To build and push your bundle operator image.
`operator-sdk run bundle <some-registry>/<project-name-bundle>:<tag>`	To deploy your bundle operator using OLM on your cluster for development purposes.
`operator-sdk run bundle private-registry.org/bundle:v1.2.3 --service-account sa-with-secret --pull-secret-name regcred --ca-secret-name cert-sec`	Configure `run bundle` (and `run bundle-upgrade`) to use an image pull secret, non-default service account configured with that secret, and custom CA certificate secret

Updating bundle channels

The following examples let you update the bundle with data-informed. For further information also check Upgrade your Operator and see Channel Naming.

NOTE: Note that it will carry over any customizations you have made and ensure a rolling update to the next version of your Operator.

make bundle CHANNELS=fast,preview DEFAULT_CHANNEL=stable VERSION=1.0.0 IMG=<some-registry>/<project-name-bundle>:<tag>

NOTE You can use environment variables to pass the values such as export CHANNELS=fast,candidate. Note that, their values will be used by make bundle command.

To test your projects

Command	Description
`operator-sdk scorecard ./bundle`	Run the Scorecard tests for your bundle.
`make test`	Run Go tests. It is valid only for Go-based operators.
`molecule test`	Run Molecule tests. It is valid only for Ansible-based operators.
`helm test`	Run Helm chart tests. It is valid only for Helm-based operators.

NOTE: This is not a comprehensive list of make targets or commands. Please see the scaffolded Makefile and make help for the full list of targets. Note that you can use operator-sdk <command> --help and check the CLI section to check all options.

Docs: Resource pruning

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Operators can create Jobs or Pods as part of their normal operation, and when those Jobs or Pods complete, they can remain on the Kubernetes cluster if not specifically removed. These resources can consume valuable cluster resources like disk storage (e.g. etcd). These resources are not tied to a Custom Resource using an ownerReference.

Operator authors have traditionally had two pruning options:

leave the resource cleanup to a system admin to perform
implement some form of pruning within their operator solution

For our purposes when we say, prune, we mean to remove a resource (e.g. kubectl delete) from a Kubernetes cluster for a given namespace.

This documentation describes the pattern and library useful for implementing a solution within an operator.

operator-lib prune library

A simple pruning implementation can be found in the operator-lib prune package. This package is written in Go and is meant to be used within Go-based operators. This package was developed to include common pruning strategies as found in common operators. The package also allow for customization of hooks and strategies.

Pruning Configuration

Users can configure the pruning library by creating code similar to this example:

cfg = Config{
	Log:           logf.Log.WithName("prune"),
	DryRun:        false,
	Clientset:     client,
	LabelSelector: "app=churro",
	Resources: []schema.GroupVersionKind{
		{Group: "", Version: "", Kind: PodKind},
	},
	Namespaces: []string{"default"},
	Strategy: StrategyConfig{
		Mode:            MaxCountStrategy,
		MaxCountSetting: 1,
	},
	PreDeleteHook: myhook,
}

Config Field	Description
Log	a logr.Logger. It is optional if a logger is provided through the context to the Execute method, which is the case with the context of the Reconcile function of operator-sdk and controller-runtime
DryRun	a boolean determines whether to actually remove resources; `true` means to execute but not to remove resources
Clientset	a client-go Kubernetes ClientSet that will be used for Kube API calls by the library
LabelSelector	Kubernetes label selector expression used to find resources to prune
Resources	Kube resource Kinds, currently PodKind and JobKind are supported by the library
Namespaces	a list of Kube Namespaces to search for resources
Strategy	specifies the pruning strategy to execute
Strategy.Mode	currently MaxCountStrategy, MaxAgeStrategy, or CustomStrategy are supported
Strategy.MaxCountSetting	integer value for maxcount strategy, specifies how many resources should remain after pruning executes
Strategy.MaxAgeSetting	golang time.Duration string value (e.g. 48h), specifies age of resources to prune
Strategy.CustomSettings	a golang map of values that can be passed into a Custom strategy function
PreDeleteHook	optionally specifies a golang function to call before pruning a resource
CustomStrategy	optionally specifies a golang function that implements a custom pruning strategy

Pruning Execution

Users can invoke the pruning by running the Execute function on the pruning configuration as follows:

err := cfg.Execute(ctx)

Users might want to implement pruning execution by means of a cron package or simply call the prune library based on some other triggering event.

If a logger has been configured in the Config structure it takes precedence on the one provided through ctx. Adding a logger.Logger to the context can be done with logr.NewContext.

Pruning Strategies

maxcount Strategy

A strategy of leaving a finite set of resources is implemented called maxcount. This strategy seeks to leave a specific number of resources, sorted by latest, on your cluster. For example, if you have 10 resources that would be pruned, and you specified a maxcount value of 4, then 6 resources would be pruned (removed) from your cluster starting with the oldest resources.

maxage Strategy

A strategy of removing resources greater than a specific time is called maxage. This strategy seeks to remove resources older than a specified maxage duration. For example, a library user might specify a value of 48h to indicate that any resource older than 48 hours would be pruned. Durations are specified using golang’s time.Duration formatting (e.g. 48h).

Pruning Customization

preDelete Hook

Users can provide a preDelete hook when using the operator-lib prune package.
This hook function will be called by the library before removing a resource. This provides a means to examine the resource logs for example, extracting any valued content, before the resource is removed from the cluster.

Here is an example of a preDelete hook:

func myhook(ctx context.Context, cfg Config, res ResourceInfo) error {
        log := prune.Logger(ctx, cfg)
        log.V(4).Info("pre-deletion", "GVK", res.GVK, "namespace", res.Namespace, "name", res.Name)
       	if res.GVK.Kind == PodKind {
                req := cfg.Clientset.CoreV1().Pods(res.Namespace).GetLogs(res.Name, &v1.PodLogOptions{})
                podLogs, err := req.Stream(context.Background())
                if err != nil {
                        return err
                }
                defer podLogs.Close()

                buf := new(bytes.Buffer)
                _, err = io.Copy(buf, podLogs)
                if err != nil {
                        return err
                }

                log.V(4).Info("pod log before removing is", "log", buf.String())
        }
	return nil
}

Note if your custom hook returns an error, then the resource will not be removed by the prune library.

Custom Strategy

Library users can also write their own custom pruning strategy function to support advanced cases. Custom strategy functions are passed in the prune configuration and a list of resources selected by the library. The custom strategy builds up a list of resources to be removed, returning the list to the prune library which performs the actual resource removal. Here is an example custom strategy:

func myStrategy(ctx context.Context, cfg Config, resources []ResourceInfo) (resourcesToRemove []ResourceInfo, err error) {
        log := Logger(ctx, cfg)
        log.V(4).Info("myStrategy called", "resources", resources, "config", cfg)
	if len(resources) != 3 {
		return resourcesToRemove, fmt.Errorf("count of resources did not equal our expectation")
	}
	return resourcesToRemove, nil
}

To have your custom strategy invoked, you will specify your function within the prune configuration as follows:

cfg.Strategy.Mode = CustomStrategy
cfg.Strategy.CustomSettings = make(map[string]interface{})
cfg.CustomStrategy = myStrategy

Notice that you can optionally pass in settings to your custom function as a map using the cfg.Strategy.CustomSettings field.

Docs: Ansible Operator Tutorial

Mon, 01 Jan 0001 00:00:00 +0000

NOTE: If your project was created with an operator-sdk version prior to v1.0.0 please migrate, or consult the legacy docs.

Prerequisites

Go through the installation guide.
Make sure your user is authorized with cluster-admin permissions.
An accessible image registry for various operator images (ex. hub.docker.com, quay.io) and be logged in to your command line environment.
- example.com is used as the registry Docker Hub namespace in these examples. Replace it with another value if using a different registry or namespace.
- Authentication and certificates if the registry is private or uses a custom CA.

Overview

We will create a sample project to let you know how it works and this sample will:

Create a Memcached Deployment if it doesn’t exist
Ensure that the Deployment size is the same as specified by the Memcached CR spec
Update the Memcached CR status using the status writer with the names of the CR’s pods

Create a new project

Use the CLI to create a new memcached-operator project:

mkdir memcached-operator
cd memcached-operator
operator-sdk init --plugins=ansible --domain example.com

Among the files generated by this command is a Kubebuilder PROJECT file. Subsequent operator-sdk commands (and help text) run from the project root read this file and are aware that the project type is Ansible.

Next, we will create a Memcached API.

operator-sdk create api --group cache --version v1alpha1 --kind Memcached --generate-role

The scaffolded operator has the following structure:

Memcached Custom Resource Definition, and a sample Memcached resource.
A “Manager” that reconciles the state of the cluster to the desired state
- A reconciler, which is an Ansible Role or Playbook.
- A watches.yaml file, which connects the Memcached resource to the memcached Ansible Role.

See scaffolded files reference and watches reference for more detailed information.

Modify the Manager

Now we need to provide the reconcile logic, in the form of an Ansible Role, which will run every time a Memcached resource is created, updated, or deleted.

Update roles/memcached/tasks/main.yml:

---
- name: start memcached
  kubernetes.core.k8s:
    definition:
      kind: Deployment
      apiVersion: apps/v1
      metadata:
        name: '{{ ansible_operator_meta.name }}-memcached'
        namespace: '{{ ansible_operator_meta.namespace }}'
      spec:
        replicas: "{{size}}"
        selector:
          matchLabels:
            app: memcached
        template:
          metadata:
            labels:
              app: memcached
          spec:
            containers:
            - name: memcached
              command:
              - memcached
              - -m=64
              - -o
              - modern
              - -v
              image: "docker.io/memcached:1.4.36-alpine"
              ports:
                - containerPort: 11211

This memcached role will:

Ensure a memcached Deployment exists
Set the Deployment size

Note that the tasks in this Ansible role file are what actually defines the behavior of the spec and status of the memcached custom resource. As Kubernetes allows entry of arbitrary fields when creating resources, we don’t need to actually create specific fields in the CRD. While we won’t be doing this in this tutorial, it is recommended to also define these fields in the CRD, so that Kubernetes users can see the fields that will be used when using the custom resource. It is also good practice to set default values for variables used in Ansible Roles, so edit roles/memcached/defaults/main.yml:

---
# defaults file for Memcached
size: 1

Finally, update the Memcached sample, config/samples/cache_v1alpha1_memcached.yaml:

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  size: 3

The key-value pairs in the Custom Resource spec are passed to Ansible as extra variables.

Note: The names of all variables in the spec field are converted to snake_case by the operator before running Ansible. For example, serviceAccount in the spec becomes service_account in Ansible. You can disable this case conversion by setting the snakeCaseParameters option to false in your watches.yaml. It is recommended that you perform some type validation in Ansible on the variables to ensure that your application is receiving the expected input.

Configure the operator’s image registry

All that remains is to build and push the operator image to the desired image registry. Your Makefile composes image tags either from values written at project initialization or from the CLI. In particular, IMAGE_TAG_BASE lets you define a common image registry, namespace, and partial name for all your image tags. Update this to another registry and/or namespace if the current value is incorrect. Afterwards you can update the IMG variable definition like so:

-IMG ?= controller:latest
+IMG ?= $(IMAGE_TAG_BASE):$(VERSION)

Once done, you do not have to set IMG or any other image variable in the CLI. The following command will build and push an operator image tagged as example.com/memcached-operator:v0.0.1 to Docker Hub:

make docker-build docker-push

Run the Operator

There are three ways to run the operator:

As a Go program outside a cluster
As a Deployment inside a Kubernetes cluster
Managed by the Operator Lifecycle Manager (OLM) in bundle format

1. Run locally outside the cluster

Execute the following command, which installs your CRDs and runs the manager locally:

make install run

2. Run as a Deployment inside the cluster

By default, a new namespace is created with name <project-name>-system, ex. memcached-operator-system, and will be used for the deployment.

Commonly, Operator authors may need to modify config/rbac in order to give their Operator the necessary permissions to reconcile.

Run the following to customize the manifests and deploy the operator.

make deploy

The scaffolded Makefile uses kustomize to apply custom configurations and generate manifests from the config/ directory, which are piped to kubectl. Run the following command to see the manifests that were applied to the cluster.

kustomize build config/default

Verify that the memcached-operator is up and running:

$ kubectl get deployment -n memcached-operator-system
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
memcached-operator-controller-manager   1/1     1            1           8m

3. Deploy your Operator with OLM

First, install OLM:

operator-sdk olm install

Bundle your operator, then build and push the bundle image. The bundle target generates a bundle in the bundle directory containing manifests and metadata defining your operator. bundle-build and bundle-push build and push a bundle image defined by bundle.Dockerfile.

make bundle bundle-build bundle-push

Finally, run your bundle. If your bundle image is hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

operator-sdk run bundle example.com/memcached-operator-bundle:v0.0.1

Check out the docs for a deep dive into operator-sdk's OLM integration.

Create a Memcached CR

Update the sample Memcached CR manifest at config/samples/cache_v1alpha1_memcached.yaml and define the spec as the following:

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  size: 3

Create the CR:

kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml

Ensure that the memcached operator creates the deployment for the sample CR with the correct size:

$ kubectl get deployment
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
memcached-sample                        3/3     3            3           1m

Check the pods and CR status to confirm the status is updated with the memcached pod names:

$ kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          1m
memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          1m
memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          1m

$ kubectl get memcached/memcached-sample -o yaml
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  creationTimestamp: "2021-03-17T19:54:42Z"
  generation: 1
  managedFields:
  - apiVersion: cache.example.com/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        .: {}
        f:conditions: {}
    manager: ansible-operator
    operation: Update
    time: "2021-03-17T19:54:42Z"
  - apiVersion: cache.example.com/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:size: {}
    manager: kubectl
    operation: Update
    time: "2021-03-17T19:54:42Z"
  name: memcached-sample
  namespace: default
  resourceVersion: "1008"
  uid: 4b023125-132a-44e3-80de-20801c7a9268
spec:
  size: 3
status:
  conditions:
  - ansibleResult:
      changed: 0
      completion: 2021-03-17T19:54:54.890394
      failures: 0
      ok: 1
      skipped: 0
    lastTransitionTime: "2021-03-17T19:54:42Z"
    message: Awaiting next reconciliation
    reason: Successful
    status: "True"
    type: Running

Update the size

Update config/samples/cache_v1alpha1_memcached.yaml to change the spec.size field in the Memcached CR from 3 to 5:

kubectl patch memcached memcached-sample -p '{"spec":{"size": 5}}' --type=merge

Confirm that the operator changes the deployment size:

$ kubectl get deployment
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
memcached-sample                        5/5     5            5           3m

Cleanup

Run the following to delete all deployed resources:

kubectl delete -f config/samples/cache_v1alpha1_memcached.yaml
make undeploy

Next Steps

We recommend reading through the Ansible development section for tips and tricks, including how to run the operator locally.

In this tutorial, the scaffolded watches.yaml could be used as-is, but has additional optional features. See the watches reference.

For brevity, some of the scaffolded files were left out of this guide. See Scaffolding Reference.

This example built a namespaced scope operator, but Ansible operators can also be used with cluster-wide scope.

OLM will manage creation of most if not all resources required to run your operator, using a bit of setup from other operator-sdk commands. Check out the OLM integration guide.

Docs: Operator Capability Levels

Mon, 01 Jan 0001 00:00:00 +0000

Operators come in different maturity levels in regards to their lifecycle management capabilities for the application or workload they deliver. The capability models aims to provide guidance in terminology to express what features users can expect from an operator.

Each capability level is associated with a certain set of management features the Operator offers around the managed workload. Operator that do not manage a workload and/or are delegating to off-clusters orchestration services would remain at Level 1. Capability levels are designated from level 1 to level 5. Each capability represents its own set of features and may be independent of each other.

Terminology

Operator - the custom controller installed on a Kubernetes cluster

Operand - the managed workload provided by the Operator as a service

Custom Resource (CR) - an instance of the CustomResourceDefinition the Operator ships that represents the Operand or an Operation on an Operand (also known as primary resources)

Managed resources - the Kubernetes objects or off-cluster services the Operator uses to constitute an Operand (also known as secondary resources)

Custom Resource Definition (CRD) - an API of the Operator, providing the blueprint and validation rules for Custom Resources.

Level 1 - Basic Install

Automated application provisioning and configuration management. This first capability level means your operator can fully provision an application through a custom resource, allowing all installation configuration details to be specified in the CR. It should also be possible to install the operator itself in multiple ways (kubectl, OLM, Catalog source). Any configuration required to make the Operand run should be configured through the CR if possible. Avoid the practice of requiring the user to create/manage configuration files outside of Kubernetes.

Installation of the workload

Operator deploys an Operand or configures off-cluster resources
Operator waits for managed resources to reach a healthy state
Operator conveys readiness of application or managed resources to the user leveraging the status block of the Custom Resource

Example: an Operator deploys a database by creating Deployment, ServiceAccount, RoleBinding, ConfigMap, PersistentVolumeClaim and Secret object, initializes an empty database schema and signals readiness of the database to accept queries.

Configuration of the workload

Operator provides configuration via the spec section of the Custom Resource
Operator reconciles configuration and updates to it with the status of the managed resources

Example: an Operator, managing a database, offers increasing the capacity of the database by resizing the underlying PersistentVolumeClaim based on changes the databases Custom Resource instance.

Guiding questions to determine Operator reaching Level 1

What installation configuration can be set in the CR?
What additional installation configuration could still be added?
Can you set operand configuration in the CR? If so, what configuration is supported for each operand?
Can you override the operand images through the CR or an environment variable of the Operator deployment?
Does the managed application / workload get updated in a non-disruptive fashion when the configuration of the CR is changed?
Does the status of the CR reflect that configuration changes are currently applied?
What additional operand configuration could still be added?
Do all of the instantiated CRs include a status block? If so, does it provide enough insight to the user about the application state?
Do all of your CRs have documentation listing valid values and mandatory fields?
If your operator is packaged for OLM, does its CSV list all images used in the CSV under spec.relatedImages?

Level 2 - Seamless Upgrades

Seamless upgrades mean the upgrade is as easy as possible for the user. You should support seamless upgrades of both your operator and operand, these would normally go hand in hand, an upgrade of the operator would automatically ensure the instantiated resources for each CR are in the new desired state and which would upgrade your operand. Upgrade may also be defined in multiple ways, such as updating the software of the operand - and other internals specific to the application - such as schema migrations. It should be very clear what is upgraded when this takes place, and what is not.

Upgrade of the managed workload

Operand can be upgraded in the process of upgrading the Operator, or
Operand can be upgraded as part of changing the CR
Operator understands how to upgrade older versions of the Operand, managed previously by an older version of the Operator

Upgrade of the Operator

Operator can be upgraded seamlessly and can either still manage older versions of the Operand or update them
Operator conveys inability to manage an unsupported version of the Operand in the status section of the CR

Example: An Operator managing a database can update an existing database from a previous to a newer version without data loss. The Operator might do so as part of a configuration change or as part of an update of the Operator itself.

Guiding questions to determine Operator reaching Level 2

Can your Operator upgrade your Operand?
Does your Operator upgrade your Operand during updates of the Operator?
Can your Operator manage older Operand version versions?
Is the Operand upgrade non disruptive?
If there is downtime during an upgrade, does the Operator convey this in the status of the CR?

Level 3 - Full Lifecycle

It should be possible to backup and restore the operand from the operator itself without any additional manual intervention other than triggering these operations. The operand data that should be backed up is any stateful data managed by the operand. You don’t need to backup the CR itself or the k8s resources created by the operator as the operator should return all resources to the same state if the CR is recreated. If your operator does not already setup the operand with other k8s resilient best practices, this should be completed to achieve this capability level. This includes liveness and readiness probes, multiple replicas, rolling deployment strategies, pod disruption budgets, CPU and memory requests and limits.

Lifecycle features

Operator provides the ability to create backups of the Operand
Operator is able to restore a backup of an Operand
Operator orchestrates complex re-configuration flows on the Operand
Operator implements fail-over and fail-back of clustered Operands
Operator supports add/removing members to a clustered Operand
Operator enables application-aware scaling of the Operand

Example: an Operator managing a database provides the ability to create an application consistent backup of the data by flushing the database log and quiescing the write activity to the database files.

Guiding questions to determine Operator reaching Level 3

Does your Operator support backing up the Operand?
Does your Operator support restoring an Operand from a backup and get it under management again?
Does your Operator wait for reconfiguration work to be finished and in the expected sequence?
Is your Operator taking cluster quorum into account, if present?
Does your Operator allow adding/removing read-only slave instances of your Operator?
Does your operand have a Liveness probe?
Does your operand have a Readiness probe which will fail if any aspect of the operand is not ready? e.g. if the connection to the database fails.
Does your operand use a rolling deployment strategy?
Does your operator create a PodDisruptionBudget resource for your operand pods?
Does your operand have CPU requests and limits set?

Level 4 - Deep Insights

Setup full monitoring and alerting for your operand. All resources such as Prometheus rules (alerts) and Grafana dashboards should be created by the operator when the operand CR is instantiated. The RED method¹ is a good place to start with knowing what metrics to expose. Aim to have as few alerts as possible, by alerting on symptoms that are associated with end-user pain rather than trying to catch every possible way that pain could be caused. Alerts should link to relevant consoles and make it easy to figure out which component is at fault Native k8s objects emit events (“Events” objects) for situations users or administrators should be alerted about. Your operator should do similar for state changes related to your operand. “Custom”, here, means that it should emit events specific to your operator/operand outside of the events already emitted by their deployment methodology. This, in conjunction with status descriptors for the CR conditions, give much needed visibility into actions taken by your operator/operand. Operators are codified domain-specific knowledge. Your end user should not need this domain-specific knowledge to gain visibility into what’s happening with their resource. Please, ensure that you look at the Kubernetes API conventions in the Events and status sections to know how to properly deal with them.

Monitoring

Operator exposing metrics about its health
Operator exposes health and performance metrics about the Operand

Alerting and Events

Operand sends useful alerts
Custom Resources emit custom events

Example: A database Operator continues to parse the logging output of the database software and understands noteworthy log events, e.g. running out of space for database files and produces alerts. The operator also instruments the database and exposes application level, e.g. database queries per second

Guiding questions to determine Operator reaching Level 4

Does your Operator expose a health metrics endpoint?
Does your Operator expose Operand alerts?
Do you have Standard Operating Procedures (SOPs) for each alert?
Does you operator create critical alerts when the service is down and warning alerts for all other alerts?
Does your Operator watch the Operand to create alerts?
Does your Operator emit custom Kubernetes events?
Does your Operator expose Operand performance metrics?

¹ The RED method The RED Method defines the three key metrics for every service in your architecture.

Rate (the number of requests per second)
Errors (the number of those requests that are failing)
Duration (the amount of time those requests take)

Note that by building projects using Operator-SDK or Kubebuilder CLI tools your solution leverages controller-runtime which provides the following reference exported by default. For further information, see the metrics documentation to understand how to enable monitoring and add custom metrics . Also, you may want to give a look at the (grafana/v1-alpha) which provides some JSON manifests to create Grafana dashboards using the default metrics exported.

Level 5 - Auto Pilot

The highest capability level aims to significantly reduce/eliminate any remaining manual intervention in managing the operand. The operator should configure the Operand to auto-scale as load picks up. The Operator should understand the application-level performance indicators and determine when it’s healthy and performing well. The operator should attempt to automatically fix an unhealthy operand. The operator should tune the operands performance, this could include scheduling on another node the pods are running on or modifying operand configuration.

Auto-scaling

Operator scales the Operand up under increased load based on Operand metric
Operator scales the Operand down below a certain load based on Operand metric

Auto-Healing

Operator can automatically heal unhealthy Operands based on Operand metrics/alerts/logs
Operator can prevent the Operand from transitioning into an unhealthy state based on Operand metrics

Auto-tuning

Operator is able to automatically tune the Operand to a certain workload pattern
Operator dynamically shifts workloads onto best suited nodes

Abnormality detection

Operator determines deviations from a standard performance profile

Example: A database operator monitors the query load of the database and automatically scales additional read-only slave replicas up and down. The operator also detects subpar index performance and automatically rebuilds the index in times of reduced load. Further, the operator understands the normal performance profile of the database and creates alerts on excessive amount of slow queries. In the event of slow queries and high disk latency the Operator automatically transitions the database files to another PersistentVolume of a higher performance class.

Guiding questions to determine Operator reaching Level 5

Can your operator read metrics such as requests per second or other relevant metrics and auto-scale horizontally or vertically, i.e., increasing the number of pods or resources used by pods?
Based on question number 1 can it scale down or decrease the number of pods or the total amount of resources used by pods?
Based on the deep insights built upon level 4 capabilities can your operator determine when an operand became unhealthy and take action such as redeploying, changing configurations, restoring backups etc.?
Again considering that with level 4 deep insights the operator has information to learn the performance baseline dynamically and can learn the best configurations for peak performance can it adjust the configurations to do so?
Can it move the workloads to better nodes, storage or networks to do so?
Can it detect and alert when anything is working below the learned performance baseline that can’t be corrected automatically?

Docs: Multi-Tenancy

Mon, 01 Jan 0001 00:00:00 +0000

NetworkPolicy

If your Operator creates or manages NetworkPolicy configurations ensure that your solution:

applies fine-grained network policies to the extent that is required for your managed application to function properly
applies fine-grained network policies to enable your managed application internal components to communicate among each other
allows users to configure your operator so it does not create or manage NetworkPolicy instances
does not create allow traffic from everywhere in the cluster type policies

NetworkPolicies are popular in multi-tenant cluster to provide an extra layer of segregation among tenants within SDN solutions. Users typically customize this extensively with goal of disallowing network traffic among unrelated tenants. Operators that deploy an accept all traffic from anywhere in the cluster style policy, are creating an obstacle in pursuing this goal, especially in instances where these policies cannot be disabled. In security-conscious environments policies like these are not allowed in production. In such cases your operator should minimally have the option to prevent NetworkPolicy objects from being created and leave this responsibility to the user. In more advanced cases, your operator should create NetworkPolicy configurations that follow the least-privilege principle, i.e. denying access to everything and from everywhere by default and only allowing access to specific authorized resources from specific authorized components.

Traffic sharding

The goal is to split or to isolated ingress traffic from certain environments, e.g. production and development environments, ending up on different routers and in this way, being managed by a different Ingress controller. This is a popular configuration option for heavily populated multi-tenant clusters, with several IngressController deployed.

If your Operator creates ingress resources the recommendation is to allow the users to customize them, through the use of a CRD. The required IngressClass needs then to be propagated to the ingress resources created so that they get picked up by the desired IngressController. Annotations are deprecated in favour of IngressClass.

Route resources

To run on the OpenShift distribution of Kubernetes you probably will use the Route API. When sharding these routes may be configured with a label selector. Based on this label selector they will amend their configuration when a route (having the label) is created or not (if the route does not have the label). The label is applied at the Route level and there is no pre-defined convention here, so users set these custom labels in accordance to how they configured their IngressController from operator.openshift.io/v1 instances.

In this way, your operator should allow the user to specify custom labels for any Route that it manages. Check the doc and this blog for an end-to-end examples and further information.

Docs: Release Guide

Mon, 01 Jan 0001 00:00:00 +0000

These steps describe how to conduct a release of the operator-sdk repo using example versions. Replace these versions with the current and new version you are releasing, respectively.

Prerequisites

The following tools and permissions are needed to conduct a release of the operator-sdk repo.

Tools

git: version 2.2+
make: version 4.2+
sed: version 4.3+

Permissions

Must be a Netlify admin
Must be an admin on the operator-sdk repo

Setting Up Tools for MacOS Users

To install the prerequisite tools on MacOS, complete the following steps:

Install GNU sed and make, which may not be installed by default:
- ```
 brew install gnu-sed make
```
Verify that the version of make is higher than 4.2 using command make --version.
Add the gnubin directory for make to your PATH from your ~/.bashrc to allow you to use gmake as make:
- ```
echo 'export PATH="/usr/local/opt/make/libexec/gnubin:$PATH"' >> ~/.bashrc
```
Verify that the version of sed is higher than 4.3 using command gnu-sed --version.
Add the gnubin directory for gnu-sed to your PATH from your ~/.bashrc to allow you to use gnu-sed as sed:
- ```
echo 'export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"' >> ~/.bashrc
```

Major and Minor Releases

We will use the v1.3.0 release version in this example.

Be sure to substitute the version you are releasing into the provided commands.

To perform a major or minor release, you must perform the following actions:

Ensure a new Netlify branch is created
Create a release branch and lock down the master branch
Create and merge a PR for the release branch
Unlock the master branch and push a release tag to it
Perform some clean up actions and announce the new release to the community

Procedure

Before creating a new release branch, it is imperative to perform the following initial setup steps:
1. In the Branches and deploy contexts pane in Netlify, click into the Additional branches list section and add v1.13.x.
  - This will watch the branch when there are changes on Github (creating the branch, or adding a commit).
  - NOTE: You must be a Netlify admin in order to edit the branches list.
Create a release branch by running the following, assuming the upstream SDK repo is the upstream remote on your machine:
- ```
git checkout master
git fetch upstream master
git pull master
git checkout -b v1.3.x
git push upstream v1.3.x
```
Make sure that the list of supported OLM versions is up to date:
1. Identify if a new version of OLM needs to be officially supported by ensuring that the latest three releases listed on the OLM release page are all listed as supported in the Overview section of the SDK docs.
2. If a new version of OLM needs to be added and an old version removed, follow the steps in the updating OLM bindata section before moving onto the next step.
Lock down the master branch to prevent further commits before the release completes:
1. Go to Settings -> Branches in the SDK repo.
2. Under Branch protection rules, click Edit on the master branch rule.
3. In section Protect matching branches of the Rule settings box, increase the number of required approving reviewers to 6.
4. Scroll down to save your changes to protect the master branch.
Create and push a release commit
1. Create a new branch to push the release commit:
  - ```
  export RELEASE_VERSION=v1.3.0
  git checkout master
  git pull master
  git checkout -b release-$RELEASE_VERSION
```
2. Update the top-level Makefile variable IMAGE_VERSION to the upcoming release tag v1.3.0. This variable ensures sample projects have been tagged correctly prior to the release commit.
  - ```
  sed -i -E 's/(IMAGE_VERSION = ).+/\1v1\.3\.0/g' Makefile
```
  If this command fails on MacOS with a warning “sed is not found”, follow the step 5 in the Setting Up Tools for MacOS Users section to map gsed to sed.
3. Run the pre-release make target:
  - ```
  make prerelease
```
The following changes should be present:
- Makefile: IMAGE_VERSION should be modified to the upcoming release tag. (This variable ensures sampleprojects have been tagged correctly prior to the release commit.)
- changelog/generated/v1.3.0.md: commit changes (created by changelog generation).
- changelog/fragments/*: commit deleted fragment files (deleted by changelog generation).
- website/content/en/docs/upgrading-sdk-version/v1.3.0.md: commit changes (created by changelog generation).
- website/config.toml: commit changes (modified by release script).
- testdata/*: Generated sample code.
4. Commit these changes and push to your remote (assuming your remote is named origin):
  - ```
  git add Makefile changelog website testdata
  git commit -sm "Release $RELEASE_VERSION"
  git push origin release-$RELEASE_VERSION
```
Create and merge a new PR for the release-v1.3.0 branch created in step 5.4.
- You can force-merge your PR to the locked-down master if you have admin access to the operator-sdk repo, or ask an administrator to do so.
- Note that the docs PR check will fail because the site isn’t published yet; the PR can be merged anyways.
Unlock the master branch
1. Go to Settings -> Branches in the SDK repo.
2. Under Branch protection rules, click Edit on the master branch rule.
3. In section Protect matching branches of the Rule settings box, reduce the number of required approving reviewers back to 1.
Create and push a release tag on master
1. Refresh your local master branch, tag the release PR commit, and push to the main operator-sdk repo (assumes the remote’s name is upstream):
  - ```
  git checkout master
  git pull master
  make tag
  git push upstream refs/tags/$RELEASE_VERSION
```
Fast-forward the latest and release branches
1. The latest branch points to the latest release tag to keep the main website subdomain up-to-date. Run the following commands to do so:
  - ```
  git checkout latest
  git reset --hard refs/tags/$RELEASE_VERSION
  git push -f upstream latest
```
2. Similarly, to update the release branch, run:
  - ```
  git checkout v1.3.x
  git reset --hard refs/tags/$RELEASE_VERSION
  git push -f upstream v1.3.x
```
Post release steps
1. Publish the new Netlify subdomain for version-specific docs.
  1. Assuming that the Netlify prestep was done before the new branch was created, a new branch option should be visible to Netlify Admins under Domain management > Branch subdomains and can be mapped to a subdomain. (Note: you may have to scroll down to the bottom of the Branch subdomains section to find the branch that is ready to be mapped.)
  2. Please test that this subdomain works by going to the link in a browser. You can use the link in the second column to jump to the docs page for this release.
2. Make an operator-framework Google Group post.
  - You can use this post as an example.
3. Post to Kubernetes slack in #kubernetes-operators and #operator-sdk-dev.
  - You can use this post as an example.
4. Clean up the GitHub milestone
  1. In the GitHub milestone, bump any open issues to the following release.
  2. Close out the milestone.
5. Update the newly unsupported branch documentation (1.1.x in this example)to mark it as archived. (Note that this step does not need to be merged before the release is complete.)
  1. Checkout the newly unsupported release branch:
    - git checkout v1.1.x
  2. Modify the website/config.toml file on lines 88-90 to be the following:
  - ```
  version = "v1.1"
  archived_version = true
  url_latest_version = "https://sdk.operatorframework.io"
```

Patch releases

We will use the v1.3.1 release version in this example.

0. Lock down release branches on GitHub

Lock down the v1.3.x branch to prevent further commits before the release completes:
1. Go to Settings -> Branches in the SDK repo.
2. Under Branch protection rules, click Edit on the v*. branch rule.
3. In section Protect matching branches of the Rule settings box, increase the number of required approving reviewers to 6.

1. Branch

Create a new branch from the release branch (v1.3.x in this example). This branch should already exist prior to cutting a patch release.

export RELEASE_VERSION=v1.3.1
git checkout v1.3.x
git pull
git checkout -b release-$RELEASE_VERSION

2. Prepare the release commit

Using the version for your release as the IMAGE_VERSION, execute the following commands from the root of the project.

# Update the IMAGE_VERSION in the Makefile
sed -i -E 's/(IMAGE_VERSION = ).+/\1v1\.3\.1/g' Makefile
#  Run the pre-release `make` target:
make prerelease

All of the following changes should be present (and no others).

Makefile: IMAGE_VERSION should be modified to the upcoming release tag. (This variable ensures sampleprojects have been tagged correctpy priror to the release commit.)
changelog/: all fragments should be deleted and consolidated into the new file changelog/generated/v1.3.1.md
docs: If there are migration steps, a new migration doc will be created. The installation docs should also contain a link update.
testdata/: Generated samples and tests should have version bumps

Commit these changes and push these changes to your fork:

git add Makefile changelog website testdata
git commit -sm "Release $RELEASE_VERSION"
git push -u origin release-$RELEASE_VERSION

3. Create and merge Pull Request

Create a pull request against the v1.3.x branch.
Once approving review is given, merge. You may have to unlock the branch by setting “required approving reviewers” to back to 1. (See step 0).

4. Create a release tag

Pull down v1.3.x and tag it.

git checkout v1.3.x
git pull upstream v1.3.x
make tag
git push upstream refs/tags/$RELEASE_VERSION

5. Fast-forward the `latest` branch

If the patch release is on the latest y-stream (in the example you would not ff latest if there was a y-stream for v1.4.x), you will need to fast-forward the latest git branch.

(The latest branch points to the latest release tag to keep the main website subdomain up-to-date.)

git checkout latest
git reset --hard tags/$RELEASE_VERSION
git push -f upstream latest

6. Post release steps

Make an operator-framework Google Group post.
Post to Kubernetes slack in #kubernetes-operators and #operator-sdk-dev.
In the GitHub milestone, bump any open issues to the following release.

Note In case there are non-transient errors while building the release job, you must:

Revert the release PR. To do so, create a PR which reverts the patch release PR created in step 3.
Fix what broke in the release branch.
Re-run the release with an incremented minor version to avoid Go module errors (ex. if v1.3.1 broke, then re-run the release as v1.3.2). Patch versions are cheap so this is not a big deal.

`scorecard-test-kuttl` image releases

The quay.io/operator-framework/scorecard-test-kuttl image is released separately from other images because it contains the kudobuilder/kuttl image, which is subject to breaking changes.

Release tags of this image are of the form: scorecard-kuttl/vX.Y.Z, where X.Y.Z is not the current operator-sdk version. For the latest version, query the operator-sdk repo tags for scorecard-kuttl/v.

The only step required is to create and push a tag. This example uses version v2.0.0, the first independent release version of this image:

export RELEASE_VERSION=scorecard-kuttl/v2.0.0
make tag
git push upstream refs/tags/$RELEASE_VERSION

The deploy/image-scorecard-test-kuttl Action workflow will build and push this image.

Helpful Tips and Information

Binaries and Signatures

Binaries will be signed using our CI system’s GPG key. Both binary and signature will be uploaded to the release.

Release Branches

Each minor release has a corresponding release branch of the form vX.Y.x, where X and Y are the major and minor release version numbers and the x is literal. This branch accepts bug fixes according to our backport policy.

Cherry-picking

Once a minor release is complete, bug fixes can be merged into the release branch for the next patch release. Fixes can be added automatically by posting a /cherry-pick v1.3.x comment in the master PR, or manually by running:

git checkout v1.3.x
git checkout -b cherrypick/some-bug
git cherry-pick <commit>
git push upstream cherrypick/some-bug

Create and merge a PR from your branch to v1.3.x.

GitHub Release Information

GitHub releases live under the Releases tab in the operator-sdk repo.

Updating OLM Bindata

Prior to an Operator SDK release, add bindata (if required) for a new OLM version by following these steps:

Add the new version to the OLM_VERSIONS variable in the Makefile.
Remove the lowest version from that variable, as operator-sdk only supports 3 versions at a time.
Run make bindata.
Check that all files were correctly updated by running this script from the root directory of the repository:
- ```
./hack/check-olm.sh
```
  If the check shows that files were missed by the make target, manually edit them to add the new version and remove the obsolete version.
Check that the list of supported OLM versions stated in the Overview section of SDK documentation is updated.
Add the changed files to ensure that they will be committed as part of the release commit:
- ```
git add -u
```

Patch Releases in Parallel

The following should be considered when doing parallel patch releases:

Releasing in order is nice but not worth the inconvenience. Release order affects the order on GitHub releases, and which is labeled “latest release”.
Do not unlock v.* branches while other releases are in progress. Instead, have an admin do the merges.
Release announcements should be consolidated.

Docs: Ansible Based Operator Testing with Molecule

Mon, 01 Jan 0001 00:00:00 +0000

Getting started

Requirements

To begin, you should have:

The latest version of the operator-sdk installed.
Docker installed and running
Molecule >= v3.0
Ansible >= v2.9
The Python Kubernetes client
An initialized Ansible Operator project, with the molecule directory present.

NOTE If you initialized a project with a previous version of operator-sdk, you can generate a new dummy project and copy in the molecule directory. Just be sure to generate the dummy project with the same api-version and kind, or some of the generated files will not work without modification. Your top-level project structure should look like this:

```
.
├── config
├── Dockerfile
├── Makefile
├── molecule
├── playbooks
├── PROJECT
├── requirements.yml
├── roles
└── watches.yaml

```

The Ansible content specified in requirements.yml will also need to be installed. You can install them with ansible-galaxy collection install -r requirements.yml

Molecule scenarios

If you look into the molecule directory, you will see four directories (default, test-local,cluster, templates). The default, test-local, and cluster directories contain a set of files that together make up what is known as a molecule scenario. The templates directory contains Jinja templates that are used by multiple scenarios to configure the Kubernetes cluster.

Our molecule scenarios have the following basic structure:

.
├── molecule.yml
├── prepare.yml
├── converge.yml
└── verify.yml

molecule.yml is a configuration file for molecule. It defines what driver to use to stand up an environment and the associated configuration, linting rules, and a variety of other configuration options. For full documentation on the options available here, see the molecule configuration documentation
prepare.yml is an Ansible playbook that is run once during the set up of a scenario. You can put any arbitrary Ansible in this playbook. It is used for one-time configuration of your test environment, for example, creating the cluster-wide CustomResourceDefinition that your Operator will watch.
converge.yml is an Ansible playbook that contains your core logic for the scenario. In a normal molecule scenario, this would import and run the associated role. For Ansible Operators, we mostly use this to create the Kubernetes resources necessary to deploy your operator into Kubernetes.

Below we will walk through the structure and function of each file for each scenario.

default

The default scenario is intended for use during the development of your Ansible role or playbook, and will run it outside of the context of an operator. You can run this scenario with molecule test or molecule converge. There is no corresponding operator-sdk command for this scenario.

The scenario has the following structure:

molecule/default
├── molecule.yml
├── prepare.yml
├── converge.yml
└── verify.yml

molecule.yml for this scenario tells molecule to use the docker driver to bring up a Kubernetes-in-Docker container, and by default exposes the API on the host’s port 9443. It also specifies a few inventory and environment variables which are used in prepare.yml and converge.yml.
prepare.yml ensures that a kubeconfig properly configured to connect to the Kubernetes-in-Docker cluster exists and is mapped to the proper port, and also waits for the Kubernetes API to become available before allowing testing to begin.
converge.yml imports and runs your role or playbook.
verify.yml is an Ansible playbook where you can put tasks to verify that the state of your cluster matches what you expect.

Configuration

There are a few parameters you can tweak at runtime to change the behavior of your molecule run. You can change these parameters by setting the environment variable before invoking molecule.

The options supported by the default scenario are:

Environment variable	Default	Purpose
KUBE_VERSION	1.17	The Kubernetes version to deploy
TEST_CLUSTER_PORT	9443	The port on the host to expose the Kubernetes API
TEST_OPERATOR_NAMESPACE	osdk-test	The namespace to run your role against

cluster

The cluster scenario runs an end-to-end test of your operator against an existing cluster. The operator image needs to be available to the cluster for this scenario to succeed. This scenario will deploy your CRDs, RBAC, and operator into the cluster, and then creates an instance of your CustomResource and runs your assertions to make sure the Operator responded properly.

You can run this scenario with molecule test or molecule converge. There is no corresponding operator-sdk command for this scenario.

The scenario has the following structure:

molecule/default
├── molecule.yml
├── create.yml
├── prepare.yml
├── converge.yml
├── verify.yml
└── destroy.yml

molecule.yml for this scenario uses the delegated driver, and does not spin up any additional infrastructure.
create.yml is a no-op, but must be present for the delegated driver to work.
prepare.yml ensures the CRD, namespace, and RBAC resources are present in the cluster.
converge.yml creates your operator deployment, based on the template in molecule/templates/operator.yaml.j2.
verify.yml is an Ansible playbook where you can put tasks to verify that the state of your cluster matches what you expect. By default, it creates a Custom Resource and waits for reconciliation to complete successfully. There is an example assertion present as well.
destroy.yml ensures that the namespace, RBAC resources, and CRD are deleted at the end of the run.

Configuration

There are a few parameters you can tweak at runtime to change the behavior of your molecule run. You can change these parameters by setting the environment variable before invoking molecule.

The options supported by the default scenario are:

Environment variable	Default	Purpose
OPERATOR_IMAGE	None	Required The image to use when deploying the operator into the cluster
OPERATOR_PULL_POLICY	Always	The pull policy to use when deploying the operator into the cluster
KUBECONFIG	~/.kube/config	The path to the Kubeconfig for the cluster to test against
TEST_OPERATOR_NAMESPACE	osdk-test	The namespace to run your role against

test-local

The test-local scenario runs a full end-to-end test of your operator that does not require an existing cluster or external registry, and can run in CI environments that allow users to run privileged containers (such as Travis). It brings up a Kubernetes-in-docker cluster, builds your Operator, deploys it into the cluster, and then creates an instance of your CustomResource and runs your assertions to make sure the Operator responded properly. You can run this scenario with molecule test -s local, or with molecule converge -s test-local which will leave the environment up afterward.

The scenario has the following structure:

molecule/test-local
├── molecule.yml
├── prepare.yml
├── converge.yml
└── verify.yml

molecule.yml for this scenario tells molecule to use the docker driver to bring up a Kubernetes-in-Docker container with the project root mounted, and exposes the API on the host’s port 10443. It also specifies a few inventory and environment variables which are used in prepare.yml and converge.yml. It is very similar to the default scenario’s configuration.
prepare.yml first runs the prepare.yml from the default scenario to ensure the kubeconfig is present and the API is up. It then runs the prepare.yml from the cluster scenario to configure your cluster’s CRDs and RBAC.
converge.yml connects to your Kubernetes-in-Docker container, and uses your mounted project root to build your Operator. This makes your Operator available to the cluster without needing to push it to an external registry. Then, it will ensure that a fresh deployment of your Operator is present in the cluster, using the template molecule/templates/operator.yaml.j2.
verify.yml will run the verify.yml from the cluster scenario, as the main difference between the test-local and cluster scenarios is the method of deployment, but not the behavior of the operator.

Configuration

There are a few parameters you can tweak at runtime to change the behavior of your molecule run. You can change these parameters by setting the environment variable before invoking molecule.

The options supported by the default scenario are:

Environment variable	Default	Purpose
KUBE_VERSION	1.17	The Kubernetes version to deploy
TEST_CLUSTER_PORT	10443	The port on the host to expose the Kubernetes API
TEST_OPERATOR_NAMESPACE	osdk-test	The namespace to deploy the operator and associated resources

converge vs test

The two most common molecule commands for testing during development are molecule test and molecule converge. molecule test performs a full loop, bringing a cluster up, preparing it, running your tasks, and tearing it down. molecule converge is more useful for iterative development, as it leaves your environment up between runs. This can cause unexpected problems if you end up corrupting your environment during testing, but running molecule destroy will reset it.

molecule test performs a full loop, bringing a cluster up, preparing it, running your tasks, and tearing it down.
molecule converge is more useful for iterative development, as it leaves your environment up between runs. This can cause unexpected problems if you end up corrupting your environment during testing, but running molecule destroy will reset it.

Writing tests

Adding a task

The default operator that is generated by operator-sdk new doesn’t do anything, so first we will need to add an Ansible task so that the Operator does something we can verify. For this example, we will create a simple ConfigMap with a single key. We’ll be adding the task to roles/example/tasks/main.yml, which should now look like this:

---
# tasks file for exampleapp
- name: create Example configmap
  kubernetes.core.k8s:
    definition:
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: 'test-data'
        namespace: '{{ ansible_operator_meta.namespace }}'
      data:
        hello: world

Adding a test

Now that our Operator actually does some work, we can add a corresponding assert to molecule/cluster/verify.yml. We’ll also add a debug message so that we can see what the ConfigMap looks like. The file should now look like this:

---

- name: Verify
  hosts: localhost
  connection: local
  tasks:
    - debug: var=cm
      vars:
        cm: '{{ lookup("kubernetes.core.k8s", api_version="v1", kind="ConfigMap", namespace=namespace, resource_name="test-data") }}'
    - assert:
        that: cm.data.hello == 'world'
      vars:
        cm: '{{ lookup("kubernetes.core.k8s", api_version="v1", kind="ConfigMap", namespace=namespace, resource_name="test-data") }}'

Now that we have a functional Operator, and an assertion of its behavior, we can verify that everything is working by running molecule test -s local.

The Ansible `assert` and `fail` modules

These modules are handy for adding assertions and failure conditions to your Ansible Operator tests:

assert
fail

Docs: Designing Lean Operators

Mon, 01 Jan 0001 00:00:00 +0000

Overview

One of the pitfalls that many operators are failing into is that they watch resources with high cardinality like secrets possibly in all namespaces. This has a massive impact on the memory used by the controller on big clusters. Such resources can be filtered by label or fields. The original doc design for Filter cache ListWatch using selectors can be accessed from here

IMPORTANT NOTE Requests to a client backed by a filtered cache for objects that do not match the filter will never return anything. In other words, filtered caches make the filtered-out objects invisible to the client.

How is this done ?

When creating the manager, you can add an Options struct to configure the Cache
Each client.Object can be filtered with labels and fields

Examples

In this scenario, the user will configure the Cache to filter the secret object by it’s label. This will return a filtered cache for objects that match the filter.

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
  Cache: cache.Options{
    ByObject: map[client.Object]cache.ByObject{
      &corev1.Secret{}: cache.ByObject{
	Label: labels.SelectorFromSet(labels.Set{"app": "app-name"}),
      },
    },
  },
})

In this scenario, the user will configure the Cache to filter the node object by it’s field name. This will return a filtered cache for objects that match the filter.

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
  Cache: cache.Options{
    ByObject: map[client.Object]cache.ByObject{
      &corev1.Node{}: cache.ByObject{
	Fields: labels.SelectorFromSet(fields.Set{"metadata.name": "node01"}),
      },
    },
  },
})

Docs: Development Tips

Mon, 01 Jan 0001 00:00:00 +0000

This document provides some useful information and tips for a developer creating an operator powered by Ansible.

Getting started with the Kubernetes Collection for Ansible

Since we are interested in using Ansible for the lifecycle management of our application on Kubernetes, it is beneficial for a developer to get a good grasp of the Kubernetes Collection for Ansible. This Ansible collection allows a developer to either leverage their existing Kubernetes resource files (written in YAML) or express the lifecycle management in native Ansible. One of the biggest benefits of using Ansible in conjunction with existing Kubernetes resource files is the ability to use Jinja templating so that you can customize deployments with the simplicity of a few variables in Ansible.

The easiest way to get started is to install the collection on your local machine and test it using a playbook.

Installing the Kubernetes Collection for Ansible

To install the Kubernetes Collection, one must first install Ansible 2.9+. For example, on Fedora/Centos:

sudo dnf install ansible

In addition to Ansible, a user must install the Python Kubernetes Client package:

pip3 install kubernetes

Finally, install the Kubernetes Collection from ansible-galaxy:

ansible-galaxy collection install kubernetes.core

Alternatively, if you’ve already initialized your operator, you may have a requirements.yml file at the top level of your project. This file specifies Ansible dependencies that need to be installed for your operator to function. By default it will install the kubernetes.core collection as well as the operator_sdk.util collection, which provides modules and plugins for operator-specific operations.

To install the dependent modules from this file, run:

ansible-galaxy collection install -r requirements.yml

Testing the Kubernetes Collection locally

Sometimes it is beneficial for a developer to run the Ansible code from their local machine as opposed to running/rebuilding the operator each time. To do this, initialize a new project:

mkdir memcached-operator && cd memcached-operator
operator-sdk init --plugins=ansible --domain=example.com --group=cache --version=v1alpha1 --kind=Memcached --generate-role
ansible-galaxy collection install -r requirements.yml

Modify roles/memcached/tasks/main.yml with desired Ansible logic. For this example we will create and delete a ConfigMap based on the value of a variable named state:

---
- name: set ConfigMap example-config to {{ state }}
  kubernetes.core.k8s:
    api_version: v1
    kind: ConfigMap
    name: example-config
    namespace: default
    state: "{{ state }}"
  ignore_errors: true

Note

Setting ignore_errors: true is done so that deleting a nonexistent ConfigMap doesn’t error out.

Modify roles/memcached/defaults/main.yml to set state to present as default.

---
state: present

Create an Ansible playbook playbook.yml in the top-level directory which includes role memcached:

---
- hosts: localhost
  roles:
    - memcached

Run the playbook:

$ ansible-playbook playbook.yml
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'


PLAY [localhost] ***************************************************************************

TASK [Gathering Facts] *********************************************************************
ok: [localhost]

Task [memcached : set ConfigMap example-config to present]
changed: [localhost]

PLAY RECAP *********************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0

Check that the ConfigMap was created:

$ kubectl get configmaps
NAME                    STATUS    AGE
example-config          Active    3s

Rerun the playbook setting state to absent:

$ ansible-playbook playbook.yml --extra-vars state=absent
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'


PLAY [localhost] ***************************************************************************

TASK [Gathering Facts] *********************************************************************
ok: [localhost]

Task [memcached : set ConfigMap example-config to absent]
changed: [localhost]

PLAY RECAP *********************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0

Check that the ConfigMap was deleted:

$ kubectl get configmaps
No resources found in default namespace.

Using Ansible inside an Operator

Now that we have demonstrated using the Kubernetes Collection, we want to trigger this Ansible logic when a custom resource changes. In the above example, we want to map a role to a specific Kubernetes resource that the operator will watch. This mapping is done in a file called watches.yaml.

Custom Resource file

The Custom Resource (CR) file format is Kubernetes resource file. The object has some mandatory fields:

apiVersion: The version of the Custom Resource that will be created.
kind: The kind of the Custom Resource that will be created
metadata: Kubernetes specific metadata to be created
spec: This is the key-value list of variables which are passed to Ansible. This field is optional and empty by default.
annotations: Kubernetes specific annotations to be appended to the CR. See the below section for Ansible Operator specific annotations. This field is optional.

Annotations for Custom Resource

This is the list of CR annotations which will modify the behavior of the operator:

ansible.operator-sdk/reconcile-period: Specifies the maximum time before a reconciliation is triggered. Note that at scale, this can reduce performance, see watches reference for more information. This value is parsed using the standard Go package time. Specifically ParseDuration is used which will apply the default suffix of s giving the value in seconds.

Example:
```
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: example
  annotations:
    ansible.operator-sdk/reconcile-period: "30s"
```

Note that a lower period will correct entropy more quickly, but reduce responsiveness to change if there are many watched resources. Typically, this option should only be used in advanced use cases where watchDependentResources is set to False and when is not possible to use the watch feature. E.g To managing external resources that don’t raise Kubernetes events.

Testing an Ansible Operator locally

Once a developer is comfortable working with the above workflow, it will be beneficial to test the logic inside an operator.

Prerequisites:

Read the Ansible Operator tutorial.
Install ansible-operator dependencies using pipenv and their OS prerequisite packages (these will differ depending on OS) locally.

The run Makefile target runs the ansible-operator binary locally, which reads from ./watches.yaml and uses ~/.kube/config to communicate with a Kubernetes cluster just as the k8s modules do. The install target registers the operator’s Memcached CustomResourceDefinition (CRD) with the apiserver.

Note

You can customize the roles path by setting the environment variable ANSIBLE_ROLES_PATH or using the flag ansible-roles-path. Note that if the role is not found in ANSIBLE_ROLES_PATH, then the operator will look for it in {{current directory}}/roles.

$ make install run
/home/user/memcached-operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.example.com created
/home/user/go/bin/ansible-operator run
{"level":"info","ts":1595899073.9861593,"logger":"cmd","msg":"Version","Go Version":"go1.13.12","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v0.19.0+git"}
{"level":"info","ts":1595899073.987384,"logger":"cmd","msg":"WATCH_NAMESPACE environment variable not set. Watching all namespaces.","Namespace":""}
{"level":"info","ts":1595899074.9504397,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1595899074.9522583,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_MEMCACHED_CACHE_EXAMPLE_COM","default":2}
{"level":"info","ts":1595899074.9524004,"logger":"cmd","msg":"Environment variable not set; using default value","Namespace":"","envVar":"ANSIBLE_DEBUG_LOGS","ANSIBLE_DEBUG_LOGS":false}
{"level":"info","ts":1595899074.9524298,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"cache.example.com","Options.Version":"v1","Options.Kind":"Memcached"}

Now that the operator is watching resource Memcached for events, the creation of a Custom Resource will trigger our Ansible Role to be executed. Take a look at config/samples/cache_v1alpha1_memcached.yaml:

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: "memcached-sample"

Since spec is not set, Ansible is invoked with no extra variables. The next section covers how extra variables are passed from a Custom Resource to Ansible. This is why it is important to set sane defaults for the operator.

Create a Custom Resource instance of Memcached with variable state default to present:

kubectl create -f config/samples/cache_v1alpha1_memcached.yaml

Check that ConfigMap example-config was created:

$ kubectl get configmaps
NAME                    STATUS    AGE
example-config          Active    3s

Modify config/samples/cache_v1alpha1_memcached.yaml to set state to absent:

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  state: absent

Apply the changes to Kubernetes and confirm that the ConfiMap is deleted:

kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
kubectl get configmaps

Testing an Ansible Operator on a cluster

Now that a developer is confident in the operator logic, testing the operator inside of a pod on a Kubernetes cluster is desired. Running as a pod inside a Kubernetes cluster is preferred for production use.

To build the memcached-operator image and push it to a registry:

make docker-build docker-push IMG=example.com/memcached-operator:v0.0.1

Deploy the memcached-operator:

make install
make deploy IMG=example.com/memcached-operator:v0.0.1

Verify that the memcached-operator is up and running:

$ kubectl get deployment -n memcached-operator-system
NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
memcached-operator       1         1         1            1           1m

Viewing the Ansible logs

In order to see the logs from a particular operator you can run:

kubectl logs deployment/memcached-operator-controller-manager -n memcached-operator-system

The logs contain the information about the Ansible run and are useful for debugging your Ansible tasks. Note that the logs may contain much more detailed information about the Ansible Operator’s internals and its interactions with Kubernetes as well.

Also, you can set the environment variable ANSIBLE_DEBUG_LOGS to True to check the full Ansible result in the logs in order to be able to debug it.

Example

In config/manager/manager.yaml and config/default/manager_metrics_patch.yaml:

...
      containers:
      - name: manager
        env:
        - name: ANSIBLE_DEBUG_LOGS
          value: "True"
...

Occasionally while developing additional debug in the Operator logs is nice to have. Using the memcached operator as an example, we can simply add the "ansible.sdk.operatorframework.io/verbosity" annotation to the Custom Resource with the desired verbosity.

apiVersion: "cache.example.com/v1alpha1"
kind: "Memcached"
metadata:
  name: "example-memcached"
  annotations:
    "ansible.sdk.operatorframework.io/verbosity": "4"
spec:
  size: 4

Custom Resource Status Management

By default, an Ansible Operator will include the generic output from previous Ansible run as the status subresource of a CR. This includes the number of successful and failed tasks and relevant error messages as shown below:

status:
  conditions:
  - ansibleResult:
      changed: 3
      completion: 2018-12-03T13:45:57.13329
      failures: 1
      ok: 6
      skipped: 0
    lastTransitionTime: 2018-12-03T13:45:57Z
    message: 'Status code was -1 and not [200]: Request failed: <urlopen error [Errno
      113] No route to host>'
    reason: Failed
    status: "True"
    type: Failure
  - lastTransitionTime: 2018-12-03T13:46:13Z
    message: Running reconciliation
    reason: Running
    status: "True"
    type: Running

An Ansible Operator also allows you to supply custom status values with the k8s_status Ansible module, which is included in operator_sdk.util collection. You can update the status from within Ansible with any key/value pairs as desired. If you do not want the operator to update the status with Ansible output, and you want to track the CR status manually from your application, you can update the watches.yaml file with manageStatus, as shown below:

- version: v1
  group: api.example.com
  kind: Memcached
  role: memcached
  manageStatus: false

The simplest way to invoke the k8s_status module is to use its fully qualified collection name (fqcn), i.e. operator_sdk.util.k8s_status. The following example updates the status subresource with key memcached and value bar:

- operator_sdk.util.k8s_status:
    api_version: app.example.com/v1
    kind: Memcached
    name: "{{ ansible_operator_meta.name }}"
    namespace: "{{ ansible_operator_meta.namespace }}"
    status:
      foo: bar

Collections can also be declared in the role’s meta/main.yml, which is included for newly scaffolded Ansible operators.

collections:
  - operator_sdk.util

Declaring collections in the role meta allows you to invoke the k8s_status module directly.

- k8s_status:
    <snip>
    status:
      foo: bar

Ansible Operator Conditions

An Ansible Operator has a set of conditions that are used during reconciliation. There are only a few main conditions:

Running - the Ansible Operator is currently running the Ansible for reconciliation.
Successful - if the run has finished and there were no errors, the Ansible Operator will be marked as Successful. It will then wait for the next reconciliation action, either the reconcile period, dependent watches triggers or the resource is updated.
Failed - if there is any error during the reconciliation run, the Ansible Operator will be marked as Failed with the error message from the error that caused this condition. The error message is the raw output from the Ansible run for reconciliation. If the Failure is intermittent, often times the situation can be resolved when the Operator reruns the reconciliation loop.

Extra vars sent to Ansible

The extra vars that are sent to Ansible are managed by the operator. The spec section will pass along the key-value pairs as extra vars. This is equivalent to how above extra vars are passed in to ansible-playbook. The operator also passes along additional variables under the ansible_operator_meta field for the name of the CR and the namespace of the CR.

For the CR example:

apiVersion: "cache.example.com/v1alpha1"
kind: "Memcached"
metadata:
  name: "memcached-sample"
spec:
  message: "Hello world 2"
  newParameter: "newParam"

The structure passed to Ansible as extra vars is:

{ "ansible_operator_meta": {
        "name": "<cr-name>",
        "namespace": "<cr-namespace>",
  },
  "message": "Hello world 2",
  "new_parameter": "newParam",
  "_app_example_com_database": {
     <Full CR>
   },
  "_app_example_com_database_spec": {
     <Full CR .spec>
   },
}

message and newParameter are set in the top level as extra variables, and ansible_operator_meta provides the relevant metadata for the Custom Resource as defined in the operator. The ansible_operator_meta fields can be accessed via dot notation in Ansible as so:

---
- debug:
    msg: "name: {{ ansible_operator_meta.name }}, {{ ansible_operator_meta.namespace }}"

Docs: Managing Resources

Mon, 01 Jan 0001 00:00:00 +0000

Managing Resources

Note that pretty much all controllers consume:

CPU: largely based on the number of reconciliations they perform, which are generally related to event activity for resources they’re watching.
Memory: largely based on the number of primary resources that exist (multiplied by some factor based on the number of operand resources they need to watch as a result) via informer caches.

And then there is a concern that one Pod or Container could monopolize all available resources and Cluster admins must consider the effects that one Pod or Container may have on other components.

In an effort to prevent a container from consuming all the resources on a cluster or affecting other workloads from being scheduled, many production clusters will define ResourceQuota configurations.

The ResourceQuota configuration also applies to tenant workloads that are managed by your Operator. Cluster administrators will typically set a ResourceQuota for each tenant’s namespace as part of the onboarding. If a LimitRange with default values has not been created in each namespace and your Operator creates Containers inside the tenant namespace without specifying at least resource requests for CPU and Memory of its Pods then, the system or quota may reject Pod creation. Check the following statements obtained from K8s docs:

“If a LimitRange is activated in a namespace for computing resources like CPU and memory, users must specify requests or limits for those values. Otherwise, the system may reject Pod creation." (Reference).
“If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation." (Reference).

In an effort to support clusters with the above configuration, to ensure safe operations and avoid negatively impacting other workloads: Operators should always include reasonable memory and CPU resource requests for their own deployment as well as for operands they deploy.

HINT Cluster admins might also able to avoid the above scenario by setting default values when they are not specified for each Pod and/or Container in a namespace.

Therefore, your Operator should always apply at least resource requests for CPU and memory for Pods/Deployments that it creates as part of the reconciliation logic. Ideally your Operator also applies memory limits to those Pods/Deployments. You may also consider CPU limits.

Resource requests and limits for the Operator Deployment can be defined by modifying theconfig/manager/manager.yaml as shown below:

  ...
  # TODO(user): Configure the resources accordingly based on the project requirements.
  # More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
  resources:
    requests:
      cpu: 10m
      memory: 64Mi
    ...

How to compute default values

IMPORTANT: A single configuration that fits all scenarios is not possible. In this way, Operators authors MUST to ensure that Cluster Admins and its users can change the resource requests/limits of the Operator/manager, and of its Operands.

However, you are able to benchmark your Operator by Monitoring the resource usage to ensure good and reasonable values for the general cases. Kubebuilder and SDK provide some metrics which can help you with.

NOTE Also, be aware that if the project was generated by Kubebuilder or SDK scaffold then some values for the Operator/manager(See config/manager/manager.yaml) are populated by default to get you started, however, you ought to optimize them based on your own tests and the specific needs of your operator.

How to change the Operator/manager resources values when under OLM management

If your operator is managed by OLM, administrators or users can configure your operator’s resource requests and limits via the subscription.

General guidelines

Following are some general recommendations to manage the resources:

MUST declare resource requests for both, CPU and Memory, for the Operator/manager and any Pod/Deployment managed by it
OPTIONALLY setting the resources limit for CPU and Memory for the Operator Pod and any Pod/Deployment managed by it.
SHOULD provide the mechanisms for Monitoring compute & memory resource usage so that, Cluster Admins can use these metrics to monitor and resize the Operator and its Operands.CAVEAT: If the Operator is integrated with OLM and the bundle has a PodMonitor or a ServiceMonitor the complete InstallPlan will fail on a cluster, which does not have these CRD/the Prometheus operator installed. In this case, you might want to ensure the dependency requirement with OLM dependency or make clear its requirement for the Operator consumers.
SHOULD allow admins to customize the requests/limits amounts defined for the Pod/Deployment created by the Operator and not hardcode these values.
SHOULD document how your Operator consumer can customize/rightsize the resources requests and limits for the Operator and Operands Pod/Deployments or describe how the solution could be configured to automatically adjust these values based on the environment. the Operator automatically adjust the values to the environment rather than asking its consumers to amend them. You might also consider leveraging the Vertical Pod Autoscaler to have the resources requested by the Operator automatically adjusted to the cluster where it is deployed. You might also look at allowing horizontal pod autoscaling for the other Pod/Deployments created by your Operator. In this case, be aware that resource requests are also required for horizontal pod autoscaling to work.CAVEAT: If you are using OpenShift as your Kubernetes distribution you might want to check the doc Automatically adjust pod resource levels with the vertical pod autoscaler. Also, if the VPA CRD is not available in the cluster where the operator gets deployed the InstallPlan will fail. Please, see how to work with OLM dependency if your project integrating with it.

Why should you set these?

Resource Requests

What happens when the resource requests are not set?

configurations made by the cluster administrators such as ResourceQuota might not work without LimitRanges. The LimitRanger admission controller can provide default values for resource requests and limits when they have not been defined.
the Operators consumers might face resource shortages on a node when resource usage increases, for example, during a daily peak in request rate.
the Operator’s consumers might be unable to successfully deploy the Operator because it does not have the minimal resources available.
the scheduler cannot make an informed placement choice when it picks the nodes the operator pods will be running on.
when there is memory contention on the node the pod is likely to get either evicted or OOM killed.
when there is CPU contention on the node the pod is likely to get starved of CPU cycles making the operator unresponsive.

Resource Limits

What happens when the resource limits are not set?

Wrong configurations or code implementations can consume all resources available, affecting other components on cluster. Also, it might leave the containers more vulnerable such as to Dos Attacks. More info. See that when there is memory contention on a node, pods will start getting evicted and possibly killed according to their OOM score. The node will be flagged with a MemoryPressure condition and eventually made unschedulable. However, when there is CPU contention on the node, neighbour pods may get slowed down to the CPU cores they requested.

However, a popular practice by cluster administrators is to leverage ResourceQuota to limit the total amount of resources that can be requested or allowed in a single namespace. This may protect against over consumption of resources by operands of a faulty or wrongly configured operator. On the other hand it also means that the operator may not be able to create additional pods, limiting its functionality when the limit has been reached.

Also, see might want to check in the K8s docs the following sections:

What happens when:

Limits reached

What happens when the resource limits have been reached?

For Memory: the container might be terminated with the reason OOM Killed. If it is restartable, the kubelet will restart it, as with any other type of runtime failure.
For CPU: the container might or might not be allowed to exceed its CPU limit for extended periods of time. However, it will not be killed for excessive CPU usage. The CPU is considered a “compressible” resource, and if the Pod starts hitting the CPU limits, Kubernetes uses kernel to starts throttling the container. That means the CPU will be artificially restricted, giving a potentially worse performance only.

You might want to check the Troubleshooting section in the Kubernetes documents to better understand how to debug these scenarios.

Limits are specified but not requests

What happens when the resource limits are defined but not the requests?

If you specify a CPU or Memory limit for a Container but do not specify a request, Kubernetes automatically assigns a CPU or Memory request that matches the limit. In this way, you will be requesting always the limit and will be allocating more resources than required. (NOT RECOMMENDED)

Values are too big

Memory and CPU requests and limits are associated with Containers, but be aware that the Memory and CPU requests and limits of a Pod are the sum of its specific computing types for all the containers in the Pod.

If you define that your Pods should have Memory or CPU request too big then, you might not only be allocating and blocking the usage of more than you ought unnecessarily. Also, your Operator consumers might be able to install your Operator via OLM, for example, but will be unable to check the Pods/Deployment running successfully when the amount defined exceeds the capacity available. In these scenarios, the Operator consumers will check that Pod(s) failed to schedule with event errors like Insufficient cpu and/or Insufficient memory.

Docs: Pod Security Standards

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The PodSecurityPolicy API is deprecated and will be removed from Kubernetes in version 1.25. This API is replaced by a new built-in admission controller (KEP-2579: Pod Security Admission Control) which allows cluster admins to enforce Pod Security Standards Labels.

What does that mean?

Namespace and Pod/Container can be defined with three different policies which are; Privileged, Baseline and Restricted. (More info). Therefore, Pod(s)/Container(s) that are not configured according to the enforced security standards defined globally or on the namespace level will not be admitted and it will not be possible to run them.

As a best practice, you must ensure that workloads (Operators and Operands) are defined to run under restricted permissions unless they need further privileges. For the cases where Pod/Container(s) requires escalating permissions, the recommendation is to use the label as described below

How should I configure my Operators and Operands to comply with the criteria?

For common cases that do not require escalating privileges: configure all containers to comply with the restricted policy as shown in the following the examples:

IMPORTANT NOTE The seccompProfile field to define that a container is restricted was introduced with K8s 1.19 and might not be supported on some vendors by default. Please, do not use this field if you are looking to build Operators that work on K8s versions < 1.19 or on vendors that do not support this field. Having this field when it is not supported can result in your Pods/Containers not being allowed to run (i.e. On Openshift versions < 4.11 with its default configuration the deployments will fail with errors like Forbidden: seccomp.)

In Kubernetes manifests:

    spec:
      securityContext:
        # WARNING: Ensure that the image used defines an UserID in the Dockerfile
        # otherwise the Pod will not run and will fail with `container has runAsNonRoot and image has non-numeric user`.
        # If you want your workloads admitted in namespaces enforced with the restricted mode in OpenShift/OKD vendors
        # then, you MUST ensure that the Dockerfile defines a User ID OR you MUST leave the `RunAsNonRoot` and
        # RunAsUser fields empty.
        runAsNonRoot: true
        # Please ensure that you can use SeccompProfile and do not use
        # if your project must work on old Kubernetes
        # versions < 1.19 or on vendors versions which
        # do NOT support this field by default (i.e. Openshift < 4.11 )
        seccompProfile:
          type: RuntimeDefault
      ...
      containers:
      - name: controller-manager
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
        ...

On Reconciliations, such as code implementation in Go:

Note: if you are setting the RunAsNonRoot value to true in the SecurityContext you will need to verify that the Pod or Container(s) are running with a numeric user that is not 0 (root). If the Pod or Container(s) do not use a non-zero numeric user, you can use the RunAsUser value to set the user to a non-zero numeric user. In this example, the memcached container does not use a non-zero numeric user and therefore the RunAsUser value is set to use a uid of 1000.

dep:= &appsv1.Deployment{
  ObjectMeta: metav1.ObjectMeta{
  ….
  },
  Spec: appsv1.DeploymentSpec{
    …
     Template: corev1.PodTemplateSpec{
       ….
        Spec: corev1.PodSpec{
           // Ensure restricted context for the Pod    
           SecurityContext: &corev1.PodSecurityContext{
              RunAsNonRoot: &[]bool{true}[0],
			  // Please ensure that you can use SeccompProfile and do NOT use
			  // this filed if your project must work on old Kubernetes
			  // versions < 1.19 or on vendors versions which 
			  // do NOT support this field by default (i.e. Openshift < 4.11)
              SeccompProfile: &corev1.SeccompProfile{
                 Type: corev1.SeccompProfileTypeRuntimeDefault,
              },
           },
           Containers: []corev1.Container{{
              Image:   "memcached:1.4.36-alpine",
              Name:    "memcached",
              // Ensure restricted context for the container  
              SecurityContext: &corev1.SecurityContext{
				 // WARNING: Ensure that the image used defines an UserID in the Dockerfile
				 // otherwise the Pod will not run and will fail with `container has runAsNonRoot and image has non-numeric user`.
				 // If you want your workloads admitted in namespaces enforced with the restricted mode in OpenShift/OKD vendors 
				 // then, you MUST ensure that the Dockerfile defines a User ID OR you MUST leave the `RunAsNonRoot` and
				 // RunAsUser fields empty. 
                 RunAsNonRoot:  &[]bool{true}[0],
                 AllowPrivilegeEscalation:  &[]bool{false}[0],
                 Capabilities: &corev1.Capabilities{
                    Drop: []corev1.Capability{
                       "ALL",
                    },
                 },
              },
           }},
        },
     },
  },
}

For Ansible and Helm language based Operators: Ansible playbooks or Helm charts MUST create manifests that comply with the requirements in the same way. You can find some examples by looking at the samples under the testdata directory.

For workloads that need elevated permissions: it is recommended that you ensure the namespace containing your solution is labeled accordingly. You can either update your operator to manage the namespace labels or include the namespace labeling as part of the manual install instructions.

It is recommended that you provide a description to help cluster admins understand why elevated permissions are required. You can add this information and the prerequisites to the description of your Operator Bundle (CSV).

Following you will find a detailed description of how to configure and test your solutions. The most straightforward way to ensure if your workloads will work in a restricted namespace is verifying if your solution can run in namespaces enforced as restricted.

NOTE: It is recommended that you test the desired behavior as part of an e2e test suite. Examples of an e2e test for this will be shown in a later section.

How the Operator bundle (CSV) must be configured to apply the standards to the Pod/Containers which are installed by OLM (Operator itself)?

For Operators integrated with OLM, there is an Operator bundle with a CSV where the spec.install.spec.deployments has a Deployment which defines the Pod/Container(s) that will be installed by OLM to get your Operator running on the cluster. In order for the security standards to be followed you will need to ensure the configurations are set correctly.

Note: Ensure the configuration is carried to the Pod/Containers on the bundle CSV after running make bundle. See that the Operator bundle generated with the target is built from the manifests under the config directory. To know more about the layout of your operator built with Operator-SDK see Project Layout.

To check an example of a CSV which complies with the restricted policy, see the Golang sample under the testdata/go/v4/memcached-operator/bundle/manifests/memcached-operator.clusterserviceversion.yaml

How can I verify my manifest?

Using Kind

To verify the policy of your Pod/Container(s) you might to use a Kind cluster as described in the K8s documentation.

Example

First lets create a namespace where we will test the files:

kubectl create ns mytest

Now, let’s label the namespace so that we can check our manifest

 kubectl label --overwrite ns mytest \
   pod-security.kubernetes.io/enforce=restricted \
   pod-security.kubernetes.io/enforce-version=v1.24 \
   pod-security.kubernetes.io/audit=restricted

Now, apply the following Pod which is not restricted on the namespace:

apiVersion: v1
kind: Pod
metadata:
 name: example
 namespace: mytest
spec:
 containers:
  - name: test
    securityContext:
     # see that we are allowing privilege escalation  
     allowPrivilegeEscalation: true 
    image: 'busybox:1.28'
    ports:
     - containerPort: 8080

Then, when we try to apply the Pod manifest above we should see an error:

$ kubectl apply -f mypodtest.yaml 
Error from server (Forbidden): error when creating "mypodtest.yaml": pods "example" is forbidden: violates PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

How do I check what policy (Privileged, Baseline and Restricted) is configured for my Operator and Operand(s)?

An easy way might be using the tool: psachecker. This tool is only able to be used to check locally the Deployments/Pods manifests and not the CSV.

Alternatively, you could test the policy enforcement by labeling the namespaces and running your operator or applying the manifests. More info.

How do I install psachecker?

The following steps will install psachecker in a Golang environment.

git clone git@github.com:stlaz/psachecker.git $GOPATH/src/github.com/stlaz/psachecker
cd $GOPATH/src/github.com/stlaz/psachecker
make build
cp kubectl-psachecker $GOPATH/bin/

How do I use psachecker to verify the policy of a manifests?

The following steps will demonstrate how we can use psachecker to verify that our Operator and deployments are properly configured in the CSV.

Create a new test.yaml file
Add Deployment schema to the file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-manifest
  namespace: test
spec:
...

Now, add the CSV deployment to this test (spec.install.spec.deployments)
Then, you can run the tool and check if its result will be restricted as expected (i.e.):

$ kubectl-psachecker inspect-workloads -f test.yaml
system: restricted

Can I use the metrics to check if my Pod/Containers are violating the PodSecurity policies?

Yes, you can. You need to label the namespaces with pod-security.kubernetes.io/audit: restricted (i.e. kubectl label --overwrite ns --all pod-security.kubernetes.io/enforce-version=v1.24 pod-security.kubernetes.io/audit=restricted). It is important to note that the results may include metrics that do not come from your Operator and Operand(s). If you are looking to use the metrics to do the checks, ensure that you check the results before and after performing the tests, for example:

kubectl get --raw /metrics | prom2json | jq '[.[] | select(.name=="pod_security_evaluations_total") ]'

[
  {
    "name": "pod_security_evaluations_total",
    "help": "[ALPHA] Number of policy evaluations that occurred, not counting ignored or exempt requests.",
    "type": "COUNTER",
    "metrics": [
      {
        "labels": {
          "decision": "allow",
          "mode": "enforce",
          "policy_level": "privileged",
          "policy_version": "latest",
          "request_operation": "create",
          "resource": "pod",
          "subresource": ""
        },
        "value": "29"
      },
      {
        "labels": {
          "decision": "allow",
          "mode": "enforce",
          "policy_level": "privileged",
          "policy_version": "latest",
          "request_operation": "update",
          "resource": "pod",
          "subresource": ""
        },
        "value": "0"
      },
      {
        "labels": {
          "decision": "deny",
          "mode": "audit",
          "policy_level": "restricted",
          "policy_version": "latest",
          "request_operation": "create",
          "resource": "controller",
          "subresource": ""
        },
        "value": "6"
      },
      {
        "labels": {
          "decision": "deny",
          "mode": "audit",
          "policy_level": "restricted",
          "policy_version": "latest",
          "request_operation": "create",
          "resource": "pod",
          "subresource": ""
        },
        "value": "7"
      },
      {
        "labels": {
          "decision": "deny",
          "mode": "warn",
          "policy_level": "restricted",
          "policy_version": "latest",
          "request_operation": "create",
          "resource": "controller",
          "subresource": ""
        },
        "value": "6"
      },
      {
        "labels": {
          "decision": "deny",
          "mode": "warn",
          "policy_level": "restricted",
          "policy_version": "latest",
          "request_operation": "create",
          "resource": "pod",
          "subresource": ""
        },
        "value": "7"
      }
    ]
  }
]

Also, you might be able to create Prometheus alerts such as:

sum (increase(pod_security_evaluations_total{decision="deny",mode="audit"}[1h])) by (policy_level)

(Recommended) How can I automate this check using e2e testing to ensure that my solutions can run under the policies?

In the tests, you can create and label a namespace with the desired policy (i.e. restricted). This would allow you to verify if any warnings/errors occur and if the Pod(s)/Container(s) are in the Running state. It can be something like:

NOTE See the test directory for the Memcached sample under the testdata/go/v3/memcached-operator to see a full example.

// namespace store the ns where the Operator and Operand(s) will be executed
const namespace = "my-operator-system"

var _ = Describe("my operator test", func() {
    BeforeEach(func() {
        ...
        
        // Where we will create the namespace for to install the Operator and Operands
        By("creating namespace")
        cmd := exec.Command("kubectl", "create", "ns", namespace)
        _, _ = utils.Run(cmd)
        
        // We will label as follows all namespaces so that we can check if warnings will be raised when 
		// a manifest be applied. (Not that some namespaces might not acced the label, i.e. if it
		// has a container running with a less restrictive policy )
        By("labeling all namespaces to warn against what can violate the restricted policy")
        cmd = exec.Command("kubectl", "label", "--overwrite", "ns", "--all",
            "pod-security.kubernetes.io/audit=restricted",
            "pod-security.kubernetes.io/enforce-version=v1.24",
            "pod-security.kubernetes.io/warn=restricted")
        _, err := utils.Run(cmd)
        ExpectWithOffset(1, err).NotTo(HaveOccurred())
        
		// We will enforce the restricted policy so that if our Operator
		// or Operand be unable to run as restricted we will be able to check
		// it by validating their status
        By("enforcing restricted to the ns where the Operator/Operand(s) will be checked")
        cmd = exec.Command("kubectl", "label", "--overwrite", "ns", namespace,
            "pod-security.kubernetes.io/audit=restricted",
            "pod-security.kubernetes.io/enforce-version=v1.24",
            "pod-security.kubernetes.io/enforce=restricted")
        _, err = utils.Run(cmd)
        Expect(err).To(Not(HaveOccurred()))
        })
    })
    
    AfterEach(func() {
        ...
    })

    It("should successfully run the Operator and Operand(s)", func() {
        // Then, here we build the operator and deploy the
        // manager as the operand in the namespaces were 
        // the policy restricted was enforced.
		
        // Therefore, we can check the Operator and Operand
        // status to ensure that all is Running.
		
        // Note that we can also verify if warns like with
        // the message Warning: would violate PodSecurity were 
        // returned when the manifest were applied on the cluster
    })
})

After following the recommendations to be restricted my workload is not running (CreateContainerConfigError). What should I do?

If you are encountering errors similar to Error: container has runAsNonRoot and image has non-numeric user or container has runAsNonRoot and image will run as root that means that the image used does not have a non-zero numeric user defined, i.e.:

USER 65532:65532 
OR
USER 1001

Due to the RunAsNonRoot field being set to true, we need to force the user in the container to a non-zero numeric user. It is recommended that the images used by your operator have a non-zero numeric user set in the image itself (similar to the example above). For further information check the note Consider an explicit UID/GID in the Dockerfile best practices guide. If your Operator will be distributed and used in vanilla Kubernetes clusters you can also fix the issue by defining the user via the security context configuration). (i.e. RunAsUser: &[]int64{1000}[0],).

NOTE If your Operator should work with specific vendors please ensure that you check if they have specific rules for Pod Security Admission. For example, we know that if you use RunAsUser on OpenShift it will disqualify the Pod from their restricted-v2 SCC. Therefore, if you want your workloads running in namespaces labeled to enforce restricted you must leave RunAsUser and RunAsNonRoot fields empty or if you want set RunAsNonRoot then, you MUST ensure that the image itself properly defines the UserID.

Docs: Reporting Issues

Mon, 01 Jan 0001 00:00:00 +0000

If any part of the operator-sdk project has bugs or documentation mistakes, please let us know by opening an issue. We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue reporting the same problem does not already exist.

To make the bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc.
Reproducible. Include the steps to reproduce the problem. We understand some issues might be hard to reproduce, please include the steps that might lead to the problem.
Isolated. Please try to isolate and reproduce the bug with minimum dependencies. It would significantly slow down the speed to fix a bug if too many dependencies are involved in a bug report. Debugging external systems that rely on operator-sdk is out of scope, but we are happy to provide guidance in the right direction or help with using operator-sdk itself.
Unique. Do not duplicate existing bug report.
Scoped. One bug per report. Do not follow up with another bug inside one report.

It may be worthwhile to read Elika Etemad’s article on filing good bug reports before creating a bug report.

We might ask for further information to locate a bug. A duplicated bug report will be closed.

Docs: Operator Observability Best Practices

Mon, 01 Jan 0001 00:00:00 +0000

Operator Observability Best Practices

In this document, we provide best practices and examples for creating metrics, recording rules and alerts. It is based on the general guidelines in Operator Capability Levels.

Note: For technical documentation of how to add metrics to your operator, please read the Metrics section of the Kubebuilder documentation.

Operator Observability Recommended Components

Health and Performance metrics - for all of the operator components
1.1. Metrics should be implemented based on the guidelines below.
1.2. Metrics Documentation - All metrics should have documentation.
1.3. Metrics Tests - Metrics should include tests that verify that they exist and that their value is correct.
Alerts for when things are not working as expected for each of the operator’s components
2.1 Alerts should be implemented based on the guidelines below.
2.2. Alerts Runbooks - Each alert can include a runbook_url annotation and an alert runbook that describes it. See additional details below.
2.3. Alerts Tests - Alerts should include E2E Testing and unit tests.
Events - Custom Resources should emit custom events for the operations taking place.

Additional components would be Dashboards, Logs and Traces, which are not addressed in this document at this point.

Operators Observability General Guidelines

Important: It is highly recommended to separate your monitoring code from your core operator code.
We recommend to create a dedicated /monitoring subfolder that will include all the code of the Operator Observability Recommended Components, that are outlined above. For example, in the memcached-operator.

In your core operator code only call the functions that will update the metrics value from your desired location. For example, in the memcached-operator.

All operators start small. This separation will help you, as a developer, with easier maintenance of both your operator core code and the monitoring code and for other stakeholders to understand your monitoring code better.

Metrics Guidelines

Metrics Naming

Kubernetes components emit metrics in Prometheus format. This format is structured plain text, designed so that people and machines can both read it.

Your operator users should get the same experience when searching for a metric across Kubernetes operators, resources and custom resources.

Check if a similar Kubernetes metric, for node, container or pod, exists and try to align to it.
The metrics search list, in the Prometheus, Grafana UI and even in the /metrics end point, is sorted in alphabetical order. When searching for a metric, it should be easy to identify metrics that are related to a specific operator. That is why we recommend that your operator metrics name will follow this format: operator name prefix + the sub-operator name or entity + metric name based on the Prometheus naming conventions. For example, in the memcached-operator.

Note: In Prometheus Node Exporter metrics are separated like this:

node_network_**receive**_packets_total
node_network_**transmit**_packets_total

In this example, based on receive and transmit.

Please follow the same principle and don’t put similar metrics details as labels, so the user experience would be fluent.
Example for this in an operator:

kubevirt_vmi_network_**receive**_errors_total
kubevirt_vmi_network_**transmit**_bytes_total
kubevirt_migrate_vmi_**data_processed**_bytes
kubevirt_migrate_vmi_**data_remaining**_bytes

Your metric suffix should indicate the metric unit. For better compatibility, Prometheus base units should be used.
Prometheus supports four metric types. Gauge,Counter,Histogram and Summary. You can read more about the different types here, Understanding metrics types.
The most common types are:

Counter - Value can only increase or reset.
Gauge Value can be increased and decreased as needed.

_total suffix should be used for accumulating count. If your metrics has labels with high cardinality, like pod/container it usually means that you can aggregate it more, thus it will not require _total suffix.

Prometheus Labels

Prometheus labels are used to differentiate the characteristics of the thing that is being measured.

Important - Be cautious when adding labels to metrics. Labels can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values. Note: There are still cases when we will still need to have a high cardinality label like the pod name, but try to keep this to the minimum.
When creating a new metric, recording rule or alert, that reports a resource like a pod or a container, make sure that the namespace is included, in order to be able to uniquely identify it.

Metrics `Help` message

Your operator metrics help message should include the following details:

What does this metric measure?
What does the output mean?
What important labels does the metric use? (Optional. If applicable).

The Help message can be used to create auto-generated documentation, like it’s done in KubeVirt and generated by the KubeVirt metrics doc generator.

We recommend to auto-generated metrics documentation and save it in your operator repository, to a location like /docs/monitoring/, so that the users can find the information about your operator metrics easily.

See Alerts, Metrics and Recording Rules Tests section for metrics testing recommendations.

Prometheus Recording Rules Naming

As per Prometheus documentation, Recording rules allow you to pre-compute frequently needed or computationally expensive expressions and save their result as a new set of time series.

Note: The Prometheus recording rules appear in Prometheus UI as metrics. Recording rule names should follow the level:metric:operations format as specified in the Prometheus recording rules best practices. This naming convention makes it clear that the metric is a recording rule and helps consumers understand they need to examine the underlying query to fully understand what the metric provides.

level: represents the aggregation level and labels of the rule output
metric: is the metric name
operations: is a list of operations that were applied to the metric, newest operation first

For example: job:up:avg_over_time or instance:node_cpu_utilisation:rate5m

In addition to this format, your operator recording rules should also follow the same naming guidelines as metrics for consistency within your operator’s observability stack.

See Alerts, Metrics and Recording Rules Tests section for recording rules testing recommendations.

Prometheus Alerts Guidelines

Clear and actionable alerts are a key component of a smooth operational experience and will result in a better experience for the end users.

The following guidances aim to align alert naming, severities, labels, etc., in order to avoid alerts fatigue for administrators.

Alert Ownership

Individual operator authors are responsible for writing and maintaining alerting rules for their components, i.e. their operators and operands.

Operator authors should also take into consideration how their components interact with existing monitoring and alerting.

As an example, if your operator deploys a service which creates one or more PersistentVolume resources, and these volumes are expected to be mostly full as part of normal operation, it’s likely that this will cause unnecessary KubePersistentVolumeFillingUp alerts to fire.

You should work to find a solution to avoid triggering these alerts if they are not actionable.

Alerts Style Guide

Alert names MUST be CamelCase, e.g.: PrometheusRuleFailures
Alert names SHOULD be prefixed with a component, e.g.: AlertmanagerFailedReload
- There may be exceptions for some broadly scoped alerts, e.g.: TargetDown
Alerts MUST include a severity label indicating the alert’s urgency.
- Valid severities are: critical, warning, or info — see below for guidelines on writing alerts of each severity.
Alerts MUST include summary and description annotations.
- Think of summary as the first line of a commit message, or an email subject line. It should be brief but informative. The description is the longer, more detailed explanation of the alert.
Alerts SHOULD include a namespace label indicating the source of the alert.
- Many alerts will include this by virtue of the fact that their PromQL expressions result in a namespace label. Others may require a static namespace label — see for example, the KubeCPUOvercommit alert.

Optional Alerts Labels and Annotations

priority label indicating the alert’s level of importance and the order in which it should be fixed.
- Valid priorities are: high, medium, or low. Higher the priority the sooner the alert should be resolved.
- If the alert doesn’t include a priority label, we can assume it is a medium priority alert. This label will usually be used for alerts with warning severity, to indicate the order in which the alert should be addressed by, even though it doesn’t require immediate action.
runbook_url annotation is a link to an alert runbook which is intended to guide a cluster owner and/or operator through the steps of fixing problems on clusters, which are surfaced by alerts.
- An example for Runbook style documentation.
- Your operator alert runbooks can be saved at your operator repository, to /docs/monitoring/runbooks/ for example, at OpenShift Runbooks if your operator is shipped with OpenShift or another location that fits your operator.
- If you are using Github, you can use Github Pages for a better view of the runbooks.
kubernetes_operator_part_of label indicating the operator name. Label name is based on the Kubernetes Recommended Labels.

Alerts Severity

Critical Alerts

For alerting current and impending disaster situations. These alerts page an SRE. The situation should warrant waking someone in the middle of the night.

Timeline: ~5 minutes.

Reserve critical level alerts only for reporting conditions that may lead to loss of data or inability to deliver service for the cluster as a whole.
Failures of most individual components should not trigger critical level alerts, unless they would result in either of those conditions.
Configure critical level alerts so they fire before the situation becomes irrecoverable.
Expect users to be notified of a critical alert within a short period of time after it fires so they can respond with corrective action quickly.

Example critical alert: KubeAPIDown

- alert: KubeAPIDown
  annotations:
    summary: Target disappeared from Prometheus target discovery.
    description: KubeAPI has disappeared from Prometheus target discovery.
    runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeAPIDown.md
  expr: |
    absent(up{job="apiserver"} == 1)
  for: 15m
  labels:
    severity: critical

This alert fires if no Kubernetes API server instance has reported metrics successfully in the last 15 minutes.
This is a clear example of a critical control-plane issue that represents a threat to the operability of the cluster as a whole, and likely warrants paging someone.
The alert has clear summary and description annotations, and it links to a runbook with information on investigating and resolving the issue.

The group of critical alerts should be small, very well defined, highly documented, polished and with a high bar set for entry.

Warning Alerts

The vast majority of alerts should use this severity.
Issues at the warning level should be addressed in a timely manner, but don’t pose an immediate threat to the operation of the cluster as a whole.

Timeline: ~60 minutes

If your alert does not meet the criteria in “Critical Alerts” above, it belongs to the warning level or lower.

Use warning level alerts for reporting conditions that may lead to inability to deliver individual features of the cluster, but not service for the cluster as a whole. Most alerts are likely to be warnings.
Configure warning level alerts so that they do not fire until components have sufficient time to try to recover from the interruption automatically.
Expect users to be notified of a warning, but for them not to respond with corrective action immediately.

Example warning alert: ClusterNotUpgradeable

- alert: ClusterNotUpgradeable
  annotations:
    summary: One or more cluster operators have been blocking minor version cluster upgrades for at least an hour.
    description: In most cases, you will still be able to apply patch releases.
      Reason {{ "{{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}}"}}
      For more information refer to 'oc adm upgrade'{{ "{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}" }}.
    expr: |
      max by (name, condition, endpoint) (cluster_operator_conditions{name="version", condition="Upgradeable", endpoint="metrics"} == 0)
    for: 60m
    labels:
      severity: warning

This alert fires if one or more operators have not reported their Upgradeable condition as true in more than an hour.
The alert has a clear name and informative summary and description annotations.
The timeline is appropriate for allowing the operator a chance to resolve the issue automatically, avoiding the need to alert an administrator.

Info Alerts

Info level alerts represent situations an administrator should be aware of, but they don’t necessarily require any action.
Use these sparingly, and consider instead reporting this information via Kubernetes events.

Example info alert: MultipleContainersOOMKilled

- alert: MultipleContainersOOMKilled
  annotations:
    description: Multiple containers were out of memory killed within the past
      15 minutes. There are many potential causes of OOM errors, however issues
      on a specific node or containers breaching their limits is common.
      summary: Containers are being killed due to OOM
  expr: sum(max by(namespace, container, pod) (increase(kube_pod_container_status_restarts_total[12m]))
    and max by(namespace, container, pod) (kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}) == 1) > 5
  for: 15m
  labels:
    namespace: kube-system
    severity: info

This alert fires if multiple containers have been terminated due to out of memory conditions in the last 15 minutes.
This is something the administrator should be aware of, but may not require immediate action.

Alerts, Metrics and Recording Rules Tests

Add tests for alerts that validate that:
- Each alert includes all mandatory fields.
- Each runbook_url link is valid.
- Each alert that includes a pod or a container also includes the namespace.
Add e2e tests that inspect the alerts during upgrade and make sure that the alerts don’t fire when they shouldn’t (Zero noise).
Add tests for metrics and recording rules that validate that:
- Metric / Recording rule exists.
- Metric / Recording rule value is as expected.
- Metric / Recording rule name follows the best practices guideline.

Docs: Testing

Mon, 01 Jan 0001 00:00:00 +0000

On all PRs, a suite of static and cluster tests is run against your changes in a CI environment. These tests can also be run locally, which is discussed below.

Static tests consist of unit, formatting, and doc link tests.

Cluster tests consist of several test types:

End-to-end (e2e): simulate the “happy path” usage of the operator-sdk binary and resulting operator project.
Integration: test components of the operator-sdk binary and features of scaffolded projects that are bound to external projects, such as OLM.
Subcommand: ensure individual subcommands function as intended with a variety of input options.

Before submitting a PR

Always run tests before submitting a PR to reduce the number of needless CI errors.

Docs only

make test-static

Code

make test-all

Local Test Environment

If running tests locally, access to a Kubernetes cluster of a compatible version is required. These tests require KUBECONFIG be set or kubeconfig file be present in a default location like $HOME/.kube/config.

You will also need to set up an envtest environment for cluster tests. Follow this doc for setup instructions.

Local clusters

A local kind cluster is used for running tests.

Running Tests

All the tests are run through the Makefile. Run make help for a full list of available tests.

Docs: Plugins

Mon, 01 Jan 0001 00:00:00 +0000

Overview

SDK uses the project Kubebuilder as a library for the CLI and plugins features. For further information see the Plugins document. To better understand its motivations, check the design documentation Integrating Kubebuilder and Operator SDK.

SDK Language-based plugins

All language-based operator projects provided by SDK (Ansible/Helm/Golang) follow the Kubebuilder standard and have a common base. The common base is generated using kustomize which is a project maintained and adopted by Kubernetes community to help work with the Kubernertes manifests (YAML files) that are used to configure the resources on the clusters.

The specific files for each language are scaffolded for the language’s plugins. The Operator SDK CLI tool can provide custom plugins and helpers which are common and useful for all language types. To check the common base, you can also run:

operator-sdk init --plugins=kustomize

Also, see the topic External Plugins to understand how it works.

Common scaffolds

Following the default common scaffolds for the projects which are built with SDK.

File/Directory	Description
Dockerfile	Defines the operator(manager) image
Makefile	Provides the helpers and options for the users. (e.g. `make bundle` which generate/update the OLM bundle manifests )
PROJECT	Project configuration. Stores the data used to do the scaffolds. For further information see Project Config
bundle.Dockerfile	Docker image which is used to provide the helpers to integrate the project with OLM. (e.g. operator-sdk run ./bundle)
config/	Directory which has all kustomize’s manifest to configure and test the project

You can check the Project Layout to better understand the files and directories scaffolded by SDK CLI, which may be common for each language-type.

Custom Plugins

By default plugins are used by the Operator SDK to provide the following features:

manifests.sdk.operatorframework.io: perform the required scaffolds to provide the helpers to allow the projects to be integrated with OLM.
scorecard.sdk.operatorframework.io: perform the required scaffolds to provide the Scorecard feature.

Optional/custom plugins

Users can also use custom plugins when they execute the SDK CLI sub-commands as a helper which includes the following options:

operator-sdk create api --plugins="go/v3,declarative"

The above example will scaffold custom code in the controllers after an API is created to allow the users to develop solutions using the kubebuilder declarative pattern. (e.g default scaffold versus example).

Note that custom plugins are called via the init sub-command to work as global plugins, and will be added in the layout field of the PROJECT file. Any sub-command executed will then also be called.

Plugins Vision

Contributors are able to create their own plugin(s) using the same standards and approach described by this document. Following them facilitates in-tree (in other words, built with the operator-sdk binary) addition of such community-driven plugins to the Operator SDK project by making review easier and code maintainable. Currently, the SDK cannot discover and use plugins that are not in-tree. However, out-of-tree plugins have been discussed in kubebuilder/issues/1378.

How to create your own plugins

If you are looking to develop similar solutions to allow users for example to create projects using other languages or that could work as helpers for the projects built with SDK/Kubebuilder projects, then see the Creating your own plugins to see how you can benefit and apply this approach.

Docs: OLM and Bundle CLI Overview

Mon, 01 Jan 0001 00:00:00 +0000

This document gives an overview of using operator-sdk to work with Operator manifests related to OLM, namely bundles and package manifests. See the manifests generation doc for an in-depth discussion of these commands.

OLM installation

The following operator-sdk subcommands manage an OLM installation:

olm install: install a particular version of OLM.
olm status: check the status of a particular version of OLM running in a cluster. This command can infer the version of an error-free OLM installation.
olm uninstall: uninstall a particular version of OLM running in a cluster. This command can infer the version of an error-free OLM installation.

Manifests and metadata

The following make recipes and operator-sdk subcommands create or interact with Operator package manifests and bundles:

generate kustomize manifests: creates kustomize bases and a kustomization.yaml in config/manifests.

Bundles

make bundle: runs the following commands:
- generate kustomize manifests: see above.
- generate bundle: creates a new or updates an existing bundle in the <project-root>/bundle directory. This command generates both manifests and metadata.
- bundle validate: validates an Operator bundle image or unpacked manifests and metadata.
make bundle-build: builds a bundle image using the bundle.Dockerfile generated by make bundle.
run bundle: runs the given Operator’s bundle image with an existing OLM installation.
run bundle-upgrade: upgrades the Operator bundle to a specified newer version.

Private bundle and catalog image registries

By default, projects are configured to push to and pull from a docker.io registry with namespace being the value passed to operator-sdk init --domain=<value>. Modify this value to push/pull bundle and catalog images with a different registry host or namespace.

All bundle and catalog image-related commands invoke opm (except for bundle image builds, for which docker is used directly). opm leverages the host’s image build/pull tools indirectly to perform various image tasks, so if your image registry is private or has a custom CA you must ensure the in-use build tool is able to push to/pull from the registry:

docker: config.json, certificate configuration
podman: auth.json, certificate configuration
none (containerd): uses docker’s config.json, certificate configuration

The run bundle or run bundle-upgrade commands use the none image tool, described above, in-cluster. These commands accept the names of secrets available in the deployment namespace that contain configuration file data. Ideally a cluster admin will provision a namespace and service account for bundle testing, such that they include and reference these secrets:

Create an image pull secret for your config.json and add it to your service account.

Create a generic secret with a cert.pem key containing root certificate(s) for your registry.

Once the above secrets have been created, run the either command with --pull-secret-name=<image pull secret> and --ca-secret-name=<certificate secret>:

operator-sdk run bundle private-custom-ca-reg.com/memcached-operator-bundle:v0.0.2 \
    --index-image private-custom-ca-reg.com/memcached-operator-catalog:v0.0.1 \
    --pull-secret-name foo-pull-sec \
    --ca-secret-name foo-cert-sec

You may have to set --namespace=<provisioned namespace> if the namespace encoded in your kubeconfig’s current context was not provisioned with these secrets.

Package Manifests

The operator-framework is removing support for the packagemanifests format in the near future, and migration efforts are currently underway. Therefore generate packagemanifests and run packagemanifests commands have been deprecated. For more details on bundle format refer here. To migrate from packagemanifests to the bundle format, use the operator-sdk pkgman-to-bundle command.

Docs: Controller Runtime Client API

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The controller-runtime library provides various abstractions to watch and reconcile resources in a Kubernetes cluster via CRUD (Create, Update, Delete, as well as Get and List in this case) operations. Operators use at least one controller to perform a coherent set of tasks within a cluster, usually through a combination of CRUD operations. The Operator SDK uses controller-runtime’s Client interface, which provides the interface for these operations.

controller-runtime defines several interfaces used for cluster interaction:

client.Client: implementers perform CRUD operations on a Kubernetes cluster.
manager.Manager: manages shared dependencies, such as Caches and Clients.
reconcile.Reconciler: compares provided state with actual cluster state and updates the cluster on finding state differences using a Client.

Clients are the focus of this document. A separate document will discuss Managers.

Note: this document uses parts of the sample memcached-operator for example code. Import paths may be different for brevity.

Client Usage

Default Client

The SDK relies on a manager.Manager to create a client.Client interface that performs Create, Update, Delete, Get, and List operations within a reconcile.Reconciler's Reconcile function. The SDK will generate code to create a Manager, which holds a Cache and a Client to be used in CRUD operations and communicate with the API server. By default a Controller’s Reconciler will be populated with the Manager’s Client which is a split-client.

The following code, found in controllers/memcached_controller.go, demonstrates how the Manager’s client is passed to a reconciler.

import (
	appsv1 "k8s.io/api/apps/v1"
	ctrl "sigs.k8s.io/controller-runtime"

	cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
)

func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr). // mgr's Client is passed to r.
		For(&cachev1alpha1.Memcached{}).
		Owns(&appsv1.Deployment{}).
		Complete(r)
}

type MemcachedReconciler struct {
    client.Client // Populated above from a manager.Manager.

    Log    logr.Logger
    Scheme *runtime.Scheme
}

A split client reads (Get and List) from the Cache and writes (Create, Update, Delete) to the API server. Reading from the Cache significantly reduces request load on the API server; as long as the Cache is updated by the API server, read operations are eventually consistent.

Non-default Client

An operator developer may wish to create their own Client that serves read requests(Get List) from the API server instead of the cache, for example. controller-runtime provides a constructor for Clients:

// New returns a new Client using the provided config and Options.
func New(config *rest.Config, options client.Options) (client.Client, error)

client.Options allow the caller to specify how the new Client should communicate with the API server.

// Options are creation options for a Client
type Options struct {
    // Scheme, if provided, will be used to map go structs to GroupVersionKinds
    Scheme *runtime.Scheme

    // Mapper, if provided, will be used to map GroupVersionKinds to Resources
    Mapper meta.RESTMapper
}

Example:

import (
    "sigs.k8s.io/controller-runtime/pkg/client/config"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

cfg, err := config.GetConfig()
...
c, err := client.New(cfg, client.Options{})
...

Note: defaults are set by client.New when Options are empty. The default scheme will have the core Kubernetes resource types registered. The caller must set a scheme that has custom operator types registered for the new Client to recognize these types.

Creating a new Client is not usually necessary nor advised, as the default Client is sufficient for most use cases.

Reconcile and the Client API

A Reconciler implements the reconcile.Reconciler interface, which exposes the Reconcile method. Reconcilers are added to a corresponding Controller for a Kind; Reconcile is called in response to cluster or external Events, with a reconcile.Request object argument, to read and write cluster state by the Controller, and returns a ctrl.Result. SDK Reconcilers have access to a Client in order to make Kubernetes API calls.

// MemcachedReconciler reconciles a Memcached object
type MemcachedReconciler struct {
    // client, initialized using mgr.Client() above, is a split client
    // that reads objects from the cache and writes to the apiserver
    client.Client

    Log    logr.Logger

    // scheme defines methods for serializing and deserializing API objects,
    // a type registry for converting group, version, and kind information
    // to and from Go schemas, and mappings between Go schemas of different
    // versions. A scheme is the foundation for a versioned API and versioned
    // configuration over time.
    Scheme *runtime.Scheme
}

// Reconcile watches for Events and reconciles cluster state with desired
// state defined in the method body.
// The Controller will requeue the Request to be processed again if an error
// is non-nil or Result.Requeue is true, otherwise upon completion it will
// remove the work from the queue.
func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile is where Controller business logic lives, i.e. where Client API calls are made via MemcachedReconciler.client. A client.Client implementer performs the following operations:

Get

// Get retrieves an API object for a given object key from the Kubernetes cluster
// and stores it in obj.
func (c Client) Get(ctx context.Context, key client.ObjectKey, obj client.Object) error

Note: A client.ObjectKey is simply an alias for types.NamespacedName.

Example:

import (
    "context"

    ctrl "sigs.k8s.io/controller-runtime"

    cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    memcached := &cachev1alpha1.Memcached{}
    err := r.Get(ctx, request.NamespacedName, memcached)

    ...
}

List

// List retrieves a list of objects for a given namespace and list options
// and stores the list in obj.
func (c Client) List(ctx context.Context, list client.Object, opts ...client.ListOption) error

A client.ListOption is an interface that sets client.ListOptions fields. A client.ListOption is created by using one of the provided implementations: MatchingLabels, MatchingFields, InNamespace.

Example:

import (
    "context"
    "fmt"

    "k8s.io/api/core/v1"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    // Return all pods in the request namespace with a label of `instance=<name>`
    // and phase `Running`.
    podList := &v1.PodList{}
    opts := []client.ListOption{
        client.InNamespace(request.NamespacedName.Namespace),
        client.MatchingLabels{"instance": request.NamespacedName.Name},
        client.MatchingFields{"status.phase": "Running"},
    }
    err := r.List(ctx, podList, opts...)

    ...
}

Create

// Create saves the object obj in the Kubernetes cluster.
// Returns an error
func (c Client) Create(ctx context.Context, obj client.Object, opts ...client.CreateOption) error

A client.CreateOption is an interface that sets client.CreateOptions fields. A client.CreateOption is created by using one of the provided implementations: DryRunAll, ForceOwnership. Generally these options are not needed.

Example:

import (
    "context"

    "k8s.io/api/apps/v1"
    ctrl "sigs.k8s.io/controller-runtime"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    dep := &v1.Deployment{ // Any cluster object you want to create.
        ...
    }
    err := r.Create(ctx, dep)

    ...
}

Update

// Update updates the given obj in the Kubernetes cluster. obj must be a
// struct pointer so that obj can be updated with the content returned
// by the API server. Update does *not* update the resource's status
// subresource
func (c Client) Update(ctx context.Context, obj client.Object, opts ...client.UpdateOption) error

A client.UpdateOption is an interface that sets client.UpdateOptions fields. A client.UpdateOption is created by using one of the provided implementations: DryRunAll, ForceOwnership. Generally these options are not needed.

Example:

import (
    "context"

    "k8s.io/api/apps/v1"
    ctrl "sigs.k8s.io/controller-runtime"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    dep := &v1.Deployment{}
    err := r.Get(ctx, request.NamespacedName, dep)

    ...

    dep.Spec.Selector.MatchLabels["is_running"] = "true"
    err := r.Update(ctx, dep)

    ...
}

Patch

// Patch patches the given obj in the Kubernetes cluster. obj must be a
// struct pointer so that obj can be updated with the content returned by the Server.
func (c Client) Patch(ctx context.Context, obj client.Object, patch client.Patch, opts ...client.PatchOption) error

A client.PatchOption is an interface that sets client.PatchOptions fields. A client.PatchOption is created by using one of the provided implementations: DryRunAll, ForceOwnership. Generally these options are not needed.

Example:

import (
    "context"

    "k8s.io/api/apps/v1"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    dep := &v1.Deployment{}
    err := r.Get(ctx, request.NamespacedName, dep)

    ...

    // A merge patch will preserve other fields modified at runtime.
    patch := client.MergeFrom(dep.DeepCopy())
    dep.Spec.Selector.MatchLabels["is_running"] = "true"
    err := r.Patch(ctx, dep, patch)

    ...
}

Updating Status Subresource

When updating the status subresource from the client, the StatusWriter must be used. The status subresource is retrieved with Status() and updated with Update() or patched with Patch().

Update() takes variadic client.UpdateOption's, and Patch() takes variadic client.PatchOption's. See Client.Update() and Client.Patch() for more details. Generally these options are not needed.

Status

// Status() returns a StatusWriter object that can be used to update the
// object's status subresource
func (c Client) Status() (client.StatusWriter, error)

Example:

import (
    "context"

    ctrl "sigs.k8s.io/controller-runtime"

    cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    mem := &cachev1alpha1.Memcached{}
    err := r.Get(ctx, request.NamespacedName, mem)

    ...

    // Update
    mem.Status.Nodes = []string{"pod1", "pod2"}
    err := r.Status().Update(ctx, mem)

    ...

    // Patch
    patch := client.MergeFrom(mem.DeepCopy())
    mem.Status.Nodes = []string{"pod1", "pod2", "pod3"}
    err := r.Status().Patch(ctx, mem, patch)

    ...
}

Delete

// Delete deletes the given obj from Kubernetes cluster.
func (c Client) Delete(ctx context.Context, obj client.Object, opts ...client.DeleteOption) error

A client.DeleteOption is an interface that sets client.DeleteOptions fields. A client.DeleteOption is created by using one of the provided implementations: GracePeriodSeconds, Preconditions, PropagationPolicy.

Example:

import (
    "context"

    "k8s.io/api/core/v1"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    pod := &v1.Pod{}
    err := r.Get(ctx, request.NamespacedName, pod)

    ...

    if pod.Status.Phase == v1.PodUnknown {
        // Delete the pod after 5 seconds.
        err := r.Delete(ctx, pod, client.GracePeriodSeconds(5))
        ...
    }

    ...
}

DeleteAllOf

// DeleteAllOf deletes all objects of the given type matching the given options.
func (c Client) DeleteAllOf(ctx context.Context, obj client.Object, opts ...client.DeleteAllOfOption) error

A client.DeleteAllOfOption is an interface that sets client.DeleteAllOfOptions fields. A client.DeleteAllOfOption wraps a client.ListOption and client.DeleteOption.

Example:

import (
    "context"
    "fmt"

    "k8s.io/api/core/v1"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    ...

    // Delete all pods in the request namespace with a label of `instance=<name>`
    // and phase `Failed`.
    pod := &v1.Pod{}
    opts := []client.DeleteAllOfOption{
        client.InNamespace(request.NamespacedName.Namespace),
        client.MatchingLabels{"instance", request.NamespacedName.Name},
        client.MatchingFields{"status.phase": "Failed"},
        client.GracePeriodSeconds(5),
    }
    err := r.DeleteAllOf(ctx, pod, opts...)

    ...
}

Example usage

import (
    "context"
    "reflect"

    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/labels"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/types"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"

    cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
)

type MemcachedReconciler struct {
    client.Client
    Log    logr.Logger
    Scheme *runtime.Scheme
}

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {

    // Fetch the Memcached instance.
    memcached := &cachev1alpha1.Memcached{}
    err := r.Get(ctx, request.NamespacedName, memcached)
    if err != nil {
        if errors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }

    // Check if the deployment already exists, if not create a new deployment.
    found := &appsv1.Deployment{}
    err = r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found)
    if err != nil {
         if errors.IsNotFound(err) {
            // Define and create a new deployment.
            dep := r.deploymentForMemcached(memcached)
            if err = r.Create(ctx, dep); err != nil {
                return ctrl.Result{}, err
            }
            return ctrl.Result{Requeue: true}, nil
        } else {
            return ctrl.Result{}, err
        }
    }

    // Ensure the deployment size is the same as the spec.
    size := memcached.Spec.Size
    if *found.Spec.Replicas != size {
        found.Spec.Replicas = &size
        if err = r.Update(ctx, found); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{Requeue: true}, nil
    }

    // Update the Memcached status with the pod names.
    // List the pods for this CR's deployment.
    podList := &corev1.PodList{}
    listOpts := []client.ListOption{
        client.InNamespace(memcached.Namespace),
        client.MatchingLabels(labelsForApp(memcached.Name)),
    }
    if err = r.List(ctx, podList, listOpts...); err != nil {
        return ctrl.Result{}, err
    }

    // Update status.Nodes if needed.
    podNames := getPodNames(podList.Items)
    if !reflect.DeepEqual(podNames, memcached.Status.Nodes) {
        memcached.Status.Nodes = podNames
        if err := r.Status().Update(ctx, memcached); err != nil {
            return ctrl.Result{}, err
        }
    }

    return ctrl.Request{}, nil
}

// deploymentForMemcached returns a Deployment object for data from m.
func (r *MemcachedReconciler) deploymentForMemcached(m *cachev1alpha1.Memcached) *appsv1.Deployment {
    lbls := labelsForApp(m.Name)
    replicas := m.Spec.Size

    dep := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      m.Name,
            Namespace: m.Namespace,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: lbls,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: lbls,
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Image:   "memcached:alpine",
                        Name:    "memcached",
                        Command: []string{"memcached", "-a=64", "-b"},
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: 10000,
                            Name:          "memcached",
                        }},
                    }},
                },
            },
        },
    }

    // Set Memcached instance as the owner and controller.memcac
    // NOTE: calling SetControllerReference, and setting owner references in
    // general, is important as it allows deleted objects to be garbage collected.
    controllerutil.SetControllerReference(m, dep, r.Scheme)
    return dep
}

// labelsForApp creates a simple set of labels for Memcached.
func labelsForApp(name string) map[string]string {
    return map[string]string{"cr_name": name}
}

Docs: Ansible Operator Advanced Options

Mon, 01 Jan 0001 00:00:00 +0000

This document shows the advanced options available to a developer of an ansible operator.

Runner Directory

The ansible runner will keep information about the ansible run in the container. This is located /tmp/ansible-operator/runner/<group>/<version>/<kind>/<namespace>/<name>. To learn more about the runner directory you can read the ansible-runner docs.

Owner Reference Injection

Owner references enable Kubernetes Garbage Collection to clean up after a CR is deleted. Owner references are injected by ansible operators by default by the proxy.

Owner references only apply to resources in the same namespace as the CR. Resources outside the namespace of the CR will automatically be annotated with operator-sdk/primary-resource and operator-sdk/primary-resource-type to track creation. These resources will not be automatically garbage collected. To handle deletion of these resources, use a finalizer.

You may want to manage what your operator watches and the owner references. This means that your operator will need to understand how to clean up after itself when your CR is deleted. To disable these features you will need to edit your Dockerfile to include the line below.

NOTE: That if you use this feature there will be a warning that dependent watches is turned off but there will be no error. WARNING: Once a CR is deployed without owner reference injection, there is no automatic way to add those references.

ENTRYPOINT ["/usr/local/bin/entrypoint", "--inject-owner-ref=false"]

If you have created resources without owner reference injection, it is possible to manually to update resources following this guide.

Max Concurrent Reconciles

Increasing the number of concurrent reconciles allows events to be processed concurrently, which can improve reconciliation performance.

The maximum number of concurrent reconciles can be set in two ways. Operator authors and admins can set the max concurrent reconciles default by including extra args to the operator container in config/manager/manager.yaml and the patch in config/default/auth_proxy_patch.yaml. (Otherwise, the default is the maximum number of logical CPUs available for the process obtained using runtime.NumCPU().)

NOTE: Admins using OLM should use the environment variable instead of the extra args.

- name: manager
  image: "quay.io/asmacdo/memcached-operator:v0.0.0"
  imagePullPolicy: "Always"
  args:
    - "--max-concurrent-reconciles"
    - "3"

Operator admins can override the value by setting an environment variable in the format MAX_CONCURRENT_RECONCILES_<kind>_<group>. This variable must be all uppercase, and periods (e.g. in the group name) are replaced with underscores.

For the memcached operator example, the component parts are retrieved with a GET on the operator:

$ kubectl get memcacheds example-memcached -o yaml

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: example-memcached
  namespace: default

From this data, we can see that the environment variable will be MAX_CONCURRENT_RECONCILES_MEMCACHED_CACHE_EXAMPLE_COM, which we can then add to config/manager/manager.yaml and config/default/auth_proxy_patch.yaml:

- name: manager
  image: "quay.io/asmacdo/memcached-operator:v0.0.0"
  imagePullPolicy: "Always"
  args:
    # This default is overridden.
    - "--max-concurrent-reconciles"
    - "3"
  env:
    # This value is used
    - name: MAX_CONCURRENT_RECONCILES_MEMCACHED_CACHE_EXAMPLE_COM
      value: "6"

Ansible Verbosity

Setting the verbosity at which ansible-runner is run controls how verbose the output of ansible-playbook will be. The normal rules for verbosity apply here, where higher values mean more output. Acceptable values range from 0 (only the most severe messages are output) to 7 (all debugging messages are output).

There are three ways to configure the verbosity argument to the ansible-runner command:

Operator authors and admins can set the Ansible verbosity by including extra args to the operator container in the operator deployment.
Operator admins can set Ansible verbosity by setting an environment variable in the format ANSIBLE_VERBOSITY_<kind>_<group>. This variable must be all uppercase and all periods (e.g. in the group name) are replaced with underscore.
Operator users, authors, and admins can set the Ansible verbosity by setting the "ansible.sdk.operatorframework.io/verbosity" annotation on the Custom Resource.

Examples

For demonstration purposes, let us assume that we have a database operator that supports two Kinds – MongoDB and PostgreSQL – in the db.example.com Group. We have only recently implemented the support for the MongoDB Kind so we want reconciles for this Kind to be more verbose. Our operator container’s spec in our config/manager/manager.yaml and config/default/auth_proxy_patch.yaml files might contain something like:

- name: manager
  image: "quay.io/example/database-operator:v1.0.0"
  imagePullPolicy: "Always"
  args:
    # This value applies to all GVKs specified in watches.yaml
    # that are not overridden by environment variables.
    - "--ansible-verbosity"
    - "1"
  env:
    # Override the verbosity for the MongoDB kind
    - name: ANSIBLE_VERBOSITY_MONGODB_DB_EXAMPLE_COM
      value: "4"

Once the Operator is deployed, the only way to change the verbosity is via the "ansible.sdk.operatorframework.io/verbosity" annotation. Continuing with our example, our CR may look like:

apiVersion: "db.example.com/v1"
kind: "PostgreSQL"
metadata:
  name: "example-db"
  annotations:
    "ansible.sdk.operatorframework.io/verbosity": "5"
spec: {}

Custom Resources with OpenAPI Validation

Currently, SDK tool does not support and will not generate automatically the CRD’s using the OpenAPI spec to perform validations.

However, it can be done manually by adding its validations as you can check in the following example.

Example

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: memcacheds.cache.example.com
spec:
  group: cache.example.com
  names:
    kind: Memcached
    listKind: MemcachedList
    plural: memcacheds
    singular: memcached
  scope: Namespaced
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      description: Memcached is the Schema for the memcacheds API
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          description: MemcachedSpec defines the desired state of Memcached
          properties:
            size:
              description: Size is the size of the memcached deployment
              format: int32
              type: integer
          required:
          - size
          type: object
        status:
          description: MemcachedStatus defines the observed state of Memcached
          properties:
            nodes:
              description: Nodes are the names of the memcached pods
              items:
                type: string
              type: array
          required:
          - nodes
          type: object
      type: object
  versions:
  - name: v1alpha1
    served: true
    storage: true

Passing Arbitrary Arguments to Ansible

You are able to use the flag --ansible-args to pass an arbitrary argument to the Ansible-based Operator. With this option we can, for example, allow a playbook to run a specific part of the configuration without running the whole playbook:

ansible-operator run --ansible-args='--tags "configuration,packages"'

ansible-operator run --ansible-args='--skip-tags "notification"'

Ansible-runner will perform the task relevant to the command specified by the user in the ---ansible-args flag.

Using Ansible-Vault

Ansible Vault allows you to keep sensitive data such as passwords or keys in encrypted files, rather than as plaintext in playbooks or roles. You can specify Ansible-Vault file via an arbitrary argument by using the --ansible-args flag. For example, let’s assume that a playbook reads in a file vars.yml which contains an encrypted text and stores it in a variable secret:

---
- name: Playbook to print debug messages
  hosts: localhost
  tasks:
    - name: Get the decrypted message variable
      include_vars:
        file: vars.yml
        name: secret
    - debug:
        msg: The decrypted value is {{secret.the_secret}}

Now, let’s also assume that we have a password file, pwd.yml, that contains the password to decrypt the encrypted text. Then, by running the command ansible-operator run --ansible-args='--vault-password-file /absolute/path/to/pwd.yml' the operator will read in the encrypted text from the file and perform decryption using the password stored in the pwd.yml file:

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ******************************** 
ok: [localhost] => {
    "msg": "The decrypted value is DECRYPTED-TEST-VALUE"
}

-------------------------------------------------------------------------------

Using Ansible Log Events

Using the --ansible-log-events CLI flag, you can determine to what degree the Ansible task logs will be outputted. The flag can take any of the following values:

Nothing - No tasks or task-related logs will be outputted.
Tasks - Only Ansible Tasks will be outputted.
Everything - All info logs and all tasks will be outputted.

If you want more control over the logs that are outputted, consider using the Zap Logger and verbosity annotations in tandem with the --ansible-log-events CLI flag.

`ansible.sdk.operatorframework.io/reconcile-period` Custom Resource Annotation

You can specify the reconcile period for an Ansible Operator by adding the ansible.sdk.operatorframework.io/reconcile-period key to the custom resource annotations. This feature specifies the maximum interval in which a cluster will get reconciled, and defaults to 10 hours if not manually set. If changes are detected in the desired state, the cluster may be reconciled sooner than the specified interval.

The reconcile period can be specified in the custom resource’s annotations in the following manner:

...
metadata:
  name: memcached-sample
  annotations:
    ansible.sdk.operatorframework.io/reconcile-period: 5s
...

The key only accepts a value in the h/m/s format, such as 1h2m4s, 3m0s, or 4s. Values such as 1x3m9s are invalid.

NOTE: Alternatively, you can specify the reconcile period for Ansible-based Operators in the following ways:

Using the --reconcile-period command-line flag
Using the ‘reconcilePeriod’ key in the watches.yaml file You should not use all three methods to specify a single reconcile period. If all three methods are used simultaneously, the order of precedence is as follows: Custom resource annotations > watches.yaml file > command-line flag.

Docs: Ansible Operator Base Images

Mon, 01 Jan 0001 00:00:00 +0000

Ansible-based operators are built on top of base images built for use with the Operator-SDK.

Ansible Versions

For Operator-SDK versions > v1.30.0 the quay.io/operator-framework/ansible-operator base image has been updated to use Ansible 2.15. The Ansible 2.11 preview base image has been removed and will no longer be built/supported past Operator-SDK v1.30.

For Operator-SDK versions <= v1.30.0, the below information applies:

There have been some major changes in the Ansible ecosystem, primarily the addition of collections and the removal of these libraries from Ansible core. Ansible 2.9 is the last release of the “old way”, with forward compatibility for collections. This is the version officially supported by the Operator-SDK.

Ansible 2.10 was a transition release, and it is NOT recommended for use with operators.

Ansible 2.11 is the future, and Operator-SDK will eventually provide only 2.11 images. Currently, 2.11 base images are in tech-preview. See: ansible-operator-2.11-preview.

Changing Ansible Operator Base Images

For Operator-SDK versions > v1.30.0 - Operators are scaffolded with the latest version of the base image (using Ansible 2.15). The base image name is the same as previous versions of the Operator-SDK.

For Operator-SDK versions <= v1.30.0, the below information applies:

Operators are scaffolded with the latest version of the base image (using Ansible 2.9) in the first line of the operator Dockerfile.

FROM quay.io/operator-framework/ansible-operator:v1.16

Operator authors who want to try out 2.11 can simply replace their FROM with: FROM quay.io/operator-framework/ansible-operator-2.11-preview:v1.16

Docs: Dependent Watches

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the watchDependentResources option in watches.yaml file. It delves into what dependent resources are, why the option is required, how it is achieved and finally gives an example.

What are dependent resources?

In most cases, an operator creates a bunch of Kubernetes resources in the cluster, that helps deploy and manage the application. For instance, the etcd-operator creates two services and a number of pods for a single EtcdCluster CR. In this case, all the Kubernetes resources created by the operator for a CR is defined as dependent resources.

Why the `watchDependentResources` option?

Often, an operator needs to watch dependent resources. To achieve this, a developer would set the field, watchDependentResources to True in the watches.yaml file. If enabled, a change in a dependent resource will trigger the reconciliation loop causing Ansible code to run.

For example, since the etcd-operator needs to ensure that all the pods are up and running, it needs to know when a pod changes. Enabling the dependent watches option would trigger the reconciliation loop to run. The Ansible logic needs to handle these cases and make sure that all the dependent resources are in the desired state as declared by the CR spec

Note: By default it is enabled when using ansible-operator

How is this achieved?

The ansible-operator base image achieves this by leveraging the concept of owner-references. Whenever a Kubernetes resource is created by Ansible code, the ansible-operator's proxy module injects owner-references into the resource being created. The owner-references means the resource is owned by the CR for which reconciliation is taking place.

Whenever the watchDependentResources field is enabled, the ansible-operator will watch all the resources owned by the CR, registering callbacks to their change events. Upon a change, the callback will enqueue a ReconcileRequest for the CR. The enqueued reconciliation request will trigger the Reconcile function of the controller which will execute the ansible logic for reconciliation.

Example

This is an example of a watches file with the watchDependentResources field set to True


- version: v1alpha1
  group: app.example.com
  kind: AppService
  playbook: playbook.yml
  maxRunnerArtifacts: 30
  reconcilePeriod: 5s
  manageStatus: False
  watchDependentResources: True

Docs: Documentation

Mon, 01 Jan 0001 00:00:00 +0000

If a contribution changes the user interface or existing APIs it must include new or updated documentation. Since the operator-sdk repository does not expose many public packages, documentation mostly comes in the form of our website’s markdown docs. Good godocs are expected nonetheless.

Likewise a changelog fragment should be added containing a summary of the change and optionally a migration guide.

Testing docs changes

This document discusses how to visually inspect documentation changes as they would be applied to the live website. All changes to documentation should be inspected locally before being pushed to a PR.

Prerequisites

The docs are built with Hugo which can be installed along with the required extensions by following the docsy install guide.

Note: Be sure to install hugo-extended.

We use git submodules to install the docsy theme. From the operator-sdk directory, update the submodules to install the theme.

git submodule update --init --recursive

Build and Serve

You can build and serve your docs to localhost:1313. From the website/ directory run:

hugo server

Any changes will be included in real time.

Check Docs

make test-docs will validate changelog fragments, build doc HTML in a container, and check its links. Please consider running this locally before creating a PR to save CI resources.

Docs: Ansible Operator Finalizers

Mon, 01 Jan 0001 00:00:00 +0000

The default behavior of an Ansible Operator is to delete all resources the operator created during reconciliation when a managed resource is marked for deletion. This behavior is usually sufficient for applications that exist only in Kubernetes, but sometimes it is necessary to perform more complex operations (for example, when your action performed against a third party API needs to be undone). These more complex cases can still be handled by Ansible Operator, through the use of a finalizer.

Finalizers allow controllers (such as an Ansible Operator) to implement asynchronous pre-delete hooks. This allows custom logic to run after a resource has been marked for deletion, but before the resource has actually been deleted from the Kubernetes cluster. For Ansible Operator, this hook takes the form of an Ansible playbook or role. You can define the mapping from your finalizer to a playbook or role by simply setting the finalizer field on the entry in your watches.yaml. You can also choose to re-run your top-level playbook or role with different variables set. The watches.yaml finalizer configuration accepts the following options:

See Ansible watches documentation for more information.

name

name is required.

This is the name of the finalizer. This is basically an arbitrary string, the existence of any finalizer string on a resource will prevent that resource from being deleted until the finalizer is removed. Ansible Operator will remove this string from the list of finalizers on successful execution of the specified role or playbook. A typical finalizer will be <qualified-group>/finalizer, where <qualified-group> is the fully qualified group of the resource being managed.

playbook

One of playbook, role, or vars must be provided. If playbook is not provided, it will default to the playbook specified at the top level of the watches.yaml entry.

This field is identical to the top-level playbook field. It requires an absolute path to a playbook on the operator’s file system.

role

One of playbook, role, or vars must be provided. If role is not provided, it will default to the role specified at the top level of the [watches.yaml][watches] entry.

This field is identical to the top-level role field.

vars

One of playbook, role, or vars must be provided.

vars is an arbitrary map of key-value pairs. The contents of vars will be passed as extra_vars to the playbook or role specified in the finalizer block, or at the top-level if neither playbook or role was set for the finalizer.

Examples

Here are a few examples of watches.yaml files that specify a finalizer:

Run top-level playbook or role with new variables

---
- version: v1alpha1
  group: app.example.com
  kind: Database
  playbook: /opt/ansible/playbook.yml
  finalizer:
    name: app.example.com/finalizer
    vars:
      state: absent

This example will run playbook.yml when the Custom Resource is deleted. Because vars is set, the playbook will be run with state set to absent. Inside the playbook, the author can check this value and perform whatever cleanup is necessary.

---
- version: v1alpha1
  group: app.example.com
  kind: Database
  role: database
  finalizer:
    name: app.example.com/finalizer
    vars:
      state: absent

This example is nearly identical to the first, except it will run the /opt/ansible/roles/database role, rather than a playbook, with the state variable set to absent.

Run a different playbook or role

---
- version: v1alpha1
  group: app.example.com
  kind: Database
  playbook: playbook.yml
  finalizer:
    name: app.example.com/finalizer
    role: teardown_database

This example will run the /opt/ansible/roles/teardown_database role when the Custom Resource is deleted.

---
- version: v1alpha1
  group: app.example.com
  kind: Database
  playbook: playbook.yml
  finalizer:
    name: app.example.com/finalizer
    playbook: destroy.yml

This example will run the /opt/ansible/destroy.yml playbook when the Custom Resource is deleted.

---
- version: v1alpha1
  group: app.example.com
  kind: Database
  playbook: playbook.yml
  finalizer:
    name: app.example.com/finalizer
    role: myNamespace.myCollection.myRole

This example will run the myRole when the Custom Resource is deleted. (The collection must have been installed.)

Run a different playbook or role with vars

You can set playbook or role and vars at the same time. This can be useful if only a small part of your logic handles interacting with the component that requires cleanup. Rather than run all the logic again, you can specify only to run the role or playbook that handled the interaction, with a different variable set.

---
- version: v1alpha1
  group: app.example.com
  kind: Database
  playbook: playbook.yml
  finalizer:
    name: app.example.com/finalizer
    role: manage_credentials
    vars:
      state: revoked

For this example, assume our application configures automated backups to a third party service. On deletion, all we want to do is revoke the credentials used to backup the data. We run just the /opt/ansible/roles/manage_credentials role, which is imported by our playbook to create the credentials in the first place, but we pass the state: revoked option, which causes the role to invalidate our credentials. For everything else in our application, automatic deletion of dependent resources will be sufficient, so we can exit successfully and let the operator remove our finalizer and allow the resource to be deleted.

Docs: Generating Manifests and Metadata

Mon, 01 Jan 0001 00:00:00 +0000

This document describes how to manage packaging and shipping your Operator in the following stages:

Generate your first release - encapsulate the metadata needed to install your Operator with the Operator Lifecycle Manager and configure the permissions it needs from the generated SDK files.
Update your Operator - apply any updates to Operator manifests made during development.
Upgrade your Operator - carry over any customizations you have made and ensure a rolling update to the next version of your Operator.

Overview

Several operator-sdk subcommands manage operator-framework manifests and metadata, in particular ClusterServiceVersion's (CSVs), for an Operator: generate bundle and generate kustomize manifests. See this CLI overview for details on each command.

Note: The packagemanifests format is deprecated and support will be removed in operator-sdk v2.0.0.

Kustomize files

operator-sdk generate kustomize manifests generates a CSV kustomize base config/manifests/bases/<project-name>.clusterserviceversion.yaml and a config/manifests/kustomization.yaml by default. These files are required as kustomize build input in downstream commands.

By default, the command starts an interactive prompt if a CSV base in config/manifests/bases is not present to collect UI metadata. You can disable the interactive prompt by passing --interactive=false.

$ operator-sdk generate kustomize manifests
INFO[0000] Generating CSV manifest version 0.1.0

Display name for the operator (required):
> memcached

Comma-separated list of keywords for your operator (required):
> app, operator
...

Once this base is written, you may modify any of the fields labeled user in the fields section below. These values will persist when generating a bundle, so make necessary metadata changes here and not the generated bundle.

For Go Operators only: the command parses CSV markers from Go API type definitions, located in ./api for single group projects and ./apis for multigroup projects, to populate certain CSV fields. You can set an alternative path to the API types root directory with --apis-dir. These markers are not available to Ansible or Helm project types.

The command attempts to process the local types defined in your API. If you import a package that uses the same name as a local type, running the command causes an infinite loop. For example:

type PodStatus struct {
  SomeField string
  // imported type with the same name will infinitely trigger
  // the parser to process the local PodStatus type
  Status v1.PodStatus 
}

To prevent an infinite loop, edit the local type definition to use a different name. For example:

type PodStatusWrapper struct {
  SomeField string
  Status v1.PodStatus 
}

ClusterServiceVersion manifests

CSV’s are manifests that define all aspects of an Operator, from what CustomResourceDefinitions (CRDs) it uses to metadata describing the Operator’s maintainers. They are typically versioned by semver, much like Operator projects themselves; this version is present in both their metadata.name and spec.version fields. The CSV generator called by generate <bundle|packagemanifests> requires certain input manifests to construct a CSV manifest; all inputs are read when either command is invoked, along with a CSV’s base, to idempotently regenerate a CSV.

The following resource kinds are typically included in a CSV, which are addressed by config/manifests/kustomization.yaml:

Role: define Operator permissions within a namespace.
ClusterRole: define cluster-wide Operator permissions.
Deployment: define how the Operator’s operand is run in pods.
ValidatingWebhookConfiguration, MutatingWebhookConfiguration: configures webhooks for your manager to handle.
CustomResourceDefinition: definitions of custom objects your Operator reconciles.
Custom resource examples: examples of objects adhering to the spec of a particular CRD.

You can optionally specify an input ClusterServiceVersion manifest to the set of manifests passed to these generate subcommands instead of having them read from the base path. This is advantageous for those who would like to take full advantage of kustomize for their base. All fields unlabeled or labeled with marker below will be overwritten by these command, so make sure you do not kustomize build those fields!

Webhooks

A CSV allows you to define both admission and conversion webhooks at spec.webhookdefinitions. The generate <bundle|packagemanifests> commands, described below, will automatically add webhooks to your CSV if the following holds true:

A webhook configuration must be associated with a Service by name and namespace, whether in a CRD or in a *WebhookConfiguration file,
The associated Service must expose one spec.ports[*].targetPort that matches both containerPort and protocol of one element in the Operator Deployment's spec.template.spec.containers[*].ports.

By default, the manager’s Deployment is configured to mount a volume containing TLS cert data created by [cert-manager][cert-manager] into the manager’s container. OLM does not yet support cert-manager, so a JSON patch was added to remove this volume and mount such that OLM can itself create and manage certs for your Operator.

Note (for Go Operators only): If targeting OLM < v0.17.0, the manager’s default webhook server is not configured with the correct cert/key paths; the correct path is /apiserver.local.config/certificates/apiserver.{cert,key}. To cover this case, make the following changes to your main.go:

import (
  ...
  ctrl "sigs.k8s.io/controller-runtime"
  "sigs.k8s.io/controller-runtime/pkg/webhook"
)

func main() {
  ...

  // Configure a webhook.Server with the correct path and file names.
  // If webhookServer is nil, which will be the case of OLM >= 0.17 is available,
  // the manager will create a server for you using Host, Port,
  // and the default CertDir, KeyName, and CertName.
  var webhookServer *webhook.Server
  const legacyOLMCertDir = "/apiserver.local.config/certificates"
  if info, err := os.Stat(legacyOLMCertDir); err == nil && info.IsDir() {
    webhookServer = &webhook.Server{
      Host:     <some host>, // Set this only if normally set in ctrl.Options below.
      Port:     <some port>, // Set this only if normally set in ctrl.Options below.
      CertDir:  legacyOLMCertDir,
      CertName: "apiserver.crt",
      KeyName:  "apiserver.key",
    }
  }

  mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    Host:          <some host>,
    Port:          <some port>,
    WebhookServer: webhookServer, // Host/Port will not be used if webhookServer is nil.
  })
 
  // Now you can register webhooks.
  ...
}

Note: The Service itself will still be placed into the manifests/ directory, in case other Operator resources require routing. Feel free to remove it otherwise.

Generate your first release

You’ve recently run operator-sdk init and created your APIs with operator-sdk create api. Now you’d like to package your Operator for deployment by OLM. Your Operator is at version v0.0.1; the Makefile variable VERSION should be set to 0.0.1. You’ve also built your operator image, example.com/memcached-operator:v0.0.1; if this image tag does not match yours, swap in the correct one in the docs below.

Bundle format

A bundle consists of manifests (CSV, CRDs, and other supported kinds) and metadata that define an Operator at a particular version, and an optional scorecard configuration file. You may have also heard of a bundle image. From the bundle docs:

An Operator Bundle is built as a scratch (non-runnable) container image that contains operator manifests and specific metadata in designated directories inside the image. Then, it can be pushed and pulled from an OCI-compliant container registry. Ultimately, an operator bundle will be used by Operator Registry and OLM to install an operator in OLM-enabled clusters.

At this stage in your Operator’s development, we only need to worry about generating bundle files; bundle images become important once you’re ready to publish your Operator.

SDK projects are scaffolded with a Makefile containing the bundle recipe by default, which wraps generate kustomize manifests, generate bundle, and other related commands.

By default make bundle will generate a CSV, copy CRDs and other supported kinds, generate metadata, and add your scorecard configuration in the bundle format:

$ make bundle
$ tree ./bundle
./bundle
├── manifests
│   ├── cache.example.com_memcacheds.yaml
│   ├── memcached-operator.clusterserviceversion.yaml
│   ├── memcached-operator-controller-manager-metrics-monitor_monitoring.coreos.com_v1_servicemonitor.yaml
│   ├── memcached-operator-controller-manager-metrics-service_v1_service.yaml
│   ├── memcached-operator-metrics-reader_rbac.authorization.k8s.io_v1beta1_clusterrole.yaml
│   └── memcached-operator-webhook-service_v1_service.yaml
├── metadata
│   └── annotations.yaml
└── tests
    └── scorecard
        └── config.yaml

Important: bundle generation is supposed to be idempotent, so any changes to CSV fields able to be persisted (marked (user) or (marker) below) must be made to the base set of manifests, typically found in config/.

Bundle metadata in bundle/metadata/annotations.yaml contains information about a particular Operator version available in a registry. OLM uses this information to install specific Operator versions and resolve dependencies. That file and bundle.Dockerfile contain the same annotations, the latter as LABELs, which do not need to be modified in most cases; if you do decide to modify them, both sets of annotations must be the same to ensure consistent Operator deployment.

Channels

Metadata for each bundle contains channel information as well:

Channels allow package authors to write different upgrade paths for different users (e.g. beta vs. stable).

Channels become important when publishing, but we should still be aware of them beforehand as they’re required values in our metadata. make bundle writes the channel alpha by default.

Validation

The bundle recipe includes a call to operator-sdk bundle validate, which runs a set of required object validators on your bundle that ensure both its format and content meet the bundle specification. These will always be run and cannot be disabled.

You may also have added CSV fields containing useful UI metadata for cluster console display, and want to ensure that metadata matches some hosted catalog’s submission requirements. The bundle validate command supports optional validators that can validate these bundle metadata. These validators are disabled by default, and can be selectively enabled with --select-optional <label-selector>. You can list all available optional validators by setting the --list-optional flag:

$ operator-sdk bundle validate --list-optional
NAME           LABELS                                                DESCRIPTION
operatorhub    name=operatorhub                                      OperatorHub.io metadata validation. 
               suite=operatorframework    
community      name=community                                        (stage: alpha) Community Operator bundle validation      
...

For example, you want to turn on the operatorhub validator shown above so you can publish the 0.0.1 operator you recently created on OperatorHub.io. To do so, you can modify your Makefile’s bundle recipe to validate any further changes you make to bundle UI metadata related to OperatorHub requirements:

bundle: ...
  ...
  operator-sdk bundle validate ./bundle --select-optional name=operatorhub

Also, see that you can test the bundle against the suite of test to ensure it against all criteria:

operator-sdk bundle validate ./bundle --select-optional suite=operatorframework

Note: The OperatorHub.io validator in the operatorframework optional suite allows you to validate that your manifests can work with a Kubernetes cluster of a particular version using the k8s-version optional key value:

operator-sdk bundle validate ./bundle --select-optional suite=operatorframework --optional-values=k8s-version=1.22

Documentation on optional validators:

operatorhub

Note: (stage: alpha) The Community validator allows you to validate your bundle.Dockerfile configuration against its specific criteria using the image-path optional key value:

operator-sdk bundle validate ./bundle --select-optional name=community --optional-values=image-path=bundle.Dockerfile

Package manifests format

A [package manifests][package-manifests] format consists of on-disk manifests (CSV, CRDs and other supported kinds) and metadata that define an Operator at all versions of that Operator. Each version is contained in its own directory, with a parent package manifest YAML file containing channel-to-version mappings, much like a bundle’s metadata.

If your Operator is already formatted as a package manifests and you do not wish to migrate to the bundle format yet, you should add the following to your Makefile to make development easier:

For Go-based Operator projects

# Options for "packagemanifests".
ifneq ($(origin FROM_VERSION), undefined)
PKG_FROM_VERSION := --from-version=$(FROM_VERSION)
endif
ifneq ($(origin CHANNEL), undefined)
PKG_CHANNELS := --channel=$(CHANNEL)
endif
ifeq ($(IS_CHANNEL_DEFAULT), 1)
PKG_IS_DEFAULT_CHANNEL := --default-channel
endif
PKG_MAN_OPTS ?= $(PKG_FROM_VERSION) $(PKG_CHANNELS) $(PKG_IS_DEFAULT_CHANNEL)

# Generate package manifests.
packagemanifests: kustomize manifests
  operator-sdk generate kustomize manifests -q
  cd config/manager && $(KUSTOMIZE) edit set image controller=$(IMG)
  $(KUSTOMIZE) build config/manifests | operator-sdk generate packagemanifests -q --version $(VERSION) $(PKG_MAN_OPTS)

For Helm/Ansible-based Operator projects

# Options for "packagemanifests".
ifneq ($(origin FROM_VERSION), undefined)
PKG_FROM_VERSION := --from-version=$(FROM_VERSION)
endif
ifneq ($(origin CHANNEL), undefined)
PKG_CHANNELS := --channel=$(CHANNEL)
endif
ifeq ($(IS_CHANNEL_DEFAULT), 1)
PKG_IS_DEFAULT_CHANNEL := --default-channel
endif
PKG_MAN_OPTS ?= $(PKG_FROM_VERSION) $(PKG_CHANNELS) $(PKG_IS_DEFAULT_CHANNEL)

# Generate package manifests.
packagemanifests: kustomize
  operator-sdk generate kustomize manifests -q
  cd config/manager && $(KUSTOMIZE) edit set image controller=$(IMG)
  $(KUSTOMIZE) build config/manifests | operator-sdk generate packagemanifests -q --version $(VERSION) $(PKG_MAN_OPTS)

By default make packagemanifests will generate a CSV, a package manifest file, and copy CRDs in the package manifests format:

$ make packagemanifests IMG=example.com/memcached-operator:v0.0.1
$ tree ./packagemanifests
./packagemanifests
├── 0.0.1
│   ├── cache.my.domain_memcacheds.yaml
│   └── memcached-operator.clusterserviceversion.yaml
└── memcached-operator.package.yaml

Update your Operator

Let’s say you added a new API App with group app and version v1alpha1 to your Operator project, and added a port to your manager Deployment in config/manager/manager.yaml.

If using a bundle format, the current version of your CSV can be updated by running:

$ make bundle IMG=example.com/memcached-operator:v0.0.1

If using a package manifests format, run:

$ make packagemanifests IMG=example.com/memcached-operator:v0.0.1

Running the command for either format will append your new CRD to spec.customresourcedefinitions.owned, replace the old data at spec.install.spec.deployments with your updated Deployment, and update your existing CSV manifest. The SDK will not overwrite user-defined fields like spec.maintainers.

Upgrade your Operator

Let’s say you’re upgrading your Operator to version v0.0.2, you’ve already updated the VERSION variable in your Makefile to 0.0.2, and built a new operator image example.com/memcached-operator:v0.0.2. You also want to add a new channel beta, and use it as the default channel.

First, update spec.replaces in your base CSV manifest to the current CSV name. In this case, the change would look like:

spec:
  ...
  replaces: memcached-operator.v0.0.1

Next, upgrade your bundle. If using a bundle format, a new version of your CSV can be created by running:

$ make bundle CHANNELS=beta DEFAULT_CHANNEL=beta IMG=example.com/memcached-operator:v0.0.2

If using a package manifests format, run:

$ make packagemanifests FROM_VERSION=0.0.1 CHANNEL=beta IS_CHANNEL_DEFAULT=1 IMG=example.com/memcached-operator:v0.0.2

Running the command for either format will persist user-defined fields, and updates spec.version and metadata.name.

For packagemanifests only The command will also populate spec.replaces with the old CSV version’s name.

CSV fields

Below are two lists of fields: the first is a list of all fields the SDK and OLM expect in a CSV, and the second are optional.

For Go Operators only: Several fields require user input (labeled user) or a CSV marker (labeled marker). This list may change as the SDK becomes better at generating CSV’s. These markers are not available to Ansible or Helm project types.

Required:

metadata.name _(user*)_: a *unique* name for this CSV of the format <project-name>.vX.Y.Z, ex. app-operator.v0.0.1.
spec.displayName (user) : a name to display for the Operator in Operator Hub.
spec.version _(user*)_: semantic version of the Operator, ex. 0.0.1.
spec.installModes (user): what mode of installation namespacing OLM should use. Currently all but MultiNamespace are supported by SDK Operators.
spec.customresourcedefinitions: any CRDs the Operator uses. Certain fields in elements of owned will be filled by the SDK.
- owned: all CRDs the Operator deploys itself from it’s bundle.
  - name: CRD’s metadata.name.
  - kind: CRD’s spec.names.kind.
  - version: CRD’s spec.version.
  - description (marker) : description of the CRD.
  - displayName (marker) : display name of the CRD.
  - resources (marker) : any Kubernetes resources used by the CRD, ex. Pod's and ConfigMap's.
  - specDescriptors (marker) : UI hints for inputs and outputs of the Operator’s spec.
  - statusDescriptors (marker) : UI hints for inputs and outputs of the Operator’s status.
  - actionDescriptors (user) : UI hints for an Operator’s in-cluster actions.
- required (user) : all CRDs the Operator expects to be present in-cluster, if any. All required element fields must be populated manually.

Optional:

spec.description (user) : a thorough description of the Operator’s functionality.
spec.keywords (user) : a list of keywords describing the Operator.
spec.maintainers (user) : a list of human or organizational entities maintaining the Operator, with a name and email.
spec.provider (user) : the Operator provider, with a name; usually an organization.
spec.labels (user) : a list of key:value pairs to be used by Operator internals.
metadata.annotations.alm-examples: CR examples, in JSON string literal format, for your CRD’s. Ideally one per CRD.
metadata.annotations.capabilities: level of Operator capability. See the Operator maturity model for a list of valid values.
spec.replaces (user): the name of the CSV being replaced by this CSV.
spec.links (user) : a list of URL’s to websites, documentation, etc. pertaining to the Operator or application being managed, each with a name and url.
spec.selector (user) : selectors by which the Operator can pair resources in a cluster.
spec.icon (user) : a base64-encoded icon unique to the Operator, set in a base64data field with a mediatype.
spec.maturity (user): the Operator’s maturity, ex. alpha.
spec.minKubeVersion (user): the minimal Kubernetes version supported by the Operator, ex. 1.16.0.
spec.webhookdefinitions: any webhooks the Operator uses.
spec.relatedImages (user): a list of image tags containing SHA digests mapped to in-CSV names that your Operator might require to perform their functions.
- To get the correct tag for an image available in some remote registry, run docker inspect --format='{{range $i, $d := .RepoDigests}}{{$d}}{{"\n"}}{{end}}' and choose the tag for the desired registry.
spec.skips (user): the names of one or more CSVs that should be skipped in a catalog’s upgrade graph.

* metadata.name and spec.version will only be automatically updated from the base CSV when you set --version when running generate <bundle|packagemanifests>.

Docs: Information Flow for Operator SDK Ansible-based Operators

Mon, 01 Jan 0001 00:00:00 +0000

Controller/Reconciler

Gets the primary resource based on NamespacedName this uses the cache from controller runtime.
Determines if the resource is deleted based on the existence of a finalizer and a metadata.DeletionTimeStamp.
- If deleted and not one of our finalizers we exit with no error.
- If finalizer is needed, but is not on the object, and is not deleted then add it and continue.
Marks the status of the CR as running if it is managing the status and continues
Creates the proxy’s kubeconfig.
Calls out Runner using the runner package.
Watches for events to come back across the results channel.
- Logs the events
- Keeps track of failure messages as well as the StatusEvent, which is the ending event that the Ansible sends to mark the playbook/role is completed.
If a finalizer exists and the resource was deleted then remove the finalizer and update the object and continue.
Update the status of the resource to mark that the run has completed

Runner Package

Create the socket that ansible will communicate over
Translate object.Spec -> ansible vars (camelCase -> camel_case) and adds them as parameters for the ansible-runner directory.
Create the directory on disk for ansible runner. or update the files on disk if they already exist.
Run the ansible runner command in a go routine, when this exits close the channel for the socket.

The Proxy

Every request to the k8s api goes through the proxy.
The owner reference is injected into the object that is being created in the same namespace as the CR.
The operator-sdk annotations are injected into the object that is being created outside of namepsace of the CR.
The proxy then adds dependent watches for the correct controller if we have not started watching the type already.
On a GET, we attempt to use the informer cache to get the resource. This will also attempt to re-add dependent watches if we find a type with an owner reference.

Ansible Runner

Ansible is run and has its own process.
Ansible-runner-http package allows runner to set us ansible events as ansible is being run. ansible runner uses the socket and the controller is watching the events as they come through the channel.

Docs: Logging

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Operator SDK-generated operators use the logr interface to log. This log interface has several backends such as zap, which the SDK uses in generated code by default. logr.Logger exposes structured logging methods that help create machine-readable logs and adding a wealth of information to log records.

Default zap logger

Operator SDK uses a zap-based logr backend when scaffolding new projects. To assist with configuring and using this logger, the SDK includes several helper functions.

In the simple example below, we add the zap flagset to the operator’s command line flags with BindFlags(), and then set the controller-runtime logger with zap.Options{}.

By default, zap.Options{} will return a logger that is ready for production use. It uses a JSON encoder, logs starting at the info level. To customize the default behavior, users can use the zap flagset and specify flags on the command line. The zap flagset includes the following flags that can be used to configure the logger:

--zap-devel: Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn) Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
--zap-encoder: Zap log encoding (‘json’ or ‘console’)
--zap-log-level: Zap Level to configure the verbosity of logging. Can be one of ‘debug’, ‘info’, ‘error’, or any integer value > 0 which corresponds to custom debug levels of increasing verbosity”)
--zap-stacktrace-level: Zap Level at and above which stacktraces are captured (one of ‘info’ or ‘error’)

Consult the controller-runtime godocs for more detailed flag information.

A simple example

Operators set the logger for all operator logging in main.go. To illustrate how this works, try out this simple example:

package main

import (
	"sigs.k8s.io/controller-runtime/pkg/log/zap"
	logf "sigs.k8s.io/controller-runtime/pkg/log"
)

var globalLog = logf.Log.WithName("global")
func main() {
	// Add the zap logger flag set to the CLI. The flag set must
	// be added before calling flag.Parse().
	opts := zap.Options{}
	opts.BindFlags(flag.CommandLine)
	flag.Parse()

	logger := zap.New(zap.UseFlagOptions(&opts))
	logf.SetLogger(logger)

	scopedLog := logf.Log.WithName("scoped")

	globalLog.Info("Printing at INFO level")
	globalLog.V(1).Info("Printing at DEBUG level")
	scopedLog.Info("Printing at INFO level")
	scopedLog.V(1).Info("Printing at DEBUG level")
}

Output using the defaults

$ go run main.go
INFO[0000] Running the operator locally in namespace default.
{"level":"info","ts":1587741740.407766,"logger":"global","msg":"Printing at INFO level"}
{"level":"info","ts":1587741740.407855,"logger":"scoped","msg":"Printing at INFO level"}

Output overriding the log level to 1 (debug)

$ go run main.go --zap-log-level=debug
INFO[0000] Running the operator locally in namespace default.
{"level":"info","ts":1587741837.602911,"logger":"global","msg":"Printing at INFO level"}
{"level":"debug","ts":1587741837.602964,"logger":"global","msg":"Printing at DEBUG level"}
{"level":"info","ts":1587741837.6029708,"logger":"scoped","msg":"Printing at INFO level"}
{"level":"debug","ts":1587741837.602973,"logger":"scoped","msg":"Printing at DEBUG level"}

Custom zap logger

In order to use a custom zap logger, zap from controller-runtime can be utilized to wrap it in a logr implementation.

Below is an example illustrating the use of zap-logfmt in logging.

Example

In your main.go file, replace the current implementation for logs inside the main function:

...
// Add the zap logger flag set to the CLI. The flag set must
// be added before calling flag.Parse().
	opts := zap.Options{}
	opts.BindFlags(flag.CommandLine)
	flag.Parse()

	logger := zap.New(zap.UseFlagOptions(&opts))
	logf.SetLogger(logger)
...

With:

	import(
	...
	zaplogfmt "github.com/sykesm/zap-logfmt"
	uzap "go.uber.org/zap"
	"go.uber.org/zap/zapcore"
	logf "sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/log/zap"
	...
)
	configLog := uzap.NewProductionEncoderConfig()
	configLog.EncodeTime = func(ts time.Time, encoder zapcore.PrimitiveArrayEncoder) {
		encoder.AppendString(ts.UTC().Format(time.RFC3339Nano))
	}
	logfmtEncoder := zaplogfmt.NewEncoder(configLog)

	// Construct a new logr.logger.
	logger := zap.New(zap.UseDevMode(true), zap.WriteTo(os.Stdout), zap.Encoder(logfmtEncoder))
	logf.SetLogger(logger)

NOTE: For this example, you will need to add the module "github.com/sykesm/zap-logfmt" to your project. Run go get -u github.com/sykesm/zap-logfmt.

Output using custom zap logger

$ go run main.go
ts=2020-04-30T20:35:59.551268Z level=info logger=global msg="Printing at INFO level"
ts=2020-04-30T20:35:59.551314Z level=debug logger=global msg="Printing at DEBUG level"
ts=2020-04-30T20:35:59.551318Z level=info logger=scoped msg="Printing at INFO level"
ts=2020-04-30T20:35:59.55132Z level=debug logger=scoped msg="Printing at DEBUG level"

By using sigs.k8s.io/controller-runtime/pkg/log, your logger is propagated through controller-runtime. Any logs produced by controller-runtime code will be through your logger, and therefore have the same formatting and destination.

Setting flags when running locally

When running locally with make run ENABLE_WEBHOOKS=false, you can use the ARGS var to pass additional flags to your operator, including the zap flags. For example:

$ make run ARGS="--zap-encoder=console" ENABLE_WEBHOOKS=false

Make sure to have your run target to take ARGS as shown below in Makefile.

# Run against the configured Kubernetes cluster in ~/.kube/config
run: manifests generate fmt vet
	go run ./main.go $(ARGS)

Setting flags when deploying to a cluster

When deploying your operator to a cluster you can set additional flags using an args array in your operator’s container spec in the file config/default/manager_metrics_patch.yaml For example:

- op: add
  path: /spec/template/spec/containers/0/args/0
  value: --zap-log-level=debug 
- op: add
  path: /spec/template/spec/containers/0/args/0
  value: --zap-encoder=console

Creating a structured log statement

There are two ways to create structured logs with logr. You can create new loggers using log.WithValues(keyValues) that include keyValues, a list of key-value pair interface{}'s, in each log record. Alternatively you can include keyValues directly in a log statement, as all logr log statements take some message and keyValues. The signature of logr.Error() has an error-type parameter, which can be nil.

An example from memcached_controller.go:

package memcached

import (
  ctrllog "sigs.k8s.io/controller-runtime/pkg/log"
)


// MemcachedReconciler reconciles a Memcached object
type MemcachedReconciler struct {
	client.Client
	Log    logr.Logger
	Scheme *runtime.Scheme
}

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := ctrllog.FromContext(ctx)

	// Fetch the Memcached instance
	memcached := &cachev1alpha1.Memcached{}
	err := r.Get(ctx, req.NamespacedName, memcached)
	if err != nil {
		if errors.IsNotFound(err) {
			// Request object not found, could have been deleted after reconcile request.
			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
			// Return and don't requeue
			log.Info("Memcached resource not found. Ignoring since object must be deleted")
			return ctrl.Result{}, nil
		}
		// Error reading the object - requeue the request.
		log.Error(err, "Failed to get Memcached")
		return ctrl.Result{}, err
	}

	// Check if the deployment already exists, if not create a new one
	found := &appsv1.Deployment{}
	err = r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found)
	if err != nil && errors.IsNotFound(err) {
		// Define a new deployment
		dep := r.deploymentForMemcached(memcached)
		log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
		err = r.Create(ctx, dep)
		if err != nil {
			log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
			return ctrl.Result{}, err
		}
		// Deployment created successfully - return and requeue
		return ctrl.Result{Requeue: true}, nil
	} else if err != nil {
		log.Error(err, "Failed to get Deployment")
		return ctrl.Result{}, err
	}

	...
}

Log records will look like the following (from log.Error() above):

2020-04-27T09:14:15.939-0400	ERROR	controllers.Memcached	Failed to create new Deployment	{"memcached": "default/memcached-sample", "Deployment.Namespace": "default", "Deployment.Name": "memcached-sample"}

Non-default logging

If you do not want to use logr as your logging tool, you can remove logr-specific statements without issue from your operator’s code, including the logr setup code in main.go, and add your own. Note that removing logr setup code will prevent controller-runtime from logging.

Docs: Internal Operator Metrics

Mon, 01 Jan 0001 00:00:00 +0000

The Ansible Operator comes with three internal metrics that provide an insight to the frequency and time of operator reconciliations. These metrics can be scraped by a Prometheus instance or any other openmetrics system. To publish operator metrics and scrape them with an openmetrics system such as Prometheus, view Kubebuilder documentation on publishing metrics.

The default metrics recorded in Operator SDK are collected in a histogram.

The following three metrics are derived from the histogram:

ansible_operator_reconciles_bucket - Each bucket in the histogram counts the number of reconciliations that have a period (in seconds) less than or equal to the upper limit of the bucket.
ansible_operator_reconciles_count - The total number of reconciliations that have occurred up to that instance of time while running an Ansible operator.
ansible_operator_reconciles_sum - The cumulative amount of time (in seconds) of all reconciliations that have occurred up to that instance of time while running an Ansible operator.

These metrics can be queried in the Prometheus UI.

Docs: Proxy Friendly Operators

Mon, 01 Jan 0001 00:00:00 +0000

Proxy-friendly Operators should inspect their environment for the standard proxy variables (HTTPS_PROXY, HTTP_PROXY, and NO_PROXY) and pass the values to Operands.

Example:

- name: start proxy-test job
  kubernetes.core.k8s:
    definition:
      apiVersion: batch/v1
      kind: Job
      metadata:
        name: env-job
        namespace: "{{ ansible_operator_meta.namespace }}"
      spec:
        template:
          spec:
            containers:
              - name: curl-example
                image: registry.access.redhat.com/ubi8/ubi:8.8
                command: ["curl"]
                args: ["http://example.com/job-request"]
                env:
                  - name: HTTP_PROXY
                    value: '{{ lookup("env", "HTTP_PROXY") | default("", True) }}'
                  - name: http_proxy
                    value: '{{ lookup("env", "HTTP_PROXY") | default("", True) }}'
            restartPolicy: Never
        backoffLimit: 4

You can set the environment variable on the Operator deployment. Using the memcached tutorial, edit config/manager/manager.yaml:

containers:
 - args:
   - --leader-elect
   - --leader-election-id=ansible-proxy-demo
   image: controller:latest
   name: manager
   env:
     - name: "HTTP_PROXY"
       value: "http_proxy_test"

Docs: Proxy Friendly Operators

Mon, 01 Jan 0001 00:00:00 +0000

Proxy-friendly Operators should inspect their environment for the standard proxy variables (HTTPS_PROXY, HTTP_PROXY, and NO_PROXY) and pass the values to Operands.

Operator-lib provides a helper function proxy.ReadProxyVarsFromEnv that does this inspection, all you need to do is append the results to the Operand environments.

Using the memcached tutorial as an example, add the following to the Reconcile loop in controllers/memcached_controller.go:

import (
  ...
   "github.com/operator-framework/operator-lib/proxy"
)


for i, container := range dep.Spec.Template.Spec.Containers {
		dep.Spec.Template.Spec.Containers[i].Env = append(container.Env, proxy.ReadProxyVarsFromEnv()...)
}

You can set the environment variable on the Operator deployment. Using the memcached tutorial, edit config/manager/manager.yaml:

containers:
 - args:
   - --leader-elect
   - --leader-election-id=go-proxy-demo
   image: controller:latest
   name: manager
   env:
     - name: "HTTP_PROXY"
       value: "http_proxy_test"

Docs: Proxy Friendly Operators

Mon, 01 Jan 0001 00:00:00 +0000

Proxy-friendly Operators should inspect their environment for the standard proxy variables (HTTPS_PROXY, HTTP_PROXY, and NO_PROXY) and pass the values to Operands.

This can be accomplished by modifying the watches.yaml to include the overrides based on an environment variable:

- group: demo.example.com
  version: v1alpha1
  kind: Nginx
  chart: helm-charts/nginx
  overrideValues:
    proxy.http: $HTTP_PROXY
#+kubebuilder:scaffold:watch

Note: This example assumes that proxy.http is included in your chart’s Values.yaml. The nginx tutorial does not have this value, but you can add to the helmcharts/nginx/Values.yaml:

proxy:
  http: ""
  https: ""
  no_proxy: ""

You will also need to make sure the chart template supports the usage of these values. Using the nginx tutorial, edit helm-charts/nginx/templates/deployment.yaml

containers:                                                                                                                                                                                                                             
  - name: {{ .Chart.Name }}                                                                                                                                                                                                             
    securityContext:                                                                                                                                                                                                                    
      {{- toYaml .Values.securityContext | nindent 12 }}                                                                                                                                                                                
    image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"                                                                                                                                         
    imagePullPolicy: {{ .Values.image.pullPolicy }}                                                                                                                                                                                     
    env:                                                                                                                                                                                                                                
      - name: http_proxy                                                                                                                                                                                                                
        value: "{{ .Values.proxy.http }}"

You can set the environment variable on the Operator deployment. Using the nginx tutorial, edit config/manager/manager.yaml:

containers:
 - args:
   - --leader-elect
   - --leader-election-id=helm-proxy-demo
   image: controller:latest
   name: manager
   env:
     - name: "HTTP_PROXY"
       value: "http_proxy_test"

Docs: Quickstart for Go-based Operators

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks through an example of building a simple memcached-operator using tools and libraries provided by the Operator SDK.

Prerequisites

Go through the installation guide.
Make sure your user is authorized with cluster-admin permissions.
An accessible image registry for various operator images (ex. hub.docker.com, quay.io) and be logged in to your command line environment.
- example.com is used as the registry Docker Hub namespace in these examples. Replace it with another value if using a different registry or namespace.
- Authentication and certificates if the registry is private or uses a custom CA.

Steps

Create a project directory for your project and initialize the project:

mkdir memcached-operator
cd memcached-operator
operator-sdk init --domain example.com --repo github.com/example/memcached-operator

Note If your local environment is Apple Silicon (darwin/arm64) use the go/v4 plugin which provides support for this platform by adding to the init subCommand the flag --plugins=go/v4

Create a simple Memcached API:

operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource --controller

Build and push your operator’s image:

make docker-build docker-push IMG="example.com/memcached-operator:v0.0.1"

OLM deployment

Install OLM:

operator-sdk olm install

Bundle your operator, then build and push the bundle image (defaults to example.com/memcached-operator-bundle:v0.0.1):

make bundle IMG="example.com/memcached-operator:v0.0.1"
make bundle-build bundle-push BUNDLE_IMG="example.com/memcached-operator-bundle:v0.0.1"

Run your bundle. If your bundle image is hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

operator-sdk run bundle <some-registry>/memcached-operator-bundle:v0.0.1

Create a sample Memcached custom resource:

$ kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com/memcached-sample created

Uninstall the operator:

operator-sdk cleanup memcached-operator

Direct deployment

Deploy your operator:

make deploy IMG="example.com/memcached-operator:v0.0.1"

Create a sample Memcached custom resource:

$ kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com/memcached-sample created

Uninstall the operator:

make undeploy

Run locally (outside the cluster)

This is recommended ONLY for development purposes

Run the operator:

make install run

In a new terminal tab/window, create a sample Memcached custom resource:

$ kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com/memcached-sample created

Stop the operator by pressing ctrl+c in the terminal tab or window the operator is running in

Next Steps

Read the full tutorial for an in-depth walkthrough of building a Go operator.

Docs: Adding Owner References for Existing Resources

Mon, 01 Jan 0001 00:00:00 +0000

Owner references are automatically injected only during creation of resources. Enabling owner reference injection will not update objects created while owner reference injection is disabled

This guide will demonstrate how to retroactively set owner references for existing resources.

A GET request to the owning resource will provide the necessary data to construct an ownerReference or an annotation.

$ kubectl get memcacheds.cache.example.com -o yaml

Example Response (Abbreviated):

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: example-memcached
  namespace: default
  uid: 2a94ff2b-84e0-40ce-8b5e-2b7e4d2bc0e2

kubectl edit can be used to update the resources by hand. See below for example ownerReference and annotations.

For objects in the same namespace as the Owner (CR)

Dependent resources within the same namespace as the owning CR are tracked with the ownerReference field.

ownerReference structure:

apiVersion: {group}/{version}
kind: {kind}
name: {metadata.name}
uid: {metadata.uid}

Example ownerReference:

metadata:
  ...(snip)
  ownerReferences:
    - apiVersion: cache.example.com/v1alpha1
      kind: Memcached
      name: example-memcached
      uid: ad834522-d9a5-4841-beac-991ff3798c00

For objects which are NOT in the same namespace as the Owner (CR)

An annotation is used instead of an ownerReference if the dependent resource is in a different namespace than the CR, or the dependent resource is a cluster level resource.

annotation structure:

operator-sdk/primary-resource: {metadata.namespace}/{metadata.name}
operator-sdk/primary-resource-type: {kind}.{group}

NOTE: The {group} can be found by splitting the apiVersion metadata of the CR, into group and version. As an example, apiVersion: cache.example.com/v1alpha1 in the config/samples directory gives us the group cache.example.com.

Example Annotation:

metadata:
  ...(snip)
  annotations:
    operator-sdk/primary-resource: default/example-memcached
    operator-sdk/primary-resource-type: Memcached.cache.example.com

Migration using Ansible assets

If you have many resources to update, it may be easier to use the following Ansible assets, which should be considered an example rather than an officially supported workflow.

To use these assets, create a vars.yml as specified below and copy playbook.yml and each_resource.yml into the same directory. Execute the playbook with:

$ ansible-playbook -i localhost playbook.yml

vars.yml

This file should be created by the user to configure the playbook, and needs to contain:

owning_resource
- apiVersion
- kind
- name
- namespace
resources_to_own (list): For each resource, specify:
- name
- namespace (if applicable)
- apiVersion
- kind

owning_resource:
  apiVersion: cache.example.com/v1alpha1
  kind: Memcached
  name: example-memcached
  namespace: default

resources_to_own:
  - name: example-memcached-memcached
    namespace: default
    apiVersion: apps/v1
    kind: Deployment
  - name: example-memcached
    apiVersion: v1
    kind: Namespace

playbook.yml

This file can be used as-is without user adjustments.

- hosts: localhost

  tasks:
    - name: Import user variables
      include_vars: vars.yml
    - name: Retrieve owning resource
      kubernetes.core.k8s_info:
        api_version: "{{ owning_resource.apiVersion }}"
        kind: "{{ owning_resource.kind }}"
        name: "{{ owning_resource.name }}"
        namespace: "{{ owning_resource.namespace }}"
      register: extra_owner_data

    - name: Ensure resources are owned
      include_tasks: each_resource.yml
      loop: "{{ resources_to_own }}"
      vars:
        to_be_owned: '{{ q("kubernetes.core.k8s",
          api_version=item.apiVersion,
          kind=item.kind,
          resource_name=item.name,
          namespace=item.namespace
        ).0 }}'
        owner_reference:
          apiVersion: "{{ owning_resource.apiVersion }}"
          kind: "{{ owning_resource.kind }}"
          name: "{{ owning_resource.name }}"
          uid: "{{ extra_owner_data.resources[0].metadata.uid }}"

`each_resource.yml`

This file can be used as-is without user adjustments.

- name: Patch resource with owner reference
  when:
    - to_be_owned.metadata.namespace is defined
    - to_be_owned.metadata.namespace == owning_resource.namespace
    - (to_be_owned.metadata.ownerReferences is not defined) or
      (owner_reference not in to_be_owned.metadata.ownerReferences)
  kubernetes.core.k8s:
    state: present
    resource_definition:
      apiVersion: "{{ to_be_owned.apiVersion }}"
      kind: "{{ to_be_owned.kind }}"
      metadata:
        name: "{{ to_be_owned.metadata.name }}"
        namespace: "{{ to_be_owned.metadata.namespace }}"
        ownerReferences: "{{ (to_be_owned.metadata.ownerReferences | default([])) + [owner_reference] }}"

- name: Patch resource with owner annotation
  when: to_be_owned.metadata.namespace is not defined or to_be_owned.metadata.namespace != owning_resource.namespace
  kubernetes.core.k8s:
    state: present
    resource_definition:
      apiVersion: "{{ to_be_owned.apiVersion }}"
      kind: "{{ to_be_owned.kind }}"
      metadata:
        name: "{{ to_be_owned.metadata.name }}"
        namespace: "{{ to_be_owned.metadata.namespace | default(omit)}}"
        annotations:
          operator-sdk/primary-resource: "{{ owning_resource.namespace }}/{{ owning_resource.name }}"
          operator-sdk/primary-resource-type: "{{ owning_resource.kind }}.{{ owning_resource.apiVersion.split('/')[0] }}"

Docs: Ansible Based Operator Scaffolding

Mon, 01 Jan 0001 00:00:00 +0000

A new Ansible operator project can be created using a command that looks like the following:

operator-sdk init --plugins ansible \
  --domain=my.domain \
  --group=apps --version=v1alpha1 --kind=AppService \
  --generate-playbook \
  --generate-role

The new project directory has many generated folders and files. For further information check the Project Layout doc.

The Deployment

The default Deployment manifest generated for the operator can be found in the config/manager/manager.yaml file. By default, the Deployment is named as ‘controller-manager’. It contains a single container named ‘manager’, and it may pick up a sidecar patch from the config/default directory. The Deployment will create a single Pod.

For the container in the Pod, there are a few things to note. The default Deployment contains a placeholder for the container image to use, so you cannot create a meaningful operator using the YAML file directly. To deploy the operator, you will run make deploy IMG=<IMG>. The image name and tag are then patched using kustomize.

The Volume Mount Path

The default EmptyDir volume mounted at /tmp/ansible-operator/runner is used to serve the input directory in ansible-runner’s terms. The mount path can NOT be changed to other paths, or else the Operator will fail to communicate with ansible-runner.

The Environment Variables

You can customize the behavior of the Ansible operator by specifying the environment variables for the container. Please refer to the Ansible Documentation for a list of environment variables that can be used to tune the behavior of the Ansible engine.

In addition to the Ansible environment variables, Operator SDK also support some special environment variables:

WATCH_NAMESPACE: This is the namespace your operator will watch for resource changes, i.e. resource create, update or delete operations. In the scaffolded operator Deployment, this is set to the same namespace in which the operator is deployed. This variable can be set to one of the following types of values:
- ‘': An empty string means that the operator will watch all namespaces. This is the default value if the WATCH_NAMESPACE environment variable is not set. It is especially useful for watching cluster-scoped resources.
- foo: The operator will watch the namespace named foo. This is the setting you will use if you are operating a namespaced resource which is deployed into a specific namespace.
- foo,bar: The operator checks the value of the environment variable and realizes that it is a comma-separated list. This means the operator will watch for resources in each of the listed namespaces.
ANSIBLE_DEBUG_LOGS: A boolean value for toggling the Ansible output during reconciliation. When set to True, the operator dumps the Ansible result into its standard output.
ANSIBLE_ROLES_PATH: The parent path(s) for the Ansible roles. When there are more than one path to set, you can use “:” to separate them. Given a path /opt/foo and a role name bar, the Ansible operator will check if the Ansible role can be found in either /opt/foo/bar or /opt/foo/roles/bar.

This value overrides the setting from the ansible-roles-path flag.
ANSIBLE_COLLECTIONS_PATH: The base path for the Ansible collections which defaults to ~/.ansible/collections or /usr/share/ansible/collections when ANSIBLE_COLLECTIONS_PATH is not explicitly specified. When a fully qualified collection name in the watches file, the Ansible operator checks if the specified collection can found under the base path that can be customized using this variable. Suppose you have ANSIBLE_COLLECTIONS_PATH set to /foo and the fully qualified collection name set to example.com.bar, the Ansible operator searches for the roles under /foo/ansible_collections/example/com/roles/bar.

This value takes precedence over the --ansible-collections-path flag.
MAX_CONCURRENT_RECONCILES_<kind>_<group>: This specifies the maximum number of concurrent reconciliations for the operator. It defaults to the number of CPUs. You can adjust this based on the cluster resources.
WORKER_<kind>_<group>: Deprecated. Use MAX_CONCURRENT_RECONCILES_<kind>_<group> instead.
ANSIBLE_VERBOSITY_<kind>_<group>: This is used to customize the verbosity of the ansible-runner command. The default value is 2. The value must be no less than 0 and no greater than 7. This value takes precedence over the global --ansible-verbosity flag, and it can be overridden by the per-resource annotation named ansible.operator-sdk/verbosity.

Docs: Ansible Operator Watches

Mon, 01 Jan 0001 00:00:00 +0000

The Watches file contains a list of mappings from custom resources, identified by it’s Group, Version, and Kind, to an Ansible Role or Playbook. The Operator expects this mapping file in a predefined location: /opt/ansible/watches.yaml These resources, as well as child resources (determined by owner references) will be monitored for updates and cached.

group: The group of the Custom Resource that you will be watching.
version: The version of the Custom Resource that you will be watching.
kind: The kind of the Custom Resource that you will be watching.
role (default): Specifies a role to be executed. This field is mutually exclusive with the “playbook” field. This field can be:
- an absolute path to a role directory.
- a relative path within one of the directories specified by ANSIBLE_ROLES_PATH environment variable or ansible-roles-path flag.
- a relative path within the current working directory, which defaults to /opt/ansible/roles.
- a fully qualified collection name of an installed Ansible collection. Ansible collections are installed to ~/.ansible/collections or /usr/share/ansible/collections by default. If they are installed elsewhere, use the ANSIBLE_COLLECTIONS_PATH environment variable or the ansible-collections-path flag
playbook: This is the playbook name that you have added to the container. This playbook is expected to be simply a way to call roles. This field is mutually exclusive with the “role” field. When running locally, the playbook is expected to be in the current project directory.
vars: This is an arbitrary map of key-value pairs. The contents will be passed as extra_vars to the playbook or role specified for this watch.
reconcilePeriod (optional): The maximum interval that the operator will wait before beginning another reconcile, even if no watched events are received. When an operator watches many resources, each reconcile can become expensive, and a low value here can actually reduce performance. Typically, this option should only be used in advanced use cases where watchDependentResources is set to False and when is not possible to use the watch feature. E.g To manage external resources that don’t emit Kubernetes events. The format for the duration string is a sequence of decimal numbers, each with optional fraction and a unit suffix, such as “300ms”, “1.5h” or “2h45m”. Valid time units are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”. Defaults to 10 hours.
manageStatus (optional): When true (default), the operator will manage the status of the CR generically. Set to false, the status of the CR is managed elsewhere, by the specified role/playbook or in a separate controller.
blacklist: A list of child resources (by GVK) that will not be watched or cached.

An example Watches file:

---
# Simple example mapping Foo to the Foo role
- version: v1alpha1
  group: foo.example.com
  kind: Foo
  role: Foo

# Simple example mapping Bar to a playbook
- version: v1alpha1
  group: bar.example.com
  kind: Bar
  playbook: playbook.yml

# More complex example for our Baz kind
# Here we will disable requeuing and be managing the CR status in the playbook,
# and specify additional variables.
- version: v1alpha1
  group: baz.example.com
  kind: Baz
  playbook: baz.yml
  manageStatus: False
  vars:
    foo: bar

# ConfigMaps owned by a Memcached CR will not be watched or cached.
- version: v1alpha1
  group: cache.example.com
  kind: Memcached
  role: /opt/ansible/roles/memcached
  blacklist:
    - group: ""
      version: v1
      kind: ConfigMap

# Example usage with a role from an installed Ansible collection
- version: v1alpha1
  group: bar.example.com
  kind: Bar
  role: myNamespace.myCollection.myRole

# Example filtering of resources with specific labels
- version: v1alpha1
  group: bar.example.com
  kind: Bar
  playbook: playbook.yml
  selector:
    matchLabels:
      foo: bar
    matchExpressions:
      - {key: foo, operator: In, values: [bar]}
      - {key: baz, operator: Exists, values: []}

The advanced features can be enabled by adding them to your watches file per GVK. They can go below the group, version, kind and playbook or role.

Some features can be overridden per resource via an annotation on that CR. The options that are overridable will have the annotation specified below.

Feature	Yaml Key	Description	Annotation for override	default	Documentation
Reconcile Period	`reconcilePeriod`	time between reconcile runs for a particular CR	ansible.sdk.operatorframework.io/reconcile-period	10h
Manage Status	`manageStatus`	Allows the ansible operator to manage the conditions section of each resource’s status section.		true
Watching Dependent Resources	`watchDependentResources`	Allows the ansible operator to dynamically watch resources that are created by ansible		true	dependent watches
Watching Cluster-Scoped Resources	`watchClusterScopedResources`	Allows the ansible operator to watch cluster-scoped resources that are created by ansible		false
Max Runner Artifacts	`maxRunnerArtifacts`	Manages the number of artifact directories that ansible runner will keep in the operator container for each individual resource.	ansible.sdk.operatorframework.io/max-runner-artifacts	20
Finalizer	`finalizer`	Sets a finalizer on the CR and maps a deletion event to a playbook or role			finalizers
Selector	`selector`	Identifies a set of objects based on their labels		None Applied	Labels and Selectors
Automatic Case Conversion	`snakeCaseParameters`	Determines whether to convert the CR spec from camelCase to snake_case before passing the contents to Ansible as extra_vars		true
Watching Annotations Changes	`watchAnnotationsChanges`	Allows the ansible operator to trigger reconciliations on annotations changes on watched resources		false

Example

---
- version: v1alpha1
  group: app.example.com
  kind: AppService
  playbook: playbook.yml
  maxRunnerArtifacts: 30
  reconcilePeriod: 5s
  manageStatus: False
  watchDependentResources: False
  snakeCaseParameters: False
  finalizer:
    name: app.example.com/finalizer
    vars:
      state: absent

Note: By using the command operator-sdk add api you are able to add additional CRDs to the project API, which can aid in designing your solution using concepts such as encapsulation, single responsibility principle, and cohesion, which could make the project easier to read, debug, and maintain. With this approach, you are able to customize and optimize the configurations more specifically per GVK via the watches.yaml file.

Example:

---
- version: v1alpha1
  group: app.example.com
  kind: AppService
  playbook: playbook.yml
  maxRunnerArtifacts: 30
  reconcilePeriod: 5s
  manageStatus: False
  watchDependentResources: False
  finalizer:
    name: app.example.com/finalizer
    vars:
      state: absent

- version: v1alpha1
  group: app.example.com
  kind: Database
  playbook: playbook.yml
  watchDependentResources: True
  manageStatus: True

Docs: Adding Admission Webhooks to an Ansible-based Operator

Mon, 01 Jan 0001 00:00:00 +0000

For general background on what admission webhooks are, why to use them, and how to build them, please refer to the official Kubernetes documentation on Extensible Admission Controllers

This guide will assume that you understand the above content, and that you have an existing admission webhook server. You will likely need to make a few modifications to the webhook server container.

When integrating an admission webhook server into your Ansible-based Operator, we recommend that you deploy it as a sidecar container alongside your operator. This allows you to make use of the proxy server that the operator deploys, as well as the cache that backs it.

Ensuring the webhook server uses the caching proxy

When an Ansible-based Operator runs, it creates a Kubernetes proxy server and serves it on http://localhost:8888. This proxy server does not require any authorization, so all you need to do to make use of the proxy is ensure that your Kubernetes client is pointing at http://localhost:8888 and that it does not attempt to verify SSL. If you use the default in-cluster configuration, you will be hitting the real API server and will not get caching for free.

Deploying the webhook server

Create a new file called config/default/manager_webhook_patch.yaml with the following content (making sure to replace the image reference placeholder string):

# This patch injects a sidecar container which is an admission webhook for the
# controller manager.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: controller-manager
  namespace: system
spec:
  template:
    spec:
      containers:
      - name: webhook
        # Replace this with the built image name
        image: "REPLACE_WEBHOOK_IMAGE"
        volumeMounts:
        - mountPath: /etc/tls/
          name: webhook-cert
      volumes:
      # This assumes there is a secret called webhook-cert containing TLS certificates
      # Projects like cert-manager can create these certificates
      - name: webhook-cert
        secret:
          secretName: webhook-cert

Then, update config/default/kustomization.yaml to include this patch:

patchesStrategicMerge:
- manager_webhook_patch.yaml # Add this line

Now, when deploying the operator with make deploy, your webhook server will run alongside the operator, but Kubernetes will not yet call the webhooks before resources can be created. In order to let Kubernetes know about your webhooks, you must create specific API resources.

Making Kubernetes call your webhooks

In order to make your webhooks callable at all, first you must create a Service that points at your webhook server. Below is a sample service that creates a Service named my-operator-webhook, that will send traffic on port 443 to port 5000 in a Pod that matches the selector name=my-operator. Modify these values to match your environment.

apiVersion: v1
kind: Service
metadata:
  name: my-operator-webhook
spec:
  ports:
  - name: webhook
    port: 443
    protocol: TCP
    # Change targetPort to match the port your server is listening on
    targetPort: 5000
  selector:
    # Change this selector to match the labels on your operator pod
    name: my-operator
  type: ClusterIP

Now that you have a Service directing traffic to your webhook server, you will need to create MutatingWebhookConfiguration or ValidatingWebhookConfiguration objects (depending on what type of webhook you have deployed), which will tell Kubernetes to send certain API requests through your webhooks before writing to etcd.

Below are examples of both MutatingWebhookConfiguration and ValidatingWebhookConfiguration objects, which will tell Kubernetes to call the my-operator-webhook service when samples.example.com Example resources are created. The mutating webhook is served on the /mutating path in my example webhook server, and the validating webhook is served on /validating. Update these values as needed to reflect your environment and desired behavior. These objects are thoroughly documented in the official Kubernetes documentation on Extensible Admission Controllers

---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: mutating.example.com
webhooks:
- name: "mutating.example.com"
  rules:
  - apiGroups:   ["samples.example.com"]
    apiVersions: ["*"]
    operations:  ["CREATE"]
    resources:   ["examples"]
    scope:       "Namespaced"
  clientConfig:
    service:
      # Replace this with the namespace your service is in
      namespace: REPLACE_NAMESPACE
      name: my-operator-webhook
      path: /mutating
  admissionReviewVersions: ["v1"]
  sideEffects: None
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: validating.example.com
webhooks:
- name: validating.example.com
  rules:
  - apiGroups:   ["samples.example.com"]
    apiVersions: ["*"]
    operations:  ["CREATE"]
    resources:   ["examples"]
    scope:       "Namespaced"
  clientConfig:
    service:
      # Replace this with the namespace your service is in
      namespace: REPLACE_NAMESPACE
      name: my-operator-webhook
      path: /validating
  admissionReviewVersions: ["v1"]
  failurePolicy: Fail
  sideEffects: None

If these resources are configured properly you will now have an admissions webhook that can reject or mutate incoming resources before they are written to the Kubernetes database.

Summary

To deploy an existing admissions webhook to validate or mutate your Kubernetes resources alongside an Ansible-based Operator, you must

Configure your admissions webhook to use the proxy server running on http://localhost:8888 in the operator pod
Add the webhook container to your operator deployment
Create a Service pointing to your webhook
Make sure your webhook is reachable via the Service over https
Create MutatingWebhookConfiguration or ValidatingWebhookConfiguration mapping the resource you want to mutate/validate to the Service you created

Docs: Changelog

Mon, 01 Jan 0001 00:00:00 +0000

The Operator SDK project tracks changes across releases by maintaining a live changelog. This changelog is compiled via fragments located in changelog/fragments

Contributors are asked to add these changelog fragments when creating pull requests by following this template.

The changlog fragments document the following information:

Description of Changes
Types of Changes
- One of: addition, change, deprecation, removal, bugfix
Migration Guide for Changes
- These are required for breaking changes, and essential (but not required) for scaffolding-related changes.

Docs: Issue Lifecycle

Mon, 01 Jan 0001 00:00:00 +0000

The Operator SDK project tracks bugs, feature requests, and questions with GitHub issues.

Triage

Each week, there is a triage meeting to review new issues. Each issue that has been filed since the previous meeting is discussed, GitHub labels are applied, and the issue is added to a Milestone. Additionally, anyone can request that a previously triaged issue can be retriaged.

Grooming

Following a release, there is a grooming meeting to review issues that are desired in the next release. Issues are discussed in the following order:

Issues in the next release milestone
Issues labeled as priority/important-soon
Issues in other milestones/backlog if specifically requested

Docs: Opening Pull Requests

Mon, 01 Jan 0001 00:00:00 +0000

Changes to Operator SDK are submitted via Github Pull Request (PR) to the relevant repository. Most PRs will be to the main Operator SDK repo.

PR Checklist

Add tests for your change, and ensure they run and pass in CI.
Add a changelog entry if necessary.
Add relevant documentation.
Rebase your commits on master and squash them into a single commit.
Write a concise commit message that references the issue number.
Push your commit to your fork and open a pull request.

Review

Before a pull request can be merged, tests must pass in CI and it must be reviewed. A PR must be approved by 2 reviewers, one of which must be at least at least a reviewer and one of which must be at least an approver, per the Operator Framework community guidelines.

Please feel free to message the developers to get eyes on your PR, whether through @‘ing on the PR itself, the #operator-sdk-dev channel on Kubernetes slack, or by attending the Operator SDK triage meeting.

Docs: Testing Operator Deployment with OLM

Mon, 01 Jan 0001 00:00:00 +0000

This document discusses the behavior of operator-sdk <run|cleanup> subcommands related to OLM deployment, and assumes you are familiar with OLM, related terminology, and have read the SDK-OLM integration design proposal.

Caveats

run bundle, run bundle-upgrade, run packagemanifests, and cleanup are intended to be used for testing purposes only, since these commands create a transient image registry that should not be used in production. Typically a registry is deployed separately and a set of catalog manifests are created in the cluster to inform OLM of that registry and which Operator versions it can deploy and where to deploy the Operator.
run bundle and run packagemanifests can only deploy one Operator and one version of that Operator at a time, and run bundle-upgrade can only upgrade one Operator and one version of that Operator at a time, hence their intended purpose being testing only.
If testing a bundle or catalog whose image will be hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

`operator-sdk run bundle` command overview

operator-sdk run bundle assumes OLM is already installed and running on your cluster. It also assumes that your Operator has a valid bundle. See the [creating a bundle][creating-bundle] guide for more information. See the CLI overview for commands to work with an OLM installation and generate a bundle.

operator-sdk run bundle <bundle-image> [--index-image=] [--kubeconfig=] [--namespace=] [--timeout=] [--install-mode=(AllNamespace|OwnNamespace|SingleNamespace=)]

Let’s look at the configuration shared between run bundle, run packagemanifests, run bundle-upgrade and cleanup:

kubeconfig: the local path to a kubeconfig. This uses well-defined default loading rules to load the config if empty.
namespace: the cluster namespace in which Operator resources are created. This namespace must already exist in the cluster. This is an optional field which will default to the kubeconfig context if not provided.
timeout: a time string dictating the maximum time that run can run. The command will return an error if the timeout is exceeded.

Let’s look at the anatomy of the run bundle configuration model:

bundle-image: specifies the Operator bundle image, this is a required parameter. The bundle image must be pullable.
index-image: specifies an index image in which to inject the given bundle. This is an optional field which will default to quay.io/operator-framework/opm:latest
install-mode: specifies which supported installMode should be used to create an OperatorGroup by configuring its spec.targetNamespaces field. The InstallModeType string passed must be marked as “supported” in the CSV being installed.
- This option understands the following strings (assuming your CSV does as well):
  - AllNamespaces: the Operator will watch all namespaces (cluster-scoped Operators). This is the default.
  - OwnNamespace: the Operator will watch its own namespace (from namespace or the kubeconfig default).
  - SingleNamespace="my-ns": the Operator will watch a namespace, not necessarily its own.
- This is an optional parameter, but if the CSV does not support AllNamespaces then this parameter becomes required to instruct run bundle with the appropriate InstallModeType.

`operator-sdk run packagemanifests` command overview

operator-sdk run packagemanifests assumes OLM is already installed and running on your cluster, and that your Operator has a valid package manifests format. See the CLI overview for commands to work with an OLM installation and generate a package manifests format.

operator-sdk run packagemanifests <packagemanifests-root-dir> [--version=] [--kubeconfig=] [--namespace=] [--timeout=] [--install-mode=(AllNamespace|OwnNamespace|SingleNamespace=)]

Let’s look at the configuration shared between run bundle, run packagemanifests and cleanup:

kubeconfig: the local path to a kubeconfig. This uses well-defined default loading rules to load the config if empty.
namespace: the cluster namespace in which Operator resources are created. This namespace must already exist in the cluster. This is an optional field which will default to the kubeconfig context if not provided.
timeout: a time string dictating the maximum time that run can run. The command will return an error if the timeout is exceeded.

Let’s look at the anatomy of the run packagemanifests configuration model:

packagemanifests-root-dir: a directory containing the Operator’s package manifests, this is a required parameter.
install-mode: specifies which supported installMode should be used to create an OperatorGroup by configuring its spec.targetNamespaces field. The InstallModeType string passed must be marked as “supported” in the CSV being installed.
- This option understands the following strings (assuming your CSV does as well):
  - AllNamespaces: the Operator will watch all namespaces (cluster-scoped Operators). This is the default.
  - OwnNamespace: the Operator will watch its own namespace (from namespace or the kubeconfig default).
  - SingleNamespace="my-ns": the Operator will watch a namespace, not necessarily its own.
- This is an optional parameter, but if the CSV does not support AllNamespaces then this parameter becomes required to instruct run packagemanifests with the appropriate InstallModeType.
version: the version of the Operator to deploy. It must be a semantic version, ex. 0.0.1. This version must match the version of the CSV manifest found in manifests-dir, e.g. packagemanifests/0.0.1 in an Operator SDK project.

`operator-sdk run bundle-upgrade` command overview

operator-sdk run bundle-upgrade assumes OLM is already installed and running on your cluster and that the Operator has a valid bundle. It also assumes that the previous version of the Operator was either deployed on the cluster using run bundle command or traditionally via OLM. Another assumption of this command is that the newer operator bundle should not exist in the index image, if the previous version of the operator bundle was installed traditionally using OLM. This will cause the registry pod to fail as the bundle is already added to the index that provides package and csv. See the CLI overview for commands to work with an OLM installation and generate a bundle.

operator-sdk run bundle-upgrade <bundle-image> [--kubeconfig=] [--namespace=] [--timeout=]

Let’s look at the anatomy of the run bundle-upgrade configuration model:

bundle-image: specifies the Operator bundle image, this is a required parameter. The bundle image must be pullable.

`operator-sdk cleanup` command overview

operator-sdk cleanup assumes an Operator was deployed using run bundle or run packagemanifests.

operator-sdk cleanup <operatorPackageName> [--delete-all=] [--delete-crds=] [--delete-operator-groups=] [--kubeconfig=] [--namespace=] [--timeout=]

Let’s look at the configuration shared between run bundle, run packagemanifests and cleanup:

kubeconfig: the local path to a kubeconfig. This uses well-defined default loading rules to load the config if empty.
namespace: the cluster namespace in which Operator resources are created. This namespace must already exist in the cluster. This is an optional field which will default to the kubeconfig context if not provided.
timeout: a time string dictating the maximum time that run can run. The command will return an error if the timeout is exceeded.

Let’s look at the anatomy of the cleanup configuration model:

operatorPackageName: the Operator’s package name which you want to remove from the cluster, e.g. memcached-operator. This is a required parameter.
delete-all: a boolean indicating to enable all the delete flags that are present. This is an optional field which will default to true if not provided. If set to true, it will enable all the delete flags to be true. If set to false, it will enable specific delete flags.
delete-crds: a boolean indicating to delete all owned CRDs and CRs. This is an optional field which will default to false if not provided. If set to true, owned CRDs and CRs will be deleted.
delete-operator-groups: a boolean indicating to delete all operator groups. This is an optional field which will default to false if not provided. If set to true, operator groups will be deleted.

Operator SDK – Operator SDK Documentation

Docs: Operator Best Practices

Development

Summary

Running On-Cluster

Summary:

Docs: OLM Integration Bundle Quickstart

Prerequisites

Steps

Next Steps

Docs: OLM Integration Bundle Tutorial

Setup

Enabling OLM

Creating a bundle

Deploying an Operator with OLM

Testing bundles

Upgrading a bundle to a newer version

Upgrading a bundle that was installed traditionally using OLM

Deploying bundles in production

Further reading

Docs: Development

Installation

Prerequisites

Download Operator SDK

Build the Operator SDK CLI

Testing

Releasing

Continuous Integration (CI)

Adding new architectures

Docs: Installation Guide

Install operator-sdk

Additional Prerequisites

Docs: Installation Guide

Install operator-sdk

Additional Prerequisites

Docs: Installation Guide

Install operator-sdk

Additional Prerequisites

Docs: Backport Policy

Process

Docs: Common recommendations and suggestions

Overview

Common Recommendations

Develop idempotent reconciliation solutions

Understanding Kubernetes APIs

Avoid a design solution where more than one Kind is reconciled by the same controller

Ideally Operators does not manage other Operators

What does it mainly mean:

Other common suggestions

Docs: OLM Integration Package Manifests Quickstart

Creating package manifests

Deploying an Operator with OLM

Testing package manifests

Migrating packagemanifests to bundles

Docs: Project Layout

Operator SDK Project Layout

Common Base

Ansible

Golang

Helm

Docs: Quickstart for Ansible-based Operators

Prerequisites

Steps

OLM deployment

Direct deployment

Next Steps

Docs: Cheat Sheet

Common commands and options

To create bundles, catalogs, and develop for OLM

Updating bundle channels

To test your projects

Docs: Resource pruning

Overview

operator-lib prune library

Pruning Configuration

Pruning Execution

Pruning Strategies

maxcount Strategy

maxage Strategy

Pruning Customization

Install `operator-sdk`

Install `operator-sdk`

Install `operator-sdk`

5. Fast-forward the `latest` branch

`scorecard-test-kuttl` image releases