Operator SDK – Helm

Docs: Installation Guide

Mon, 01 Jan 0001 00:00:00 +0000

Install `operator-sdk`

Follow the steps in the installation guide to learn how to install the operator-sdk CLI tool.

Additional Prerequisites

docker version 17.03+.
kubectl and access to a Kubernetes cluster of a compatible version.

Docs: Quickstart for Helm-based Operators

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks through an example of building a simple nginx-operator powered by Helm using tools and libraries provided by the Operator SDK.

Prerequisites

Go through the installation guide.
Make sure your user is authorized with cluster-admin permissions.
An accessible image registry for various operator images (ex. hub.docker.com, quay.io) and be logged in to your command line environment.
- example.com is used as the registry Docker Hub namespace in these examples. Replace it with another value if using a different registry or namespace.
- Authentication and certificates if the registry is private or uses a custom CA.

Steps

Create a project directory for your project and initialize the project:

mkdir nginx-operator
cd nginx-operator
operator-sdk init --domain example.com --plugins helm

Create a simple nginx API using Helm’s built-in chart boilerplate (from helm create):

operator-sdk create api --group demo --version v1alpha1 --kind Nginx

Build and push your operator’s image:

make docker-build docker-push IMG="example.com/nginx-operator:v0.0.1"

OLM deployment

Install OLM:

operator-sdk olm install

Bundle your operator, then build and push the bundle image (defaults to example.com/nginx-operator-bundle:v0.0.1):

make bundle IMG="example.com/nginx-operator:v0.0.1"
make bundle-build bundle-push IMG="example.com/nginx-operator:v0.0.1"

Run your bundle. If your bundle image is hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

operator-sdk run bundle example.com/nginx-operator-bundle:v0.0.1

Create a sample Nginx custom resource:

$ kubectl apply -f config/samples/demo_v1alpha1_nginx.yaml
nginx.demo.example.com/nginx-sample created

Uninstall the operator:

operator-sdk cleanup nginx-operator

Direct deployment

Deploy your operator:

make deploy IMG="example.com/nginx-operator:v0.0.1"

Create a sample Nginx custom resource:

$ kubectl apply -f config/samples/demo_v1alpha1_nginx.yaml
nginx.demo.example.com/nginx-sample created

Uninstall the operator:

make undeploy

Next Steps

Read the full tutorial for an in-depth walkthrough of building a Helm operator.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The motivations for the new layout are related to bringing more flexibility to users and part of the process to Integrating Kubebuilder and Operator SDK. Because of this integration you may be referred to the Kubebuilder documentation https://book.kubebuilder.io/ for more information about certain topics. When using this document just remember to replace $ kubebuilder <command> with $ operator-sdk <command>.

Note: It is recommended that you have your project upgraded to the latest SDK v1.y release version before following the steps in this guide to migrate to the new layout. However, the steps might work from previous versions as well. In this case, if you find an issue which is not covered here then check the previous Migration Guides which might help out.

What was changed

The deploy directory was replaced with the config directory including a new layout of Kubernetes manifests files:
- CRD manifests in deploy/crds/ are now in config/crd/bases
- CR manifests in deploy/crds/ are now in config/samples
- Controller manifest deploy/operator.yaml is now in config/manager/manager.yaml
- RBAC manifests in deploy are now in config/rbac/
build/Dockerfile is moved to Dockerfile in the project root directory

What is new

Scaffolded projects now use:

kustomize to manage Kubernetes resources needed to deploy your operator
A Makefile with helpful targets for build, test, and deployment, and to give you flexibility to tailor things to your project’s needs
Updated metrics configuration using kube-auth-proxy, a --metrics-bind-address flag, and kustomize-based deployment of a Kubernetes Service and prometheus operator ServiceMonitor
Preliminary support for CLI plugins. For more info see the [plugins design document][plugins-phase1-design-doc]
A PROJECT configuration file to store information about GVKs, plugins, and help the CLI make decisions

Generated files with the default API versions:

apiextensions/v1 for generated CRDs (apiextensions/v1beta1 was deprecated in Kubernetes 1.16 and will be removed in 1.22)
admissionregistration.k8s.io/v1 for webhooks (admissionregistration.k8s.io/v1beta1 was deprecated in Kubernetes 1.16 and will be removed in 1.22 )

How to migrate

The easy migration path is to initialize a new project, re-recreate APIs, then copy pre-v1.0.0 configuration files into the new project.

Prerequisites

Go through the installation guide.
Make sure your user is authorized with cluster-admin permissions.
An accessible image registry for various operator images (ex. hub.docker.com, quay.io) and be logged in to your command line environment.
- example.com is used as the registry Docker Hub namespace in these examples. Replace it with another value if using a different registry or namespace.
- Authentication and certificates if the registry is private or uses a custom CA.

Creating a new project

In Kubebuilder-style projects, CRD groups are defined using two different flags (--group and --domain).

When we initialize a new project, we need to specify the domain that all APIs in our project will share, so before creating the new project, we need to determine which domain we’re using for the APIs in our existing project.

To determine the domain, look at the spec.group field in your CRDs in the deploy/crds directory.

The domain is everything after the first DNS segment. Using demo.example.com as an example, the --domain would be example.com.

So let’s create a new project with the same domain (example.com):

mkdir nginx-operator
cd nginx-operator
operator-sdk init --plugins=helm --domain=example.com

Now that we have our new project initialized, we need to re-create each of our APIs. Using our API example from earlier (demo.example.com), we’ll use demo for the --group flag.

For --version and --kind, we use spec.versions[0].name and spec.names.kind, respectively.

For each API in the existing project, run:

operator-sdk create api \
    --group=demo \
    --version=<version> \
    --kind=<Kind> \
    --helm-chart=<path_to_existing_project>/helm-charts/<chart>

Migrating your Custom Resource samples

Update the CR manifests in config/samples with the values of the CRs in your existing project which are in deploy/crds/<group>_<version>_<kind>_cr.yaml

Migrating `watches.yaml`

Check if you have custom options in the watches.yaml file of your existing project. If so, update the new watches.yaml file to match. In our example, it will look like:

# Use the 'create api' subcommand to add watches to this file.
- group: example.com
  version: v1alpha1
  kind: Nginx
  chart: helm-charts/nginx
#+kubebuilder:scaffold:watch

NOTE: Do not remove the +kubebuilder:scaffold:watch marker. It allows the tool to update the watches file when new APIs are created.

Checking RBAC Permissions

In your new project, roles are automatically generated in config/rbac/role.yaml. If you modified these permissions manually in deploy/role.yaml in your existing project, you need to re-apply them in config/rbac/role.yaml.

New projects are configured to watch all namespaces by default, so they need a ClusterRole to have the necessary permissions. Ensure that config/rbac/role.yaml remains a ClusterRole if you want to retain the default behavior of the new project conventions.

The following rules were used in earlier versions of helm-operator to automatically create and manage services and servicemonitors for metrics collection. If your operator’s charts don’t require these rules, they can safely be left out of the new config/rbac/role.yaml file:

  - apiGroups:
    - monitoring.coreos.com
    resources:
    - servicemonitors
    verbs:
    - get
    - create
  - apiGroups:
    - apps
    resourceNames:
    - nginx-operator
    resources:
    - deployments/finalizers
    verbs:
    - update

Updating your ServiceAccount

New Helm projects come with a ServiceAccount controller-manager in config/rbac/service_account.yaml. Your project’s RoleBinding and ClusterRoleBinding subjects, and Deployments spec.template.spec.serviceAccountName that reference a ServiceAccount already refer to this new name. When you run make deploy, your project’s name will be prepended to controller-manager, making it unique within a namespace, much like your old deploy/service_account.yaml. If you wish to use the old ServiceAccount, make sure to update all RBAC bindings and your manager Deployment.

Configuring your Operator

If your existing project has customizations in deploy/operator.yaml, they need to be ported to config/manager/manager.yaml. If you are passing custom arguments in your deployment, make sure to also update config/default/auth_proxy_patch.yaml.

Note that the following environment variables are no longer used.

OPERATOR_NAME is deprecated. It is used to define the name for a leader election config map. Operator authors should begin using --leader-election-id instead.
POD_NAME was used to enable a particular pod to hold the leader election lock when the Helm operator used the leader for life mechanism. Helm operator now uses controller-runtime’s leader with lease mechanism, and POD_NAME is no longer necessary.

Exporting metrics

If you are using metrics and would like to keep them exported you will need to configure it in the config/default/kustomization.yaml. Please see the metrics doc to know how you can perform this setup.

The default port used by the metric endpoint binds to was changed from :8383 to :8080. To continue using port 8383, specify --metrics-bind-address=:8383 when you start the operator.

Verify the migration

The project can now be deployed on cluster by running the command:

make deploy IMG=example.com/nginx-operator:v0.0.1

You can troubleshoot your deployment by checking container logs:

kubectl logs deployment.apps/nginx-operator-controller-manager -n nginx-operator-system -c manager

For further steps regarding the deployment of the operator, creation of custom resources, and cleaning up of resources, see the tutorial.

Docs: Helm Operator Tutorial

Mon, 01 Jan 0001 00:00:00 +0000

NOTE: If your project was created with an operator-sdk version prior to v1.0.0 please migrate, or consult the legacy docs.

Prerequisites

Go through the installation guide.
Make sure your user is authorized with cluster-admin permissions.
An accessible image registry for various operator images (ex. hub.docker.com, quay.io) and be logged in to your command line environment.
- example.com is used as the registry Docker Hub namespace in these examples. Replace it with another value if using a different registry or namespace.
- Authentication and certificates if the registry is private or uses a custom CA.

Overview

We will create a sample project to let you know how it works and this sample will:

Create an Nginx Deployment if it doesn’t exist
Ensure that the Deployment size is the same as specified by the Nginx CR spec

Create a new project

Use the CLI to create a new Helm-based nginx-operator project:

mkdir nginx-operator
cd nginx-operator
operator-sdk init --plugins helm --domain example.com --group demo --version v1alpha1 --kind Nginx

This creates the nginx-operator project specifically for watching the Nginx resource with APIVersion demo.example.com/v1alpha1 and Kind Nginx.

For Helm-based projects, operator-sdk init also generates the RBAC rules in config/rbac/role.yaml based on the resources that would be deployed by the chart’s default manifest. Be sure to double check that the rules generated in config/rbac/role.yaml meet the operator’s permission requirements.

To learn more about the project directory structure, see the project layout doc.

Use an existing chart

Instead of creating your project with a boilerplate Helm chart, you can also use --helm-chart, --helm-chart-repo, and --helm-chart-version to use an existing chart, either from your local filesystem or a remote chart repository.

If --helm-chart is specified, the --group, --version, and --kind flags become optional. If left unset, the default will be:

Flag	Value
domain	my.domain
group	charts
kind	deduce from the specified chart
version	v1alpha1

If --helm-chart is a local chart archive (e.g example-chart-1.2.0.tgz) or directory, it will be validated and unpacked or copied into the project.

Otherwise, the SDK will attempt to fetch the specified helm chart from a remote repository.

If a custom repository URL is not specified by --helm-chart-repo, the following chart reference formats are supported:

<repoName>/<chartName>: Fetch the helm chart named chartName from the helm chart repository named repoName, as specified in the $HELM_HOME/repositories/repositories.yaml file. Use helm repo add to configure this file.
<url>: Fetch the helm chart archive at the specified URL.

If a custom repository URL is specified by --helm-chart-repo, the only supported format for --helm-chart is:

<chartName>: Fetch the helm chart named chartName in the helm chart repository specified by the --helm-chart-repo URL.

If --helm-chart-version is not set, the SDK will fetch the latest available version of the helm chart. Otherwise, it will fetch the specified version. The option --helm-chart-version is not used when --helm-chart itself refers to a specific version, for example when it is a local path or a URL.

Note: For more details and examples run operator-sdk init --plugins helm --help.

Customize the operator logic

For this example the nginx-operator will execute the following reconciliation logic for each Nginx Custom Resource (CR):

Create an nginx Deployment if it doesn’t exist
Create an nginx Service if it doesn’t exist
Create an nginx Ingress if it is enabled and doesn’t exist
Ensure that the Deployment, Service, and optional Ingress match the desired configuration (e.g. replica count, image, service type, etc) as specified by the Nginx CR

Watch the Nginx CR

By default, the nginx-operator watches Nginx resource events as shown in watches.yaml and executes Helm releases using the specified chart:

# Use the 'create api' subcommand to add watches to this file.
- group: demo
  version: v1alpha1
  kind: Nginx
  chart: helm-charts/nginx
#+kubebuilder:scaffold:watch

Reviewing the Nginx Helm Chart

When a Helm operator project is created, the SDK creates an example Helm chart that contains a set of templates for a simple Nginx release.

For this example, we have templates for deployment, service, and ingress resources, along with a NOTES.txt template, which Helm chart developers use to convey helpful information about a release.

If you aren’t already familiar with Helm Charts, take a moment to review the Helm Chart developer documentation.

Understanding the Nginx CR spec

Helm uses a concept called values to provide customizations to a Helm chart’s defaults, which are defined in the Helm chart’s values.yaml file.

Overriding these defaults is as simple as setting the desired values in the CR spec. Let’s use the number of replicas as an example.

First, inspecting helm-charts/nginx/values.yaml, we see that the chart has a value called replicaCount and it is set to 1 by default. If we want to have 2 nginx instances in our deployment, we would need to make sure our CR spec contained replicaCount: 2.

Update config/samples/demo_v1alpha1_nginx.yaml to look like the following:

apiVersion: demo.example.com/v1alpha1
kind: Nginx
metadata:
  name: nginx-sample
spec:
  replicaCount: 2

Similarly, we see that the default service port is set to 80, but we would like to use 8080, so we’ll again update config/samples/demo_v1alpha1_nginx.yaml by adding the service port override:

apiVersion: demo.example.com/v1alpha1
kind: Nginx
metadata:
  name: nginx-sample
spec:
  replicaCount: 2
  service:
    port: 8080

As you may have noticed, the Helm operator simply applies the entire spec as if it was the contents of a values file, just like helm install -f ./overrides.yaml works.

Configure the operator’s image registry

All that remains is to build and push the operator image to the desired image registry. Your Makefile composes image tags either from values written at project initialization or from the CLI. In particular, IMAGE_TAG_BASE lets you define a common image registry, namespace, and partial name for all your image tags. Update this to another registry and/or namespace if the current value is incorrect. Afterwards you can update the IMG variable definition like so:

-IMG ?= controller:latest
+IMG ?= $(IMAGE_TAG_BASE):$(VERSION)

Once done, you do not have to set IMG or any other image variable in the CLI. The following command will build and push an operator image tagged as example.com/nginx-operator:v0.0.1 to Docker Hub:

make docker-build docker-push

Run the operator

There are three ways to run the operator:

As a Go program outside a cluster
As a Deployment inside a Kubernetes cluster
Managed by the Operator Lifecycle Manager (OLM) in bundle format

1. Run locally outside the cluster

Execute the following command, which installs your CRDs and runs the manager locally:

make install run

2. Run as a Deployment inside the cluster

By default, a new namespace is created with the name <project-name>-system, ex. nginx-operator-system, and will be used for the deployment.

Run the following to deploy the operator. This will also install the RBAC manifests from config/rbac.

make deploy

Verify that the nginx-operator is up and running:

$ kubectl get deployment -n nginx-operator-system
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
nginx-operator-controller-manager   1/1     1            1           8m

3. Deploy your Operator with OLM

First, install OLM:

operator-sdk olm install

Bundle your operator, then build and push the bundle image. The bundle target generates a [bundle][doc-bundle] in the bundle directory containing manifests and metadata defining your operator. bundle-build and bundle-push build and push a bundle image defined by bundle.Dockerfile.

make bundle bundle-build bundle-push

Finally, run your bundle. If your bundle image is hosted in a registry that is private and/or has a custom CA, these configuration steps must be completed.

operator-sdk run bundle example.com/nginx-operator-bundle:v0.0.1

Check out the docs for a deep dive into the operator-sdk's OLM integration.

Create a Nginx CR

Create the nginx CR that we modified earlier:

kubectl apply -f config/samples/demo_v1alpha1_nginx.yaml

Ensure that the nginx-operator creates the deployment for the CR:

$ kubectl get deployment
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
nginx-sample   2/2     2            2           2m13s

Check the pods to confirm 2 replicas were created:

$ kubectl get pods
NAME                                                   READY   STATUS    RESTARTS   AGE
nginx-sample-c786bfdcf-4g6md                           1/1     Running   0          81s
nginx-sample-c786bfdcf-6bhmx                           1/1     Running   0          81s

Check that the service port is set to 8080:

$ kubectl get service
NAME                                      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
nginx-sample                              ClusterIP   10.96.26.3   <none>        8080/TCP   1m

Update the replicaCount and remove the port

Change the spec.replicaCount field from 2 to 3, remove the spec.service field:

$ cat config/samples/demo_v1alpha1_nginx.yaml
apiVersion: demo.example.com/v1alpha1
kind: Nginx
metadata:
  name: nginx-sample
spec:
  replicaCount: 3

And apply the change:

kubectl apply -f config/samples/demo_v1alpha1_nginx.yaml

Confirm that the operator changes the deployment size:

$ kubectl get deployment
NAME                                           DESIRED   CURRENT   UP-TO-DATE     AGE
nginx-sample                                   3/3       3            3           7m29s

Check that the service port is set to the default (80):

$ kubectl get service
NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)  AGE
nginx-sample                              ClusterIP   10.96.152.76    <none>        80/TCP   7m54s

Troubleshooting

Use the following command to check the operator logs.

kubectl logs deployment.apps/nginx-operator-controller-manager  -n nginx-operator-system -c manager

Use the following command to check the CR status and events.

kubectl describe nginxes.demo.example.com

Cleanup

Clean up the resources:

kubectl delete -f config/samples/demo_v1alpha1_nginx.yaml

Note: Make sure the above custom resource has been deleted before proceeding to run make undeploy, as helm-operator’s controller adds finalizers to the custom resources. Otherwise your cluster may have dangling custom resource objects that cannot be deleted.

make undeploy

Next steps

Next, check out the following:

Operator packaging and distribution with OLM.
The advanced features doc for more use cases and under-the-hood details.

Docs: Reference for Helm-based Operators

Mon, 01 Jan 0001 00:00:00 +0000

Operator SDK – Helm

Docs: Installation Guide

Install operator-sdk

Additional Prerequisites

Docs: Quickstart for Helm-based Operators

Prerequisites

Steps

OLM deployment

Direct deployment

Next Steps

Docs:

Overview

What was changed

What is new

How to migrate

Prerequisites

Creating a new project

Migrating your Custom Resource samples

Migrating watches.yaml

Checking RBAC Permissions

Updating your ServiceAccount

Configuring your Operator

Exporting metrics

Verify the migration

Docs: Helm Operator Tutorial

Prerequisites

Overview

Create a new project

Use an existing chart

Customize the operator logic

Watch the Nginx CR

Reviewing the Nginx Helm Chart

Understanding the Nginx CR spec

Configure the operator’s image registry

Run the operator

1. Run locally outside the cluster

2. Run as a Deployment inside the cluster

3. Deploy your Operator with OLM

Create a Nginx CR

Update the replicaCount and remove the port

Troubleshooting

Cleanup

Next steps

Docs: Reference for Helm-based Operators

Install `operator-sdk`

Migrating `watches.yaml`