Operator SDK – Advanced Features of Helm-based Operators

Docs: Setting Override Values in Helm-based Operators

Mon, 01 Jan 0001 00:00:00 +0000

Sometimes it is useful to pass down environment variables from the Operators Deployment all the way to the helm charts templates. This allows the Operator to be configured at a global level at runtime. This is new compared to dealing with the helm CLI as they usually don’t have access to any environment variables in the context of Tiller (helm v2) or the helm binary (helm v3) for security reasons.

With the helm Operator this becomes possible by override values. This enforces that certain template values provided by the chart’s default values.yaml or by a CR spec are always set when rendering the chart. If the value is set by a CR it gets overridden by the global override value. The override value can be static but can also refer to an environment variable and use go templates. Using override values is currently the only way to pass down environment variables to the chart.

An example use case of this is when your helm chart references container images by chart variables, which is a good practice. If your Operator is deployed in a disconnected environment (no network access to the default images location) you can use this mechanism to set them globally at the Operator level using environment variables versus individually per CR / chart release.

Note that it is strongly recommended to reference container images in your chart by helm variables and then also associate these with an environment variable of your Operator like shown below. This allows your Operator to be mirrored for offline usage when packaged for OLM.

Basic usage

To configure your operator with override values, add an overrideValues map to your watches.yaml file for the GVK and chart you need to override. For example, to change the repository used by the nginx chart, you would update your watches.yaml to the following:

# Use the 'create api' subcommand to add watches to this file.
- group: example.com
  version: v1alpha1
  kind: Nginx
  chart: helm-charts/nginx
  overrideValues:
    image.repository: quay.io/mycustomrepo/myimage

By setting image.repository to quay.io/mycustomrepo/myimage you are ensuring that quay.io/mycustomrepo/myimage will always be used instead of the chart’s default repository (nginx). If the CR attempts to set this value, it will be ignored.

Using environment variables

It is also possible to reference environment variables in the overrideValues section:

  overrideValues:
    image.repository: $IMAGE_REPOSITORY # or ${IMAGE_REPOSITORY}

By using an environment variable reference in overrideValues you enable these override values to be set at runtime by configuring the environment variable on the operator deployment. For example, in config/manager/manager.yaml you could add the following snippet to the container spec:

env:
  - name: IMAGE_REPOSITORY
    value: quay.io/mycustomrepo/myimage

If an environment variable reference is listed in overrideValues, but is not present in the environment when the operator runs, it will resolve to an empty string and override all other values. Therefore, these environment variables should always be set. It is suggested to update the Dockerfile to set these environment variables to the same defaults that are defined by the chart.

Using Go templates

Lastly, you can use Go text/template strings along with slim-sprig functions to provide even more flexibility when building override values.

For example, consider a situation where your operator has an environment variable, $IMAGE, set to quay.io/mycustomrepo/myimage:latest. You can use sprig template functions to split that environment variable into its repo and tag:

  overrideValues:
    image.repository: '{{ ("$IMAGE" | split ":")._0 }}'
    image.tag: '{{ ("$IMAGE" | split ":")._1 }}'

The resulting override values sent to the helm installation would look like:

  overrideValues:
    image.repository: quay.io/mycustomrepo/myimage
    image.tag: latest

Event generation

To warn users that their CR settings may be ignored, the Helm operator creates events on the CR that include the name and value of each overridden value. For example:

$ kubectl describe nginxes.example.com
...
Events:
  Type     Reason               Age   From              Message
  ----     ------               ----  ----              -------
  Warning  OverrideValuesInUse  1m    nginx-controller  Chart value "image.repository" overridden to "quay.io/mycustomrepo" by operator's watches.yaml

Docs: Maximum Concurrent Reconciles in Helm-based Operators

Mon, 01 Jan 0001 00:00:00 +0000

Depending on the number of CRs your operator is managing, it might be necessary to tune the maximum number of concurrent reconciles to ensure timely reconciliations. The --max-concurrent-reconciles flag can be used to override the default max concurrent reconciles, which by default is the number of CPUs on the node on which the operator is running. For example:

$ cat config/manager/manager.yaml
...
    spec:
      containers:
      - args:
        - manager
        - --max-concurrent-reconciles=10
...

While running locally, this flag can also be added to the helm binary. For example, running helm-operator binary with the above mentioned flag would give us a similar result:

helm-operator --max-concurrent-reconciles=10

NOTE: If you’re using the default scaffolding, it is necessary to also apply this change to the config/default/manager_metrics_patch.yaml file. This file is a kustomize patch to the operator deployment that configures metrics to require authorization for accessing your operator metrics. When kustomize applies this patch, it overrides the args defined in config/manager/manager.yaml

Docs: Custom Resource Annotations in Helm-based Operators

Mon, 01 Jan 0001 00:00:00 +0000

`helm.sdk.operatorframework.io/upgrade-force`

This annotation can be set to "true" on custom resources to enable the chart to be upgraded with the helm upgrade --force option. For more info see the Helm Upgrade documentation and this explanation of --force behavior.

Example

apiVersion: example.com/v1alpha1
kind: Nginx
metadata:
  name: nginx-sample
  annotations:
    helm.sdk.operatorframework.io/upgrade-force: "true"
spec:
  replicaCount: 2
  service:
    port: 8080

Setting this annotation to true and making a change to trigger an upgrade (e.g. setting spec.replicaCount: 3) will cause the custom resource to be reconciled and upgraded with the force option. This can be verified in the log message when an upgrade succeeds:

{"level":"info","ts":1591198931.1703992,"logger":"helm.controller","msg":"Upgraded release","namespace":"helm-nginx","name":"example-nginx","apiVersion":"cache.example.com/v1alpha1","kind":"Nginx","release":"example-nginx","force":true}

`helm.sdk.operatorframework.io/uninstall-wait`

This annotation can be set to "true" on custom resources to enable the deletion to wait until all the resources in the status.deployedRelease.manifest are deleted.

Example

apiVersion: example.com/v1alpha1
kind: Nginx
metadata:
  name: nginx-sample
  annotations:
    helm.sdk.operatorframework.io/uninstall-wait: "true"
spec:
...
status:
  ...
  deployedRelease:
    manifest: |
      ---
      # Source: nginx/templates/serviceaccount.yaml
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: nginx-sample
        labels:
          helm.sh/chart: nginx-0.1.0
          app.kubernetes.io/name: nginx
          app.kubernetes.io/instance: nginx-sample
          app.kubernetes.io/version: "1.16.0"
          app.kubernetes.io/managed-by: Helm
      ---
      # Source: nginx/templates/service.yaml
      apiVersion: v1
      kind: Service
      metadata:
        name: nginx-sample
        labels:
          helm.sh/chart: nginx-0.1.0
          app.kubernetes.io/name: nginx
          app.kubernetes.io/instance: nginx-sample
          app.kubernetes.io/version: "1.16.0"
          app.kubernetes.io/managed-by: Helm
      spec:
        type: ClusterIP
        ports:
          - port: 80
            targetPort: http
            protocol: TCP
            name: http
        selector:
          app.kubernetes.io/name: nginx
          app.kubernetes.io/instance: nginx-sample
      ---
      # Source: nginx/templates/deployment.yaml
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: nginx-sample
        labels:
          helm.sh/chart: nginx-0.1.0
          app.kubernetes.io/name: nginx
          app.kubernetes.io/instance: nginx-sample
          app.kubernetes.io/version: "1.16.0"
          app.kubernetes.io/managed-by: Helm
      spec:
        replicas: 1
        selector:
          matchLabels:
            app.kubernetes.io/name: nginx
            app.kubernetes.io/instance: nginx-sample
        template:
          metadata:
            labels:
              app.kubernetes.io/name: nginx
              app.kubernetes.io/instance: nginx-sample
          spec:
            serviceAccountName: nginx-sample
            securityContext:
              {}
            containers:
              - name: nginx
                securityContext:
                  {}
                image: "nginx:1.16.0"
                imagePullPolicy: IfNotPresent
                ports:
                  - name: http
                    containerPort: 80
                    protocol: TCP
                livenessProbe:
                  httpGet:
                    path: /
                    port: http
                readinessProbe:
                  httpGet:
                    path: /
                    port: http
                resources:
                  {}

Setting this annotation to true and deleting the custom resource will cause the custom resource to be reconciled continuously until all the resources in status.deployedRelease.manifest are deleted. This can be verified in the log message when a delete has been triggered:

{"level":"info","ts":1612294054.5845876,"logger":"helm.controller","msg":"Uninstall wait","namespace":"default","name":"nginx-sample","apiVersion":"example.com/v1alpha1","kind":"Nginx","release":"nginx-sample"}

`helm.sdk.operatorframework.io/reconcile-period`

While running a Helm-based operator, the reconcile-period can be specified through the custom resource’s annotations under the helm.sdk.operatorframework.io/reconcile-period key. This feature guarantees that an operator will get reconciled, at minimum, in the specified interval of time. In other words, it ensures that the cluster will not go longer than the specified reconcile-period without being reconciled. However, the cluster may be reconciled at any moment if there are changes detected in the desired state.

The reconcile period can be specified in the custom resource’s annotations in the following manner:

...
metadata:
  name: nginx-sample
  annotations:
    helm.sdk.operatorframework.io/reconcile-period: 5s
...

The value that is present under this key must be in the h/m/s format. For example, 1h2m4s, 3m0s, 4s are all valid values, but 1x3m9s is invalid.

NOTE: This is just one way of specifying the reconcile period for Helm-based operators. There are two other ways: using the --reconcile-period command-line flag and under the ‘reconcilePeriod’ key in the watches.yaml file. If these three methods are used simultaneously to specify reconcile period (which they should not be), the order of precedence is as follows: Custom Resource Annotations > watches.yaml > command-line flag.

`helm.sdk.operatorframework.io/rollback-force`

Whenever a helm-based operator encounters an error during reconcilliation, by default, it would attempt to perform a rollback with the --force option. While this works as expected in most scenarios, there are a few edge cases where performing a rollback with --force could have undesired side effects.

...
metadata:
  name: nginx-sample
  annotations:
    helm.sdk.operatorframework.io/rollback-force: false
...

Adding annotation to the custom resource, helm.sdk.operatorframework.io/rollback-force: false therefore allows a user, to change the default behavior of the helm-based operator whereby, rollbacks will be performed without the --force option whenever an error is encountered.