Operator SDK – Advanced Topics

Docs: Custom Bundle Validation

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Operator authors can now use “external” validators with the operator-sdk bundle validate command by using the --alpha-select-external flag. This feature enables Operator authors, users, and registry pipelines to use custom validators. These custom validators can be written in any language.

Usage

External validators can be used by specifying a list of local filepaths to executables using colons as path separators:

$ operator-sdk bundle validate \
--alpha-select-external path/validator1:path/validator2

Writing a Custom Validator

For a validator to work with operator-sdk bundle validate each of the files must:

Be executable with appropriate permissions
Return JSON to STDOUT in the ManifestResult format.

Custom Validator from Scratch

Using the errors package from operator-framework/api, we can start by validating the correct number of arguments and marshaling a ManifestResult into STDOUT.

myvalidator/main.go

package main

import (
	"encoding/json"
	"fmt"
	"os"

	"github.com/operator-framework/api/pkg/validation/errors"
)

func main() {

	// we expect a single argument which is the bundle root.
	// usage: validator-poc <bundle root>
	if len(os.Args) < 2 {
		fmt.Printf("usage: %s <bundle root>\n", os.Args[0])
		os.Exit(1)
	}

	var validatorErrors []errors.Error
	var validatorWarnings []errors.Error
	result := errors.ManifestResult{
		Name:     "Always Green Example",
		Errors:   validatorErrors,
		Warnings: validatorWarnings,
	}
	prettyJSON, err := json.MarshalIndent(result, "", "    ")
	if err != nil {
		fmt.Println("Invalid json")
		os.Exit(1)
	}
	fmt.Printf("%s\n", string(prettyJSON))
}

When executed on its own, this validator prints a JSON representation of ManifestResult.

go build -o myvalidator/main myvalidator/main.go && ./myvalidator/main

{
    "Name": "Always Green Example",
    "Errors": null,
    "Warnings": null
}

$ go build -o myvalidator/main myvalidator/main.go
$ operator-sdk bundle validate ./bundle --alpha-select-external ./myvalidator/main

INFO[0000] All validation tests have completed successfully

From here, custom validator authors can read in the bundle and make any assertions necessary.

Errors and Warnings are both implementations of the error interface and need ErrorType, Level, Field, BadValue, and Detail, which are all initialized by arbitrary strings. When using Golang, validator authors can use the operator-framework/api impementation of errors and warnings

validatorErrors = []errors.Error{errors.Error{"someErrorType", "somelevel", "somefield", "somebadvalue", "somedetail"}}
validatorWarnings = []errors.Error{errors.Error{"someWarningType", "somelevel", "somefield", "somebadvalue", "somedetail"}}

We can now rebuild and run the validator, which now shows errors.

$ go build -o myvalidator/main myvalidator/main.go
$ operator-sdk bundle validate ./bundle --alpha-select-external ./myvalidator/main

WARN[0000] somelevel: Field somefield, Value somebadvalue: somedetail
ERRO[0000] somelevel: Field somefield, Value somebadvalue: somedetail

Composing Validators

For users wishing to use validators from operator-framework/api without being restricted to the version that is built into the operator-sdk binary, it is possible to create a main.go that makes use of the validation package at an arbitrary version.

Currently, some of the code necessary requires copying code from internal packages, which may someday become a library.

myvalidator/main.go

package main

import (
	"encoding/json"
	"errors"
	"fmt"
	"os"
	"path/filepath"
	"strings"

	apimanifests "github.com/operator-framework/api/pkg/manifests"
	apivalidation "github.com/operator-framework/api/pkg/validation"
	registrybundle "github.com/operator-framework/operator-registry/pkg/lib/bundle"
	log "github.com/sirupsen/logrus"

	"github.com/spf13/afero"
	"sigs.k8s.io/yaml"
)

func main() {

	// we expect a single argument which is the bundle root.
	// usage: validator-poc <bundle root>
	if len(os.Args) < 2 {
		fmt.Printf("usage: %s <bundle root>\n", os.Args[0])
		os.Exit(1)
	}

	// Read the bundle object and metadata from the passed in directory.
	bundle, _, err := getBundleDataFromDir(os.Args[1])
	if err != nil {
		fmt.Printf("problem getting bundle [%s] data, %v\n", os.Args[1], err)
		os.Exit(1)
	}

	// pass the objects to the validator
	objs := bundle.ObjectsToValidate()
	for _, obj := range bundle.Objects {
		objs = append(objs, obj)
	}
	results := apivalidation.GoodPracticesValidator.Validate(objs...)

	// take each of the ManifestResults and print to STDOUT
	for _, result := range results {
		prettyJSON, err := json.MarshalIndent(result, "", "    ")
		if err != nil {
			// should output JSON so that the call knows how to parse it
			fmt.Printf("XXX ERROR: %v\n", err)
		}
		fmt.Printf("%s\n", string(prettyJSON))
	}
}

// getBundleDataFromDir returns the bundle object and associated metadata from dir, if any.
func getBundleDataFromDir(dir string) (*apimanifests.Bundle, string, error) {
	// Gather bundle metadata.
	metadata, _, err := FindBundleMetadata(dir)
	if err != nil {
		return nil, "", err
	}
	manifestsDirName, hasLabel := metadata.GetManifestsDir()
	if !hasLabel {
		manifestsDirName = registrybundle.ManifestsDir
	}
	manifestsDir := filepath.Join(dir, manifestsDirName)
	// Detect mediaType.
	mediaType, err := registrybundle.GetMediaType(manifestsDir)
	if err != nil {
		return nil, "", err
	}
	// Read the bundle.
	bundle, err := apimanifests.GetBundleFromDir(manifestsDir)
	if err != nil {
		return nil, "", err
	}
	return bundle, mediaType, nil
}

// -------------------------------------------------------
// Everything below this line was copied code from the internal Operator SDK
// registry package operator-sdk/internal/registry/labels.go. If this is
// generally useful please file an issue to move this to a reuable library.
// to make this a library or other reusable code.
// -------------------------------------------------------

type MetadataNotFoundError string

func (e MetadataNotFoundError) Error() string {
	return fmt.Sprintf("metadata not found in %s", string(e))
}

// Labels is a set of key:value labels from an operator-registry object.
type Labels map[string]string

// GetManifestsDir returns the manifests directory name in ls using
// a predefined key, or false if it does not exist.
func (ls Labels) GetManifestsDir() (string, bool) {
	value, hasKey := ls[registrybundle.ManifestsLabel]
	return filepath.Clean(value), hasKey
}

// FindBundleMetadata walks bundleRoot searching for metadata (ex. annotations.yaml),
// and returns metadata and its path if found. If one is not found, an error is returned.
func FindBundleMetadata(bundleRoot string) (Labels, string, error) {
	return findBundleMetadata(afero.NewOsFs(), bundleRoot)
}

func findBundleMetadata(fs afero.Fs, bundleRoot string) (Labels, string, error) {
	// Check the default path first, and return annotations if they were found or an error if that error
	// is not because the path does not exist (it exists or there was an unmarshalling error).
	annotationsPath := filepath.Join(bundleRoot, registrybundle.MetadataDir, registrybundle.AnnotationsFile)
	annotations, err := readAnnotations(fs, annotationsPath)
	if (err == nil && len(annotations) != 0) || (err != nil && !errors.Is(err, os.ErrNotExist)) {
		return annotations, annotationsPath, err
	}

	// Annotations are not at the default path, so search recursively.
	annotations = make(Labels)
	annotationsPath = ""
	err = afero.Walk(fs, bundleRoot, func(path string, info os.FileInfo, err error) error {
		if err != nil {
			return err
		}
		// Skip directories and hidden files, or if annotations were already found.
		if len(annotations) != 0 || info.IsDir() || strings.HasPrefix(path, ".") {
			return nil
		}

		annotationsPath = path
		// Ignore this error, since we only care if any annotations are returned.
		if annotations, err = readAnnotations(fs, path); err != nil {
			log.Debug(err)
		}
		return nil
	})
	if err != nil {
		return nil, "", err
	}

	if len(annotations) == 0 {
		return nil, "", MetadataNotFoundError(bundleRoot)
	}

	return annotations, annotationsPath, nil
}

// readAnnotations reads annotations from file(s) in bundleRoot and returns them as Labels.
func readAnnotations(fs afero.Fs, annotationsPath string) (Labels, error) {
	// The annotations file is well-defined.
	b, err := afero.ReadFile(fs, annotationsPath)
	if err != nil {
		return nil, err
	}

	// Use the arbitrarily-labelled bundle representation of the annotations file
	// for forwards and backwards compatibility.
	annotations := registrybundle.AnnotationMetadata{
		Annotations: make(Labels),
	}
	if err = yaml.Unmarshal(b, &annotations); err != nil {
		return nil, fmt.Errorf("error unmarshalling potential bundle metadata %s: %v", annotationsPath, err)
	}

	return annotations.Annotations, nil
}

The main.go is then built into a binary and used with operator-sdk bundle validate

$ go build -o myvalidator/main myvalidator/main.go
$ operator-sdk bundle validate ./bundle --alpha-select-external ./myvalidator/main

WARN[0000] Warning: Value sandbox-op.v0.0.1: owned CRD "sandboxes.sandbox.example.come" has an empty description
INFO[0000] All validation tests have completed successfully

Docs: Multiple Architectures

Mon, 01 Jan 0001 00:00:00 +0000

Supporting Multiple Architectures

Authors may decide to distribute their bundles for various architectures: x86_64, aarch64, ppc64le, s390x, etc, to accommodate the diversity of Kubernetes clusters and reach a larger number of potential users. Each supported architecture requires compatible binaries. Binary compatibility is based on the platform, which is generally comprised of an operating system, a CPU architecture, and other architecture variants. For this guide, we will treat the CPU architecture as the platform differentiator.

That said, the same general advice would apply to supporting workloads on clusters of the same architecture but different operating systems (e.g. linux/amd64, windows/amd64). In this guide, we assume all workloads will be using the linux operating system.

Fundamentals

The basic principle of supporting multiple architectures is to ensure that each of your operator images is built for each of the architectures to be supported. From there, the images should be hosted in image registries as manifest lists. Finally, you’ll need to update your distribution configuration to set which architectures are supported. This section explains each of these concepts in turn.

Building an Operator for Multiple Architectures

Kubebuilder explains how you can use docker buildx to build images for multiple architectures. Operator SDK leverages KubeBuilder to ensure that builds can be cross-platform from the start.

Manifest lists

The most straightforward way of building operators and operands supporting multiple architectures is to leverage manifest lists, specified by Image Manifest V2, Schema 2 or OCI Image Index. A manifest list points to specific image manifests for one or more architectures.

For convenience tools like buildah allow to cross-build and manifest multi-arch containers on one host. For instance with buildah:

for a in amd64 arm64 ppc64le s390x; do \
  buildah bud --manifest registry/username/repo:v1 --arch $a; \
done

This creates the manifest list, builds each image, and adds them to the manifest list.

The result can then be pushed to the desired registry.

buildah push registry/username/repo:v1

Docker with buildx provides similar capabilities.

docker buildx build --push --platform linux/amd64,linux/arm64,linux/ppc64le,linux/s390x --tag registry/username/repo:v1 .

See docker documentation for additional options.

Caveat: the Dockerfile generated by the SDK for the operator explicitly references GOARCH=amd64 for go build. This can be amended to GOARCH=$TARGETARCH. Docker will automatically set the environment variable to the value specified by –platform. With buildah –build-arg will need to be used for the purpose.

Caveat: When mirroring registries for disconnected installations (environments without internet connection) all the images referenced by a manifest list need to be copied, including images for architectures that may not be used in the environment.

Operator Lifecycle Manager

For operators distributed through the Operator Lifecycle Manager (OLM):

Bundle images are not architecture-specific. They contain only plaintext Kubernetes manifests and operator metadata.
All image references in the ClusterServiceVersion should be manifest lists containing the pointers to the image manifests for the supported architectures.
Labels for OS and architectures can be set in the CSV. Please refer to the Operator Lifecycle Management Documentation for details.

Supporting Clusters with Multi-Architecture Compute Nodes

The Fundamentals above aim to guide authors on the key steps to building and distributing operators that can run on multiple architectures. These instructions are sufficient when your cluster’s compute nodes share the same architecture. However, operator authors should also understand the implications of running their operators in a cluster with multi-architecture compute nodes since it is not always guaranteed that the architectures of the compute nodes will match the architectures supported by the operator.

Safe Scheduling Using Node Affinity

Node affinity is a mechanism exposed in a Kubernetes pod template that allows a PodSpec author to instruct the scheduler to restrict a pod to run only on (or with a preference for) nodes that meet specific criteria. To ensure that pods are always scheduled to nodes of compatible architecture, it is a best practice for authors to set node affinity requirements to ensure their operators and operands will only schedule to the nodes with architectures available to the images in the pod. If you don’t do this, a container scheduled to an incompatible node will immediately crash with an exec format error, which will ultimately lead to an ImagePullBackoff event as the pod is restarted only to crash again with the same error.

Determining the Architectures Supported by an Image

For a given container image, you can check which architectures are supported by listing them by inspecting the manifest. Piping the output to the python json.tool module enables pretty-printed JSON output.

$ skopeo inspect --raw <image> | python -m 'json.tool'

Here’s an example of architectures distributed for the Alpine Linux container image. Notice that both linux/amd64 and linux/arm64 are supported.

$ skopeo inspect --raw docker://alpine:latest | python -m 'json.tool'
{
    "manifests": [
        {
            "digest": "sha256:c0669ef34cdc14332c0f1ab0c2c01acb91d96014b172f1a76f3a39e63d1f0bda",
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            },
            "size": 528
        },
        ...
        {
            "digest": "sha256:30e6d35703c578ee703230b9dc87ada2ba958c1928615ac8a674fcbbcbb0f281",
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "arm64",
                "os": "linux",
                "variant": "v8"
            },
            "size": 528
        },
      ...

You can also use docker to inspect the manifest, but beware that it will return no such manifest error if the referenced image is not actually a manifest list.

$ docker manifest inspect <image>

If the above commands do not produce output with manifests, then it may be because the referenced name points to a single image rather than a manifest list. In this case, you can find the architecture by inspecting the image directly.

$ skopeo inspect <image>

This example shows how to inspect a local image to determine the architecture.

$ skopeo inspect docker://alpine
{
    "Name": "docker.io/library/alpine",
    ...
    "Architecture": "amd64",
    "Os": "linux",
    ...
}

Alternatively you can pull the image down and inspect it with docker.

$ docker pull <image> && docker inspect <image>

Setting Node-Affinity Criteria for Operators & Operands

Kubernetes provides a mechanism called nodeAffinity which can be used to limit the possible node targets where a pod can be scheduled. The following example can be used to update a PodSpec or PodTemplateSpec to prevent the scheduling of pods on nodes of incompatible architecture. Here we compare the kubernetes.io/arch and kubernetes.io/os keys set on the node to ensure that the values match one of the supported OS/architecture pairs. This assumes that the referenced image points to a manifest list with references to images for each of the supported architectures. It is important to remember that nodeAffinity should be set anywhere a container reference is defined, including Pod, Deployment, DaemonSet, StatefulSet, or any other object that defines a PodSpec or PodTemplateSpec.

The list of architecture values should only include the architectures the operator supports. The full list of possible GOARCH values is available here. It should be noted that Kubernetes allows a user to specify both requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution. The syntax for each block is the same. At a minimum, an operator author should set the required terms since they protect against a pod being scheduled on an incompatible node. If the operator in question has much better performance on a subset of the supported architectures, it may also be prudent to set preferredDuringSchedulingIgnoredDuringExecution so that the optimal arches are selected first if they are available.

Setting Node Affinity in a Kubernetes Manifest

To update the PodSpec and PodTemplateSpec objects of an operator and its operands, you will need to scan your operator for instances of these objects. Most of the time, these objects will be defined directly in a Kubernetes manifest yaml; however, for dynamically created workloads, these are sometimes embedded in the operator’s logic directly.

The most common directory with PodSpec and PodTemplateSpec objects that will need to be updated will be in the operator’s config directory. Additionally, if the operator is configured for OLM, there will likely be a deployment object in the bundle/manifests/*.clusterserviceversion.yaml.

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/arch
          operator: In
          values:
          - amd64
          - arm64
          - ppc64le
          - s390x
        - key: kubernetes.io/os
            operator: In
            values:
              - linux

Setting Node Affinity in Golang

While Ansible and Helm operators will usually define their operators and operands according to the syntax defined above directly in yaml files or directly in a role or variable template file, Go operators will sometimes embed the logic directly in the operator itself. Here is an example of how a PodSpec would be updated according to the syntax of the Go API.

Template: corev1.PodTemplateSpec{
    ...
    Spec: corev1.PodSpec{
        Affinity: &corev1.Affinity{
            NodeAffinity: &corev1.NodeAffinity{
                RequiredDuringSchedulingIgnoredDuringExecution: &corev1.NodeSelector{
                    NodeSelectorTerms: []corev1.NodeSelectorTerm{
                        {
                            MatchExpressions: []corev1.NodeSelectorRequirement{
                                {
                                    Key:      "kubernetes.io/arch",
                                    Operator: "In",
                                    Values:   []string{"amd64"},
                                },
                                {
                                    Key:      "kubernetes.io/os",
                                    Operator: "In",
                                    Values:   []string{"linux"},
                                },
                            },
                        },
                    },
                },
            },
        },
        SecurityContext: &corev1.PodSecurityContext{
            ...
        },
        Containers: []corev1.Container{{
            ...
        }},
    },

Updating Operator Lifecycle Manager Configurations for Multi-Architecture Compute Nodes

The Operator Lifecycle Manager (OLM) is often used to distribute operators via operator catalogs. The operator deployment object in an OLM integrated operator is defined in the ClusterServiceVersion yaml. It is important to remember to set the node affinity block in each of the spec.template.spec under spec.install.deployments.

If you’re planning to distribute the operator via OLM, you can find more information in the OLM guide for supporting multiarch.

Overriding Affinity for an Operator Pod as a Cluster Admin

A cluster admin might have some context that can be used to refine the scheduling requirements for the operator images specified in the deployment spec of the Cluster Service Version. To ensure the fine control of scheduling remains with the cluster admin, OLM provides a mechanism for overriding the operator pod affinity configuration as part of the Subscription object.

Caveat: No built-in affinity override controls exist that allow cluster admins to override affinity for operands. It is up to the operator author to determine whether this would be appropriate for one or more of their operands and what configuration options should be allowed.

Validating Your Operator’s Multi-Architecture Readiness

A validator is available to help authors ensure that operators are defined according to the best practices. Because operands can be declared and defined in a variety of ways depending on the language used and structure of the operator, this validator focuses on verifying that the images defined in the CSV deployment spec are compliant with the fundamentals and best practices. It is up to the operator author to ensure that affinity best practices are also being followed for each operand’s PodSpec and PodTemplateSpec definitions.

$ operator-sdk bundle validate ./bundle --select-optional name=multiarch

Docs: Multiple Service Accounts

Mon, 01 Jan 0001 00:00:00 +0000

Using Multiple Service Accounts

There may be a need to have multiple service accounts to provide only the necessary permissions to various objects that the operator creates on a Kubernetes cluster.

This can be accomplished by using the --extra-service-accounts flag when generating the bundle with make bundle.

Updating the `Makefile` to use `--extra-service-accounts`

Update the bundle target in the Makefile to add the --extra-service-accounts flag with the name of the desired service account. This ensures that the permissions and configurations do not get overwritten by make bundle. For example, modify the line that contains operator-sdk generate bundle similar to below replacing myOperator-name-additional-service-account to the desired service account name appended to the operator name.

bundle: manifests kustomize operator-sdk ## Generate bundle manifests and metadata, then validate generated files.
	operator-sdk generate kustomize manifests -q
	cd config/manager && $(KUSTOMIZE) edit set image controller=$(IMG)
	$(KUSTOMIZE) build config/manifests | operator-sdk generate bundle -q --overwrite --extra-service-accounts myOperator-name-additional-service-account --version $(VERSION) $(BUNDLE_METADATA_OPTS)
	operator-sdk bundle validate ./bundle

The --extra-service-accounts flag takes a comma-separated list of strings, so you can add more than a single service account name if desired.

Add RBAC configurations for `--extra-service-accounts`

These steps will need to be followed for every additional service account.

Create a new service account file. For example:

cat << EOF > config/rbac/additional_service_account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: additional-service-account
  namespace: system
EOF

Create a role binding. In this example, it is a ClusterRoleBinding:

cat << EOF > config/rbac/additional_role_binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: additional-service-account-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: additional-service-account-role
subjects:
- kind: ServiceAccount
  name: additional-service-account
  namespace: system
EOF

Create a role with desired permissions. In this example, it is a ClusterRole that provides permission to the privileged SecurityContextConstraint (SCC).

cat << EOF > config/rbac/additional_role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: additional-service-account-role
rules:
- apiGroups:
  - security.openshift.io
  resourceNames:
  - privileged
  resources:
  - securitycontextconstraints
  verbs:
  - use
EOF

Update the RBAC `kustomization.yaml`

Make sure to update the RBAC configuration kustomization.yaml file with the previously created RBAC yaml files. For example:

cat << EOF >> config/rbac/kustomization.yaml

# Add MyCustomObject service account
- additional_service_account.yaml
- additional_role.yaml
- additional_role_binding.yaml
EOF

Operator SDK – Advanced Topics

Docs: Custom Bundle Validation

Summary

Usage

Writing a Custom Validator

Custom Validator from Scratch

Composing Validators

Docs: Multiple Architectures

Supporting Multiple Architectures

Fundamentals

Building an Operator for Multiple Architectures

Manifest lists

Operator Lifecycle Manager

Supporting Clusters with Multi-Architecture Compute Nodes

Safe Scheduling Using Node Affinity

Determining the Architectures Supported by an Image

Setting Node-Affinity Criteria for Operators & Operands

Setting Node Affinity in a Kubernetes Manifest

Setting Node Affinity in Golang

Updating Operator Lifecycle Manager Configurations for Multi-Architecture Compute Nodes

Overriding Affinity for an Operator Pod as a Cluster Admin

Validating Your Operator’s Multi-Architecture Readiness

Docs: Multiple Service Accounts

Using Multiple Service Accounts

Updating the Makefile to use --extra-service-accounts

Add RBAC configurations for --extra-service-accounts

Update the RBAC kustomization.yaml

Updating the `Makefile` to use `--extra-service-accounts`

Add RBAC configurations for `--extra-service-accounts`

Update the RBAC `kustomization.yaml`