Operator SDK – Overview

Docs: Project Layout

Mon, 01 Jan 0001 00:00:00 +0000

Operator SDK Project Layout

All projects initialized with operator-sdk init have a common base structure which builds on kubebuilder’s project layout. Each project type is customized further with code of that type’s language.

Common Base

The common structure contains the following items:

File/Directory	Description
`Dockerfile`	The Dockerfile of your operator project, used to build the image with `make docker-build`.
`Makefile`	Build file with helper targets to help you work with your project.
`PROJECT`	This file represents the project’s configuration and is used to track useful information for the CLI and plugins.
`bin/`	This directory contains useful binaries such as the `manager` which is used to run your project locally and the `kustomize` utility used for the project configuration. For other language types, it might have other binaries useful for developing your operator.
`bundle/`	This directory contains all the files used to integrate your project with OLM with the bundle format. It is built from the Makefile target `make bundle`.
`bundle/manifests/`	This directory has the OLM manifests of your bundle.
`bundle/metadata/`	This directory has the OLM metadata of your bundle e.g the index image annotations.
`bundle/tests/`	This directory has the Scorecard tests shipped with your operator bundle.
`config/`	Contains configuration files to launch your project on a cluster. Plugins might use it to provide functionality. For example, for the CLI to help create your operator bundle it will look for the CRD’s and CR’s which are scaffolded in this directory. You will also find all Kustomize YAML definitions as well.
`config/crd/`	Contains the Custom Resources Definitions.
`config/default/`	Contains a Kustomize base for launching the controller in a standard configuration.
`config/manager/`	Contains the manifests to launch your operator project as pods on the cluster.
`config/manifests/`	Contains the base to generate your OLM manifests in the bundle directory.
`config/prometheus/`	Contains the manifests required to enable project to serve metrics to Prometheus such as the `ServiceMonitor` resource.
`config/scorecard/`	Contains the manifests required to allow you test your project with Scorecard.
`config/rbac/`	Contains the RBAC permissions required to run your project.
`config/samples/`	Contains the Custom Resources.
`bundle.Dockerfile`	The Dockerfile to build the bundle image. Used to build the operator bundle image with `make bundle-build. \|

Ansible

Now, let’s look at the files and directories specific to Ansible-based operators.

File/Directory	Description
`config/testing/`	Manifest files to help you test your project. For example, to change the image policy for your Molecule tests or to enable debug level in the Ansible logs.
`molecule/`	Contain the manifests for your Molecule tests.
`molecule/default`	Contains the default Molecule task.
`molecule/kind`	Contains the Molecule task to be executed on the cluster.
`playbooks/`	Contains the Ansible playbooks.
`roles/`	Contains the Ansible role files for each Kind scaffold.
`requirements.yml`	This file specifies Ansible dependencies that need to be installed for your operator to function.
`watches.yaml`	Contains Group, Version, Kind, and the playbooks and rules location. Used to configure the Ansible watches.

Golang

Now, let’s look at the files and directories specific to Go-based operators.

File/Directory	Description
`api/`	Contains the api definition
`config/certmanager`	Contains the Kustomize manifests which configure the cert-manager by the Webhooks.
`config/webhook`	Contains the Kustomize manifests to configure the Webhook.
`controllers`	Contains the controllers.
`main.go`	Implements the project initialization.
`hack/`	Contains utility files, e.g. the file used to scaffold the license header for your project files.

Helm

Now, let’s look at the files and directories specific to Helm-based operators.

File/Directory	Description
`helm-charts`	Contains the Helm charts for each Kind scaffold which can be initialized with `operator-sdk init --plugins=helm [options]` or `operator-sdk create api [options]` .
`watches.yaml`	Contains Group, Version, Kind, and Helm chart location. Used to configure the Helm watches.

Docs: Cheat Sheet

Mon, 01 Jan 0001 00:00:00 +0000

Below you will find a cheat sheet with options and helpers for projects, which are built with the SDK and are respecting its proposed layout.

Common commands and options

Command	Description
`operator-sdk init`	To initialize an operator project in the current directory.
`operator-sdk init --plugins=<plugin-key>`	To initialize an operator project in the current directory using a specific plugin. To check the available plugins you can run `operator-sdk --help`. E.g (`operator-sdk init --plugins=helm`).
`operator-sdk create api [flags]`	Lets you create your own APIs with its GKV by Extending the Kubernetes API with CustomResourceDefinitions, or lets you use external/core-types. Also generates their respective controllers.
`operator-sdk create webhook [flags]`	To scaffold Webhooks for the APIs declared in the project. Currently, only the Go-based project supports this option.
`make docker-build IMG=<some-registry>/<project-name>:<tag>`	Build the operator image.
`make docker-build docker-push IMG=<some-registry>/<project-name>:<tag>`	Build and push the operator image for your registry.
`make install`	Install the CRDs into the cluster.
`make uninstall`	Uninstall the CRDs into the cluster.
`make run`	Run your controller locally and outside of the cluster. Note that this will run in the foreground, so switch to a new terminal if you want to leave it running.
`make deploy`	Deploy your project on the cluster.
`make undeploy`	Undeploy your project on the cluster.

To create bundles, catalogs, and develop for OLM

For further information check Operator SDK Integration with Operator Lifecycle Manager.

Command	Description
`make bundle`	Create/update the bundle based on the project manifests in the `bundle/` directory. For more info see Create a bundle.
`operator-sdk bundle validate ./bundle`	To validate your bundle spec definition.
`operator-sdk bundle validate ./bundle --select-optional suite=operatorframework`	Validate your bundle against OperatorHub.io criteria. For further information use the flag `--help`.
`operator-sdk olm install`	To install OLM on your cluster for development purposes.
`operator-sdk olm uninstall`	To uninstall OLM from your cluster.
`make bundle-build BUNDLE_IMG=<some-registry>/<project-name-bundle>:<tag>`	To build your bundle operator image.
`make bundle-build bundle-push BUNDLE_IMG=<some-registry>/<project-name-bundle>:<tag>`	To build and push your bundle operator image.
`operator-sdk run bundle <some-registry>/<project-name-bundle>:<tag>`	To deploy your bundle operator using OLM on your cluster for development purposes.
`operator-sdk run bundle private-registry.org/bundle:v1.2.3 --service-account sa-with-secret --pull-secret-name regcred --ca-secret-name cert-sec`	Configure `run bundle` (and `run bundle-upgrade`) to use an image pull secret, non-default service account configured with that secret, and custom CA certificate secret

Updating bundle channels

The following examples let you update the bundle with data-informed. For further information also check Upgrade your Operator and see Channel Naming.

NOTE: Note that it will carry over any customizations you have made and ensure a rolling update to the next version of your Operator.

make bundle CHANNELS=fast,preview DEFAULT_CHANNEL=stable VERSION=1.0.0 IMG=<some-registry>/<project-name-bundle>:<tag>

NOTE You can use environment variables to pass the values such as export CHANNELS=fast,candidate. Note that, their values will be used by make bundle command.

To test your projects

Command	Description
`operator-sdk scorecard ./bundle`	Run the Scorecard tests for your bundle.
`make test`	Run Go tests. It is valid only for Go-based operators.
`molecule test`	Run Molecule tests. It is valid only for Ansible-based operators.
`helm test`	Run Helm chart tests. It is valid only for Helm-based operators.

NOTE: This is not a comprehensive list of make targets or commands. Please see the scaffolded Makefile and make help for the full list of targets. Note that you can use operator-sdk <command> --help and check the CLI section to check all options.

Docs: Operator Capability Levels

Mon, 01 Jan 0001 00:00:00 +0000

Operators come in different maturity levels in regards to their lifecycle management capabilities for the application or workload they deliver. The capability models aims to provide guidance in terminology to express what features users can expect from an operator.

Each capability level is associated with a certain set of management features the Operator offers around the managed workload. Operator that do not manage a workload and/or are delegating to off-clusters orchestration services would remain at Level 1. Capability levels are designated from level 1 to level 5. Each capability represents its own set of features and may be independent of each other.

Terminology

Operator - the custom controller installed on a Kubernetes cluster

Operand - the managed workload provided by the Operator as a service

Custom Resource (CR) - an instance of the CustomResourceDefinition the Operator ships that represents the Operand or an Operation on an Operand (also known as primary resources)

Managed resources - the Kubernetes objects or off-cluster services the Operator uses to constitute an Operand (also known as secondary resources)

Custom Resource Definition (CRD) - an API of the Operator, providing the blueprint and validation rules for Custom Resources.

Level 1 - Basic Install

Automated application provisioning and configuration management. This first capability level means your operator can fully provision an application through a custom resource, allowing all installation configuration details to be specified in the CR. It should also be possible to install the operator itself in multiple ways (kubectl, OLM, Catalog source). Any configuration required to make the Operand run should be configured through the CR if possible. Avoid the practice of requiring the user to create/manage configuration files outside of Kubernetes.

Installation of the workload

Operator deploys an Operand or configures off-cluster resources
Operator waits for managed resources to reach a healthy state
Operator conveys readiness of application or managed resources to the user leveraging the status block of the Custom Resource

Example: an Operator deploys a database by creating Deployment, ServiceAccount, RoleBinding, ConfigMap, PersistentVolumeClaim and Secret object, initializes an empty database schema and signals readiness of the database to accept queries.

Configuration of the workload

Operator provides configuration via the spec section of the Custom Resource
Operator reconciles configuration and updates to it with the status of the managed resources

Example: an Operator, managing a database, offers increasing the capacity of the database by resizing the underlying PersistentVolumeClaim based on changes the databases Custom Resource instance.

Guiding questions to determine Operator reaching Level 1

What installation configuration can be set in the CR?
What additional installation configuration could still be added?
Can you set operand configuration in the CR? If so, what configuration is supported for each operand?
Can you override the operand images through the CR or an environment variable of the Operator deployment?
Does the managed application / workload get updated in a non-disruptive fashion when the configuration of the CR is changed?
Does the status of the CR reflect that configuration changes are currently applied?
What additional operand configuration could still be added?
Do all of the instantiated CRs include a status block? If so, does it provide enough insight to the user about the application state?
Do all of your CRs have documentation listing valid values and mandatory fields?
If your operator is packaged for OLM, does its CSV list all images used in the CSV under spec.relatedImages?

Level 2 - Seamless Upgrades

Seamless upgrades mean the upgrade is as easy as possible for the user. You should support seamless upgrades of both your operator and operand, these would normally go hand in hand, an upgrade of the operator would automatically ensure the instantiated resources for each CR are in the new desired state and which would upgrade your operand. Upgrade may also be defined in multiple ways, such as updating the software of the operand - and other internals specific to the application - such as schema migrations. It should be very clear what is upgraded when this takes place, and what is not.

Upgrade of the managed workload

Operand can be upgraded in the process of upgrading the Operator, or
Operand can be upgraded as part of changing the CR
Operator understands how to upgrade older versions of the Operand, managed previously by an older version of the Operator

Upgrade of the Operator

Operator can be upgraded seamlessly and can either still manage older versions of the Operand or update them
Operator conveys inability to manage an unsupported version of the Operand in the status section of the CR

Example: An Operator managing a database can update an existing database from a previous to a newer version without data loss. The Operator might do so as part of a configuration change or as part of an update of the Operator itself.

Guiding questions to determine Operator reaching Level 2

Can your Operator upgrade your Operand?
Does your Operator upgrade your Operand during updates of the Operator?
Can your Operator manage older Operand version versions?
Is the Operand upgrade non disruptive?
If there is downtime during an upgrade, does the Operator convey this in the status of the CR?

Level 3 - Full Lifecycle

It should be possible to backup and restore the operand from the operator itself without any additional manual intervention other than triggering these operations. The operand data that should be backed up is any stateful data managed by the operand. You don’t need to backup the CR itself or the k8s resources created by the operator as the operator should return all resources to the same state if the CR is recreated. If your operator does not already setup the operand with other k8s resilient best practices, this should be completed to achieve this capability level. This includes liveness and readiness probes, multiple replicas, rolling deployment strategies, pod disruption budgets, CPU and memory requests and limits.

Lifecycle features

Operator provides the ability to create backups of the Operand
Operator is able to restore a backup of an Operand
Operator orchestrates complex re-configuration flows on the Operand
Operator implements fail-over and fail-back of clustered Operands
Operator supports add/removing members to a clustered Operand
Operator enables application-aware scaling of the Operand

Example: an Operator managing a database provides the ability to create an application consistent backup of the data by flushing the database log and quiescing the write activity to the database files.

Guiding questions to determine Operator reaching Level 3

Does your Operator support backing up the Operand?
Does your Operator support restoring an Operand from a backup and get it under management again?
Does your Operator wait for reconfiguration work to be finished and in the expected sequence?
Is your Operator taking cluster quorum into account, if present?
Does your Operator allow adding/removing read-only slave instances of your Operator?
Does your operand have a Liveness probe?
Does your operand have a Readiness probe which will fail if any aspect of the operand is not ready? e.g. if the connection to the database fails.
Does your operand use a rolling deployment strategy?
Does your operator create a PodDisruptionBudget resource for your operand pods?
Does your operand have CPU requests and limits set?

Level 4 - Deep Insights

Setup full monitoring and alerting for your operand. All resources such as Prometheus rules (alerts) and Grafana dashboards should be created by the operator when the operand CR is instantiated. The RED method¹ is a good place to start with knowing what metrics to expose. Aim to have as few alerts as possible, by alerting on symptoms that are associated with end-user pain rather than trying to catch every possible way that pain could be caused. Alerts should link to relevant consoles and make it easy to figure out which component is at fault Native k8s objects emit events (“Events” objects) for situations users or administrators should be alerted about. Your operator should do similar for state changes related to your operand. “Custom”, here, means that it should emit events specific to your operator/operand outside of the events already emitted by their deployment methodology. This, in conjunction with status descriptors for the CR conditions, give much needed visibility into actions taken by your operator/operand. Operators are codified domain-specific knowledge. Your end user should not need this domain-specific knowledge to gain visibility into what’s happening with their resource. Please, ensure that you look at the Kubernetes API conventions in the Events and status sections to know how to properly deal with them.

Monitoring

Operator exposing metrics about its health
Operator exposes health and performance metrics about the Operand

Alerting and Events

Operand sends useful alerts
Custom Resources emit custom events

Example: A database Operator continues to parse the logging output of the database software and understands noteworthy log events, e.g. running out of space for database files and produces alerts. The operator also instruments the database and exposes application level, e.g. database queries per second

Guiding questions to determine Operator reaching Level 4

Does your Operator expose a health metrics endpoint?
Does your Operator expose Operand alerts?
Do you have Standard Operating Procedures (SOPs) for each alert?
Does you operator create critical alerts when the service is down and warning alerts for all other alerts?
Does your Operator watch the Operand to create alerts?
Does your Operator emit custom Kubernetes events?
Does your Operator expose Operand performance metrics?

¹ The RED method The RED Method defines the three key metrics for every service in your architecture.

Rate (the number of requests per second)
Errors (the number of those requests that are failing)
Duration (the amount of time those requests take)

Note that by building projects using Operator-SDK or Kubebuilder CLI tools your solution leverages controller-runtime which provides the following reference exported by default. For further information, see the metrics documentation to understand how to enable monitoring and add custom metrics . Also, you may want to give a look at the (grafana/v1-alpha) which provides some JSON manifests to create Grafana dashboards using the default metrics exported.

Level 5 - Auto Pilot

The highest capability level aims to significantly reduce/eliminate any remaining manual intervention in managing the operand. The operator should configure the Operand to auto-scale as load picks up. The Operator should understand the application-level performance indicators and determine when it’s healthy and performing well. The operator should attempt to automatically fix an unhealthy operand. The operator should tune the operands performance, this could include scheduling on another node the pods are running on or modifying operand configuration.

Auto-scaling

Operator scales the Operand up under increased load based on Operand metric
Operator scales the Operand down below a certain load based on Operand metric

Auto-Healing

Operator can automatically heal unhealthy Operands based on Operand metrics/alerts/logs
Operator can prevent the Operand from transitioning into an unhealthy state based on Operand metrics

Auto-tuning

Operator is able to automatically tune the Operand to a certain workload pattern
Operator dynamically shifts workloads onto best suited nodes

Abnormality detection

Operator determines deviations from a standard performance profile

Example: A database operator monitors the query load of the database and automatically scales additional read-only slave replicas up and down. The operator also detects subpar index performance and automatically rebuilds the index in times of reduced load. Further, the operator understands the normal performance profile of the database and creates alerts on excessive amount of slow queries. In the event of slow queries and high disk latency the Operator automatically transitions the database files to another PersistentVolume of a higher performance class.

Guiding questions to determine Operator reaching Level 5

Can your operator read metrics such as requests per second or other relevant metrics and auto-scale horizontally or vertically, i.e., increasing the number of pods or resources used by pods?
Based on question number 1 can it scale down or decrease the number of pods or the total amount of resources used by pods?
Based on the deep insights built upon level 4 capabilities can your operator determine when an operand became unhealthy and take action such as redeploying, changing configurations, restoring backups etc.?
Again considering that with level 4 deep insights the operator has information to learn the performance baseline dynamically and can learn the best configurations for peak performance can it adjust the configurations to do so?
Can it move the workloads to better nodes, storage or networks to do so?
Can it detect and alert when anything is working below the learned performance baseline that can’t be corrected automatically?