Operator SDK – Ansible

Docs: Installation Guide

Mon, 01 Jan 0001 00:00:00 +0000

Install `operator-sdk`

Follow the steps in the installation guide to learn how to install the operator-sdk CLI tool.

Additional Prerequisites

docker version 17.03+
python version 3.8.6+
ansible version v2.9.0+
ansible-runner version 2.0.2+
ansible-runner-http version v1.0.0+
openshift version v0.12.0+
kubectl and access to a Kubernetes cluster of a compatible version.

Docs: Quickstart for Ansible-based Operators

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks through an example of building a simple memcached-operator powered by Ansible using tools and libraries provided by the Operator SDK.

Prerequisites

Go through the installation guide.
Make sure your user is authorized with cluster-admin permissions.
Have an accessible image registry for various operator images (ex. hub.docker.com, quay.io) and be logged in to your command line environment.
- example.com is used as the registry Docker Hub namespace in these examples. Replace it with another value if using a different registry or namespace.
- Authentication and certificates if the registry is private or uses a custom CA.

Steps

Create a project directory for your project and initialize the project:

mkdir memcached-operator
cd memcached-operator
operator-sdk init --domain example.com --plugins ansible

Create a simple Memcached API:

operator-sdk create api --group cache --version v1alpha1 --kind Memcached --generate-role

Build and push your operator’s image:

make docker-build docker-push IMG="example.com/memcached-operator:v0.0.1"

OLM deployment

Install OLM:

operator-sdk olm install

Bundle your operator, then build and push the bundle image (defaults to example.com/memcached-operator-bundle:v0.0.1):

make bundle IMG="example.com/memcached-operator:v0.0.1"
make bundle-build bundle-push

Run your bundle. If your bundle image is hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

operator-sdk run bundle example.com/memcached-operator-bundle:v0.0.1

Create a sample Memcached custom resource:

$ kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com/memcached-sample created

Uninstall the operator:

operator-sdk cleanup memcached-operator

Direct deployment

Deploy your operator:

make deploy IMG="example.com/memcached-operator:v0.0.1"

Create a sample Memcached custom resource:

$ kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com/memcached-sample created

Uninstall the operator:

make undeploy

Next Steps

Read the full tutorial for an in-depth walkthrough of building an Ansible operator.

Docs: Ansible Operator Tutorial

Mon, 01 Jan 0001 00:00:00 +0000

NOTE: If your project was created with an operator-sdk version prior to v1.0.0 please migrate, or consult the legacy docs.

Prerequisites

Go through the installation guide.
Make sure your user is authorized with cluster-admin permissions.
An accessible image registry for various operator images (ex. hub.docker.com, quay.io) and be logged in to your command line environment.
- example.com is used as the registry Docker Hub namespace in these examples. Replace it with another value if using a different registry or namespace.
- Authentication and certificates if the registry is private or uses a custom CA.

Overview

We will create a sample project to let you know how it works and this sample will:

Create a Memcached Deployment if it doesn’t exist
Ensure that the Deployment size is the same as specified by the Memcached CR spec
Update the Memcached CR status using the status writer with the names of the CR’s pods

Create a new project

Use the CLI to create a new memcached-operator project:

mkdir memcached-operator
cd memcached-operator
operator-sdk init --plugins=ansible --domain example.com

Among the files generated by this command is a Kubebuilder PROJECT file. Subsequent operator-sdk commands (and help text) run from the project root read this file and are aware that the project type is Ansible.

Next, we will create a Memcached API.

operator-sdk create api --group cache --version v1alpha1 --kind Memcached --generate-role

The scaffolded operator has the following structure:

Memcached Custom Resource Definition, and a sample Memcached resource.
A “Manager” that reconciles the state of the cluster to the desired state
- A reconciler, which is an Ansible Role or Playbook.
- A watches.yaml file, which connects the Memcached resource to the memcached Ansible Role.

See scaffolded files reference and watches reference for more detailed information.

Modify the Manager

Now we need to provide the reconcile logic, in the form of an Ansible Role, which will run every time a Memcached resource is created, updated, or deleted.

Update roles/memcached/tasks/main.yml:

---
- name: start memcached
  kubernetes.core.k8s:
    definition:
      kind: Deployment
      apiVersion: apps/v1
      metadata:
        name: '{{ ansible_operator_meta.name }}-memcached'
        namespace: '{{ ansible_operator_meta.namespace }}'
      spec:
        replicas: "{{size}}"
        selector:
          matchLabels:
            app: memcached
        template:
          metadata:
            labels:
              app: memcached
          spec:
            containers:
            - name: memcached
              command:
              - memcached
              - -m=64
              - -o
              - modern
              - -v
              image: "docker.io/memcached:1.4.36-alpine"
              ports:
                - containerPort: 11211

This memcached role will:

Ensure a memcached Deployment exists
Set the Deployment size

Note that the tasks in this Ansible role file are what actually defines the behavior of the spec and status of the memcached custom resource. As Kubernetes allows entry of arbitrary fields when creating resources, we don’t need to actually create specific fields in the CRD. While we won’t be doing this in this tutorial, it is recommended to also define these fields in the CRD, so that Kubernetes users can see the fields that will be used when using the custom resource. It is also good practice to set default values for variables used in Ansible Roles, so edit roles/memcached/defaults/main.yml:

---
# defaults file for Memcached
size: 1

Finally, update the Memcached sample, config/samples/cache_v1alpha1_memcached.yaml:

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  size: 3

The key-value pairs in the Custom Resource spec are passed to Ansible as extra variables.

Note: The names of all variables in the spec field are converted to snake_case by the operator before running Ansible. For example, serviceAccount in the spec becomes service_account in Ansible. You can disable this case conversion by setting the snakeCaseParameters option to false in your watches.yaml. It is recommended that you perform some type validation in Ansible on the variables to ensure that your application is receiving the expected input.

Configure the operator’s image registry

All that remains is to build and push the operator image to the desired image registry. Your Makefile composes image tags either from values written at project initialization or from the CLI. In particular, IMAGE_TAG_BASE lets you define a common image registry, namespace, and partial name for all your image tags. Update this to another registry and/or namespace if the current value is incorrect. Afterwards you can update the IMG variable definition like so:

-IMG ?= controller:latest
+IMG ?= $(IMAGE_TAG_BASE):$(VERSION)

Once done, you do not have to set IMG or any other image variable in the CLI. The following command will build and push an operator image tagged as example.com/memcached-operator:v0.0.1 to Docker Hub:

make docker-build docker-push

Run the Operator

There are three ways to run the operator:

As a Go program outside a cluster
As a Deployment inside a Kubernetes cluster
Managed by the Operator Lifecycle Manager (OLM) in bundle format

1. Run locally outside the cluster

Execute the following command, which installs your CRDs and runs the manager locally:

make install run

2. Run as a Deployment inside the cluster

By default, a new namespace is created with name <project-name>-system, ex. memcached-operator-system, and will be used for the deployment.

Commonly, Operator authors may need to modify config/rbac in order to give their Operator the necessary permissions to reconcile.

Run the following to customize the manifests and deploy the operator.

make deploy

The scaffolded Makefile uses kustomize to apply custom configurations and generate manifests from the config/ directory, which are piped to kubectl. Run the following command to see the manifests that were applied to the cluster.

kustomize build config/default

Verify that the memcached-operator is up and running:

$ kubectl get deployment -n memcached-operator-system
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
memcached-operator-controller-manager   1/1     1            1           8m

3. Deploy your Operator with OLM

First, install OLM:

operator-sdk olm install

Bundle your operator, then build and push the bundle image. The bundle target generates a bundle in the bundle directory containing manifests and metadata defining your operator. bundle-build and bundle-push build and push a bundle image defined by bundle.Dockerfile.

make bundle bundle-build bundle-push

Finally, run your bundle. If your bundle image is hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

operator-sdk run bundle example.com/memcached-operator-bundle:v0.0.1

Check out the docs for a deep dive into operator-sdk's OLM integration.

Create a Memcached CR

Update the sample Memcached CR manifest at config/samples/cache_v1alpha1_memcached.yaml and define the spec as the following:

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  size: 3

Create the CR:

kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml

Ensure that the memcached operator creates the deployment for the sample CR with the correct size:

$ kubectl get deployment
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
memcached-sample                        3/3     3            3           1m

Check the pods and CR status to confirm the status is updated with the memcached pod names:

$ kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          1m
memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          1m
memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          1m

$ kubectl get memcached/memcached-sample -o yaml
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  creationTimestamp: "2021-03-17T19:54:42Z"
  generation: 1
  managedFields:
  - apiVersion: cache.example.com/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        .: {}
        f:conditions: {}
    manager: ansible-operator
    operation: Update
    time: "2021-03-17T19:54:42Z"
  - apiVersion: cache.example.com/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:size: {}
    manager: kubectl
    operation: Update
    time: "2021-03-17T19:54:42Z"
  name: memcached-sample
  namespace: default
  resourceVersion: "1008"
  uid: 4b023125-132a-44e3-80de-20801c7a9268
spec:
  size: 3
status:
  conditions:
  - ansibleResult:
      changed: 0
      completion: 2021-03-17T19:54:54.890394
      failures: 0
      ok: 1
      skipped: 0
    lastTransitionTime: "2021-03-17T19:54:42Z"
    message: Awaiting next reconciliation
    reason: Successful
    status: "True"
    type: Running

Update the size

Update config/samples/cache_v1alpha1_memcached.yaml to change the spec.size field in the Memcached CR from 3 to 5:

kubectl patch memcached memcached-sample -p '{"spec":{"size": 5}}' --type=merge

Confirm that the operator changes the deployment size:

$ kubectl get deployment
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
memcached-sample                        5/5     5            5           3m

Cleanup

Run the following to delete all deployed resources:

kubectl delete -f config/samples/cache_v1alpha1_memcached.yaml
make undeploy

Next Steps

We recommend reading through the Ansible development section for tips and tricks, including how to run the operator locally.

In this tutorial, the scaffolded watches.yaml could be used as-is, but has additional optional features. See the watches reference.

For brevity, some of the scaffolded files were left out of this guide. See Scaffolding Reference.

This example built a namespaced scope operator, but Ansible operators can also be used with cluster-wide scope.

OLM will manage creation of most if not all resources required to run your operator, using a bit of setup from other operator-sdk commands. Check out the OLM integration guide.

Docs: Ansible Based Operator Testing with Molecule

Mon, 01 Jan 0001 00:00:00 +0000

Getting started

Requirements

To begin, you should have:

The latest version of the operator-sdk installed.
Docker installed and running
Molecule >= v3.0
Ansible >= v2.9
The Python Kubernetes client
An initialized Ansible Operator project, with the molecule directory present.

NOTE If you initialized a project with a previous version of operator-sdk, you can generate a new dummy project and copy in the molecule directory. Just be sure to generate the dummy project with the same api-version and kind, or some of the generated files will not work without modification. Your top-level project structure should look like this:

```
.
├── config
├── Dockerfile
├── Makefile
├── molecule
├── playbooks
├── PROJECT
├── requirements.yml
├── roles
└── watches.yaml

```

The Ansible content specified in requirements.yml will also need to be installed. You can install them with ansible-galaxy collection install -r requirements.yml

Molecule scenarios

If you look into the molecule directory, you will see four directories (default, test-local,cluster, templates). The default, test-local, and cluster directories contain a set of files that together make up what is known as a molecule scenario. The templates directory contains Jinja templates that are used by multiple scenarios to configure the Kubernetes cluster.

Our molecule scenarios have the following basic structure:

.
├── molecule.yml
├── prepare.yml
├── converge.yml
└── verify.yml

molecule.yml is a configuration file for molecule. It defines what driver to use to stand up an environment and the associated configuration, linting rules, and a variety of other configuration options. For full documentation on the options available here, see the molecule configuration documentation
prepare.yml is an Ansible playbook that is run once during the set up of a scenario. You can put any arbitrary Ansible in this playbook. It is used for one-time configuration of your test environment, for example, creating the cluster-wide CustomResourceDefinition that your Operator will watch.
converge.yml is an Ansible playbook that contains your core logic for the scenario. In a normal molecule scenario, this would import and run the associated role. For Ansible Operators, we mostly use this to create the Kubernetes resources necessary to deploy your operator into Kubernetes.

Below we will walk through the structure and function of each file for each scenario.

default

The default scenario is intended for use during the development of your Ansible role or playbook, and will run it outside of the context of an operator. You can run this scenario with molecule test or molecule converge. There is no corresponding operator-sdk command for this scenario.

The scenario has the following structure:

molecule/default
├── molecule.yml
├── prepare.yml
├── converge.yml
└── verify.yml

molecule.yml for this scenario tells molecule to use the docker driver to bring up a Kubernetes-in-Docker container, and by default exposes the API on the host’s port 9443. It also specifies a few inventory and environment variables which are used in prepare.yml and converge.yml.
prepare.yml ensures that a kubeconfig properly configured to connect to the Kubernetes-in-Docker cluster exists and is mapped to the proper port, and also waits for the Kubernetes API to become available before allowing testing to begin.
converge.yml imports and runs your role or playbook.
verify.yml is an Ansible playbook where you can put tasks to verify that the state of your cluster matches what you expect.

Configuration

There are a few parameters you can tweak at runtime to change the behavior of your molecule run. You can change these parameters by setting the environment variable before invoking molecule.

The options supported by the default scenario are:

Environment variable	Default	Purpose
KUBE_VERSION	1.17	The Kubernetes version to deploy
TEST_CLUSTER_PORT	9443	The port on the host to expose the Kubernetes API
TEST_OPERATOR_NAMESPACE	osdk-test	The namespace to run your role against

cluster

The cluster scenario runs an end-to-end test of your operator against an existing cluster. The operator image needs to be available to the cluster for this scenario to succeed. This scenario will deploy your CRDs, RBAC, and operator into the cluster, and then creates an instance of your CustomResource and runs your assertions to make sure the Operator responded properly.

You can run this scenario with molecule test or molecule converge. There is no corresponding operator-sdk command for this scenario.

The scenario has the following structure:

molecule/default
├── molecule.yml
├── create.yml
├── prepare.yml
├── converge.yml
├── verify.yml
└── destroy.yml

molecule.yml for this scenario uses the delegated driver, and does not spin up any additional infrastructure.
create.yml is a no-op, but must be present for the delegated driver to work.
prepare.yml ensures the CRD, namespace, and RBAC resources are present in the cluster.
converge.yml creates your operator deployment, based on the template in molecule/templates/operator.yaml.j2.
verify.yml is an Ansible playbook where you can put tasks to verify that the state of your cluster matches what you expect. By default, it creates a Custom Resource and waits for reconciliation to complete successfully. There is an example assertion present as well.
destroy.yml ensures that the namespace, RBAC resources, and CRD are deleted at the end of the run.

Configuration

There are a few parameters you can tweak at runtime to change the behavior of your molecule run. You can change these parameters by setting the environment variable before invoking molecule.

The options supported by the default scenario are:

Environment variable	Default	Purpose
OPERATOR_IMAGE	None	Required The image to use when deploying the operator into the cluster
OPERATOR_PULL_POLICY	Always	The pull policy to use when deploying the operator into the cluster
KUBECONFIG	~/.kube/config	The path to the Kubeconfig for the cluster to test against
TEST_OPERATOR_NAMESPACE	osdk-test	The namespace to run your role against

test-local

The test-local scenario runs a full end-to-end test of your operator that does not require an existing cluster or external registry, and can run in CI environments that allow users to run privileged containers (such as Travis). It brings up a Kubernetes-in-docker cluster, builds your Operator, deploys it into the cluster, and then creates an instance of your CustomResource and runs your assertions to make sure the Operator responded properly. You can run this scenario with molecule test -s local, or with molecule converge -s test-local which will leave the environment up afterward.

The scenario has the following structure:

molecule/test-local
├── molecule.yml
├── prepare.yml
├── converge.yml
└── verify.yml

molecule.yml for this scenario tells molecule to use the docker driver to bring up a Kubernetes-in-Docker container with the project root mounted, and exposes the API on the host’s port 10443. It also specifies a few inventory and environment variables which are used in prepare.yml and converge.yml. It is very similar to the default scenario’s configuration.
prepare.yml first runs the prepare.yml from the default scenario to ensure the kubeconfig is present and the API is up. It then runs the prepare.yml from the cluster scenario to configure your cluster’s CRDs and RBAC.
converge.yml connects to your Kubernetes-in-Docker container, and uses your mounted project root to build your Operator. This makes your Operator available to the cluster without needing to push it to an external registry. Then, it will ensure that a fresh deployment of your Operator is present in the cluster, using the template molecule/templates/operator.yaml.j2.
verify.yml will run the verify.yml from the cluster scenario, as the main difference between the test-local and cluster scenarios is the method of deployment, but not the behavior of the operator.

Configuration

There are a few parameters you can tweak at runtime to change the behavior of your molecule run. You can change these parameters by setting the environment variable before invoking molecule.

The options supported by the default scenario are:

Environment variable	Default	Purpose
KUBE_VERSION	1.17	The Kubernetes version to deploy
TEST_CLUSTER_PORT	10443	The port on the host to expose the Kubernetes API
TEST_OPERATOR_NAMESPACE	osdk-test	The namespace to deploy the operator and associated resources

converge vs test

The two most common molecule commands for testing during development are molecule test and molecule converge. molecule test performs a full loop, bringing a cluster up, preparing it, running your tasks, and tearing it down. molecule converge is more useful for iterative development, as it leaves your environment up between runs. This can cause unexpected problems if you end up corrupting your environment during testing, but running molecule destroy will reset it.

molecule test performs a full loop, bringing a cluster up, preparing it, running your tasks, and tearing it down.
molecule converge is more useful for iterative development, as it leaves your environment up between runs. This can cause unexpected problems if you end up corrupting your environment during testing, but running molecule destroy will reset it.

Writing tests

Adding a task

The default operator that is generated by operator-sdk new doesn’t do anything, so first we will need to add an Ansible task so that the Operator does something we can verify. For this example, we will create a simple ConfigMap with a single key. We’ll be adding the task to roles/example/tasks/main.yml, which should now look like this:

---
# tasks file for exampleapp
- name: create Example configmap
  kubernetes.core.k8s:
    definition:
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: 'test-data'
        namespace: '{{ ansible_operator_meta.namespace }}'
      data:
        hello: world

Adding a test

Now that our Operator actually does some work, we can add a corresponding assert to molecule/cluster/verify.yml. We’ll also add a debug message so that we can see what the ConfigMap looks like. The file should now look like this:

---

- name: Verify
  hosts: localhost
  connection: local
  tasks:
    - debug: var=cm
      vars:
        cm: '{{ lookup("kubernetes.core.k8s", api_version="v1", kind="ConfigMap", namespace=namespace, resource_name="test-data") }}'
    - assert:
        that: cm.data.hello == 'world'
      vars:
        cm: '{{ lookup("kubernetes.core.k8s", api_version="v1", kind="ConfigMap", namespace=namespace, resource_name="test-data") }}'

Now that we have a functional Operator, and an assertion of its behavior, we can verify that everything is working by running molecule test -s local.

The Ansible `assert` and `fail` modules

These modules are handy for adding assertions and failure conditions to your Ansible Operator tests:

assert
fail

Docs: Development Tips

Mon, 01 Jan 0001 00:00:00 +0000

This document provides some useful information and tips for a developer creating an operator powered by Ansible.

Getting started with the Kubernetes Collection for Ansible

Since we are interested in using Ansible for the lifecycle management of our application on Kubernetes, it is beneficial for a developer to get a good grasp of the Kubernetes Collection for Ansible. This Ansible collection allows a developer to either leverage their existing Kubernetes resource files (written in YAML) or express the lifecycle management in native Ansible. One of the biggest benefits of using Ansible in conjunction with existing Kubernetes resource files is the ability to use Jinja templating so that you can customize deployments with the simplicity of a few variables in Ansible.

The easiest way to get started is to install the collection on your local machine and test it using a playbook.

Installing the Kubernetes Collection for Ansible

To install the Kubernetes Collection, one must first install Ansible 2.9+. For example, on Fedora/Centos:

sudo dnf install ansible

In addition to Ansible, a user must install the Python Kubernetes Client package:

pip3 install kubernetes

Finally, install the Kubernetes Collection from ansible-galaxy:

ansible-galaxy collection install kubernetes.core

Alternatively, if you’ve already initialized your operator, you may have a requirements.yml file at the top level of your project. This file specifies Ansible dependencies that need to be installed for your operator to function. By default it will install the kubernetes.core collection as well as the operator_sdk.util collection, which provides modules and plugins for operator-specific operations.

To install the dependent modules from this file, run:

ansible-galaxy collection install -r requirements.yml

Testing the Kubernetes Collection locally

Sometimes it is beneficial for a developer to run the Ansible code from their local machine as opposed to running/rebuilding the operator each time. To do this, initialize a new project:

mkdir memcached-operator && cd memcached-operator
operator-sdk init --plugins=ansible --domain=example.com --group=cache --version=v1alpha1 --kind=Memcached --generate-role
ansible-galaxy collection install -r requirements.yml

Modify roles/memcached/tasks/main.yml with desired Ansible logic. For this example we will create and delete a ConfigMap based on the value of a variable named state:

---
- name: set ConfigMap example-config to {{ state }}
  kubernetes.core.k8s:
    api_version: v1
    kind: ConfigMap
    name: example-config
    namespace: default
    state: "{{ state }}"
  ignore_errors: true

Note

Setting ignore_errors: true is done so that deleting a nonexistent ConfigMap doesn’t error out.

Modify roles/memcached/defaults/main.yml to set state to present as default.

---
state: present

Create an Ansible playbook playbook.yml in the top-level directory which includes role memcached:

---
- hosts: localhost
  roles:
    - memcached

Run the playbook:

$ ansible-playbook playbook.yml
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'


PLAY [localhost] ***************************************************************************

TASK [Gathering Facts] *********************************************************************
ok: [localhost]

Task [memcached : set ConfigMap example-config to present]
changed: [localhost]

PLAY RECAP *********************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0

Check that the ConfigMap was created:

$ kubectl get configmaps
NAME                    STATUS    AGE
example-config          Active    3s

Rerun the playbook setting state to absent:

$ ansible-playbook playbook.yml --extra-vars state=absent
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'


PLAY [localhost] ***************************************************************************

TASK [Gathering Facts] *********************************************************************
ok: [localhost]

Task [memcached : set ConfigMap example-config to absent]
changed: [localhost]

PLAY RECAP *********************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0

Check that the ConfigMap was deleted:

$ kubectl get configmaps
No resources found in default namespace.

Using Ansible inside an Operator

Now that we have demonstrated using the Kubernetes Collection, we want to trigger this Ansible logic when a custom resource changes. In the above example, we want to map a role to a specific Kubernetes resource that the operator will watch. This mapping is done in a file called watches.yaml.

Custom Resource file

The Custom Resource (CR) file format is Kubernetes resource file. The object has some mandatory fields:

apiVersion: The version of the Custom Resource that will be created.
kind: The kind of the Custom Resource that will be created
metadata: Kubernetes specific metadata to be created
spec: This is the key-value list of variables which are passed to Ansible. This field is optional and empty by default.
annotations: Kubernetes specific annotations to be appended to the CR. See the below section for Ansible Operator specific annotations. This field is optional.

Annotations for Custom Resource

This is the list of CR annotations which will modify the behavior of the operator:

ansible.operator-sdk/reconcile-period: Specifies the maximum time before a reconciliation is triggered. Note that at scale, this can reduce performance, see watches reference for more information. This value is parsed using the standard Go package time. Specifically ParseDuration is used which will apply the default suffix of s giving the value in seconds.

Example:
```
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: example
  annotations:
    ansible.operator-sdk/reconcile-period: "30s"
```

Note that a lower period will correct entropy more quickly, but reduce responsiveness to change if there are many watched resources. Typically, this option should only be used in advanced use cases where watchDependentResources is set to False and when is not possible to use the watch feature. E.g To managing external resources that don’t raise Kubernetes events.

Testing an Ansible Operator locally

Once a developer is comfortable working with the above workflow, it will be beneficial to test the logic inside an operator.

Prerequisites:

Read the Ansible Operator tutorial.
Install ansible-operator dependencies using pipenv and their OS prerequisite packages (these will differ depending on OS) locally.

The run Makefile target runs the ansible-operator binary locally, which reads from ./watches.yaml and uses ~/.kube/config to communicate with a Kubernetes cluster just as the k8s modules do. The install target registers the operator’s Memcached CustomResourceDefinition (CRD) with the apiserver.

Note

You can customize the roles path by setting the environment variable ANSIBLE_ROLES_PATH or using the flag ansible-roles-path. Note that if the role is not found in ANSIBLE_ROLES_PATH, then the operator will look for it in {{current directory}}/roles.

$ make install run
/home/user/memcached-operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.example.com created
/home/user/go/bin/ansible-operator run
{"level":"info","ts":1595899073.9861593,"logger":"cmd","msg":"Version","Go Version":"go1.13.12","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v0.19.0+git"}
{"level":"info","ts":1595899073.987384,"logger":"cmd","msg":"WATCH_NAMESPACE environment variable not set. Watching all namespaces.","Namespace":""}
{"level":"info","ts":1595899074.9504397,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1595899074.9522583,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_MEMCACHED_CACHE_EXAMPLE_COM","default":2}
{"level":"info","ts":1595899074.9524004,"logger":"cmd","msg":"Environment variable not set; using default value","Namespace":"","envVar":"ANSIBLE_DEBUG_LOGS","ANSIBLE_DEBUG_LOGS":false}
{"level":"info","ts":1595899074.9524298,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"cache.example.com","Options.Version":"v1","Options.Kind":"Memcached"}

Now that the operator is watching resource Memcached for events, the creation of a Custom Resource will trigger our Ansible Role to be executed. Take a look at config/samples/cache_v1alpha1_memcached.yaml:

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: "memcached-sample"

Since spec is not set, Ansible is invoked with no extra variables. The next section covers how extra variables are passed from a Custom Resource to Ansible. This is why it is important to set sane defaults for the operator.

Create a Custom Resource instance of Memcached with variable state default to present:

kubectl create -f config/samples/cache_v1alpha1_memcached.yaml

Check that ConfigMap example-config was created:

$ kubectl get configmaps
NAME                    STATUS    AGE
example-config          Active    3s

Modify config/samples/cache_v1alpha1_memcached.yaml to set state to absent:

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  state: absent

Apply the changes to Kubernetes and confirm that the ConfiMap is deleted:

kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
kubectl get configmaps

Testing an Ansible Operator on a cluster

Now that a developer is confident in the operator logic, testing the operator inside of a pod on a Kubernetes cluster is desired. Running as a pod inside a Kubernetes cluster is preferred for production use.

To build the memcached-operator image and push it to a registry:

make docker-build docker-push IMG=example.com/memcached-operator:v0.0.1

Deploy the memcached-operator:

make install
make deploy IMG=example.com/memcached-operator:v0.0.1

Verify that the memcached-operator is up and running:

$ kubectl get deployment -n memcached-operator-system
NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
memcached-operator       1         1         1            1           1m

Viewing the Ansible logs

In order to see the logs from a particular operator you can run:

kubectl logs deployment/memcached-operator-controller-manager -n memcached-operator-system

The logs contain the information about the Ansible run and are useful for debugging your Ansible tasks. Note that the logs may contain much more detailed information about the Ansible Operator’s internals and its interactions with Kubernetes as well.

Also, you can set the environment variable ANSIBLE_DEBUG_LOGS to True to check the full Ansible result in the logs in order to be able to debug it.

Example

In config/manager/manager.yaml and config/default/manager_metrics_patch.yaml:

...
      containers:
      - name: manager
        env:
        - name: ANSIBLE_DEBUG_LOGS
          value: "True"
...

Occasionally while developing additional debug in the Operator logs is nice to have. Using the memcached operator as an example, we can simply add the "ansible.sdk.operatorframework.io/verbosity" annotation to the Custom Resource with the desired verbosity.

apiVersion: "cache.example.com/v1alpha1"
kind: "Memcached"
metadata:
  name: "example-memcached"
  annotations:
    "ansible.sdk.operatorframework.io/verbosity": "4"
spec:
  size: 4

Custom Resource Status Management

By default, an Ansible Operator will include the generic output from previous Ansible run as the status subresource of a CR. This includes the number of successful and failed tasks and relevant error messages as shown below:

status:
  conditions:
  - ansibleResult:
      changed: 3
      completion: 2018-12-03T13:45:57.13329
      failures: 1
      ok: 6
      skipped: 0
    lastTransitionTime: 2018-12-03T13:45:57Z
    message: 'Status code was -1 and not [200]: Request failed: <urlopen error [Errno
      113] No route to host>'
    reason: Failed
    status: "True"
    type: Failure
  - lastTransitionTime: 2018-12-03T13:46:13Z
    message: Running reconciliation
    reason: Running
    status: "True"
    type: Running

An Ansible Operator also allows you to supply custom status values with the k8s_status Ansible module, which is included in operator_sdk.util collection. You can update the status from within Ansible with any key/value pairs as desired. If you do not want the operator to update the status with Ansible output, and you want to track the CR status manually from your application, you can update the watches.yaml file with manageStatus, as shown below:

- version: v1
  group: api.example.com
  kind: Memcached
  role: memcached
  manageStatus: false

The simplest way to invoke the k8s_status module is to use its fully qualified collection name (fqcn), i.e. operator_sdk.util.k8s_status. The following example updates the status subresource with key memcached and value bar:

- operator_sdk.util.k8s_status:
    api_version: app.example.com/v1
    kind: Memcached
    name: "{{ ansible_operator_meta.name }}"
    namespace: "{{ ansible_operator_meta.namespace }}"
    status:
      foo: bar

Collections can also be declared in the role’s meta/main.yml, which is included for newly scaffolded Ansible operators.

collections:
  - operator_sdk.util

Declaring collections in the role meta allows you to invoke the k8s_status module directly.

- k8s_status:
    <snip>
    status:
      foo: bar

Ansible Operator Conditions

An Ansible Operator has a set of conditions that are used during reconciliation. There are only a few main conditions:

Running - the Ansible Operator is currently running the Ansible for reconciliation.
Successful - if the run has finished and there were no errors, the Ansible Operator will be marked as Successful. It will then wait for the next reconciliation action, either the reconcile period, dependent watches triggers or the resource is updated.
Failed - if there is any error during the reconciliation run, the Ansible Operator will be marked as Failed with the error message from the error that caused this condition. The error message is the raw output from the Ansible run for reconciliation. If the Failure is intermittent, often times the situation can be resolved when the Operator reruns the reconciliation loop.

Extra vars sent to Ansible

The extra vars that are sent to Ansible are managed by the operator. The spec section will pass along the key-value pairs as extra vars. This is equivalent to how above extra vars are passed in to ansible-playbook. The operator also passes along additional variables under the ansible_operator_meta field for the name of the CR and the namespace of the CR.

For the CR example:

apiVersion: "cache.example.com/v1alpha1"
kind: "Memcached"
metadata:
  name: "memcached-sample"
spec:
  message: "Hello world 2"
  newParameter: "newParam"

The structure passed to Ansible as extra vars is:

{ "ansible_operator_meta": {
        "name": "<cr-name>",
        "namespace": "<cr-namespace>",
  },
  "message": "Hello world 2",
  "new_parameter": "newParam",
  "_app_example_com_database": {
     <Full CR>
   },
  "_app_example_com_database_spec": {
     <Full CR .spec>
   },
}

message and newParameter are set in the top level as extra variables, and ansible_operator_meta provides the relevant metadata for the Custom Resource as defined in the operator. The ansible_operator_meta fields can be accessed via dot notation in Ansible as so:

---
- debug:
    msg: "name: {{ ansible_operator_meta.name }}, {{ ansible_operator_meta.namespace }}"

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The motivations for the new layout are related to bringing more flexibility to users and part of the process to Integrating Kubebuilder and Operator SDK. Because of this integration you may be referred to the Kubebuilder documentation https://book.kubebuilder.io/ for more information about certain topics. When using this document just remember to replace $ kubebuilder <command> with $ operator-sdk <command>.

Note: It is recommended that you have your project upgraded to the latest SDK v1.y release version before following the steps in this guide to migrate to the new layout. However, the steps might work from previous versions as well. In this case, if you find an issue which is not covered here then check the previous Migration Guides which might help out.

What was changed

The deploy directory was replaced with the config directory including a new layout of Kubernetes manifests files:
- CRD manifests in deploy/crds/ are now in config/crd/bases
- CR manifests in deploy/crds/ are now in config/samples
- Controller manifest deploy/operator.yaml is now in config/manager/manager.yaml
- RBAC manifests in deploy are now in config/rbac/
build/Dockerfile is moved to Dockerfile in the project root directory
The molecule/ directory is now more aligned to Ansible and the new Layout

What is new

Scaffolded projects now use:

kustomize to manage Kubernetes resources needed to deploy your operator
A Makefile with helpful targets for build, test, and deployment, and to give you flexibility to tailor things to your project’s needs
Updated metrics configuration using kube-auth-proxy, a --metrics-bind-address flag, and kustomize-based deployment of a Kubernetes Service and prometheus operator ServiceMonitor
Preliminary support for CLI plugins. For more info see the [plugins design document][plugins-phase1-design-doc]
A PROJECT configuration file to store information about GVKs, plugins, and help the CLI make decisions.

Generated files with the default API versions:

apiextensions/v1 for generated CRDs (apiextensions/v1beta1 was deprecated in Kubernetes 1.16 and will be removed in 1.22)
admissionregistration.k8s.io/v1 for webhooks (admissionregistration.k8s.io/v1beta1 was deprecated in Kubernetes 1.16 and will be removed in 1.22 )

How to migrate

The easy migration path is to initialize a new project, re-recreate APIs, then copy pre-v1.0.0 configuration files into the new project.

Prerequisites

Go through the installation guide.
Make sure your user is authorized with cluster-admin permissions.
An accessible image registry for various operator images (ex. hub.docker.com, quay.io) and be logged in to your command line environment.
- example.com is used as the registry Docker Hub namespace in these examples. Replace it with another value if using a different registry or namespace.
- Authentication and certificates if the registry is private or uses a custom CA.

Creating a new project

In Kubebuilder-style projects, CRD groups are defined using two different flags (--group and --domain).

When we initialize a new project, we need to specify the domain that all APIs in our project will share, so before creating the new project, we need to determine which domain we’re using for the APIs in our existing project.

To determine the domain, look at the spec.group field in your CRDs in the deploy/crds directory.

The domain is everything after the first DNS segment. Using cache.example.com as an example, the --domain would be example.com.

So let’s create a new project with the same domain (example.com):

mkdir memcached-operator
cd memcached-operator
operator-sdk init --plugins=ansible --domain=example.com

Now that we have our new project initialized, we need to recreate each of our APIs. Using our API example from earlier (cache.example.com), we’ll use cache for the --group flag.

For --version and --kind, we use spec.versions[0].name and spec.names.kind, respectively.

For each API in the existing project, run:

operator-sdk create api \
    --group=cache \
    --version=v1 \
    --kind=Memcached

Running the above command creates an empty roles/<kind>. We can copy over the content of our old roles/<kind> to the new one.

Migrating your Custom Resource samples

Update the CR manifests in config/samples with the values of the CRs in your existing project which are in deploy/crds/<group>_<version>_<kind>_cr.yaml In our example the config/samples/cache_v1alpha1_memcached.yaml will look like:

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  # TODO(user): Add fields here
  size: 3

Migrating `watches.yaml`

Update the watches.yaml file with your roles/playbooks and check if you have custom options in the watches.yaml file of your existing project. If so, update the new watches.yaml file to match.

In our example, we will replace # FIXME: Specify the role or playbook for this resource. with our previous role and it will look like:

---
# Use the 'create api' subcommand to add watches to this file.
- version: v1alpha1
  group: cache.example.com
  kind: Memcached
  role: memcached
#+kubebuilder:scaffold:watch

NOTE: Do not remove the +kubebuilder:scaffold:watch marker. It allows the tool to update the watches file when new APIs are created.

Additionally pre-1.0 the reconcilePeriod parameter was an integer representing the maximum time in seconds before a reconcile would be triggered. With 1.0, it was changed to a string representing the maximum duration before a reconcile will be triggered. Appending an s to your reconcilePeriod will set the duration unit to seconds and match the old behavior.

so for example a resource set to requeue every hour:

---
# Use the 'create api' subcommand to add watches to this file.
- version: v1alpha1
  group: cache.example.com
  kind: Memcached
  role: memcached
  reconcilePeriod: 3600
#+kubebuilder:scaffold:watch

would become

---
# Use the 'create api' subcommand to add watches to this file.
- version: v1alpha1
  group: cache.example.com
  kind: Memcached
  role: memcached
  reconcilePeriod: 3600s
#+kubebuilder:scaffold:watch

and the values 60m and 1h would be equivalent to the 3600s that is used.

Migrating your Molecule tests

If you are using Molecule in your project will be required to port the tests for the new layout.

See that default structure changed from:

├── cluster
│   ├── converge.yml
│   ├── create.yml
│   ├── destroy.yml
│   ├── molecule.yml
│   ├── prepare.yml
│   └── verify.yml
├── default
│   ├── converge.yml
│   ├── molecule.yml
│   ├── prepare.yml
│   └── verify.yml
├── templates
│   └── operator.yaml.j2
└── test-local
    ├── converge.yml
    ├── molecule.yml
    ├── prepare.yml
    └── verify.yml

To:

├── default
│   ├── converge.yml
│   ├── create.yml
│   ├── destroy.yml
│   ├── kustomize.yml
│   ├── molecule.yml
│   ├── prepare.yml
│   ├── tasks
│   │   └── foo_test.yml
│   └── verify.yml
└── kind
    ├── converge.yml
    ├── create.yml
    ├── destroy.yml
    └── molecule.yml

Ensure that the provisioner.host_vars.localhost has the following host_vars:

....
    host_vars:
      localhost:
        ansible_python_interpreter: '{{ ansible_playbook_python }}'
        config_dir: ${MOLECULE_PROJECT_DIRECTORY}/config
        samples_dir: ${MOLECULE_PROJECT_DIRECTORY}/config/samples
        operator_image: ${OPERATOR_IMAGE:-""}
        operator_pull_policy: ${OPERATOR_PULL_POLICY:-"Always"}
        kustomize: ${KUSTOMIZE_PATH:-kustomize}
...

For more information read the Testing with Molecule.

Checking RBAC Permissions

In your new project, roles are automatically generated in config/rbac/role.yaml. If you modified these permissions manually in deploy/role.yaml in your existing project, you need to re-apply them in config/rbac/role.yaml.

New projects are configured to watch all namespaces by default, so they need a ClusterRole to have the necessary permissions. Ensure that config/rbac/role.yaml remains a ClusterRole if you want to retain the default behavior of the new project conventions.

The following rules were used in earlier versions of ansible-operator to automatically create and manage services and servicemonitors for metrics collection. If your operator’s don’t require these rules, they can safely be left out of the new config/rbac/role.yaml file:

  - apiGroups:
    - monitoring.coreos.com
    resources:
    - servicemonitors
    verbs:
    - get
    - create
  - apiGroups:
    - apps
    resourceNames:
    - memcached-operator
    resources:
    - deployments/finalizers
    verbs:
    - update

Updating your ServiceAccount

New Ansible projects come with a ServiceAccount controller-manager in config/rbac/service_account.yaml. Your project’s RoleBinding and ClusterRoleBinding subjects, and Deployment’s spec.template.spec.serviceAccountName that reference a ServiceAccount already refer to this new name. When you run make deploy, your project’s name will be prepended to controller-manager, making it unique within a namespace, much like your old deploy/service_account.yaml. If you wish to use the old ServiceAccount, make sure to update all RBAC bindings and your manager Deployment.

Configuring your Operator

If your existing project has customizations in deploy/operator.yaml, they need to be ported to config/manager/manager.yaml. If you are passing custom arguments in your deployment, make sure to also update config/default/auth_proxy_patch.yaml.

Note that the following environment variables are no longer used.

OPERATOR_NAME is deprecated. It is used to define the name for a leader election config map. Operator authors should begin using --leader-election-id instead.
POD_NAME has been removed. It was used to enable a particular pod to hold the leader election lock when the Ansible operator used the leader for life mechanism. Ansible operator now uses controller-runtime’s leader with lease mechanism.

Exporting metrics

If you are using metrics and would like to keep them exported you will need to configure it in the config/default/kustomization.yaml. Please see the metrics doc to know how you can perform this setup.

The default port used by the metric endpoint binds to was changed from :8383 to :8080. To continue using port 8383, specify --metrics-bind-address=:8383 when you start the operator.

Verify the migration

The project can now be deployed on the cluster by running the command:

make deploy IMG=example.com/memcached-operator:v0.0.1

You can troubleshoot your deployment by checking the container logs:

kubectl logs deployment.apps/memcached-operator-controller-manager -n memcached-operator-system -c manager

For further steps regarding the deployment of the operator, creation of custom resources, and cleaning up of resources, see the tutorial.

Docs: Ansible Based Operator Reference

Mon, 01 Jan 0001 00:00:00 +0000

Operator SDK – Ansible

Docs: Installation Guide

Install operator-sdk

Additional Prerequisites

Docs: Quickstart for Ansible-based Operators

Prerequisites

Steps

OLM deployment

Direct deployment

Next Steps

Docs: Ansible Operator Tutorial

Prerequisites

Overview

Create a new project

Modify the Manager

Configure the operator’s image registry

Run the Operator

1. Run locally outside the cluster

2. Run as a Deployment inside the cluster

3. Deploy your Operator with OLM

Create a Memcached CR

Update the size

Cleanup

Next Steps

Docs: Ansible Based Operator Testing with Molecule

Getting started

Requirements

Molecule scenarios

default

Configuration

cluster

Configuration

test-local

Configuration

converge vs test

Writing tests

Adding a task

Adding a test

The Ansible assert and fail modules

Docs: Development Tips

Getting started with the Kubernetes Collection for Ansible

Installing the Kubernetes Collection for Ansible

Testing the Kubernetes Collection locally

Note

Using Ansible inside an Operator

Custom Resource file

Annotations for Custom Resource

Testing an Ansible Operator locally

Note

Testing an Ansible Operator on a cluster

Viewing the Ansible logs

Custom Resource Status Management

Ansible Operator Conditions

Extra vars sent to Ansible

Docs:

Overview

What was changed

What is new

How to migrate

Prerequisites

Creating a new project

Migrating your Custom Resource samples

Migrating watches.yaml

Migrating your Molecule tests

Checking RBAC Permissions

Updating your ServiceAccount

Configuring your Operator

Exporting metrics

Verify the migration

Docs: Ansible Based Operator Reference

Install `operator-sdk`

The Ansible `assert` and `fail` modules

Migrating `watches.yaml`