Operator SDK – Ansible Based Operator Reference

Docs: Ansible Operator Advanced Options

Mon, 01 Jan 0001 00:00:00 +0000

This document shows the advanced options available to a developer of an ansible operator.

Runner Directory

The ansible runner will keep information about the ansible run in the container. This is located /tmp/ansible-operator/runner/<group>/<version>/<kind>/<namespace>/<name>. To learn more about the runner directory you can read the ansible-runner docs.

Owner Reference Injection

Owner references enable Kubernetes Garbage Collection to clean up after a CR is deleted. Owner references are injected by ansible operators by default by the proxy.

Owner references only apply to resources in the same namespace as the CR. Resources outside the namespace of the CR will automatically be annotated with operator-sdk/primary-resource and operator-sdk/primary-resource-type to track creation. These resources will not be automatically garbage collected. To handle deletion of these resources, use a finalizer.

You may want to manage what your operator watches and the owner references. This means that your operator will need to understand how to clean up after itself when your CR is deleted. To disable these features you will need to edit your Dockerfile to include the line below.

NOTE: That if you use this feature there will be a warning that dependent watches is turned off but there will be no error. WARNING: Once a CR is deployed without owner reference injection, there is no automatic way to add those references.

ENTRYPOINT ["/usr/local/bin/entrypoint", "--inject-owner-ref=false"]

If you have created resources without owner reference injection, it is possible to manually to update resources following this guide.

Max Concurrent Reconciles

Increasing the number of concurrent reconciles allows events to be processed concurrently, which can improve reconciliation performance.

The maximum number of concurrent reconciles can be set in two ways. Operator authors and admins can set the max concurrent reconciles default by including extra args to the operator container in config/manager/manager.yaml and the patch in config/default/auth_proxy_patch.yaml. (Otherwise, the default is the maximum number of logical CPUs available for the process obtained using runtime.NumCPU().)

NOTE: Admins using OLM should use the environment variable instead of the extra args.

- name: manager
  image: "quay.io/asmacdo/memcached-operator:v0.0.0"
  imagePullPolicy: "Always"
  args:
    - "--max-concurrent-reconciles"
    - "3"

Operator admins can override the value by setting an environment variable in the format MAX_CONCURRENT_RECONCILES_<kind>_<group>. This variable must be all uppercase, and periods (e.g. in the group name) are replaced with underscores.

For the memcached operator example, the component parts are retrieved with a GET on the operator:

$ kubectl get memcacheds example-memcached -o yaml

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: example-memcached
  namespace: default

From this data, we can see that the environment variable will be MAX_CONCURRENT_RECONCILES_MEMCACHED_CACHE_EXAMPLE_COM, which we can then add to config/manager/manager.yaml and config/default/auth_proxy_patch.yaml:

- name: manager
  image: "quay.io/asmacdo/memcached-operator:v0.0.0"
  imagePullPolicy: "Always"
  args:
    # This default is overridden.
    - "--max-concurrent-reconciles"
    - "3"
  env:
    # This value is used
    - name: MAX_CONCURRENT_RECONCILES_MEMCACHED_CACHE_EXAMPLE_COM
      value: "6"

Ansible Verbosity

Setting the verbosity at which ansible-runner is run controls how verbose the output of ansible-playbook will be. The normal rules for verbosity apply here, where higher values mean more output. Acceptable values range from 0 (only the most severe messages are output) to 7 (all debugging messages are output).

There are three ways to configure the verbosity argument to the ansible-runner command:

Operator authors and admins can set the Ansible verbosity by including extra args to the operator container in the operator deployment.
Operator admins can set Ansible verbosity by setting an environment variable in the format ANSIBLE_VERBOSITY_<kind>_<group>. This variable must be all uppercase and all periods (e.g. in the group name) are replaced with underscore.
Operator users, authors, and admins can set the Ansible verbosity by setting the "ansible.sdk.operatorframework.io/verbosity" annotation on the Custom Resource.

Examples

For demonstration purposes, let us assume that we have a database operator that supports two Kinds – MongoDB and PostgreSQL – in the db.example.com Group. We have only recently implemented the support for the MongoDB Kind so we want reconciles for this Kind to be more verbose. Our operator container’s spec in our config/manager/manager.yaml and config/default/auth_proxy_patch.yaml files might contain something like:

- name: manager
  image: "quay.io/example/database-operator:v1.0.0"
  imagePullPolicy: "Always"
  args:
    # This value applies to all GVKs specified in watches.yaml
    # that are not overridden by environment variables.
    - "--ansible-verbosity"
    - "1"
  env:
    # Override the verbosity for the MongoDB kind
    - name: ANSIBLE_VERBOSITY_MONGODB_DB_EXAMPLE_COM
      value: "4"

Once the Operator is deployed, the only way to change the verbosity is via the "ansible.sdk.operatorframework.io/verbosity" annotation. Continuing with our example, our CR may look like:

apiVersion: "db.example.com/v1"
kind: "PostgreSQL"
metadata:
  name: "example-db"
  annotations:
    "ansible.sdk.operatorframework.io/verbosity": "5"
spec: {}

Custom Resources with OpenAPI Validation

Currently, SDK tool does not support and will not generate automatically the CRD’s using the OpenAPI spec to perform validations.

However, it can be done manually by adding its validations as you can check in the following example.

Example

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: memcacheds.cache.example.com
spec:
  group: cache.example.com
  names:
    kind: Memcached
    listKind: MemcachedList
    plural: memcacheds
    singular: memcached
  scope: Namespaced
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      description: Memcached is the Schema for the memcacheds API
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          description: MemcachedSpec defines the desired state of Memcached
          properties:
            size:
              description: Size is the size of the memcached deployment
              format: int32
              type: integer
          required:
          - size
          type: object
        status:
          description: MemcachedStatus defines the observed state of Memcached
          properties:
            nodes:
              description: Nodes are the names of the memcached pods
              items:
                type: string
              type: array
          required:
          - nodes
          type: object
      type: object
  versions:
  - name: v1alpha1
    served: true
    storage: true

Passing Arbitrary Arguments to Ansible

You are able to use the flag --ansible-args to pass an arbitrary argument to the Ansible-based Operator. With this option we can, for example, allow a playbook to run a specific part of the configuration without running the whole playbook:

ansible-operator run --ansible-args='--tags "configuration,packages"'

ansible-operator run --ansible-args='--skip-tags "notification"'

Ansible-runner will perform the task relevant to the command specified by the user in the ---ansible-args flag.

Using Ansible-Vault

Ansible Vault allows you to keep sensitive data such as passwords or keys in encrypted files, rather than as plaintext in playbooks or roles. You can specify Ansible-Vault file via an arbitrary argument by using the --ansible-args flag. For example, let’s assume that a playbook reads in a file vars.yml which contains an encrypted text and stores it in a variable secret:

---
- name: Playbook to print debug messages
  hosts: localhost
  tasks:
    - name: Get the decrypted message variable
      include_vars:
        file: vars.yml
        name: secret
    - debug:
        msg: The decrypted value is {{secret.the_secret}}

Now, let’s also assume that we have a password file, pwd.yml, that contains the password to decrypt the encrypted text. Then, by running the command ansible-operator run --ansible-args='--vault-password-file /absolute/path/to/pwd.yml' the operator will read in the encrypted text from the file and perform decryption using the password stored in the pwd.yml file:

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ******************************** 
ok: [localhost] => {
    "msg": "The decrypted value is DECRYPTED-TEST-VALUE"
}

-------------------------------------------------------------------------------

Using Ansible Log Events

Using the --ansible-log-events CLI flag, you can determine to what degree the Ansible task logs will be outputted. The flag can take any of the following values:

Nothing - No tasks or task-related logs will be outputted.
Tasks - Only Ansible Tasks will be outputted.
Everything - All info logs and all tasks will be outputted.

If you want more control over the logs that are outputted, consider using the Zap Logger and verbosity annotations in tandem with the --ansible-log-events CLI flag.

`ansible.sdk.operatorframework.io/reconcile-period` Custom Resource Annotation

You can specify the reconcile period for an Ansible Operator by adding the ansible.sdk.operatorframework.io/reconcile-period key to the custom resource annotations. This feature specifies the maximum interval in which a cluster will get reconciled, and defaults to 10 hours if not manually set. If changes are detected in the desired state, the cluster may be reconciled sooner than the specified interval.

The reconcile period can be specified in the custom resource’s annotations in the following manner:

...
metadata:
  name: memcached-sample
  annotations:
    ansible.sdk.operatorframework.io/reconcile-period: 5s
...

The key only accepts a value in the h/m/s format, such as 1h2m4s, 3m0s, or 4s. Values such as 1x3m9s are invalid.

NOTE: Alternatively, you can specify the reconcile period for Ansible-based Operators in the following ways:

Using the --reconcile-period command-line flag
Using the ‘reconcilePeriod’ key in the watches.yaml file You should not use all three methods to specify a single reconcile period. If all three methods are used simultaneously, the order of precedence is as follows: Custom resource annotations > watches.yaml file > command-line flag.

Docs: Ansible Operator Base Images

Mon, 01 Jan 0001 00:00:00 +0000

Ansible-based operators are built on top of base images built for use with the Operator-SDK.

Ansible Versions

For Operator-SDK versions > v1.30.0 the quay.io/operator-framework/ansible-operator base image has been updated to use Ansible 2.15. The Ansible 2.11 preview base image has been removed and will no longer be built/supported past Operator-SDK v1.30.

For Operator-SDK versions <= v1.30.0, the below information applies:

There have been some major changes in the Ansible ecosystem, primarily the addition of collections and the removal of these libraries from Ansible core. Ansible 2.9 is the last release of the “old way”, with forward compatibility for collections. This is the version officially supported by the Operator-SDK.

Ansible 2.10 was a transition release, and it is NOT recommended for use with operators.

Ansible 2.11 is the future, and Operator-SDK will eventually provide only 2.11 images. Currently, 2.11 base images are in tech-preview. See: ansible-operator-2.11-preview.

Changing Ansible Operator Base Images

For Operator-SDK versions > v1.30.0 - Operators are scaffolded with the latest version of the base image (using Ansible 2.15). The base image name is the same as previous versions of the Operator-SDK.

For Operator-SDK versions <= v1.30.0, the below information applies:

Operators are scaffolded with the latest version of the base image (using Ansible 2.9) in the first line of the operator Dockerfile.

FROM quay.io/operator-framework/ansible-operator:v1.16

Operator authors who want to try out 2.11 can simply replace their FROM with: FROM quay.io/operator-framework/ansible-operator-2.11-preview:v1.16

Docs: Dependent Watches

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the watchDependentResources option in watches.yaml file. It delves into what dependent resources are, why the option is required, how it is achieved and finally gives an example.

What are dependent resources?

In most cases, an operator creates a bunch of Kubernetes resources in the cluster, that helps deploy and manage the application. For instance, the etcd-operator creates two services and a number of pods for a single EtcdCluster CR. In this case, all the Kubernetes resources created by the operator for a CR is defined as dependent resources.

Why the `watchDependentResources` option?

Often, an operator needs to watch dependent resources. To achieve this, a developer would set the field, watchDependentResources to True in the watches.yaml file. If enabled, a change in a dependent resource will trigger the reconciliation loop causing Ansible code to run.

For example, since the etcd-operator needs to ensure that all the pods are up and running, it needs to know when a pod changes. Enabling the dependent watches option would trigger the reconciliation loop to run. The Ansible logic needs to handle these cases and make sure that all the dependent resources are in the desired state as declared by the CR spec

Note: By default it is enabled when using ansible-operator

How is this achieved?

The ansible-operator base image achieves this by leveraging the concept of owner-references. Whenever a Kubernetes resource is created by Ansible code, the ansible-operator's proxy module injects owner-references into the resource being created. The owner-references means the resource is owned by the CR for which reconciliation is taking place.

Whenever the watchDependentResources field is enabled, the ansible-operator will watch all the resources owned by the CR, registering callbacks to their change events. Upon a change, the callback will enqueue a ReconcileRequest for the CR. The enqueued reconciliation request will trigger the Reconcile function of the controller which will execute the ansible logic for reconciliation.

Example

This is an example of a watches file with the watchDependentResources field set to True


- version: v1alpha1
  group: app.example.com
  kind: AppService
  playbook: playbook.yml
  maxRunnerArtifacts: 30
  reconcilePeriod: 5s
  manageStatus: False
  watchDependentResources: True

Docs: Ansible Operator Finalizers

Mon, 01 Jan 0001 00:00:00 +0000

The default behavior of an Ansible Operator is to delete all resources the operator created during reconciliation when a managed resource is marked for deletion. This behavior is usually sufficient for applications that exist only in Kubernetes, but sometimes it is necessary to perform more complex operations (for example, when your action performed against a third party API needs to be undone). These more complex cases can still be handled by Ansible Operator, through the use of a finalizer.

Finalizers allow controllers (such as an Ansible Operator) to implement asynchronous pre-delete hooks. This allows custom logic to run after a resource has been marked for deletion, but before the resource has actually been deleted from the Kubernetes cluster. For Ansible Operator, this hook takes the form of an Ansible playbook or role. You can define the mapping from your finalizer to a playbook or role by simply setting the finalizer field on the entry in your watches.yaml. You can also choose to re-run your top-level playbook or role with different variables set. The watches.yaml finalizer configuration accepts the following options:

See Ansible watches documentation for more information.

name

name is required.

This is the name of the finalizer. This is basically an arbitrary string, the existence of any finalizer string on a resource will prevent that resource from being deleted until the finalizer is removed. Ansible Operator will remove this string from the list of finalizers on successful execution of the specified role or playbook. A typical finalizer will be <qualified-group>/finalizer, where <qualified-group> is the fully qualified group of the resource being managed.

playbook

One of playbook, role, or vars must be provided. If playbook is not provided, it will default to the playbook specified at the top level of the watches.yaml entry.

This field is identical to the top-level playbook field. It requires an absolute path to a playbook on the operator’s file system.

role

One of playbook, role, or vars must be provided. If role is not provided, it will default to the role specified at the top level of the [watches.yaml][watches] entry.

This field is identical to the top-level role field.

vars

One of playbook, role, or vars must be provided.

vars is an arbitrary map of key-value pairs. The contents of vars will be passed as extra_vars to the playbook or role specified in the finalizer block, or at the top-level if neither playbook or role was set for the finalizer.

Examples

Here are a few examples of watches.yaml files that specify a finalizer:

Run top-level playbook or role with new variables

---
- version: v1alpha1
  group: app.example.com
  kind: Database
  playbook: /opt/ansible/playbook.yml
  finalizer:
    name: app.example.com/finalizer
    vars:
      state: absent

This example will run playbook.yml when the Custom Resource is deleted. Because vars is set, the playbook will be run with state set to absent. Inside the playbook, the author can check this value and perform whatever cleanup is necessary.

---
- version: v1alpha1
  group: app.example.com
  kind: Database
  role: database
  finalizer:
    name: app.example.com/finalizer
    vars:
      state: absent

This example is nearly identical to the first, except it will run the /opt/ansible/roles/database role, rather than a playbook, with the state variable set to absent.

Run a different playbook or role

---
- version: v1alpha1
  group: app.example.com
  kind: Database
  playbook: playbook.yml
  finalizer:
    name: app.example.com/finalizer
    role: teardown_database

This example will run the /opt/ansible/roles/teardown_database role when the Custom Resource is deleted.

---
- version: v1alpha1
  group: app.example.com
  kind: Database
  playbook: playbook.yml
  finalizer:
    name: app.example.com/finalizer
    playbook: destroy.yml

This example will run the /opt/ansible/destroy.yml playbook when the Custom Resource is deleted.

---
- version: v1alpha1
  group: app.example.com
  kind: Database
  playbook: playbook.yml
  finalizer:
    name: app.example.com/finalizer
    role: myNamespace.myCollection.myRole

This example will run the myRole when the Custom Resource is deleted. (The collection must have been installed.)

Run a different playbook or role with vars

You can set playbook or role and vars at the same time. This can be useful if only a small part of your logic handles interacting with the component that requires cleanup. Rather than run all the logic again, you can specify only to run the role or playbook that handled the interaction, with a different variable set.

---
- version: v1alpha1
  group: app.example.com
  kind: Database
  playbook: playbook.yml
  finalizer:
    name: app.example.com/finalizer
    role: manage_credentials
    vars:
      state: revoked

For this example, assume our application configures automated backups to a third party service. On deletion, all we want to do is revoke the credentials used to backup the data. We run just the /opt/ansible/roles/manage_credentials role, which is imported by our playbook to create the credentials in the first place, but we pass the state: revoked option, which causes the role to invalidate our credentials. For everything else in our application, automatic deletion of dependent resources will be sufficient, so we can exit successfully and let the operator remove our finalizer and allow the resource to be deleted.

Docs: Information Flow for Operator SDK Ansible-based Operators

Mon, 01 Jan 0001 00:00:00 +0000

Controller/Reconciler

Gets the primary resource based on NamespacedName this uses the cache from controller runtime.
Determines if the resource is deleted based on the existence of a finalizer and a metadata.DeletionTimeStamp.
- If deleted and not one of our finalizers we exit with no error.
- If finalizer is needed, but is not on the object, and is not deleted then add it and continue.
Marks the status of the CR as running if it is managing the status and continues
Creates the proxy’s kubeconfig.
Calls out Runner using the runner package.
Watches for events to come back across the results channel.
- Logs the events
- Keeps track of failure messages as well as the StatusEvent, which is the ending event that the Ansible sends to mark the playbook/role is completed.
If a finalizer exists and the resource was deleted then remove the finalizer and update the object and continue.
Update the status of the resource to mark that the run has completed

Runner Package

Create the socket that ansible will communicate over
Translate object.Spec -> ansible vars (camelCase -> camel_case) and adds them as parameters for the ansible-runner directory.
Create the directory on disk for ansible runner. or update the files on disk if they already exist.
Run the ansible runner command in a go routine, when this exits close the channel for the socket.

The Proxy

Every request to the k8s api goes through the proxy.
The owner reference is injected into the object that is being created in the same namespace as the CR.
The operator-sdk annotations are injected into the object that is being created outside of namepsace of the CR.
The proxy then adds dependent watches for the correct controller if we have not started watching the type already.
On a GET, we attempt to use the informer cache to get the resource. This will also attempt to re-add dependent watches if we find a type with an owner reference.

Ansible Runner

Ansible is run and has its own process.
Ansible-runner-http package allows runner to set us ansible events as ansible is being run. ansible runner uses the socket and the controller is watching the events as they come through the channel.

Docs: Internal Operator Metrics

Mon, 01 Jan 0001 00:00:00 +0000

The Ansible Operator comes with three internal metrics that provide an insight to the frequency and time of operator reconciliations. These metrics can be scraped by a Prometheus instance or any other openmetrics system. To publish operator metrics and scrape them with an openmetrics system such as Prometheus, view Kubebuilder documentation on publishing metrics.

The default metrics recorded in Operator SDK are collected in a histogram.

The following three metrics are derived from the histogram:

ansible_operator_reconciles_bucket - Each bucket in the histogram counts the number of reconciliations that have a period (in seconds) less than or equal to the upper limit of the bucket.
ansible_operator_reconciles_count - The total number of reconciliations that have occurred up to that instance of time while running an Ansible operator.
ansible_operator_reconciles_sum - The cumulative amount of time (in seconds) of all reconciliations that have occurred up to that instance of time while running an Ansible operator.

These metrics can be queried in the Prometheus UI.

Docs: Proxy Friendly Operators

Mon, 01 Jan 0001 00:00:00 +0000

Proxy-friendly Operators should inspect their environment for the standard proxy variables (HTTPS_PROXY, HTTP_PROXY, and NO_PROXY) and pass the values to Operands.

Example:

- name: start proxy-test job
  kubernetes.core.k8s:
    definition:
      apiVersion: batch/v1
      kind: Job
      metadata:
        name: env-job
        namespace: "{{ ansible_operator_meta.namespace }}"
      spec:
        template:
          spec:
            containers:
              - name: curl-example
                image: registry.access.redhat.com/ubi8/ubi:8.8
                command: ["curl"]
                args: ["http://example.com/job-request"]
                env:
                  - name: HTTP_PROXY
                    value: '{{ lookup("env", "HTTP_PROXY") | default("", True) }}'
                  - name: http_proxy
                    value: '{{ lookup("env", "HTTP_PROXY") | default("", True) }}'
            restartPolicy: Never
        backoffLimit: 4

You can set the environment variable on the Operator deployment. Using the memcached tutorial, edit config/manager/manager.yaml:

containers:
 - args:
   - --leader-elect
   - --leader-election-id=ansible-proxy-demo
   image: controller:latest
   name: manager
   env:
     - name: "HTTP_PROXY"
       value: "http_proxy_test"

Docs: Adding Owner References for Existing Resources

Mon, 01 Jan 0001 00:00:00 +0000

Owner references are automatically injected only during creation of resources. Enabling owner reference injection will not update objects created while owner reference injection is disabled

This guide will demonstrate how to retroactively set owner references for existing resources.

A GET request to the owning resource will provide the necessary data to construct an ownerReference or an annotation.

$ kubectl get memcacheds.cache.example.com -o yaml

Example Response (Abbreviated):

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: example-memcached
  namespace: default
  uid: 2a94ff2b-84e0-40ce-8b5e-2b7e4d2bc0e2

kubectl edit can be used to update the resources by hand. See below for example ownerReference and annotations.

For objects in the same namespace as the Owner (CR)

Dependent resources within the same namespace as the owning CR are tracked with the ownerReference field.

ownerReference structure:

apiVersion: {group}/{version}
kind: {kind}
name: {metadata.name}
uid: {metadata.uid}

Example ownerReference:

metadata:
  ...(snip)
  ownerReferences:
    - apiVersion: cache.example.com/v1alpha1
      kind: Memcached
      name: example-memcached
      uid: ad834522-d9a5-4841-beac-991ff3798c00

For objects which are NOT in the same namespace as the Owner (CR)

An annotation is used instead of an ownerReference if the dependent resource is in a different namespace than the CR, or the dependent resource is a cluster level resource.

annotation structure:

operator-sdk/primary-resource: {metadata.namespace}/{metadata.name}
operator-sdk/primary-resource-type: {kind}.{group}

NOTE: The {group} can be found by splitting the apiVersion metadata of the CR, into group and version. As an example, apiVersion: cache.example.com/v1alpha1 in the config/samples directory gives us the group cache.example.com.

Example Annotation:

metadata:
  ...(snip)
  annotations:
    operator-sdk/primary-resource: default/example-memcached
    operator-sdk/primary-resource-type: Memcached.cache.example.com

Migration using Ansible assets

If you have many resources to update, it may be easier to use the following Ansible assets, which should be considered an example rather than an officially supported workflow.

To use these assets, create a vars.yml as specified below and copy playbook.yml and each_resource.yml into the same directory. Execute the playbook with:

$ ansible-playbook -i localhost playbook.yml

vars.yml

This file should be created by the user to configure the playbook, and needs to contain:

owning_resource
- apiVersion
- kind
- name
- namespace
resources_to_own (list): For each resource, specify:
- name
- namespace (if applicable)
- apiVersion
- kind

owning_resource:
  apiVersion: cache.example.com/v1alpha1
  kind: Memcached
  name: example-memcached
  namespace: default

resources_to_own:
  - name: example-memcached-memcached
    namespace: default
    apiVersion: apps/v1
    kind: Deployment
  - name: example-memcached
    apiVersion: v1
    kind: Namespace

playbook.yml

This file can be used as-is without user adjustments.

- hosts: localhost

  tasks:
    - name: Import user variables
      include_vars: vars.yml
    - name: Retrieve owning resource
      kubernetes.core.k8s_info:
        api_version: "{{ owning_resource.apiVersion }}"
        kind: "{{ owning_resource.kind }}"
        name: "{{ owning_resource.name }}"
        namespace: "{{ owning_resource.namespace }}"
      register: extra_owner_data

    - name: Ensure resources are owned
      include_tasks: each_resource.yml
      loop: "{{ resources_to_own }}"
      vars:
        to_be_owned: '{{ q("kubernetes.core.k8s",
          api_version=item.apiVersion,
          kind=item.kind,
          resource_name=item.name,
          namespace=item.namespace
        ).0 }}'
        owner_reference:
          apiVersion: "{{ owning_resource.apiVersion }}"
          kind: "{{ owning_resource.kind }}"
          name: "{{ owning_resource.name }}"
          uid: "{{ extra_owner_data.resources[0].metadata.uid }}"

`each_resource.yml`

This file can be used as-is without user adjustments.

- name: Patch resource with owner reference
  when:
    - to_be_owned.metadata.namespace is defined
    - to_be_owned.metadata.namespace == owning_resource.namespace
    - (to_be_owned.metadata.ownerReferences is not defined) or
      (owner_reference not in to_be_owned.metadata.ownerReferences)
  kubernetes.core.k8s:
    state: present
    resource_definition:
      apiVersion: "{{ to_be_owned.apiVersion }}"
      kind: "{{ to_be_owned.kind }}"
      metadata:
        name: "{{ to_be_owned.metadata.name }}"
        namespace: "{{ to_be_owned.metadata.namespace }}"
        ownerReferences: "{{ (to_be_owned.metadata.ownerReferences | default([])) + [owner_reference] }}"

- name: Patch resource with owner annotation
  when: to_be_owned.metadata.namespace is not defined or to_be_owned.metadata.namespace != owning_resource.namespace
  kubernetes.core.k8s:
    state: present
    resource_definition:
      apiVersion: "{{ to_be_owned.apiVersion }}"
      kind: "{{ to_be_owned.kind }}"
      metadata:
        name: "{{ to_be_owned.metadata.name }}"
        namespace: "{{ to_be_owned.metadata.namespace | default(omit)}}"
        annotations:
          operator-sdk/primary-resource: "{{ owning_resource.namespace }}/{{ owning_resource.name }}"
          operator-sdk/primary-resource-type: "{{ owning_resource.kind }}.{{ owning_resource.apiVersion.split('/')[0] }}"

Docs: Ansible Based Operator Scaffolding

Mon, 01 Jan 0001 00:00:00 +0000

A new Ansible operator project can be created using a command that looks like the following:

operator-sdk init --plugins ansible \
  --domain=my.domain \
  --group=apps --version=v1alpha1 --kind=AppService \
  --generate-playbook \
  --generate-role

The new project directory has many generated folders and files. For further information check the Project Layout doc.

The Deployment

The default Deployment manifest generated for the operator can be found in the config/manager/manager.yaml file. By default, the Deployment is named as ‘controller-manager’. It contains a single container named ‘manager’, and it may pick up a sidecar patch from the config/default directory. The Deployment will create a single Pod.

For the container in the Pod, there are a few things to note. The default Deployment contains a placeholder for the container image to use, so you cannot create a meaningful operator using the YAML file directly. To deploy the operator, you will run make deploy IMG=<IMG>. The image name and tag are then patched using kustomize.

The Volume Mount Path

The default EmptyDir volume mounted at /tmp/ansible-operator/runner is used to serve the input directory in ansible-runner’s terms. The mount path can NOT be changed to other paths, or else the Operator will fail to communicate with ansible-runner.

The Environment Variables

You can customize the behavior of the Ansible operator by specifying the environment variables for the container. Please refer to the Ansible Documentation for a list of environment variables that can be used to tune the behavior of the Ansible engine.

In addition to the Ansible environment variables, Operator SDK also support some special environment variables:

WATCH_NAMESPACE: This is the namespace your operator will watch for resource changes, i.e. resource create, update or delete operations. In the scaffolded operator Deployment, this is set to the same namespace in which the operator is deployed. This variable can be set to one of the following types of values:
- ‘': An empty string means that the operator will watch all namespaces. This is the default value if the WATCH_NAMESPACE environment variable is not set. It is especially useful for watching cluster-scoped resources.
- foo: The operator will watch the namespace named foo. This is the setting you will use if you are operating a namespaced resource which is deployed into a specific namespace.
- foo,bar: The operator checks the value of the environment variable and realizes that it is a comma-separated list. This means the operator will watch for resources in each of the listed namespaces.
ANSIBLE_DEBUG_LOGS: A boolean value for toggling the Ansible output during reconciliation. When set to True, the operator dumps the Ansible result into its standard output.
ANSIBLE_ROLES_PATH: The parent path(s) for the Ansible roles. When there are more than one path to set, you can use “:” to separate them. Given a path /opt/foo and a role name bar, the Ansible operator will check if the Ansible role can be found in either /opt/foo/bar or /opt/foo/roles/bar.

This value overrides the setting from the ansible-roles-path flag.
ANSIBLE_COLLECTIONS_PATH: The base path for the Ansible collections which defaults to ~/.ansible/collections or /usr/share/ansible/collections when ANSIBLE_COLLECTIONS_PATH is not explicitly specified. When a fully qualified collection name in the watches file, the Ansible operator checks if the specified collection can found under the base path that can be customized using this variable. Suppose you have ANSIBLE_COLLECTIONS_PATH set to /foo and the fully qualified collection name set to example.com.bar, the Ansible operator searches for the roles under /foo/ansible_collections/example/com/roles/bar.

This value takes precedence over the --ansible-collections-path flag.
MAX_CONCURRENT_RECONCILES_<kind>_<group>: This specifies the maximum number of concurrent reconciliations for the operator. It defaults to the number of CPUs. You can adjust this based on the cluster resources.
WORKER_<kind>_<group>: Deprecated. Use MAX_CONCURRENT_RECONCILES_<kind>_<group> instead.
ANSIBLE_VERBOSITY_<kind>_<group>: This is used to customize the verbosity of the ansible-runner command. The default value is 2. The value must be no less than 0 and no greater than 7. This value takes precedence over the global --ansible-verbosity flag, and it can be overridden by the per-resource annotation named ansible.operator-sdk/verbosity.

Docs: Ansible Operator Watches

Mon, 01 Jan 0001 00:00:00 +0000

The Watches file contains a list of mappings from custom resources, identified by it’s Group, Version, and Kind, to an Ansible Role or Playbook. The Operator expects this mapping file in a predefined location: /opt/ansible/watches.yaml These resources, as well as child resources (determined by owner references) will be monitored for updates and cached.

group: The group of the Custom Resource that you will be watching.
version: The version of the Custom Resource that you will be watching.
kind: The kind of the Custom Resource that you will be watching.
role (default): Specifies a role to be executed. This field is mutually exclusive with the “playbook” field. This field can be:
- an absolute path to a role directory.
- a relative path within one of the directories specified by ANSIBLE_ROLES_PATH environment variable or ansible-roles-path flag.
- a relative path within the current working directory, which defaults to /opt/ansible/roles.
- a fully qualified collection name of an installed Ansible collection. Ansible collections are installed to ~/.ansible/collections or /usr/share/ansible/collections by default. If they are installed elsewhere, use the ANSIBLE_COLLECTIONS_PATH environment variable or the ansible-collections-path flag
playbook: This is the playbook name that you have added to the container. This playbook is expected to be simply a way to call roles. This field is mutually exclusive with the “role” field. When running locally, the playbook is expected to be in the current project directory.
vars: This is an arbitrary map of key-value pairs. The contents will be passed as extra_vars to the playbook or role specified for this watch.
reconcilePeriod (optional): The maximum interval that the operator will wait before beginning another reconcile, even if no watched events are received. When an operator watches many resources, each reconcile can become expensive, and a low value here can actually reduce performance. Typically, this option should only be used in advanced use cases where watchDependentResources is set to False and when is not possible to use the watch feature. E.g To manage external resources that don’t emit Kubernetes events. The format for the duration string is a sequence of decimal numbers, each with optional fraction and a unit suffix, such as “300ms”, “1.5h” or “2h45m”. Valid time units are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”. Defaults to 10 hours.
manageStatus (optional): When true (default), the operator will manage the status of the CR generically. Set to false, the status of the CR is managed elsewhere, by the specified role/playbook or in a separate controller.
blacklist: A list of child resources (by GVK) that will not be watched or cached.

An example Watches file:

---
# Simple example mapping Foo to the Foo role
- version: v1alpha1
  group: foo.example.com
  kind: Foo
  role: Foo

# Simple example mapping Bar to a playbook
- version: v1alpha1
  group: bar.example.com
  kind: Bar
  playbook: playbook.yml

# More complex example for our Baz kind
# Here we will disable requeuing and be managing the CR status in the playbook,
# and specify additional variables.
- version: v1alpha1
  group: baz.example.com
  kind: Baz
  playbook: baz.yml
  manageStatus: False
  vars:
    foo: bar

# ConfigMaps owned by a Memcached CR will not be watched or cached.
- version: v1alpha1
  group: cache.example.com
  kind: Memcached
  role: /opt/ansible/roles/memcached
  blacklist:
    - group: ""
      version: v1
      kind: ConfigMap

# Example usage with a role from an installed Ansible collection
- version: v1alpha1
  group: bar.example.com
  kind: Bar
  role: myNamespace.myCollection.myRole

# Example filtering of resources with specific labels
- version: v1alpha1
  group: bar.example.com
  kind: Bar
  playbook: playbook.yml
  selector:
    matchLabels:
      foo: bar
    matchExpressions:
      - {key: foo, operator: In, values: [bar]}
      - {key: baz, operator: Exists, values: []}

The advanced features can be enabled by adding them to your watches file per GVK. They can go below the group, version, kind and playbook or role.

Some features can be overridden per resource via an annotation on that CR. The options that are overridable will have the annotation specified below.

Feature	Yaml Key	Description	Annotation for override	default	Documentation
Reconcile Period	`reconcilePeriod`	time between reconcile runs for a particular CR	ansible.sdk.operatorframework.io/reconcile-period	10h
Manage Status	`manageStatus`	Allows the ansible operator to manage the conditions section of each resource’s status section.		true
Watching Dependent Resources	`watchDependentResources`	Allows the ansible operator to dynamically watch resources that are created by ansible		true	dependent watches
Watching Cluster-Scoped Resources	`watchClusterScopedResources`	Allows the ansible operator to watch cluster-scoped resources that are created by ansible		false
Max Runner Artifacts	`maxRunnerArtifacts`	Manages the number of artifact directories that ansible runner will keep in the operator container for each individual resource.	ansible.sdk.operatorframework.io/max-runner-artifacts	20
Finalizer	`finalizer`	Sets a finalizer on the CR and maps a deletion event to a playbook or role			finalizers
Selector	`selector`	Identifies a set of objects based on their labels		None Applied	Labels and Selectors
Automatic Case Conversion	`snakeCaseParameters`	Determines whether to convert the CR spec from camelCase to snake_case before passing the contents to Ansible as extra_vars		true
Watching Annotations Changes	`watchAnnotationsChanges`	Allows the ansible operator to trigger reconciliations on annotations changes on watched resources		false

Example

---
- version: v1alpha1
  group: app.example.com
  kind: AppService
  playbook: playbook.yml
  maxRunnerArtifacts: 30
  reconcilePeriod: 5s
  manageStatus: False
  watchDependentResources: False
  snakeCaseParameters: False
  finalizer:
    name: app.example.com/finalizer
    vars:
      state: absent

Note: By using the command operator-sdk add api you are able to add additional CRDs to the project API, which can aid in designing your solution using concepts such as encapsulation, single responsibility principle, and cohesion, which could make the project easier to read, debug, and maintain. With this approach, you are able to customize and optimize the configurations more specifically per GVK via the watches.yaml file.

Example:

---
- version: v1alpha1
  group: app.example.com
  kind: AppService
  playbook: playbook.yml
  maxRunnerArtifacts: 30
  reconcilePeriod: 5s
  manageStatus: False
  watchDependentResources: False
  finalizer:
    name: app.example.com/finalizer
    vars:
      state: absent

- version: v1alpha1
  group: app.example.com
  kind: Database
  playbook: playbook.yml
  watchDependentResources: True
  manageStatus: True

Docs: Adding Admission Webhooks to an Ansible-based Operator

Mon, 01 Jan 0001 00:00:00 +0000

For general background on what admission webhooks are, why to use them, and how to build them, please refer to the official Kubernetes documentation on Extensible Admission Controllers

This guide will assume that you understand the above content, and that you have an existing admission webhook server. You will likely need to make a few modifications to the webhook server container.

When integrating an admission webhook server into your Ansible-based Operator, we recommend that you deploy it as a sidecar container alongside your operator. This allows you to make use of the proxy server that the operator deploys, as well as the cache that backs it.

Ensuring the webhook server uses the caching proxy

When an Ansible-based Operator runs, it creates a Kubernetes proxy server and serves it on http://localhost:8888. This proxy server does not require any authorization, so all you need to do to make use of the proxy is ensure that your Kubernetes client is pointing at http://localhost:8888 and that it does not attempt to verify SSL. If you use the default in-cluster configuration, you will be hitting the real API server and will not get caching for free.

Deploying the webhook server

Create a new file called config/default/manager_webhook_patch.yaml with the following content (making sure to replace the image reference placeholder string):

# This patch injects a sidecar container which is an admission webhook for the
# controller manager.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: controller-manager
  namespace: system
spec:
  template:
    spec:
      containers:
      - name: webhook
        # Replace this with the built image name
        image: "REPLACE_WEBHOOK_IMAGE"
        volumeMounts:
        - mountPath: /etc/tls/
          name: webhook-cert
      volumes:
      # This assumes there is a secret called webhook-cert containing TLS certificates
      # Projects like cert-manager can create these certificates
      - name: webhook-cert
        secret:
          secretName: webhook-cert

Then, update config/default/kustomization.yaml to include this patch:

patchesStrategicMerge:
- manager_webhook_patch.yaml # Add this line

Now, when deploying the operator with make deploy, your webhook server will run alongside the operator, but Kubernetes will not yet call the webhooks before resources can be created. In order to let Kubernetes know about your webhooks, you must create specific API resources.

Making Kubernetes call your webhooks

In order to make your webhooks callable at all, first you must create a Service that points at your webhook server. Below is a sample service that creates a Service named my-operator-webhook, that will send traffic on port 443 to port 5000 in a Pod that matches the selector name=my-operator. Modify these values to match your environment.

apiVersion: v1
kind: Service
metadata:
  name: my-operator-webhook
spec:
  ports:
  - name: webhook
    port: 443
    protocol: TCP
    # Change targetPort to match the port your server is listening on
    targetPort: 5000
  selector:
    # Change this selector to match the labels on your operator pod
    name: my-operator
  type: ClusterIP

Now that you have a Service directing traffic to your webhook server, you will need to create MutatingWebhookConfiguration or ValidatingWebhookConfiguration objects (depending on what type of webhook you have deployed), which will tell Kubernetes to send certain API requests through your webhooks before writing to etcd.

Below are examples of both MutatingWebhookConfiguration and ValidatingWebhookConfiguration objects, which will tell Kubernetes to call the my-operator-webhook service when samples.example.com Example resources are created. The mutating webhook is served on the /mutating path in my example webhook server, and the validating webhook is served on /validating. Update these values as needed to reflect your environment and desired behavior. These objects are thoroughly documented in the official Kubernetes documentation on Extensible Admission Controllers

---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: mutating.example.com
webhooks:
- name: "mutating.example.com"
  rules:
  - apiGroups:   ["samples.example.com"]
    apiVersions: ["*"]
    operations:  ["CREATE"]
    resources:   ["examples"]
    scope:       "Namespaced"
  clientConfig:
    service:
      # Replace this with the namespace your service is in
      namespace: REPLACE_NAMESPACE
      name: my-operator-webhook
      path: /mutating
  admissionReviewVersions: ["v1"]
  sideEffects: None
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: validating.example.com
webhooks:
- name: validating.example.com
  rules:
  - apiGroups:   ["samples.example.com"]
    apiVersions: ["*"]
    operations:  ["CREATE"]
    resources:   ["examples"]
    scope:       "Namespaced"
  clientConfig:
    service:
      # Replace this with the namespace your service is in
      namespace: REPLACE_NAMESPACE
      name: my-operator-webhook
      path: /validating
  admissionReviewVersions: ["v1"]
  failurePolicy: Fail
  sideEffects: None

If these resources are configured properly you will now have an admissions webhook that can reject or mutate incoming resources before they are written to the Kubernetes database.

Summary

To deploy an existing admissions webhook to validate or mutate your Kubernetes resources alongside an Ansible-based Operator, you must

Configure your admissions webhook to use the proxy server running on http://localhost:8888 in the operator pod
Add the webhook container to your operator deployment
Create a Service pointing to your webhook
Make sure your webhook is reachable via the Service over https
Create MutatingWebhookConfiguration or ValidatingWebhookConfiguration mapping the resource you want to mutate/validate to the Service you created

Operator SDK – Ansible Based Operator Reference

Docs: Ansible Operator Advanced Options

Runner Directory

Owner Reference Injection

Max Concurrent Reconciles

Ansible Verbosity

Examples

Custom Resources with OpenAPI Validation

Passing Arbitrary Arguments to Ansible

Using Ansible-Vault

Using Ansible Log Events

ansible.sdk.operatorframework.io/reconcile-period Custom Resource Annotation

Docs: Ansible Operator Base Images

Ansible Versions

Changing Ansible Operator Base Images

Docs: Dependent Watches

What are dependent resources?

Why the watchDependentResources option?

How is this achieved?

Example

Docs: Ansible Operator Finalizers

name

playbook

role

vars

Examples

Run top-level playbook or role with new variables

Run a different playbook or role

Run a different playbook or role with vars

Docs: Information Flow for Operator SDK Ansible-based Operators

Controller/Reconciler

Runner Package

The Proxy

Ansible Runner

Docs: Internal Operator Metrics

Docs: Proxy Friendly Operators

Docs: Adding Owner References for Existing Resources

For objects in the same namespace as the Owner (CR)

For objects which are NOT in the same namespace as the Owner (CR)

Migration using Ansible assets

vars.yml

playbook.yml

each_resource.yml

Docs: Ansible Based Operator Scaffolding

The Deployment

The Volume Mount Path

The Environment Variables

Docs: Ansible Operator Watches

Example

Docs: Adding Admission Webhooks to an Ansible-based Operator

Ensuring the webhook server uses the caching proxy

Deploying the webhook server

Making Kubernetes call your webhooks

Summary

`ansible.sdk.operatorframework.io/reconcile-period` Custom Resource Annotation

Why the `watchDependentResources` option?

`each_resource.yml`