Updating your operators for OpenShift when Kubernetes changes APIs

Beginning with Kubernetes 1.16, the Kubernetes project periodically deprecates and removes APIs from the platform in order to maintain a clean, high functioning set of APIs.

The table below shows recent releases of Kubernetes, the version of OpenShift in which it is implemented, and a link to the removed APIs:

Operators utilizing the deprecated APIs will not work in releases of OpenShift in which the APIs are removed. Customers will subsequently be blocked from upgrading their OpenShift clusters unless the actions summarized in the below sections are taken.

To aid in this transition, OpenShift raises events and alerts for cluster administrators when an existing operator within the cluster is incompatible with the next OpenShift release. 

For more information on how Red Hat monitors the usage of deprecated APIs, the OpenShift API compatibility policy, as well as tools available to help cluster administrators identify deprecated APIs, review the Planning for and Reviewing Recent Kubernetes API Deprecations blog post.

Updating my Operator to support OpenShift after the API removal

To ensure a smooth upgrade path for users, you must specify the OpenShift version compatibility to include the latest version that includes the deprecated APIs, at a minimum, or the earliest version your operator will support. For example, for operators updating for the OpenShift 4.9 API removals, they must specify at least v4.8, where the removed APIs were still present, in the annotations.yaml file:

annotations:
com.redhat.openshift.versions: “v4.8”

Users can then upgrade the operator to a version no longer using the removed APIs prior to upgrading their OpenShift cluster to a release that no longer supports the APIs.

Ahead of any OpenShift release with removed APIs, we remove all incompatible operators from the certified operator indexes. This means that your operator will not be available to customers. Therefore, we recommend updating the APIs in your operator and recertifying ahead of your operator being removed to minimize friction 

This removal takes place close to the OpenShift version release date; we recommend you update and recertify at least one month prior to the scheduled version release date.

Once your operator is removed from an OpenShift version, there are several additional steps and considerations that must be taken to update your operator properly.

NOTE: The following examples explain how to update an operator to work with OpenShift 4.12, a version containing removed APIs. The same process may be used for any release in which Kubernetes APIs are removed.

Let’s begin with the simplest way to update and recertify: 

If you do not need, or want, users to install the previous versions of your operator and your operator does not contain any controller logic requiring users to follow a specific upgrade path, such as migrating from version A -> B -> C, follow these steps:

1. Since we officially removed incompatible operators entirely, you cannot publish new bundles with the replaces field configuration in your CSVs, because no previous version exists to replace. However, you can still publish your new bundle using the skips or skipRange options instead of replaces. 

By using these field options, you allow your users to upgrade their installed operators by skipping a version or a range of versions:

metadata:
         name: mynewoperator.v1.0.3
      annotations:
              olm.skipRange: "<1.0.3"

2. Set the com.redhat.openshift.versions annotation in the annotations.yaml file of your updated operator metadata bundle to “v4.11”:

annotations: 
    # This annotation will allow your bundle to be published on         
    # 4.11 and beyond
    com.redhat.openshift.versions: "v4.10"

Note: For further information please reference Controlling Operator compatibility with OpenShift Container Platform versions 

Additional steps

More steps and special considerations might be required in specific scenarios.  For example, you may want or need the previous versions of your operator, including the deprecated APIs, to be installable AND you want to allow upgrades of your operator after the customer has upgraded to OpenShift 4.12. 

In this scenario, the following steps must be followed:

Create two separate operator metadata bundles, each with the same structure and data you typically include, but with two differences:

a. The first bundle must be published only for OpenShift 4.11

b. The second bundle must skip previous versions and publish only to OpenShift 4.12 or 4.12+

Bundle A (targeting 4.11)

1. Configure CSV as normal, using replaces:

metadata:
   name: mynewbundle.v1.0.3
spec:
   replaces: myoldbundle.v1.0.2

2. Properly configure your annotations.yaml to ensure distribution only on OpenShift 4.11:

(annotations.yaml)
annotations: 
    # The following annotation will allow your bundle to be published on         
    # 4.11 only
    com.redhat.openshift.versions: "=v4.11"

Bundle B (targeting 4.12)

1. Configure CSV to use  skipRange instead ofreplaces:

name: mynewoperator.v1.0.3
   annotations:
      olm.skipRange: "<1.0.3"

2. Configure the annotations.yaml labels as above, only for 4.12.

 com.redhat.openshift.versions: "=v4.12"

If you DO NOT want to use skips or skipRange, use the following publishing option:

Publish two separate bundles. 

The first bundle may have any configuration you normally create, however it must only be published to OpenShift 4.11. 

The second bundle will not have any replaces, skips, or skipRange configuration and will only be published to 4.12. By removing this field, it will be as if it is the first publication made for the project.

Bundle A (targeting 4.11)

This bundle configuration will be the same as in the previous example.

1. Configure CSV as normal, using replaces:

metadata:
   name: mynewbundle.v1.0.3
spec:
   replaces: myoldbundle.v1.0.2

2. Properly configure your annotations.yaml to ensure distribution only on OpenShift 4.11:

(annotations.yaml)
annotations: 
    # The following annotation will allow your bundle to be published on         
    # 4.11 only
    com.redhat.openshift.versions: "=v4.11"

Bundle B (targeting 4.12)

1. Ensure this bundle does not use replaces, skips, or skipRange in the CSV

2. Configure the annotations.yaml as you did above, to indicate distribution only to OpenShift 4.12.

FAQ

Why can’t I use replaces to publish my new operator version? 

When using replaces, the bundle version you are replacing must exist in that specific version of the OpenShift catalog. Skips and skipRange options work similarly to replaces, but they do not have the same constraints.

However, skips and skipRange can get a bit tricky in the following circumstances:

Example:

v4 replaces v3 v3 skips v2 v2 replaces v1

In this scenario, users will be stuck at version 1. This is because OLM will never upgrade to a version that is skipped by another version. Since v2 is skipped by v3, OLM will not upgrade to v2, thus prohibiting an upgrade to v3 and beyond.

skips is intended to be used in the following way:

v4 replaces v3 v3 skips v2 and v3 replaces v1 v2 replaces v1

The original intent of skips was for when a version (v2 in this example) is released but has problems or defects, v3 is then shipped to correct the defects in v2. Therefore, the project owner “skips” v2 when possible, only allowing an upgrade from v1 to v3, and if v2 is already installed, upgrading to v3.

Why do I need to add the OCP labels to the annotations.yaml file? 

Our pipelines will only look for this label in the bundle image, which is automatically generated by the operator certification pipeline, to know what version should be added in each OCP/OLM index catalog. Ensure the value is properly added to the annotations.yaml file so the bundle image generates properly. 

If you would like more information, please reach out to epm@redhat.com