API deprecation in Kubernetes 1.22 that will impact your operators

The APIs being removed are the same as those removed in Kubernetes v1.22 (read more on OpenShift blog). These APIs have previously been marked as deprecated for multiple releases. 

Certified Operators that use the beta versions of the following APIs will accordingly need to be updated to use the stable APIs. These APIs are commonly used by Operators:

• CustomResourceDefinition (CRD)
• ValidatingWebhookConfiguration
• MutatingWebhookConfiguration
• CertificateSigningRequest

The CRD update is expected to be the change that will affect most partners. The good news is that it’s really easy to make this update because the stable API is almost identical to the one you’re already using – it just requires a small code update. 

• ValidatingWebhookConfiguration and MutatingWebhookConfiguration are both automatically addressed by OLM and will be created using the correct API version if they are managed as part of the operator packaging format.
• CertificateSigningRequest APIs are less commonly used. Most of the changes are strengthening validation and requiring additional fields, which you can review more here.
• OpenShift will alert users that deprecated APIs are in use and which Operators are using them. Customer clusters will be blocked from upgrading past OpenShift 4.8 if they are using your products and your Operators are using the retired APIs.

Keep in mind this change is not OpenShift-specific, the entire Kubernetes ecosystem is required to make this change and you may already be hearing about it from your customers.

Checking for usage of Deprecated APIs in your product

The fastest check is to grep your codebase and metadata for all usage of “apiextensions.k8s.io/v1beta1” and verify the above APIs aren’t being used. If you developed your operator with the Operator-SDK you can also run a compatibility test suite on the operator metadata:

$ operator-sdk bundle validate ./bundle --select-optional suite=operatorframework --optional-values=k8s-version=1.22

Inside of an OpenShift cluster, you can also navigate to the Alerts section:

Image

Information about the Users and Service Accounts using an API is available with this CLI command:

$ oc get apirequestcount.apiserver.openshift.io

Updating your Operator before OpenShift 4.9 is GA

Uploading a new version of your Operator by September 30th is key to supporting a smooth upgrade path for your customers and preventing confusion.

OpenShift 4.8 (Kubernetes 1.21) already understands the updated APIs so you may publish your updated Operator whenever you’d like. Be sure to set the metadata to indicate that it works on both 4.8 and 4.9+. For example, version 1.0.3 has been updated to use new APIs and replaces the previous version 1.0.2:

metadata:

   name: myoperator.v1.0.3

spec:

   replaces:

   - myoperator.v1.0.2

It is important that you publish an updated version of your operator to at least the OpenShift 4.8 catalog, so that users have a chance to upgrade before they upgrade their cluster to 4.9. The OLM packaging format allows to specify with which OpenShift releases your operator is compatible and to which OpenShift versions it should be published.

As a reminder, you can restrict which versions of your Operator target particular OpenShift versions.

Updating your Operator after OpenShift 4.9 is GA

If you do not push new versions compatible with 4.9 by September 30th, your package will be completely removed from OpenShift 4.9 until you have an updated version. To ensure an 4.8 to 4.9 upgrade path, also release a compatible version targeting OpenShift 4.8, which will allow you to publish an upgrade path using the skips and skipsRange configuration.

If you are unable to solve this scenario by using these options due to a complex update graph, you will need to raise a ticket asking for manual intervention and we will help you out. An example of this situation would be if you will also skip the distribution on the previous OCP version(s) and their specific constraints.

Testing your changes

After you have packaged a new version, test out your Operator on both OpenShift 4.8 and if available, an OpenShift 4.9 release candidate.

Testing bundles

To test and deploy the Operator with OLM, make sure the bundle image is present in a registry, operator-sdk run bundle can create a pod to serve that bundle to OLM via a Subscription, along with other OLM objects, ephemerally, e.g.:

$ operator-sdk run bundle <some-registry>/memcached-operator-bundle:v0.0.1

Note: If testing a bundle whose image will be hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

Upgrading a bundle to a newer version

You can also use the operator-sdk run bundle-upgrade command with your newer version of bundle image to upgrade an existing operator bundle deployed on cluster. For example, to upgrade the previously deployed memcached-operator bundle from version 0.0.1 to 0.0.2, run the following:

$ operator-sdk run bundle-upgrade <some-registry>/memcached-operator-bundle:v0.0.2

Upgrading a bundle that was installed traditionally using OLM

If you want to test your Operator bundle upgrade even if it was originally deployed using OLM without using the run bundle command. You can create a CatalogSource and deploy an Operator bundle traditionally using OLM and then upgrade the Operator bundle to a newer version. See more details.

Double check your API usage

To ensure you are not using a deprecated API anymore, you should see the alerts go away and the oc command should show no usage:

$ oc get apirequestcount.apiserver.openshift.io/validatingwebhookconfigurations.v1beta1.admissionregistration.k8s.io

NAME                              REMOVEDINRELEASE   CURRENTHOUR   LAST24H

validatingwebhookconfigurations   1.22               4             362

What if I ship a single product version to all OpenShift versions

If your product has a single version that supports all OpenShift versions, all you need to do is update to the new v1 APIs.

For CustomResourceDefinitions, which is the most commonly used API by partners, all supported versions of OpenShift (4.6, 4.7, 4.8 as of writing) contain this updated API. In fact, support goes back further than OpenShift 4.6.

How to migrate from v1beta1 to v1 CRDs (for example)

Due to the number of options available to build Operators, it is hard to provide direct guidance on how to update your CRDs (docs on v1 format). Recent versions of the OperatorSDK greater than 1.0.0 and Kubebuilder greater than 3.0.0 scaffold your project with the correct CRD version. You can also check this doc on how to migrate your project to use the latest versions of OperatorSDK.

qIf you are using OperatorSDK CLI between v0.18.x-v1.0.0 you can generate new CRDs easily:

$ operator-sdk generate crds --crd-version=v1

INFO[0000] Running CRD generator.                       

INFO[0000] CRD generation complete.

If you have been generating the manifests with controller-gen or manually then, you can just use a version greater than v0.4.1 and re-generate:

$ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role paths="./..." output:crd:artifacts:config=<add here the output-path>

Here’s a full example:

$ go get sigs.k8s.io/controller-tools/cmd/controller-gen@v0.4.1

$ controller-gen --version

Version: v0.4.1

$ controller-gen crd:crdVersions=v1,preserveUnknownFields=false rbac:roleName=manager-role paths="./..." output:crd:artifacts:config=deploy/crds/

In a live cluster, you can invoke Kubernetes’ conversion functionality by applying a crd.v1beta1 and then kubectl get a crd.v1 to view it’s converted format.

However, if your project has been using Webhooks:

Add the makers sideEffects and admissionReviewVersions to your webhook (e.g. `memcached-operator/api/v1alpha1/memcached_webhook.go`):

Example:

//+kubebuilder:webhook:path=/mutate-cache-example-com-v1alpha1-memcached,mutating=true,failurePolicy=fail,sideEffects=None,groups=cache.example.com,resources=memcacheds,verbs=create;update,versions=v1alpha1,name=mmemcached.kb.io,admissionReviewVersions={v1,v1beta1}

Then, run the command:

 controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=<add here the output-path>

Help from Red Hat

If you have any questions, please open a ticket through the Partner Connect portal and we will be happy to help.