Container Image Registries Changes Part 3 - Consequences

As covered in previous in part 1 and 2 of this blog series , there are changes coming to the Red Hat container image registries. Here we will explore why it is important to be prepared for them.

These changes will start being deployed May 1, 2023. It will not be applied to all repositories at once but will occur in phases over the subsequent weeks.

Take action now to avoid problems

As a reminder, if you are pulling container images from registry.redhat.io or registry.access.redhat.com in an environment that restricts outbound network connections, be sure to adjust the firewall configuration to allow connections to the Quay.io CDN hosts. More information about this can be found in this article.

When you have made changes but want to verify they will be sufficient, pull the registry.redhat.io/redhat/redhat-operator-index:v4.12 image – this already utilizes Quay.io for image pulls.

In general, wherever outbound connections to registry.redhat.io or registry.access.redhat.com are allowed, so too should connections to the Quay.io CDN hosts.

Let's look at some of the types of failure you might face if you leave this too late.

Failing image builds

There are many ways of building container images, but let's focus on one example, that of building from a Containerfile using buildah. A common image to use in the FROM line is the Red Hat Universal Base Image, ubi:

FROM ubi8/ubi

...

Without taking action, building this image in an environment that restricts outbound network access may result in errors like this:

$ buildah bud -f Containerfile

STEP 1/2: FROM ubi8/ubi

Resolved "ubi8/ubi" as an alias

(/etc/containers/registries.conf.d/000-shortnames.conf)

Trying to pull registry.access.redhat.com/ubi8/ubi:latest...

WARN[0006] Failed, retrying in 2s ... (1/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp 18.165.98.111:443: connect: connection refused

In this case, cdn02.quay.io is the Quay.io CDN host used to serve one of the filesystem layer blobs, but the restrictive firewall has prevented access to it by rejecting the connection. The resolution will be to allow access to all of the Quay.io CDN hosts.

The same problem may be seen when building an image using any other Red Hat base image (such as RHEL8 or RHEL9).

Failing to run images

Not just building images, but running images may be affected, if those images have not already been pulled locally. For example, when running a web server like this:

$ podman run registry.access.redhat.com/ubi9/httpd-24

Trying to pull registry.access.redhat.com/ubi9/httpd-24:latest...

WARN[0005] Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp 143.204.176.119:443: connect: connection refused

Failing to mirror images

Don't forget about any local mirrors of Red Hat container images you have.

In order to keep them in sync the original content needs to be accessible from Red Hat container image registries, whether you are using 'oc', 'oc-mirror', 'skopeo sync' or some other method. If the environment performing the sync restricts outbound network access, make sure it can access the Quay.io CDN hosts as well as the Red Hat container image registries.