Container Image Registries Changes Part 3 - Consequences
As covered in previous in part 1 and 2 of this blog series , there are changes coming to the Red Hat container image registries. Here we will explore why it is important to be prepared for them.
These changes will start being deployed May 1, 2023. It will not be applied to all repositories at once but will occur in phases over the subsequent weeks.
Take action now to avoid problems
As a reminder, if you are pulling container images from registry.redhat.io or registry.access.redhat.com in an environment that restricts outbound network connections, be sure to adjust the firewall configuration to allow connections to the Quay.io CDN hosts. More information about this can be found in this article.
When you have made changes but want to verify they will be sufficient, pull the
registry.redhat.io/redhat/redhat-operator-index:v4.12 image – this already utilizes Quay.io for image pulls.
In general, wherever outbound connections to registry.redhat.io or registry.access.redhat.com are allowed, so too should connections to the Quay.io CDN hosts.
Let's look at some of the types of failure you might face if you leave this too late.
Failing image builds
There are many ways of building container images, but let's focus on one example, that of building from a Containerfile using buildah. A common image to use in the FROM line is the Red Hat Universal Base Image, ubi:
Without taking action, building this image in an environment that restricts outbound network access may result in errors like this:
$ buildah bud -f Containerfile
STEP 1/2: FROM ubi8/ubi
Resolved "ubi8/ubi" as an alias
Trying to pull registry.access.redhat.com/ubi8/ubi:latest...
WARN Failed, retrying in 2s ... (1/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp 184.108.40.206:443: connect: connection refused
In this case, cdn02.quay.io is the Quay.io CDN host used to serve one of the filesystem layer blobs, but the restrictive firewall has prevented access to it by rejecting the connection. The resolution will be to allow access to all of the Quay.io CDN hosts.
The same problem may be seen when building an image using any other Red Hat base image (such as RHEL8 or RHEL9).
Failing to run images
Not just building images, but running images may be affected, if those images have not already been pulled locally. For example, when running a web server like this:
$ podman run registry.access.redhat.com/ubi9/httpd-24
Trying to pull registry.access.redhat.com/ubi9/httpd-24:latest...
WARN Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp 220.127.116.11:443: connect: connection refused
Failing to mirror images
Don't forget about any local mirrors of Red Hat container images you have.
In order to keep them in sync the original content needs to be accessible from Red Hat container image registries, whether you are using 'oc', 'oc-mirror', 'skopeo sync' or some other method. If the environment performing the sync restricts outbound network access, make sure it can access the Quay.io CDN hosts as well as the Red Hat container image registries.