Technical resources
Container image registry changes impacting your customers
Starting May 1, 2023 Red Hat will implement changes to our container image registry. Your customers who often pull container images will be impacted by this because they may not be able to download the content after the change goes live. In this post we will go into detail about what will be changing and why so that you can properly guide any of your customers who will need to adjust their firewall changes. This will also help you prepare responses to customers who will reach out with questions about how and what to do.
Why is this change happening?
Red Hat OpenShift’s operator index images are already served from registry.redhat.io using Quay.io as the backend. This is being extended to all Red Hat container images. The extension allows your customers to benefit from the high availability of the Quay.io registry while simplifying the way Red Hat delivers container images and paving the way for future enhancements. Non-OCP (OpenShift Container Platform) customers that are using container image content from our registries will also be impacted. They could either be consuming the content directly (building/running their own apps on our images) or consuming the content as part of a RH-layered product.
What’s Changing
At the present time, all image manifests and filesystem blobs are served directly from registry.redhat.io and registry.access.redhat.com. The upcoming change will mean filesystem blobs are served from Quay.io instead. To avoid problems pulling container images, your customers will need to allow outbound connections to these hostnames:
cdn.quay.io
cdn01.quay.io
cdn02.quay.io
cdn03.quay.io
This change should be made to any firewall configuration that specifically allows outbound connections to registry.redhat.io or registry.access.redhat.com. After making this change your customers will be able to continue pulling images from registry.redhat.io and registry.access.redhat.com as before.
It will not require a Quay.io login, or to interact with the Quay.io registry directly in any way, in order to continue pulling Red Hat container images.
Outbound connections to these hosts may already be allowed in your customer’s firewall configuration as a result of having previously followed the OpenShift installation instructions, or due to otherwise needing to use the Quay.io registry. Other products synchronizing or downloading container images from the Red Hat registry, such as Red Hat Ansible Automation Platform (AAP) or Red Hat Satellite, may also need changes to their relevant firewall or proxy to allow outbound access to the hosts listed above.
It is highly recommended that your customers use the hostnames instead of IP addresses when configuring firewall rules. This article explains this in greater detail.
Testing
Your customers can confirm image pulls will continue to work ahead of the registry change. To make this happen, they will need to pull the registry.redhat.io/redhat/redhat-operator-index:v4.12 image, which already has filesystem blobs hosted on Quay.io. Have your customers run the following commands using your customer portal credentials:
podman login registry.redhat.iopodman pull registry.redhat.io/redhat/redhat-operator-index:v4.12 echo $?
If the image was pulled successfully the "echo $?" command will show "0".
Additional information
Other than this major change, many things will not change or be impacted. This means:
- Red Hat container images will continue to be signed in the same way, and with the same keys.
- Container image manifests are served directly from registry.redhat.io and registry.access.redhat.com. Redirects to the Quay.io CDN are only for config and filesystem blobs.
- Pulling an image by its sha256 digest must still be done using its schema 2 digest (see this earlier article).
- Image tags, schema 2 digests, image IDs, and signatures will remain unchanged
- Images pulled before the change is made will remain valid and do not need to be re-pulled
- No changes relating to ImageContentSourcePolicy are needed for OpenShift or Kubernetes clusters
- For OpenShift or Kubernetes clusters, no node restarts, cache changes, or upgrades of any kind are needed
Allowing outbound connections to the hostnames mentioned above may resolve the following issues, depending on the characteristics of the firewall you use:
- Connection refused when pulling images
- I/O timeout when pulling images
- ImagePullBackOff status when pulling images within an OpenShift or Kubernetes cluster
Here are example errors you might see from "podman pull" with different firewall configurations:
Trying to pull [...]... WARN[0033] Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: i/o timeout WARN[0065] Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: i/o timeout WARN[0099] Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: i/o timeout Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: i/o timeout
Trying to pull [...]... WARN[0033] Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: connect: connection refused WARN[0065] Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: connect: connection refused WARN[0099] Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: connect: connection refused Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: connect: connection refused
Getting help
Your Red Hat account team or Red Hat partner is available for guidance. Alternatively, reach out to our support experts: https://access.redhat.com/support/.
