Technical resources
Container Image Registries Changes Part 2
As covered in our first blog , there are changes coming to the Red Hat container image registries. Here we will dive a little deeper into what those changes are, and why you need to take action to adapt to them.
Pulling container images
What happens when you run a command like "podman pull"? Let's take a look at an example which has already been changed to use the Quay.io CDN:
podman pull registry.redhat.io/redhat/redhat-operator/index:v4.12
Pulling a container image from a registry means following the registry HTTP specification, specifically the Docker HTTP Registry API for the Red Hat registries. For simplicity, we'll ignore the OAuth2 authentication flow and API version check, and focus on the actual image pull. We'll also ignore manifest lists, as they are not relevant to the discussion.
There are several objects to be fetched from the registry in order to pull the container image: the image manifest, the 'config' blob, and the filesystem layer blobs.
The filesystem layer blobs are the actual content of the image in the form of compressed archives of the filesystem layers in the image. The 'config' blob is a description of the labels, entrypoint, working directory, and other metadata relating to the image, in JSON format. The image manifest ties the 'config' blob together with the filesystem layer blobs (in the right order), also in JSON format.
Assembling the image
First, podman will fetch the image manifest with an HTTP request like this:
GET /v2/redhat/redhat-operator-index/manifests/v4.12
The response will contain a list of the filesystem layer blob sha256 digests, as well as the sha256 digest of a 'config' blob. Next, podman will fetch each of those blobs by their digests. Each request will look like this:
GET /v2/redhat/redhat-operator-index/blobs/sha256:…
Each request will result in an HTTP 302 response, which details a different URL from which to fetch the content. The response will have a Location header, something like this:
Location: https://cdn02.quay.io/sha256/…
Podman follows these redirect responses, issuing new GET requests to the indicated URL each time. Once it has received all the blobs it can begin assembling them to form a local copy of the image.
Manifests and blobs
While the image manifests are returned directly in responses from the Red Hat registries, requests for the blobs (which make up the image content and metadata) result in responses that redirect to another host. In the example above, the redirect location header points to cdn02.quay.io. What is this?
Quay.io's content delivery network uses AWS CloudFront. The blobs for container images are stored there, rather than in the Quay.io registry itself. For the image in this example, registry.redhat.io acts as a proxy to Quay.io. Customers can pull images using registry.redhat.io as the registry, but with the data coming from a private Quay.io organization.
Firewall configuration
This arrangement, of using Quay.io CDN hosts for blobs, will be extended to all container images on the Red Hat registries soon (see this article). When this happens, all customer systems which pull images from Red Hat registries registry.redhat.io or redhat.access.redhat.com will also need to be able to make outbound TCP connections to the Quay.io CDN hosts. If they cannot, pulling images will fail.
Customers should configure their firewalls to allow connections to the hostnames listed in the article mentioned above, rather than CIDR IP address ranges. The reason for this is to avoid the potential need to update them in the future. For a customer using CIDR ranges instead of hostnames in the firewall configuration, there are a couple of situations where they may need to reconfigure them in the future. They are:
- AWS CloudFront may update, add or remove address ranges (current ranges are here)
- Red Hat may choose a different CDN provider for the Quay.io in future
Trust
What prevents non-Red Hat containers from Quay.io being pulled, now that connections to the Quay.io CDN are allowed?
The Red Hat registries, registry.redhat.io and registry.access.redhat.com, return a header in the HTTP 302 redirect response which allows access only to specific content in the Quay.io CDN for a short period of time. Due to this mechanism, no image content can be pulled directly from the Quay.io CDN hosts – it must instead be pulled using a container registry. The Red Hat registries only provide access to Red Hat container images.
