Technical resources
Speed up preflight container certification for large telco workloads by leveraging Ansible parallelization
Introduction
The Red Hat software certification program includes various workflows organized by certification type. It’s designed to help telecommunications service providers, (telco) partners, and users align their Containerized Network Functions (CNFs) with Kubernetes best practice principles. The process for CNF certification incorporates container and operator bundle images certification and is managed by the Preflight certification tool, and Red Hat Connect Portal, and Red Hat Best Practice Certification for Kubernetes.
When certifying CNFs at scale, especially those involving a large number of containers, the process can be time-consuming. Even with the use of automation via the Distributed-CI (DCI) tool, managing tasks including project creation, testing, report submissions, Common Vulnerabilities and Exposures (CVE) scanning, and publishing to Red Hat’s ecosystem catalog, can become overwhelming. In one example, scanning 70 containers sequentially in a single process took approximately 4 hours. To address this challenge, we explore how leveraging Ansible's parallel execution, combined with asynchronous processing and polling, can significantly reduce the time required for container certification tasks.
Leveraging Ansible async with batching
In our test case, we used Ansible async with batching to launch multiple preflight processes in parallel. By waiting for results with async_status, we significantly reduced certification time.
To maximize efficiency in Ansible, you can use the async feature to set a maximum task time and control status checks with the poll option. A poll value of 0 allows Ansible to move on without waiting for task completion, while async_status monitors progress and waits until all tasks are finished. Teams can parallelize large workload container certifications, for first-time certifications or re-certifications, saving up to 74% of testing time compared to single-process testing.
Learn more about async, poll, and async_status in our guidebook.
Ansible asynchronous batch processing in container test
Use Case 1: New CNF Container Certifications
For our first example, let's use a case where we’re working with 10 new containers with a goal to run the certification workflow in parallel. We configured Ansible to run the scans in two batches, processing 10 containers in parallel in the first batch.
Here's how we set up our Ansible playbook to handle this scenario:
In this example, we configure 10 containers per batch where users can define up to the amount of overhead allowed). We reduced overhead by running only as many scans as the system can handle without becoming overloaded.
In general, scanning more containers through parallel processing saves more time. The feature can be turned on and off by updating the parameter below: do_container_parallel_test to true/false.
We configure each container image with relevant metadata, including product lists, short descriptions and other mandatory parameters for the Red Hat portal.
Use Case 2: CNF Container Recertification
For the CNF container recertification case, the CNF and its containers have already been certified to the catalog in the past. When a new release comes out, the certification workflow needs to rerun again to ensure it still complies with CNF and container certification standards. In this scenario, we have existing project identifiers linked to containers from previous releases. Ansible's batch processing handles the task of running the preflight check and submitting new results in parallel.
Here’s an example playbook for re-certifying containers with new releases:
In this example, the container identifiers are passed to Ansible for each container, allowing it to locate and update the existing projects in the Red Hat partner portal.
The containers are recertified in a parallel process, significantly reducing the time required to update large sets of containers.
Use Case 3: CNF container certification preflight check
For container certification checks, the process involves pre-scanning containers to see if the new release can pass the test. Unlike more detailed use cases, this check focuses only on running the preflight certification test without submitting results or performing other tasks. The main goal is to quickly give partners feedback on whether the new release meets certification requirements and to identify any issues if it doesn't. This check is also part of the one-shot certification workflow as a prerequisite.
In this scenario, Ansible's batch processing is used to run the preflight tests in parallel, providing output results for each tested container.
Here’s an example playbook for launching a preflight check in parallel:
Example of the preflight checking test results from DCI Job Server:
Time savings calculation
To illustrate the difference between the parallel and sequential preflight scans, we recorded a couple examples of the time saved using our method when applying it to use case 1; use cases 2 and 3 show similar results.
Total time saved with 20 containers and 15 images per batch:
Description | Time (minutes) |
| Total time for scanning container images one at a time | 27 |
| Total time for scanning container images in parallel | 7 |
| Total time saved | 20 |
Total time saved with 30 containers and 20 images per batch:
| Description | Time (minutes) |
| Total time for scanning container images one at a time | 40 |
| Total time for scanning container images in parallel | 10 |
| Total time saved | 30 |
Plot total time comparison with or without parallelization
Total percentage saved with 20 containers and 15 images per batch:
| Description | Percentage |
| Percentage saved =(20 minutes / 27 minutes) × 100 | 74% |
Total percentage saved with 30 containers and 20 images per batch:
| Description | Percentage |
| Percentage saved =(30 minutes / 40 minutes) × 100 | 75% |
By running the preflight to scan and submit the containers in parallel, we saved 20 minutes, equating to a time savings of approximately 74%.
The results show significant time savings and performance improvements. By using Ansible’s async and batch processing, we achieved substantial improvements in certification workflows.
Summary of the key benefits:
Time Savings: Processing container certifications in parallel reduced the overall time required by more than 74% (for 20 containers and 15 containers per batch), especially for large CNFs with hundreds of containers.
Plot histogram with 20 containers and 15 per batch
Plot histogram with 30 containers and 20 per batch
Performance efficiency: With proper batch sizing, we were able to make better use of system resources, avoiding unnecessary overhead while maximizing throughput.
Scalability: This approach scales easily to handle both small and large container certification requests, making it ideal for Red Hat partners with growing CNF workloads.
Conclusion
By adding Ansible's parallelism and async features to the DCI tool, it enables the tool to invoke multiple preflight instances to handle multiple container certificate tests, projects, and certificate publications simultaneously. This optimization streamlines the end-to-end certification process for CNF new container certification and certified CNF container renewal. This method reduces the time required to complete Red Hat certification and improves efficiency, ensuring that our partners can maintain CNF certification without sacrificing productivity.
