Technical resources
Keysight Telco Partner Experience: Red Hat Certification for Open RAN SIM CE
This post explores Keysight Technologies' journey to achieve Red Hat certification for its Telco workload Keysight Open RAN Simulators, Cloud Edition. Keysight's process involved container and operator assessments, automated using DCI (Distributed CI). Keysight completed two certifications, leading to unexpected improvements in development and deployment processes. We will highlight benefits of Preflight certification, emphasize security enhancements, and begin the discussion about cloud-native function (CNF) certification and CNF workloads at Keysight.
Keysight's Commitment to 5G Innovation
Keysight is an American company founded in 2014 as a spin-off of Agilent Technologies. Keysight products include hardware and software for benchtop, modular, and field instruments. Keysight began investing in preparations for 5G architecture and the accompanying demands in 2013. The needs of 5G are inherently different from 4G, as 5G aims to address existing and emerging mobile network architecture requirements for consolidated services, greater scale, and lower latency communications.
The 5G Core introduces new protocols, service-based architecture, virtualization, and network slicing, among other things. The elastic nature of the 5G Core network creates new challenges for testing the core network elements, both in isolation and in end-to-end setups.
Open RAN Simulators, Cloud Edition: Simulating 5G Core Telco Traffic
Figure 1. Open RAN Architecture
One of Keysight's portfolios includes 5G telco solutions for measuring different scenarios ― Keysight Open RAN Simulators, Cloud Edition. The product aims to produce a cloud-native application for testing the 5G Core. The product can validate 5G nodes and interfaces via specialized Keysight hardware or Virtual Edition (VE), validate complex scenarios for service-based architectures. It is used to enforce and validate multiple user-plane quality of service characteristics per flow or session, control test traffic mix and intensity using network objectives to independently manage control and user-planes and simulate UE behavior in 5G use case deployments.
With 5G development and adoption progressing rapidly, cloud-native is central to 5G Core architecture. Open RAN Simulators, Cloud Edition aligns fully with the new paradigm, which includes microservice architecture and options for deployment either as a virtual machine (VM) or container.
Keysight and Red Hat: A Technology Partnership
Figure 2. Keysight - Red Hat Partnership
The need for simulating various scenarios for 5G Core traffic has resulted in a distributed microservices architecture for the Open RAN Simulators, Cloud Edition. The architecture includes over 20 Go and Helm operators and over 60 container images. Keysight has joined the Red Hat Telco Partnership Program to ensure the workload aligns with cloud-native principles and adheres to best practices.
The goal was to Preflight and CNF-certify the Open RAN Simulators software. The ongoing partnership program is a collaborative effort involving Keysight's Wireless team and Red Hat's Telco Partner CI Team. We achieved certification with the help of DCI, a distributed CI system written in Ansible, which facilitated the automation of the certification process.
Open RAN Simulators: On the Road to Red Hat Certification for Telco Workload
Figure 3. Keysight certification process
Red Hat provides a range of certifications tailored to specific workloads. In the case of Keysight, we pursued the most comprehensive certification, covering all aspects of a CNF-native workload. The certification process involved several key steps:
1. Containers:
1.1. Preflight for Containers: This step involved checking for basic best practices, including ensuring containers do not run as root, are based on Universal Base Image (UBI), provide proper licensing, and limit layer count, among other factors.
1.2. OSCAP-Podman Scans: Conducting vulnerability scans using OSCAP-podman, encompassing over a thousand bi-weekly-updated tests.
2. Operators:
2.1. Preflight for General-Purpose Operators: This phase included four tests. Three of these tests focused on basic formatting validations provided by the operator-sdk. The fourth, DeployableByOLM, verified if the operator could be deployed by Operator Lifecycle Management (OLM) and ensured that its subscription and Cluster Service Version (CSV) were up and running.
2.2. CNF for Telco Operators: This step evaluated compliance with the best practices outlined in the CNF requirements document published by Red Hat.
Figure 4. Detailed overview of Keysight certification process
The certification process was automated with DCI:
- At first, we used the Ansible CI for testing while ensuring all certification tests passed. DCI automated the installation of Openshift Container Platform (OCP) in a private Keysight lab, deployed and set up the Open RAN Sim CE workload, and ran all the necessary certification tests, such as Preflight, OSCAP-podman, and CNF.
- Once the tests were green, DCI submitted the results and handled all certification-related tasks. These tasks included creating projects on Red Hat Partner Connect, generating and submitting test results, making pull requests in the certified operators' GitHub repository, and publishing containers and operators in the catalog.
Keysight Journey to Red Hat Certification
During the partnership, Keysight underwent two certifications: Preflight certification on OCP-4.8 in December 2022 and Preflight and CNF certification on OCP-4.11 in September 2023. The benefits were predictable at the beginning of the Preflight certification for containers. The OSCAP-podman vulnerability check meant transitioning to the latest base images and ensuring overall application security. Preflight tests like HasLicense, HasUniqueTag, HasRequiredLabel, and RunAsNonRoot reinforced fundamental best practices.
As we progressed, we needed to change the way Keysight built the images. For example, the rapid pace of the vulnerability race demanded updating the base images for more than 60 containers approximately every two weeks. Manual builds were no longer effective, prompting the Keysight development team to develop an image-build batch automation solution to bridge this gap.
Additional unexpected improvements emerged during the workload certification. The initial deployment of Open RAN Simulators (ORAN Sim), managed by Helm charts and custom Golang scripts, did not meet Red Hat's certification process. This issue led to a substantial refactoring effort to transition to a deployment that relied on a combination of Helm and Golang operators, allowing for a standard deployment with OLM.
This new deployment approach required design decisions regarding image hosting and authentication, ultimately requiring the creation of an internal, secure Harbor registry to host Keysight images.
During the second certification, we initially believed that all the benefits from the Preflight certification had already been applied to the Open RAN SIM workload. However, we unexpectedly encountered a new test, HasModifiedFiles, introduced by the Preflight team for containers. The concept behind this test is straightforward: It validates that all installed packages come from officially signed repositories and are installed in a standard 'dnf-install' manner. As designed originally, the test checked for potential malicious software and ensured the applications' vulnerability-free status. But it revealed much more in our case. The Keysight team needed to conduct rework on some Containerfiles, which involved implementing multi-stage builds, simplifying the package installation process, addressing permission issues, and cleaning up application caches.
Conclusion
In this post we primarily discussed the advantages of Preflight certification, which apply to various types of workloads rather than being Telco-specific. However, we discovered that even a basic certification prompted significant changes in the development process and how Keysight packages, delivers, and deploys its workloads. In a future post, we'll explore the CNF certification process and delve into Keysight's progress in cloud-native workloads.
Helpful Resources
Please explore the following links to learn more:
[3] Exploring Certification Test Suites Integrated in DCI
[4] Red Hat Connect Partner Community
