Technical resources
Keysight Telco Partner Experience: Addressing CNF Certification for Open RAN SIM CE
Introduction
In a previous post, we discussed the beginning of Keysight Technologies’ journey to achieve Red Hat certification for its Telco workload called Open RAN SIM CE. We described the Preflight certification process, which resulted in the certification of Keysight’s containers and operators.
This post continues the journey, focusing on the Red Hat OpenShift CNF Certification for Keysight Open RAN Simulators, Cloud Edition (Open RAN SIM CE). We will describe the changes and improvements made in the product to adapt to the certification requirements. We’ll also talk about the workflow followed in collaboration with Red Hat’s Telco Partner CI team using Distributed CI (DCI). We intend to share the experience during the process and a compilation of good practices for partners that want to include cloud-native functions (CNFs) certification in their CNF CI process. We will also share the final certification results and conclusions.
Open RAN SIM CE’s Breakdown in Microservices
Keysight has tried to break its Open RAN SIM CE into as many services as possible in order to adhere to the microservices architecture. The microservices architecture enables an organization to deliver large, complex applications rapidly, frequently, reliably, and sustainably. With that being said, Open RAN SIM CE is divided into more than 20 operators, each responsible for managing a specific aspect of the microservices ecosystem. This segmentation ensures a distributed governance model where different operators manage different tasks, such as load balancing, service discovery, and monitoring. In order to handle the complexities of the microservices ecosystem, the delegation of responsibilities among numerous operators enables a cohesive yet distributed management approach. This method also allows for targeted attention, robustness, and scalability.
Open RAN SIM CE, as mentioned in the previous blog post, provides a user interface from which you can create multiple sessions. Each session simulates a different use case with the help of the agents that contain all the protocols from the 5G architecture. From here, we can easily see a chain of operators on which the application is based: UI, SSO, NATS (to connect the agent to the middleware), session-manager, agent-management (to configure the agent based on the test objective), test-result and license-server operator (to run any type of scenario). With a microservices architecture, the implementation of this kind of workflow becomes easier and more accessible for implementing improvements, changes, and updates.
The Automation Journey Toward CNF Certification
CNF Certification verifies that OpenShift applications comply with commonly recognized industry standards and best practices. It is not enough to describe a set of workloads as CNFs (which is the scope of the Vendor-Validation process). These workloads must meet strict requirements to qualify as CNFs within the scope of Red Hat CNF Certification.
To address CNF certification, Keysight, in collaboration with Red Hat’s Telco Partner CI team, used DCI as a support tool. After achieving Keysight vendor validation for its microservices, seen as CNFs within this scope, we followed the recommended good practices for CNF certification with DCI, already published in this blog (see part 1 and part 2), to get the most from this experience.
In particular, DCI helped Keysight automate the onboarding of the OpenShift cluster and the deployment and testing of Open RAN SIM CE’s CNFs. This approach is crucial since the CNF Certification test suite requires the workloads under test to be up and running in the OCP cluster, as it runs autodiscovery mechanisms to detect the resources to test (and to ignore). The suite, also known as CNF Cert Suite, tests the workloads to be certified and generates the results to be submitted for certification. DCI also provided capabilities to Keysight to properly configure the CNF Cert Suite to Autodiscover the identified CNFs from Open RAN SIM CE.
As recommended in the CNF certification good practices, Keysight and Red Hat drafted a test plan to identify the CNF scenarios to target. In our case, we covered the entire telco scenario, including the most basic tests from the non-telco scenario. By running the CNF Cert Suite for the first time, we identified the test cases that we needed to fix.
After this, we established daily jobs. We launched Open RAN SIM CE deployment on a daily basis. Included in each iteration were the modifications Keysight implemented to fix the failed test cases. We monitored this process through a dashboard made by the Red Hat Telco Partner CI team, where we analyzed the number of CNF tests that passed, skipped, or failed, sending a daily email to report the results. Here is an example of an extraction of the daily dashboard during the last certification process we addressed together:
Figure 1. An example of the dashboard used in the certification process
During this CI testing workflow, we worked with two living documents to analyze the evolution of the CNF test cases:
- An Excel file where we recorded the results of all CNF Certification test cases on each iteration, updating the results with the changes made to the product and the new results obtained in the DCI jobs. Figure 2 is an example of the Excel file we used to order the test cases and label them as reviewed. The file shows the classification we used to determine the order of the test cases, as well as the labeling system we used to indicate which test cases had been reviewed.
Figure 2. An example of the Excel file used to order and label the test cases
- A document to report the exceptions for the certification process. Any test case that cannot pass because of the CNF’s code and behavior requires an exception request that explains why it is failing or being skipped. Consequently, while we analyzed the evolution of the test cases in the Excel file, we included in this other document the reasons some tests were failed or skipped.
We completed our iterative work once we had successfully passed all the test cases, except those we deemed impossible to fix. We submitted exception requests for those cases. Additionally, we did not run certification-related tests as the CNF certification requires the certification of container and operator images with Preflight before submitting the request. Since Keysight was making changes to the images to fix the failed tests from the CNF Cert Suite, we had to wait until the end of the CNF certification testing process to run Preflight certification. Once we verified that the containers and operators with the changes for CNF certification were Preflight certified, we could submit the CNF certification request. With all this, we were ready to submit the CNF certification test results, again following the best practices, which eventually resulted in the direct acceptance of the certification request without any additional iteration.
Keysight’s product now appears in the catalog as CNF Certified in OCP 4.11 for partner product version 1.0. Also, Red Hat published a KB article to report the results of the CNF certification, including the certified versions and exceptions that were accepted. Note: You need a Red Hat account to access these resources.
Next, we’ll look at what kinds of changes Keysight made to fix all possible CNF Cert Suite test cases.
Improvements Required to Address CNF Certification
During the CNF certification, the Keysight development team concluded that some important changes were necessary to reach the best practices proposed by the CNF certification framework.
First, we started modifying the charts. The charts were initially set to use cluster role bindings, but they only needed role binding capabilities, according to best practices. This enhancement was important for Open RAN SIM CE as some of the pods requested unnecessary privileges. The addition of the liveness, readiness, and startup probes — which give a status on the running pod — was another significant development.
In the end, all the capabilities introduced to the application benefit end users, enabling them to readily debug any issues that may arise using the good practices proposed by the community.
Conclusions and lessons learned
In this blog post, we have seen how Keysight’s identified microservices (that is, the CNF workloads) in Open RAN SIM CE passed Red Hat’s CNF certification on the first attempt. That success stems in part from following good practices defined by the Red Hat Telco Partner CI team and using Distributed CI as a support tool to run the CNF Cert Suite. Apart from the workflow followed, we have seen the improvements Keysight had to make in the product to reach the certification goal. We hope this serves as a reference for future CNF certification cases.
