Technology certification
Suggested workflow to properly prepare your OpenShift workloads for Certification
Introduction
The Red Hat Software Certification program involves multiple workflows, depending on the workload to be certified. Partners and users can accommodate their software to follow Red Hat best practices for containerized applications, managed by preflight, using Red Hat® OpenShift®, Container, and Operator certification. Users can also utilize Meets Best Practices and CNF Certification and Vendor Validation, all managed by Red Hat Best Practices Test Suite for Kubernetes, also known as Certsuite.
However, the certification journey usually implies modifying the software prior to certifying it for compliance with best practices proposed by each certification workflow. Following a proper workflow and having the proper support tools to quickly detect and determine what’s failing and what’s been fixed on a continuous integration (CI) basis becomes necessary to reach the software certification with the desired result.
In this blog, we’ll share recommendations and best practices from the Telco Partner CI and internal Red Hat teams when working with partners. Our goal is to help you prepare Red Hat OpenShift workloads for certification, using a process or workflow that you can easily replicate in collaboration with partners and Red Hat teams. As you implement this workflow, we’ll show you how to automate preflight and CNF Certification tests report and analysis launched with Distributed-CI (DCI) using spreadsheets to gather all the results— with the ability to check for changes in the test results. These tools will be useful for giving Red Hat teams the means to follow up the certification process with a Red Hat partner and to justify exception requests based on what is observed in the support tools.
Suggested pre-certification workflow
Before starting with your certification journey, you, along with your partners and the Red Hat team, will make decisions that will likely impact your experience in trying to certify the target software. Consider, among other things:
- What kind of certification do I need to run for the Red Hat OpenShift workload? Are there ways of simplifying the execution of all potential certification tools by using a single entry point?
- If there are multiple certification tools to run, is there any dependency between them?
- How should I handle the failures or unexpected results I may obtain when executing the certification tools?
Developing a test plan will explicitly define the tests you need to run, the execution order of each test along with the dependencies between them, and the tools needed to run the execution and to gather the results for subsequent reporting and troubleshooting. You can also exchange information between all parties working on the pre-certification process, making your test plan a support tool for proper decision making during the certification journey.
Pre-certification work is transformed into an iterative process, usually supported by CI mechanisms. You can run certification tests in the order you have defined, retrieve the results, and decide on how to proceed in situations where the results are unexpected. Teams are encouraged to discuss test execution irregularities, the potential solutions to remedy the problem, and troubleshooting and subsequent executions until the issues are found.
This isn’t a simple process and depends on the workload to certify. In fact, it isn’t always possible to pass all certification tests. There may be exceptions because of the specifications of the given software but will likely be accepted as long as they are properly documented. A good test plan with proper documentation of each exception found throughout the feedback loop when administering certification tests will improve the chances of the certification team accepting the exceptions.
How can we implement this workflow?
In the Telco Partner CI team, we work with partners and internal teams using Distributed-CI (DCI) as our main CI system to support an iterative feedback loop. It can, from a single point, execute all possible certification suites, unifying how the results are reported making it easier to troubleshoot potential issues that may be found while working in the pre-certification work and follow up with partners.
All certification tests launched with DCI using preflight and Certsuite provide a JUnit file as output, and report the result of the certification execution. View an example (note: you need a Red Hat account to gain access) of the look and feel of the JUnit results from DCI, using our example-cnf workload as example:
Figure 1. Example of JUnit files visualized through Distributed-CI GUI
The JUnit files provide simplified yet powerful information to really understand what tests are passing, skipped over (Certsuite will skip over test suites where the conditions to execute the test are not met), and failing. This can be an entry point to start deciding what next steps can look like on a partner’s pre-certification journey. Typically, we want to fix tests that are not working as expected but in a specific order; always try to fix the Certsuite tests first since they may impact the container, operator, and/or Helm chart images for posterior certification for these other certification programs.
This is the input information that is used by the Telco Partner CI team to start building the test plan and to gather certification execution results. In our case, we define the test plan with spreadsheets. By extracting and parsing the content from the JUnit files, we can export the data to a tabular format, ready for processing by our spreadsheets. We can automatically update the report, ensuring the correct JUnit files are parsed. We update the report to make it visually appealing and easy to read for partners so they can follow the defined objectives and progress of test results, all while sharing feedback between partners and Red Hat teams.
This schema, shown below, starts with the DCI execution, followed by a job results extraction to a spreadsheet, and then the evolution of migrating successful jobs into an iterative loop until reaching the test plan target.
Figure 2. Pre-certification workflow to review all certification tests
Examples
For the example-cnf workload, the Telco Partner CI team worked on a Google spreadsheet to extract and parse the data from preflight and Certsuite. The process to build this spreadsheet consists on several steps:
- Firstly, select the proper JUnit files to extract the certification results, using DCI API.
- In the case of Certsuite, only one JUnit file needs to be analyzed.
- In the case of Preflight, it depends on how many container and operator images have been tested.
- Parse the JUnit files with some formulas to tabulate the input data, obtaining the information below using Certsuite:
Figure 3. JUnit file data transformed into tabular data
- Then, in a new sheet, make a link between the tabular data and the feedback to provide. We visually touched up the spreadsheet to ensure the results are clearly presented. Any individual can easily provide feedback.
Here’s an example of a Cersuite report:
Image
Figure 4. Example of Certsuite report on a spreadsheet
And here’s the case for Preflight:
Image
Figure 5. Example of Preflight report on a spreadsheet
- Do a first review round of the current results. Typically, tests that are passing could be marked as reviewed in the Reviewed column; it is always worth taking a look at the results to confirm all requirements for each test are assessed to avoid false positives. Tests that do not pass are marked as not reviewed.
- For each certification tool, negotiate with the partner; what is the order in which tests will be reviewed and fixed? You can use the feedback column to write any information that may be useful for the testing process, and include an estimated time to completion column to define a tentative date to fix the issue. We recommend:
- Typically, first tests to fix are the ones for Certsuite, since there are some tests that check if the workloads are certified. It is better to fix all Certsuite issues before starting certifying your workloads with other tools such as preflight or Helm chart certification.
- In the case of Certsuite, there may be tests that cannot be fixed. For them, an exception request has to be made. Information included in the feedback fields could be useful to document all the exception requests.
- Finally, iterate over the test plan defined in the spreadsheet in a continuous integration loop. As presented in the previous figures, you should fill in the feedback fields, so that partners and Red Hat teams can document whatever information that can be interesting to try to fix the target test, and then use it consequently.
Conclusions
The expertise shared will help your team with automation that you can use to enhance the partner experience during pre-certification. You can learn more about the utility of this tool in our blog focused on CNF Certification. To better understand how partners are using our spreadsheets, you can read more on how Keysight, a Red Hat partner, developed a workflow to reach CNF Certification for their Open RAN SIM workload.
If you’re interested in learning more about how to prepare workflows for certification on your team, contact the Telco Partner CI team.
