Technology certification
Good practices to generate the best CNF Certification with DCI part 1
Introduction
The information that follows aims at specifying how DCI can help you in the road to achieve CNF Certification, understanding what configuration you should provide to DCI to correctly execute CNF Cert Suite and obtain the required (and desired) results. We will also provide some useful tips and good practices to have in mind when dealing with the submission of the CNF Certification test results, with the intention of minimizing the number of iterations to do before achieving the certification.
This content will be divided into two parts. This first part will firstly review the two certification levels that are the object of study: CNF Certification and Vendor Validation, including the prerequisites you need to fulfill before starting with the certification effort, finally providing some tips to get the most out of the execution of the CNF Certification tests. We will analyze what you need to submit for certification evaluation in the second part.
A brief overview about CNF Certification and Vendor Validation
Both CNF Certification and Vendor Validation are specialized processes that belong to the Red Hat OpenShift Certification program, available for products that implement a network function delivered in a container format with Red Hat OpenShift as the deployment platform. This web resource from Red Hat Customer Portal (you need a valid Red Hat account to access there), provides all the steps that you need to follow to achieve both certification badges.
In particular, the main difference between these two initiatives is the level of commitment achieved. While Vendor Validation can be reached by just testing the product internally, the CNF Certification, apart from achieving Vendor Validation firstly, requires the product to pass a specific set of tests, defined in the cnf-certification-test project (also called CNF Cert Suite; we’ll be using that term going forward t to refer to that project). In our case, we will focus on CNF Certification for the scope of this post.
To test your workloads with CNF Cert Suite, you have multiple alternatives. A suitable way of launching the suite in an automated way is to leverage on Distributed-CI’s capabilities. Note that the review and explanation of how to launch the CNF Cert Suite with DCI is out of the scope of this post; in fact, there’s already a previous blog post that describes this process. However, here we will explain the main configuration to be provided when launching the DCI jobs to obtain the proper CNF test results, with the intention of generating what we really need to have to directly submit all this data for certification, minimizing the number of iterations with the reviewers to achieve CNF Certification.
What do you need to know to get the most from this post?
This following information is recommended for people that are used to running CNF Cert Suite with Distributed-CI. If not, please check the post referenced in the previous section to understand how you can use DCI to launch CNF Cert Suite.
Moreover, even though it is not a hard requirement to get to understand the topics we are about to review, the general understanding of Vendor Validation and CNF Certification workflows (remember you can check it here) is a plus.
Prerequisites for CNF Certification
Before starting with the CNF Cert Suite execution with DCI to accomplish the CNF Certification badge, you need to fulfill the following prerequisites (among others already commented in the CNF Certification and Vendor Validation workflow):
- You have to create a CNF project in the Red Hat Partner Connect portal. This step can be automated with DCI by using the openshift_cnf role, where you have some examples to properly create a project automatically. The same CNF project is used for the Vendor Validation and CNF Certification processes.
- You need to achieve Vendor Validation for your product.
- You must determine what CNF scenario(s) you are going to target for certification, and determine what are the test cases that you have to pass, since there would be test cases that would be mandatory or optional depending on the selected CNF scenario.
- All your workloads (containers, operators, Helm charts) must be Red Hat certified before proceeding for CNF Certification. The good news is that DCI offers support for the certification of all these workloads. You have a brief description of each case in this blog post. Furthermore, this post focuses on container and operator certification with DCI using Preflight certification suites, which is the most typical case for Red Hat Software Certification on OpenShift.
- You should pass all the CNF Cert Suite tests for the CNF scenario(s) you are targeting. If there are tests that are not passing, you should apply the proper fixes in your workloads to make them pass. If it is impossible to apply any fix to some tests, it must be properly justified why the tests are not passing, so that exception requests will be made for these tests.
The two last items could result in the most time-consuming tasks, since it may require changes in your workloads to address both objectives. This blog post will provide some useful tips for the last item, helping you to organize the CNF testing plan.
Understanding how CNF Cert Suite’s tests are organized
As we have already mentioned, CNF Cert Suite encompasses a set of CNF test suites. The description of all available test suites, together with the CNF scenarios that can be covered, can be found in the cnf-certification-test’s Test Catalog. You need to understand the difference between these two concepts:
- The test suites group test cases by specific categories, depending on what they are testing: networking aspects, certification checks, operator configuration, etc.
- Each test case can be mandatory or not depending on the CNF scenario (or CNF type) you want to target in your CNF certification. Most typical ones are non-telco (which includes the most basic test that has to be passed) and telco (which includes test cases oriented to telco workloads) scenarios, but you also have the extended and far-edge scenarios, more focused on specific types of deployments. Remember the CNF scenarios are cumulative: for example, to address the telco scenario, you have to pass the non-telco scenario too, since it’s the most basic one.
We recommend to get the CNF Cert Suite execution tracked with a specific document (could be a text document, a spreadsheet, whatever option that better suits you), where you specify the CNF scenario(s) you are targeting (non-telco, telco, extended, far-edge) and the test cases that are mandatory or optional for your case. You should address from most basic tests (so, firstly, focus on non-telco category and work on making all the mandatory tests in this category to pass) to most specific ones (in case you are targeting more specific categories).
With this idea in mind, you would have a test plan to follow when validating your workloads with CNF Cert Suite. Now, you will have to test your workloads with CNF Cert Suite. If using DCI properly, you should obtain an output like this one (this example is based on our tnf_test_example), where you can see the test results, including the tests that are passing, being skipped or failing, with the reason for failure for the last ones.
In the files section from your DCI job, you will have available the files that you will need for the certification submission, and that will be explained in the second part of this blog series. You can see an extract of this section in the next screenshot, which directly comes from the example job we have provided in the previous paragraph:
After a successful execution, your next task is to analyze the result and check if all the mandatory tests for the categories you are addressing are passing or not. If any of the test cases are not passing, you will need to apply the proper fixes in your workload to make them pass, and then recheck again. Note that, if this implies a need to modify a container image or an operator definition, you will need to recertify again with Preflight, so it is important to fix all possible test cases before addressing a new container/operator certification process, since it is time-consuming and may delay your trip to CNF Certification.
Of course, there may be test cases that cannot be fixed and have to fail, or some others that do not apply to your case because you do not need the configuration under test (so that the test case is skipped). In that case, an exception can be requested when submitting the certification results. We recommend documenting these exceptions in the same document you defined your test plan for later submission. Actually, these exceptions are eventually documented in a knowledge base article that summarizes the certification result.
Conclusions
With all this in mind, you should be ready to address CNF Certification, but you are just missing the last step of the workflow: to submit the result and get the acceptance from Red Hat to achieve your CNF Certification badge.
Let’s examine what you have to do when you have everything ready for certification (i.e. you have achieved to fix the test cases that were failing, and in case some failures are unavoidable, you have a proper justification for requesting the exception, and all the images are Red Hat certified) in the second part of this blog series.
