Good practices to generate the best CNF Certification with DCI part 2

This post continues from part 1, providing useful information to address the CNF Certification process. In this case, we will focus on the submission of certification results, checking the files that you should submit to the certification project when you have finished the test execution and you have all in place to start with the last stage of this journey.

What to submit to CNF Certification

The last step to complete the certification checklist to certify your already Vendor Validated CNF project involves the submission of the CNF certification test results in the Red Hat Partner Certification portal. As the workflow specifies, you have to submit the following results, that can be extracted from your DCI job, which was reviewed in the first part of this blog series:

• tnf_config.yml, describing how CNF Cert Suite has been configured to test the workloads.
• claim.json, which summarizes the results of the CNF Cert Suite execution

But to these two files, you will add two more files to complete your report. What you want to include is, mainly, the justification for the test cases that are failing or being skipped and are mandatory for your CNF category.

For this, you need to include the tnf_feedback variable to your pipeline configuration. In this variable, you need to include the test cases that require exception and the proper justification. This link will off you some guidelines to properly fill it.

By including this, if you run  DCI again, you will have a job like this one; where, in the files section, you will have available a file called -cnf-test-results.tar.gz, which includes some of the outputs from the CNF Cert Suite execution.

To confirm the result contains everything  we need for certification, just extract the content of the tar.gz file in your computer and open results.html. In the upper right corner, in "Choose a scenario", select the CNF scenario you are addressing (e.g. "Telco"; remember they are cumulative).

Image

Then, in the main menu, push on "Results", and leave the default options that appear (only marking "Show Mandatory Results for that scenario", and displaying all the test results). You should see something like this, with the test feedback of all the test cases you have executed:

Image

At this point, you need to check the following:

• Confirm that the number of failed + skipped tests are equal to the number of variables related to test cases defined in the tnf_feedback variable.
• Open the failed/skipped tests and confirm the content you've included in the pipeline is correct.

This is an example for the second case. If this is the variable defined under tnf_feedback:

    tnf_feedback:

platform-alteration-tainted-node-kernel: "Appearing in OCP 4.12 because of OCPBUGS-3083 and CILAB-1179 (but it is whitelisted), and in OCP \         

   4.13 because of OCPBUGS-13330.\\nOur cluster10 is also hitting a kernel taint."

you should see the following in the HTML report:

Image

When confirming all the tests are correctly documented, push in the button you have in the upper right corner, called "Download Results". A new file will be generated, called results-feedback.html. Check that it is including correctly the feedback for each failed/skipped test.

With all this,  you are now ready for certification. To tnf_config.yml and claim.json files, you just need to add the tar.gz file we have generated in the DCI job, together with the new results-feedback.html file generated previously. Moreover, including in a comment the link to the final DCI job from which you have retrieved the submitted files is a plus, also saying that results-feedback.html contains the report for the exception request, following the new format defined in the CNF Cert Suite.

Just finish with the steps suggested in the workflow, then you will have to wait for the review and be prepared to reply to suggested feedback or modifications if they are provided. If everything is fine, you will finish the CNF Certification process and will achieve the CNF Certified badge for your product.

Conclusions

We hope these posts are useful for you when dealing with the CNF Certification process. Remember, this is an iterative process where you would need to fix issues in your workloads before proceeding for certification. By defining a proper test plan and having clear the different steps you need to follow, your journey to CNF Certification will be simpler and clearer.