Testing With OSD-E2E

All Add-Ons must have a reference to a test harness container in a publicly available repository. The Add-On development team is responsible for creating and maintaining the test harness image. That image is generated by the OSD e2e process.

The test harness will be tested against OCP nightly and OSD next.

Please refer to the OSD-E2E Add-On Documentation for more details on how this test harness will be run and how it is expected to report results.

Primer into OSD E2E tests and prow jobs

To ensure certain things such as validating that the addon can be easily and successfully installed on a customer’s cluster, we have prow jobs setup which run e2e tests (one test suite per addon) every 12 hours. If the e2e tests corresponding to any addon fail, then automated alerts/notifications are sent to the addon team. Every addon’s e2e tests are packaged in an image called “testHarness”, which is built and pushed to quay.ioby the team maintaining the addon. Once the “testHarness” image is built and pushed, the team must register their addon to testHarness image’s e2e tests by making a PR against this file.

You can access the portal for prow jobs here. The prow jobs follow the below steps to run the e2e tests. For every e2e test defined inside this file:

An OSD cluster is created and the addon, which is being tested, is installed. Openshift API is used to perform these operations via the API definition provided at https://api.openshift.com
The e2e prow job definition, specifically for the addon from this file, is parsed and hence, the parameters required to run its e2e tests will be recognized as well.
The “testHarness” image for the addon is parsed and executed against the parameters fetched from the above step.
If an MT-SRE team member notices those tests failing, they should notify the respective team to take a look at them and fix them.

Last modified January 31, 2023: line length (7f2a39dd)