This the multi-page printable view of this section. Click here to print.
Testing Addons
1 - Installing a specific version of an Addon in a staging environment
Add-on services are typically installed using the OpenShift Cluster Manager web console, by selecting the specific addon from the Add-ons tab and clicking Install. However, only the latest version of an addon service can be installed using the OpenShift Cluster Manager console.
In some cases, you might need to install an older version of an addon, for example, to test the upgrade of an addon from one version to the next. Follow this procedure to install a specific version of an addon service in a staging environment.
IMPORTANT: Installing an addon service using this procedure is only recommended for testing upgrades in a staging environment and is not supported for customer-facing production workloads.
Prerequisites
You have the
version_select
capability added to your organization by creating a merge request to the ocm-resources respository.For more information about assigning capabilities to an organization, see Customer Capabilities Management. For more information about enabling the
version_select
capability, see organization YAML example and merge request example.
Procedure
Create a JSON file with the addon service and addon version that you want to install. In this example, the JSON file is
install-payload.json
, the addon id isreference-addon
, and the version we want to install is0.6.7
.Example
{ "addon": { "id": "reference-addon" }, "addon_version": { "id": "0.6.7" } }
NOTE: If the addon that you are installing has a required parameter, ensure that you add it to the JSON file. For instance, the
managed-odh
addon, which is shown in the example below, requires the parameternotification-email
to be included.Example
{ "addon": { "id": "managed-odh" }, "addon_version": { "id": "1.23.0" }, "parameters": { "items": [ { "id": "notification-email", "value": "me@somewhere.com" } ] } }
Set the
CLUSTER_ID
environment variable:export CLUSTER_ID=<your_cluster_internal_id>
Run the following API request to install the addon:
ocm post /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addons --body install-payload.json
Verify the addon installation:
Log into your cluster:
oc login
Run the
oc get addons
command to view the addon installation status:$ oc get addons NAME STATUS AGE reference-addon Pending 10m
Optionally, run the
watch
command to watch the addon installation status:$ watch oc get addons NAME STATUS AGE reference-addon Ready 32m
If you do not want the addon to automatically upgrade to the latest version after installation, delete the addon upgrade policy before the addon installation completes.
List the upgrade policies:
Example
$ ocm get /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies { "kind": "AddonUpgradePolicyList", "page": 1, "size": 1, "total": 1, "items": [ { "kind": "AddonUpgradePolicy", "id": "991a69a5-ce33-11ed-9dda-0a580a8308f5", "href": "/api/clusters_mgmt/v1/clusters/22ogsfo8kd36bk280b6bqbi7l03micmm/addon_upgrade_policies/991a69a5-ce33-11ed-9dda-0a580a8308f5", "schedule": "0,15,30,45 * * * *", "schedule_type": "automatic", "upgrade_type": "ADDON", "version": "", "next_run": "2023-03-29T19:30:00Z", "cluster_id": "22ogsfo8kd36bk280b6bqbi7l03micmm", "addon_id": "reference-addon" } ] }
Delete the addon upgrade policy:
Syntax
ocm delete /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies/<addon_upgrade_policy_id>
Example
ocm delete /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies/991a69a5-ce33-11ed-9dda-0a580a8308f5
Verify the upgrade policy no longer exists:
Syntax
ocm get /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies | grep <addon_upgrade_policy_id>
Example
ocm get /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies | grep 991a69a5-ce33-11ed-9dda-0a580a8308f5
Review the addon installation status and version:
Example
$ oc get addons reference-addon -o yaml apiVersion: addons.managed.openshift.io/v1alpha1 kind: Addon metadata: annotations: ... creationTimestamp: "2023-03-20T19:07:08Z" finalizers: - addons.managed.openshift.io/cache ... spec: displayName: Reference Addon ... pause: false version: 0.6.7 status: conditions: - lastTransitionTime: "2023-03-20T19:08:10Z" message: "" observedGeneration: 2 reason: FullyReconciled status: "True" type: Available - lastTransitionTime: "2023-03-20T19:08:10Z" message: Addon has been successfully installed. observedGeneration: 2 reason: AddonInstalled status: "True" type: Installed lastObservedAvailableCSV: redhat-reference-addon/reference-addon.v0.6.7 observedGeneration: 2 observedVersion: 0.6.7 phase: Ready
In this example, you can see the addon version is set to
0.6.7
andAddonInstalled
status isTrue
.(Optional) If needed, recreate the addon upgrade policy manually.
Create a JSON file with the addon upgrade policy information.
Example of automatic upgrade
{ "kind": "AddonUpgradePolicy", "addon_id": "reference-addon", "cluster_id": "$CLUSTER_ID", "schedule_type": "automatic", "upgrade_type": "ADDON" }
Example of manual upgrade
{ "kind": "AddonUpgradePolicy", "addon_id": "reference-addon", "cluster_id": "$CLUSTER_ID", "schedule_type": "manual", "upgrade_type": "ADDON", "version": "0.7.0" }
In the example above, the schedule_type for the
reference-addon
is set tomanual
and the version to upgrade to is set0.7.0
. The upgrade policy will execute once and the addon will upgrade to version0.7.0
.Run the following API request to install the addon upgrade policy:
Syntax
ocm post /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies --body <your_json_filename>
Example
ocm post /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies --body reference-upgrade-policy.json
Verify the upgrade policy exists:
Syntax
ocm get /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies | jq '.items[] | select(.addon_id=="<addon_id>")'
Example
ocm get /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies | jq '.items[] | select(.addon_id=="reference-addon")'
Useful commands
Get a list of available addons:
ocm get /api/clusters_mgmt/v1/addons | jq '.items[].id'
Get a list of available versions to install for a given addon id:
Syntax
ocm get /api/clusters_mgmt/v1/addons/<addon-id>/versions | jq '.items[].id'
Example
$ ocm get /api/clusters_mgmt/v1/addons/reference-addon/versions | jq '.items[].id' "0.0.0" "0.1.5" "0.1.6" "0.2.2" "0.3.0" "0.3.1" "0.3.2" "0.4.0" "0.4.1" "0.5.0" "0.5.1" "0.6.0" "0.6.1" "0.6.2" "0.6.3" "0.6.4" "0.6.5" "0.6.6" "0.6.7" "0.7.0"
2 - Testing With OCP (Without OCM)
Testing Without OCM
During the development process, it might be useful (and cheaper) to run your addon on an OCP cluster.
You can spin up an OCP cluster on your local machine using CRC.
OCP and OSD differ in one important aspect: OCP gives you full access, while OSD restricts the administrative actions. But Managed Tenants will apply resources as unrestricted admin to OSD, just like you can do with your OCP, so OCP is a good OSD mockup for our use case.
By doing this, you’re skipping:
- OCM and SKU management
- Hive
First, you have to build your catalog. Let take the managed-odh
as example:
$ managedtenants --environment=stage --addons-dir addons --dry-run run --debug tasks/deploy/10_build_push_catalog.py:managed-odh
Loading stage...
Loading stage OK
== TASKS =======================================================================
tasks/deploy/10_build_push_catalog.py:BuildCatalog:managed-odh:stage...
-> creating the temporary directory
-> /tmp/managed-odh-stage-1bkjtsea
-> generating the bundle directory
-> generating the bundle package.yaml
-> building the docker image
-> ['docker', 'build', '-f', PosixPath('/home/apahim/git/managed-tenants/Dockerfile.catalog'), '-t', 'quay.io/osd-addons/opendatahub-operator:stage-91918fe', PosixPath('/tmp/managed-odh-stage-1bkjtsea')]
tasks/deploy/10_build_push_catalog.py:BuildCatalog:managed-odh:stage OK
tasks/deploy/10_build_push_catalog.py:PushCatalog:managed-odh:stage...
-> pushing the docker image
-> ['docker', '--config', '/home/apahim/.docker', 'push', 'quay.io/osd-addons/opendatahub-operator:stage-91918fe']
tasks/deploy/10_build_push_catalog.py:PushCatalog:managed-odh:stage OK
That command has built the image
quay.io/osd-addons/opendatahub-operator:stage-91918fe
on your local machine.
You can inspect the image with:
$ docker run --rm -it --entrypoint "bash" quay.io/osd-addons/opendatahub-operator:stage-91918fe -c "ls manifests/"
0.8.0 1.0.0-experiment managed-odh.package.yml
$ docker run --rm -it --entrypoint "bash" quay.io/osd-addons/opendatahub-operator:stage-91918fe -c "cat manifests/managed-odh.package.yml"
channels:
- currentCSV: opendatahub-operator.1.0.0-experiment
name: beta
defaultChannel: beta
packageName: managed-odh
Next, you have to tag/push that image to some public registry repository of yours:
$ docker tag quay.io/osd-addons/opendatahub-operator:stage-91918fe quay.io/<my-repository>/opendatahub-operator:stage-91918fe
$ docker push quay.io/<my-repository>/opendatahub-operator:stage-91918fe
Getting image source signatures
Copying blob 9fbc4a1ed0b0 done
Copying blob c4d8f7894b7d skipped: already exists
Copying blob 61598d8d1b24 skipped: already exists
Copying blob 38ada4bcd26f skipped: already exists
Copying blob d5fdf1f627c8 skipped: already exists
Copying blob 2bf094d88b12 skipped: already exists
Copying blob 8a6c7bacb5db done
Copying config 3088e48540 done
Writing manifest to image destination
Copying config 3088e48540 [--------------------------------------] 0.0b / 3.6KiB
Writing manifest to image destination
Writing manifest to image destination
Storing signatures
Now we have to apply the OpenShift resources that will install the operator
in the OCP cluster. You can use the managedtenants
command to generate
the stage SelectorSyncSet
and look at it for reference:
$ managedtenants --environment=stage --addons-dir addons --dry-run run --debug tasks/generate/99_generate_SelectorSyncSet.py
Loading stage...
Loading stage OK
== POSTTASKS ===================================================================
tasks/generate/99_generate_SelectorSyncSet.py:GenerateSSS:stage...
-> Generating SSS template /home/apahim/git/managed-tenants/openshift/stage.yaml
tasks/generate/99_generate_SelectorSyncSet.py:GenerateSSS:stage OK
Here’s the SelectorSyncSet snippet we are interested in:
---
- apiVersion: hive.openshift.io/v1
kind: SelectorSyncSet
metadata:
name: addon-managed-odh
spec:
clusterDeploymentSelector:
matchLabels:
api.openshift.com/addon-managed-odh: "true"
resourceApplyMode: Sync
resources:
- apiVersion: v1
kind: Namespace
metadata:
annotations:
openshift.io/node-selector: ""
labels: null
name: redhat-opendatahub
- apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: addon-managed-odh-catalog
namespace: openshift-marketplace
spec:
displayName: Managed Open Data Hub Operator
image: quay.io/osd-addons/opendatahub-operator:stage-${IMAGE_TAG}
publisher: OSD Red Hat Addons
sourceType: grpc
- apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
name: redhat-layered-product-og
namespace: redhat-opendatahub
- apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: addon-managed-odh
namespace: redhat-opendatahub
spec:
channel: beta
name: managed-odh
source: addon-managed-odh-catalog
sourceNamespace: openshift-marketplace
Our OpenShift manifest to be applied to the OCP cluster looks as follows:
kind: List
metadata: {}
apiVersion: v1
items:
- apiVersion: v1
kind: Namespace
metadata:
name: redhat-opendatahub
- apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: addon-managed-odh-catalog
spec:
displayName: Managed Open Data Hub Operator
image: quay.io/<my-repository>/opendatahub-operator:stage-91918fe
publisher: OSD Red Hat Addons
sourceType: grpc
- apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
name: redhat-layered-product-og
- apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: addon-managed-odh
spec:
channel: beta
name: managed-odh
source: addon-managed-odh-catalog
sourceNamespace: openshift-marketplace
Finally, apply it to the OCP cluster:
$ oc apply -f manifest.yaml
Namespace/redhat-opendatahub created
CatalogSource/addon-managed-odh-catalog created
Subscription/addon-managed-odh created
OperatorGroup/redhat-layered-product-og created
Your operator should be installed in the cluster.
3 - Testing With OSD-E2E
Testing With OSD-E2E
All Add-Ons must have a reference to a test harness container in a publicly available repository. The Add-On development team is responsible for creating and maintaining the test harness image. That image is generated by the OSD e2e process.
The test harness will be tested against OCP nightly and OSD next.
Please refer to the OSD-E2E Add-On Documentation for more details on how this test harness will be run and how it is expected to report results.
Primer into OSD E2E tests and prow jobs
To ensure certain things such as validating that the addon can be easily and successfully installed on a customer’s cluster, we have prow jobs setup which run e2e tests (one test suite per addon) every 12 hours. If the e2e tests corresponding to any addon fail, then automated alerts/notifications are sent to the addon team. Every addon’s e2e tests are packaged in an image called “testHarness”, which is built and pushed to quay.ioby the team maintaining the addon. Once the “testHarness” image is built and pushed, the team must register their addon to testHarness image’s e2e tests by making a PR against this file.
You can access the portal for prow jobs here. The prow jobs follow the below steps to run the e2e tests. For every e2e test defined inside this file:
- An OSD cluster is created and the addon, which is being tested, is installed. Openshift API is used to perform these operations via the API definition provided at https://api.openshift.com
- The e2e prow job definition, specifically for the addon from this file, is parsed and hence, the parameters required to run its e2e tests will be recognized as well.
- The “testHarness” image for the addon is parsed and executed against the parameters fetched from the above step.
- If an MT-SRE team member notices those tests failing, they should notify the respective team to take a look at them and fix them.