This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Testing Addons

How to test addons.

1: Installing a specific version of an Addon in a staging environment
2: Testing With OCP (Without OCM)
3: Testing With OSD-E2E

1 - Installing a specific version of an Addon in a staging environment

Add-on services are typically installed using the OpenShift Cluster Manager web console, by selecting the specific addon from the Add-ons tab and clicking Install. However, only the latest version of an addon service can be installed using the OpenShift Cluster Manager console.

In some cases, you might need to install an older version of an addon, for example, to test the upgrade of an addon from one version to the next. Follow this procedure to install a specific version of an addon service in a staging environment.

IMPORTANT: Installing an addon service using this procedure is only recommended for testing upgrades in a staging environment and is not supported for customer-facing production workloads.

Prerequisites

You have the version_select capability added to your organization by creating a merge request to the ocm-resources respository.
For more information about assigning capabilities to an organization, see Customer Capabilities Management. For more information about enabling the version_select capability, see organization YAML example and merge request example.

Procedure

Create a JSON file with the addon service and addon version that you want to install. In this example, the JSON file is install-payload.json, the addon id is reference-addon, and the version we want to install is 0.6.7.
Example
```
{
 "addon": {
   "id": "reference-addon"
 },
 "addon_version": {
   "id": "0.6.7"
 }
}
```
NOTE: If the addon that you are installing has a required parameter, ensure that you add it to the JSON file. For instance, the managed-odh addon, which is shown in the example below, requires the parameter notification-email to be included.
Example
```
{
  "addon": {
    "id": "managed-odh"
},
  "addon_version": {
    "id": "1.23.0"
},
  "parameters": {
        "items": [
        {
          "id": "notification-email",
          "value": "me@somewhere.com"
        }
      ]
    }
}
```

Set the CLUSTER_ID environment variable:

export CLUSTER_ID=<your_cluster_internal_id>

Run the following API request to install the addon:

ocm post /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addons --body install-payload.json

Verify the addon installation:
1. Log into your cluster:
```
oc login
```
2. Run the oc get addons command to view the addon installation status:
```
$ oc get addons
NAME              STATUS   AGE
reference-addon   Pending  10m
```
3. Optionally, run the watch command to watch the addon installation status:
```
$ watch oc get addons
NAME                 STATUS    AGE
reference-addon      Ready     32m
```

If you do not want the addon to automatically upgrade to the latest version after installation, delete the addon upgrade policy before the addon installation completes.

List the upgrade policies:

Example

$ ocm get /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies
{
"kind": "AddonUpgradePolicyList",
"page": 1,
"size": 1,
"total": 1,
"items": [
 {
  "kind": "AddonUpgradePolicy",
  "id": "991a69a5-ce33-11ed-9dda-0a580a8308f5",
  "href": "/api/clusters_mgmt/v1/clusters/22ogsfo8kd36bk280b6bqbi7l03micmm/addon_upgrade_policies/991a69a5-ce33-11ed-9dda-0a580a8308f5",
  "schedule": "0,15,30,45 * * * *",
  "schedule_type": "automatic",
  "upgrade_type": "ADDON",
  "version": "",
  "next_run": "2023-03-29T19:30:00Z",
  "cluster_id": "22ogsfo8kd36bk280b6bqbi7l03micmm",
  "addon_id": "reference-addon"
 }
]
}

Delete the addon upgrade policy:

Syntax

ocm delete /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies/<addon_upgrade_policy_id>

Example

ocm delete /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies/991a69a5-ce33-11ed-9dda-0a580a8308f5

Verify the upgrade policy no longer exists:

Syntax

ocm get /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies | grep <addon_upgrade_policy_id>

Example

ocm get /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies | grep 991a69a5-ce33-11ed-9dda-0a580a8308f5

Review the addon installation status and version:

Example

$ oc get addons reference-addon -o yaml
apiVersion: addons.managed.openshift.io/v1alpha1
kind: Addon
metadata:
  annotations:
 ...
 creationTimestamp: "2023-03-20T19:07:08Z"
 finalizers:
 - addons.managed.openshift.io/cache
 ...
spec:
displayName: Reference Addon
 ...
 pause: false
 version: 0.6.7
status:
  conditions:
  - lastTransitionTime: "2023-03-20T19:08:10Z"
    message: ""
    observedGeneration: 2
    reason: FullyReconciled
    status: "True"
    type: Available
  - lastTransitionTime: "2023-03-20T19:08:10Z"
    message: Addon has been successfully installed.
    observedGeneration: 2
    reason: AddonInstalled
    status: "True"
    type: Installed
 lastObservedAvailableCSV: redhat-reference-addon/reference-addon.v0.6.7
 observedGeneration: 2
 observedVersion: 0.6.7
 phase: Ready

In this example, you can see the addon version is set to 0.6.7 and AddonInstalled status is True.

(Optional) If needed, recreate the addon upgrade policy manually.

Create a JSON file with the addon upgrade policy information.

Example of automatic upgrade

{
  "kind": "AddonUpgradePolicy",
  "addon_id": "reference-addon",
  "cluster_id": "$CLUSTER_ID",
  "schedule_type": "automatic",
  "upgrade_type": "ADDON"
}

Example of manual upgrade

{
  "kind": "AddonUpgradePolicy",
  "addon_id": "reference-addon",
  "cluster_id": "$CLUSTER_ID",
  "schedule_type": "manual",
  "upgrade_type": "ADDON",
  "version": "0.7.0"
}

In the example above, the schedule_type for the reference-addon is set to manual and the version to upgrade to is set 0.7.0. The upgrade policy will execute once and the addon will upgrade to version 0.7.0.

Run the following API request to install the addon upgrade policy:

Syntax

ocm post /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies --body <your_json_filename>

Example

ocm post /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies --body reference-upgrade-policy.json

Verify the upgrade policy exists:

Syntax

 ocm get /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies | jq '.items[] | select(.addon_id=="<addon_id>")'

Example

ocm get /api/clusters_mgmt/v1/clusters/$CLUSTER_ID/addon_upgrade_policies | jq '.items[] | select(.addon_id=="reference-addon")'

Useful commands

Get a list of available addons:

ocm get /api/clusters_mgmt/v1/addons | jq '.items[].id'

Get a list of available versions to install for a given addon id:

Syntax

ocm get /api/clusters_mgmt/v1/addons/<addon-id>/versions | jq '.items[].id'

Example

$ ocm get /api/clusters_mgmt/v1/addons/reference-addon/versions | jq '.items[].id'
"0.0.0"
"0.1.5"
"0.1.6"
"0.2.2"
"0.3.0"
"0.3.1"
"0.3.2"
"0.4.0"
"0.4.1"
"0.5.0"
"0.5.1"
"0.6.0"
"0.6.1"
"0.6.2"
"0.6.3"
"0.6.4"
"0.6.5"
"0.6.6"
"0.6.7"
"0.7.0"

2 - Testing With OCP (Without OCM)

Testing Without OCM

During the development process, it might be useful (and cheaper) to run your addon on an OCP cluster.

You can spin up an OCP cluster on your local machine using CRC.

OCP and OSD differ in one important aspect: OCP gives you full access, while OSD restricts the administrative actions. But Managed Tenants will apply resources as unrestricted admin to OSD, just like you can do with your OCP, so OCP is a good OSD mockup for our use case.

By doing this, you’re skipping:

OCM and SKU management
Hive

First, you have to build your catalog. Let take the managed-odh as example:

$ managedtenants --environment=stage --addons-dir addons --dry-run run --debug tasks/deploy/10_build_push_catalog.py:managed-odh
Loading stage...
Loading stage OK
== TASKS =======================================================================
tasks/deploy/10_build_push_catalog.py:BuildCatalog:managed-odh:stage...
 -> creating the temporary directory
 -> /tmp/managed-odh-stage-1bkjtsea
 -> generating the bundle directory
 -> generating the bundle package.yaml
 -> building the docker image
 -> ['docker', 'build', '-f', PosixPath('/home/apahim/git/managed-tenants/Dockerfile.catalog'), '-t', 'quay.io/osd-addons/opendatahub-operator:stage-91918fe', PosixPath('/tmp/managed-odh-stage-1bkjtsea')]
tasks/deploy/10_build_push_catalog.py:BuildCatalog:managed-odh:stage OK
tasks/deploy/10_build_push_catalog.py:PushCatalog:managed-odh:stage...
 -> pushing the docker image
 -> ['docker', '--config', '/home/apahim/.docker', 'push', 'quay.io/osd-addons/opendatahub-operator:stage-91918fe']
tasks/deploy/10_build_push_catalog.py:PushCatalog:managed-odh:stage OK

That command has built the image quay.io/osd-addons/opendatahub-operator:stage-91918fe on your local machine.

You can inspect the image with:

$ docker run --rm -it --entrypoint "bash"  quay.io/osd-addons/opendatahub-operator:stage-91918fe -c "ls manifests/"
0.8.0  1.0.0-experiment  managed-odh.package.yml

$ docker run --rm -it --entrypoint "bash"  quay.io/osd-addons/opendatahub-operator:stage-91918fe -c "cat manifests/managed-odh.package.yml"
channels:
- currentCSV: opendatahub-operator.1.0.0-experiment
  name: beta
defaultChannel: beta
packageName: managed-odh

Next, you have to tag/push that image to some public registry repository of yours:

$ docker tag quay.io/osd-addons/opendatahub-operator:stage-91918fe quay.io/<my-repository>/opendatahub-operator:stage-91918fe
$ docker push quay.io/<my-repository>/opendatahub-operator:stage-91918fe
Getting image source signatures
Copying blob 9fbc4a1ed0b0 done
Copying blob c4d8f7894b7d skipped: already exists
Copying blob 61598d8d1b24 skipped: already exists
Copying blob 38ada4bcd26f skipped: already exists
Copying blob d5fdf1f627c8 skipped: already exists
Copying blob 2bf094d88b12 skipped: already exists
Copying blob 8a6c7bacb5db done
Copying config 3088e48540 done
Writing manifest to image destination
Copying config 3088e48540 [--------------------------------------] 0.0b / 3.6KiB
Writing manifest to image destination
Writing manifest to image destination
Storing signatures

Now we have to apply the OpenShift resources that will install the operator in the OCP cluster. You can use the managedtenants command to generate the stage SelectorSyncSet and look at it for reference:

$ managedtenants --environment=stage --addons-dir addons --dry-run run --debug tasks/generate/99_generate_SelectorSyncSet.py
Loading stage...
Loading stage OK
== POSTTASKS ===================================================================
tasks/generate/99_generate_SelectorSyncSet.py:GenerateSSS:stage...
 -> Generating SSS template /home/apahim/git/managed-tenants/openshift/stage.yaml
tasks/generate/99_generate_SelectorSyncSet.py:GenerateSSS:stage OK

Here’s the SelectorSyncSet snippet we are interested in:

---
- apiVersion: hive.openshift.io/v1
  kind: SelectorSyncSet
  metadata:
    name: addon-managed-odh
  spec:
    clusterDeploymentSelector:
      matchLabels:
        api.openshift.com/addon-managed-odh: "true"
    resourceApplyMode: Sync
    resources:
      - apiVersion: v1
        kind: Namespace
        metadata:
          annotations:
            openshift.io/node-selector: ""
          labels: null
          name: redhat-opendatahub
      - apiVersion: operators.coreos.com/v1alpha1
        kind: CatalogSource
        metadata:
          name: addon-managed-odh-catalog
          namespace: openshift-marketplace
        spec:
          displayName: Managed Open Data Hub Operator
          image: quay.io/osd-addons/opendatahub-operator:stage-${IMAGE_TAG}
          publisher: OSD Red Hat Addons
          sourceType: grpc
      - apiVersion: operators.coreos.com/v1alpha2
        kind: OperatorGroup
        metadata:
          name: redhat-layered-product-og
          namespace: redhat-opendatahub
      - apiVersion: operators.coreos.com/v1alpha1
        kind: Subscription
        metadata:
          name: addon-managed-odh
          namespace: redhat-opendatahub
        spec:
          channel: beta
          name: managed-odh
          source: addon-managed-odh-catalog
          sourceNamespace: openshift-marketplace

Our OpenShift manifest to be applied to the OCP cluster looks as follows:

kind: List
metadata: {}
apiVersion: v1
items:
  - apiVersion: v1
    kind: Namespace
    metadata:
      name: redhat-opendatahub
  - apiVersion: operators.coreos.com/v1alpha1
    kind: CatalogSource
    metadata:
      name: addon-managed-odh-catalog
    spec:
      displayName: Managed Open Data Hub Operator
      image: quay.io/<my-repository>/opendatahub-operator:stage-91918fe
      publisher: OSD Red Hat Addons
      sourceType: grpc
  - apiVersion: operators.coreos.com/v1alpha2
    kind: OperatorGroup
    metadata:
      name: redhat-layered-product-og
  - apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: addon-managed-odh
    spec:
      channel: beta
      name: managed-odh
      source: addon-managed-odh-catalog
      sourceNamespace: openshift-marketplace

Finally, apply it to the OCP cluster:

$ oc apply -f manifest.yaml
Namespace/redhat-opendatahub created
CatalogSource/addon-managed-odh-catalog created
Subscription/addon-managed-odh created
OperatorGroup/redhat-layered-product-og created

Your operator should be installed in the cluster.

3 - Testing With OSD-E2E

Testing With OSD-E2E

All Add-Ons must have a reference to a test harness container in a publicly available repository. The Add-On development team is responsible for creating and maintaining the test harness image. That image is generated by the OSD e2e process.

The test harness will be tested against OCP nightly and OSD next.

Please refer to the OSD-E2E Add-On Documentation for more details on how this test harness will be run and how it is expected to report results.

Primer into OSD E2E tests and prow jobs

To ensure certain things such as validating that the addon can be easily and successfully installed on a customer’s cluster, we have prow jobs setup which run e2e tests (one test suite per addon) every 12 hours. If the e2e tests corresponding to any addon fail, then automated alerts/notifications are sent to the addon team. Every addon’s e2e tests are packaged in an image called “testHarness”, which is built and pushed to quay.ioby the team maintaining the addon. Once the “testHarness” image is built and pushed, the team must register their addon to testHarness image’s e2e tests by making a PR against this file.

You can access the portal for prow jobs here. The prow jobs follow the below steps to run the e2e tests. For every e2e test defined inside this file:

An OSD cluster is created and the addon, which is being tested, is installed. Openshift API is used to perform these operations via the API definition provided at https://api.openshift.com
The e2e prow job definition, specifically for the addon from this file, is parsed and hence, the parameters required to run its e2e tests will be recognized as well.
The “testHarness” image for the addon is parsed and executed against the parameters fetched from the above step.
If an MT-SRE team member notices those tests failing, they should notify the respective team to take a look at them and fix them.