Operator-sdk run bundle failed registry pod due to missing root schema field

[bitscuit]

Bug Report

What did you do?

Run operator-sdk run bundle quay.io/henry_h_li/common-service-operator-bundle:a9101de0-dirty --install-mode OwnNamespace

What did you expect to see?

Successful run with registry pod running and operator pod running

What did you see instead? Under which circumstances?

The operator never gets installed because the registry pod is in crash loop.

$ oc get pods
NAME                                                              READY   STATUS   RESTARTS      AGE
uay-io-henry-h-li-common-service-operator-bundle-a9101de0-dirty   0/1     Error    2 (22s ago)   24s

Logs

time="2023-09-26T19:27:53Z" level=info msg="starting pprof endpoint" address="localhost:6060"
time="2023-09-26T19:27:53Z" level=fatal msg="object '{\"image\":\"quay.io/henry_h_li/common-service-operator-bundle:a9101de0-dirty\",\"name\":\"ibm-common-service-operator.v4.3.0\",\"package\":\"ibm-common-service-operator\",\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"CommonService\",\"version\":\"v3\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"ibm-common-service-operator\",\"version\":\"4.3.0\"}},{\"type\":\"olm.csv.metadata\",\"value\":{\"annotations\":{\"alm-examples\":\"[\\n  {\\n    \\\"apiVersion\\\": \\\"operator.ibm.com/v3\\\",\\n    \\\"kind\\\": \\\"CommonService\\\",\\n    \\\"metadata\\\": {\\n      \\\"labels\\\": {\\n        \\\"app.kubernetes.io/instance\\\": \\\"ibm-common-service-operator\\\",\\n        \\\"app.kubernetes.io/managed-by\\\": \\\"ibm-common-service-operator\\\",\\n        \\\"app.kubernetes.io/name\\\": \\\"ibm-common-service-operator\\\"\\n      },\\n      \\\"name\\\": \\\"example-commonservice\\\"\\n    },\\n    \\\"spec\\\": {\\n      \\\"size\\\": \\\"starterset\\\"\\n    }\\n  }\\n]\",\"capabilities\":\"Seamless Upgrades\",\"containerImage\":\"icr.io/cpopen/common-service-operator:latest\",\"createdAt\":\"2023-09-26T19:21:40Z\",\"description\":\"The IBM Common Service Operator is used to deploy IBM Common Services\",\"nss.operator.ibm.com/managed-operators\":\"ibm-common-service-operator\",\"nss.operator.ibm.com/managed-webhooks\":\"\",\"olm.skipRange\":\">=3.3.0 <4.3.0\",\"operatorChannel\":\"v4.3\",\"operatorVersion\":\"4.3.0\",\"operators.openshift.io/infrastructure-features\":\"[\\\"disconnected\\\"]\",\"operators.operatorframework.io/builder\":\"operator-sdk-v1.30.0\",\"operators.operatorframework.io/project_layout\":\"go.kubebuilder.io/v3\",\"repository\":\"https://github.com/IBM/ibm-common-service-operator\",\"support\":\"IBM\"},\"apiServiceDefinitions\":{},\"crdDescriptions\":{\"owned\":[{\"description\":\"CommonService is the Schema for the commonservices API. Documentation For additional details regarding install parameters check https://ibm.biz/icpfs39install. License By installing this product you accept the license terms https://ibm.biz/icpfs39license\",\"displayName\":\"CommonService\",\"kind\":\"CommonService\",\"name\":\"commonservices.operator.ibm.com\",\"specDescriptors\":[{\"description\":\"Accepting the license - URL: https://ibm.biz/integration-licenses\",\"displayName\":\"Accept\",\"path\":\"license.accept\",\"value\":[false],\"x-descriptors\":[\"urn:alm:descriptor:com.tectonic.ui:checkbox\"]},{\"description\":\"The size of deployment profile for IBM Cloud Pak foundational services\",\"displayName\":\"Size\",\"path\":\"size\",\"x-descriptors\":[\"urn:alm:descriptor:com.tectonic.ui:select:starterset\",\"urn:alm:descriptor:com.tectonic.ui:select:starter\",\"urn:alm:descriptor:com.tectonic.ui:select:small\",\"urn:alm:descriptor:com.tectonic.ui:select:medium\",\"urn:alm:descriptor:com.tectonic.ui:select:large\",\"urn:alm:descriptor:com.tectonic.ui:select:production\"]},{\"description\":\"The profile controller for IBM Cloud Pak foundational services\",\"displayName\":\"ProfileController\",\"path\":\"profileController\",\"x-descriptors\":[\"urn:alm:descriptor:com.tectonic.ui:select:default\",\"urn:alm:descriptor:com.tectonic.ui:select:commonservice\",\"urn:alm:descriptor:com.tectonic.ui:select:turbonomic\",\"urn:alm:descriptor:com.tectonic.ui:select:vpa\"]}],\"statusDescriptors\":[{\"description\":\"Installed Bedrock Operator Name\",\"displayName\":\"Name\",\"path\":\"bedrockOperators[0].name\"},{\"description\":\"Installed Bedrock Operator InstallPlan Name\",\"displayName\":\"InstallPlanName\",\"path\":\"bedrockOperators[0].installPlanName\"},{\"description\":\"Installed Bedrock Operator Status\",\"displayName\":\"OperatorStatus\",\"path\":\"bedrockOperators[0].operatorStatus\"},{\"description\":\"Installed Bedrock Operator Subscription Status\",\"displayName\":\"SubscriptionStatus\",\"path\":\"bedrockOperators[0].subscriptionStatus\"},{\"description\":\"Installed Bedrock Operator Version\",\"displayName\":\"Version\",\"path\":\"bedrockOperators[0].version\"},{\"description\":\"Installed Bedrock Operator Troubleshooting\",\"displayName\":\"Troubleshooting\",\"path\":\"bedrockOperators[0].troubleshooting\"},{\"description\":\"The Overall Status Of All Installed Bedrock Operators\",\"displayName\":\"OverallStatus\",\"path\":\"overallStatus\"}],\"version\":\"v3\"}]},\"description\":\"# Introduction\\n\\nThe `IBM Cloud Pak foundational services` is a bridge to connect IBM Cloud Paks and Operand Deployment Lifecycle Manager (ODLM) with IBM Cloud Pak foundational services. You can also install the `IBM Cloud Pak foundational services` in stand-alone mode.\\n\\nWhen you install this operator, the operator completes the following tasks:\\n\\n  - Installs ODLM in all namespaces mode\\n  - Creates the `ibm-common-services` namespace\\n  - Creates the Common Services `OperandRegistry` and `OperandConfig` in the `ibm-common-services` namespace\\n\\n# Details\\nFor more information about installing this operator and other Common Services operators, see [Installer documentation](http://ibm.biz/cpcs_opinstall). If you are using this operator as part of an IBM Cloud Pak, see the documentation for that IBM Cloud Pak to learn more about how to install and use the operator service. For more information about IBM Cloud Paks, see [IBM Cloud Paks that use Common Services](http://ibm.biz/cpcs_cloudpaks).\\n\\nFor more information about the available IBM Cloud Pak foundational services, see the [IBM Knowledge Center](http://ibm.biz/cpcsdocs).\\n\\n## Supported platforms\\n\\nRed Hat OpenShift Container Platform 4.5 or newer installed on one of the following platforms:\\n\\n - Linux x86_64\\n - Linux on Power (ppc64le)\\n - Linux on IBM Z and LinuxONE\\n\\n## Operator versions\\n\\n - 4.3.0\\n\\n## Prerequisites\\n\\nBefore you install this operator, you need to first install the operator prerequisites:\\n\\n- For the list of prerequisites for installing the operator, see the IBM Knowledge Center [Preparing to install services documentation](http://ibm.biz/cpcs_opinstprereq).\\n\\n### Resources Required\\n\\n#### Minimum scheduling capacity\\n\\n| Software                   | Memory (MB) | CPU (cores) | Disk (GB) | Nodes  |\\n|\"}}]}\n' is missing root schema field"

Not sure why there is a missing q from quay in the pod name either.

Environment

Operator type:

/language go

Kubernetes cluster type:

OpenShift

$ operator-sdk version

$ go version (if language is Go)

go version go1.21.0 linux/amd64

$ kubectl version

Possible Solution

Additional context

I hit this error when using operator-sdk v1.30.0 and v1.31.0.

I did not see this issue when using operator-sdk v1.29.0 or v1.28.0, so I do not think it's because of opm since the same default index image was used for all runs.

The only change related to operator-sdk run bundle I could find between v1.29.0 and v1.30.0 is this one #6477, but I do not see how this change would cause the error log above.

I also have another operator where operator-sdk run bundle runs successfully even at v1.30.0 or v1.31.0, https://github.com/IBM/ibm-cert-manager-operator

[bitscuit]

/kind bug

[bitscuit]

/triage support

[everettraven]

This is interesting. At first glance of the error it seems like there is no schema: field in the FBC object that is being parsed, but running opm render on the image provided in the issue description that field does exist. I haven't had time to try and replicate this, but will swing back around when I have some time if someone else hasn't already taken a look at this.

Thanks for filing the issue!

[bitscuit]

Let me know how I can help debug the issue. It probably is something specifically with common-service-operator since cert-manager-operator did not see issues.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.