[nri-bundle][pixie-chart] unable to deploy the chart

[prabhakardinvicara]

Hi,

Am facing the below error while deploy nri-bundle/pixie-chart on Kubernetes cluster (AWS EKS) with helm version 4.3.2

Error: template: nri-bundle/charts/pixie-chart/templates/00_olm.yaml:1:41: executing "nri-bundle/charts/pixie-chart/templates/00_olm.yaml" at <lookup "operators.coreos.com/v1" "OperatorGroup" "" "">: error calling lookup: unable to get apiresource from unstructured: operators.coreos.com/v1, Kind=OperatorGroup: the server could not find the requested resource

please suggest

[guhcampos]

I get the same error on GKE clusters, tried up to 4.4.8 with no success.

This seems related to missing the CRDs for the Pixie operator. I recall their installation was required and documented previously, but the new Pixie installation docs do not mention this at all, presumably because the Helm 3 only version of the charts were supposed to install them? (but apparently don't )

[guhcampos]

Confirming this was the case, after running:

kubectl apply -f https://raw.githubusercontent.com/pixie-labs/pixie/main/k8s/operator/crd/base/px.dev_viziers.yaml
kubectl apply -f https://raw.githubusercontent.com/pixie-labs/pixie/main/k8s/operator/helm/crds/olm_crd.yaml

The chart applies correctly. The only place I still found these steps was https://docs.newrelic.com/docs/kubernetes-pixie/auto-telemetry-pixie/manage-pixie-data/ as they were removed from the installation docs when the charts got updated.

To clarify a bit, this was a helm upgrade from 3.6.6 to 4.x.x, on a cluster where Pixie was disabled, so my guess is the charts should install correctly on brand new clusters or clusters where Pixie was already present, and we are hitting a corner case where we want to upgrade from a non-pixie environment.

I would suggest that the Helm Charts and Pixie Installation Docs should be updated to reflect this.

[calinciobanu]

Also running into this issue with a customer when upgrading from a non-pixie to a pixie environment. Using AWS EKS and helm version 3.4.1. I think we'll have them uninstall and reinstall with Pixie.

lookup_func.go:111: [ERROR] unable to retrieve resource list for: [operators.coreos.com/v1](http://operators.coreos.com/v1) , error: the server could not find the requested resource

lookup_func.go:85: [ERROR] unable to get apiresource from unstructured: [operators.coreos.com/v1](http://operators.coreos.com/v1), Kind=OperatorGroup , error the server could not find the requested resource

Error: UPGRADE FAILED: template: nri-bundle/charts/pixie-chart/templates/00_olm.yaml:1:41: executing "nri-bundle/charts/pixie-chart/templates/00_olm.yaml" at <lookup "[operators.coreos.com/v1](http://operators.coreos.com/v1)" "OperatorGroup" "" "">: error calling lookup: unable to get apiresource from unstructured: [operators.coreos.com/v1](http://operators.coreos.com/v1), Kind=OperatorGroup: the server could not find the requested resource

[philomory]

Helm 3 includes the ability to bundle CRDs into a chart (in the crds directory), but, it handles management of CRDs very poorly; in particular, CRDs are only ever touched during helm install (or helm upgrade --install if it's acting as an install rather than an upgrade, e.g., the helm release containing the CRDs was not previously present). This brings with it a couple of limitations:

1. If a chart updates their CRDs, helm upgrade will just leave the old versions in place and you'll never get any updates
2. If you uninstall a release, the CRDs will remain (this, at least, is arguably the right call)
3. If a chart containing CRDs is used as a subchart of an umbrella chart (as in the case of nri-bundle bundling pixie-operator-chart), and a flag is used to control whether the subchart is installed (such as pixie-chart.enabled), then, on initial installation of the umbrella chart, the CRDs from the subchart will only be installed if the subchart itself is installed; if, at a later time, the umbrella chart is upgraded and the flag is changed, the subchart will now be installed for the first time... but this is still considered an "upgrade" by Helm, so the CRDs in the crds folder are skipped entirely (as they always are for upgrades).

For these reasons, a lot of projects that need CRDs seem to opt to package the CRDs as regular resources in the templates folder, but in a dedicated <project>-crds chart, allowing them to be installed before the main chart, upgraded when needed, left in-place when the main chart is uninstalled (to avoid data loss), still uninstalled via helm if you're really sure you don't need them any more, etc.

Ultimately, I think the solution will have to involve packaging the Pixie CRDs in their own chart, pixie-crds or pixie-operator-crds or similar; and then, unfortunately, asking the user to install the pixie-crds chart before installing the nri-bundle chart (because, as noted here by the KEDA project, subchart dependency ordering is not possible in Helm, so if you made pixie-crds a dependency of pixie-operator or nri-bundle, helm would still try to validate the parent chart before installing the CRD chart).