update image with emqx-operator has a problem

[neverlaughtingboy]

my emqx version is 5.0.16，and operator verion is 2.1.1
2023-03-02T09:10:31Z ERROR Reconciler error {"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX", "eMQX": {"name":"emqx","namespace":"emqx-by-operator-test"}, "namespace": "emqx-by-operator-test", "name": "emqx", "reconcileID": "2b6ebb89-bdf7-475c-aaba-ac60a865f355", "error": "failed to update StatefulSet emqx-core: StatefulSet.apps "emqx-core" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden", "errorVerbose": "StatefulSet.apps "emqx-core" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden\nfailed to update StatefulSet emqx-core\ngithub.com/emqx/emqx-operator/internal/handler.(*Handler).Update\n\t/workspace/internal/handler/handler.go:153\ngithub.com/emqx/emqx-operator/internal/handler.(*Handler).CreateOrUpdate\n\t/workspace/internal/handler/handler.go:126\ngithub.com/emqx/emqx-operator/internal/handler.(*Handler).CreateOrUpdateList\n\t/workspace/internal/handler/handler.go:61\ngithub.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*EMQXReconciler).Reconcile\n\t/workspace/controllers/apps/v2alpha1/emqx_controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234

[Rory-Z]

Hi, @neverlaughtingboy It looks like you modified other fields besides the image, could you please describe your steps?

[Rory-Z]

"error": "failed to update StatefulSet emqx-core: StatefulSet.apps "emqx-core" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"
Typically, this is an error caused by modifying some of the StatefulSet's fields, which is usually caused by modifying the wrong EMQX Custom Resource field.
If you make sure you just change the image, could you please use kubectl retry it, if this issue is still, please share your Yaml file.

[neverlaughtingboy]

use kubectl still has this issue,but when i tried deploy it in to new namespace start over again,it seems work properly.
I think i figured out what's causing it,I didn’t know how much space to allocate to the core node at first, so I patch the pv of the core node, does it make sense?

[Rory-Z]

so I patch the pv of the core node

Did you edit the volumeClaimTemplates for EMQX CR when it is running ? If yes, I think this is reason, because the statefulSet is not allow update volumeClaimTemplates when it running.

[neverlaughtingboy]

I patch pvc use this command when it is running
kubectl patch pvc emqx-core-data-emqx-core-0 -n emqx-by-operator-test -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
If I still need to expand pvc in the future,should I stop the emqx cluster first and edit emqx cr to achieve?

[Rory-Z]

You can patch the PVC, that's all right, but you can't edit the volumeClaimTemplates, because the StatefulSet not allow update this field.

If you want update volumeClaimTemplates, you need stop the EMQX cluster and delete the PVC resource, and re-create EMQX cluster