Quickstart broken: modelmesh-serving svc not created in --namespace-scope-mode: secret "model-serving-etcd" not found

[planetA]

Describe the bug

The quickstart install instructions no longer work correctly. After deploying a model, the InferenceService does not get into READY state. Inference requests cannot be performed.

The behavior is similar to #433 but the reason seems to be different. In logs I see:

Normal   Scheduled    1s    default-scheduler  Successfully assigned modelmesh-serving/modelmesh-serving-mlserver-1.x-66bfb5668f-zlcbv to mplaneta-cluster
Warning  FailedMount  1s    kubelet            MountVolume.SetUp failed for volume "etcd-config" : secret "model-serving-etcd" not found

The secret does not exist and if I try to create it manually, it is deleted immediately. If I recreate the cluster without --namespace-scope-mode, the erroneous behavior disappears.

To Reproduce

Prepare:

RELEASE="release-0.12"
git clone -b $RELEASE --depth 1 --single-branch https://github.com/kserve/modelmesh-serving.git
cd modelmesh-serving

Install

kubectl create namespace modelmesh-serving
./scripts/install.sh --namespace-scope-mode --namespace modelmesh-serving --quickstart -d


kubectl apply -f - <<EOF
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: example-mnist-isvc
  annotations:
    serving.kserve.io/deploymentMode: ModelMesh
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storage:
        key: localMinIO
        path: sklearn/mnist-svm.joblib
EOF

Environment (please complete the following information):

release-0.12

kubectl version
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

[haiminh2001]

if you use --namespace-scope-mode, you must label your namespace with modelmesh-enabled=true
For example:

kubectl label ns my-namespace modelmesh-enabled=true

The secret should be created right after.