502 Bad gateway: Hazelcast Management Center via GCE Ingress

[maslick]

For those of you who will be exposing Hazelcast Management Center (MC) via Google's Ingress GCE controller - be aware that GCE Ingress is expecting a working (should be 200 OK) readiness probe from the Service backing the MC Deployment.

I tried with the defaults and got 502 Bad gateway when accessing the exposed MC service. You can read about similar experience here.

Note also, that the default endpoint /hazelcast-mancenter returns 302 Found, which is ok for the readiness probe, but apparently not for GCE Ingress. The quick hack would be to use either /hazelcast-mancenter/index.html or /hazelcast-mancenter/. But in the long term I would propose to implement a dedicated health probe, e.g. /health or /healthz, etc.

So 302 Found is ok for checking the readiness of the Service, but if you check the status of the ingress, you will probably see the UNHEALTHY status:

k describe ing mc-ingress
Name:             mc-ingress
Namespace:        default
Address:          34.98.YYY.2
Default backend:  mc-service:8080 (10.52.2.23:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     mc-service:8080 (10.52.2.23:8080)
Annotations:  
  ingress.kubernetes.io/backends:  {"k8s-be-32517--47d3253409234a5d":"UNHEALTHY"}
Events:                            <none>

Here is a working yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mc-volume
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mc
  labels:
    app: hazelcast
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hazelcast
  template:
    metadata:
      labels:
        app: hazelcast
    spec:
      containers:
        - name: mc
          image: hazelcast/management-center
          resources:
            limits:
              memory: 1Gi
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /hazelcast-mancenter/
              port: 8080
          readinessProbe:
            httpGet:
              path: /hazelcast-mancenter/
              port: 8080
          volumeMounts:
            - name: mc-storage
              mountPath: /data
      volumes:
        - name: mc-storage
          persistentVolumeClaim:
            claimName: mc-volume
---
apiVersion: v1
kind: Service
metadata:
  name: mc-service
spec:
  type: NodePort
  selector:
    app: hazelcast
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: mc-ingress
  labels:
    app: hazelcast
spec:
  backend:
    serviceName: mc-service
    servicePort: 8080

P.S. Nginx Ingress doesn't have such issue. But a dedicated health endpoint is a nice to have non the less.

[maslick]

Not a big issue though, but can be an obstacle for newcomers. Perhaps, the Enterprise version of Hazelcast has it fixed. I don't know if anyone who will be using GCE Ingress in production, though. I am personally in favor of Nginx Ingress. Just observations :)

[maslick]

Thanks to @puzpuzpuz who pointed me in the right direction, I have enabled the /health endpoint. See the yaml below:

---
# ==================================================================
# Hazelcast management center (PVC + Deployment + NodePort service)
# ==================================================================
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mc-volume
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mc
  labels:
    app: hazelcast
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hazelcast
  template:
    metadata:
      labels:
        app: hazelcast
    spec:
      containers:
        - name: mc
          image: hazelcast/management-center
          resources:
            limits:
              memory: 1Gi
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /hazelcast-mancenter/health
              port: 8081
            initialDelaySeconds: 30
            periodSeconds: 15
          readinessProbe:
            httpGet:
              path: /hazelcast-mancenter/health
              port: 8081
            initialDelaySeconds: 30
            periodSeconds: 15
          volumeMounts:
            - name: mc-storage
              mountPath: /data
          env:
            - name: JAVA_OPTS
              value: "-Dhazelcast.mc.healthCheck.enable=true"
      volumes:
        - name: mc-storage
          persistentVolumeClaim:
            claimName: mc-volume
---
apiVersion: v1
kind: Service
metadata:
  name: mc-service
spec:
  type: NodePort
  selector:
    app: hazelcast
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: mc-ingress
  labels:
    app: hazelcast
spec:
  backend:
    serviceName: mc-service
    servicePort: 8080

However, it doesn't go through:

k describe ing mc-ingress
...
ingress.kubernetes.io/backends:         {"k8s-be-31175--3c74f75351609d43":"UNHEALTHY"}
...

I guess the reason is that the health endpoint is exposed on a different port - 8081, whereas GCE Ingress is checking the readiness endpoint through the exposed service (which has target port 8080, not 8081).