[frontend] Error on Kubeflow Pipelines Dashboard: 'No Healthy Upstream', '503 UC upstream_reset_before_response_started{connection_termination}' #11260

ChaeSeungJi · 2024-10-01T11:19:36Z

Environment

How did you deploy Kubeflow Pipelines (KFP)?

Kubeflow 1.9.1 Release Candidate 1
while ! kustomize build example | kubectl delete -f -; do echo "Retrying to delete resources"; sleep 20; done

KFP version: 1.19.1

Kubernetes version

Steps to reproduce

Deployed Kubeflow Pipelines (KFP) using Kubeflow 1.9.1 Release Candidate 1.
The installation process completed successfully without any errors.
After the installation, I opened the Kubeflow dashboard.
When accessing the Kubeflow Pipelines dashboard, an error occurred.
Other components like Notebook, Volume, etc., are working fine, but the issue is specific to the KFP dashboard.

Expected result

ml-pipeline-ui error : Error: failed to retrieve list of pipelines. Click Details for more information.

Details : An error occurred upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: 111
When refreshing the page (F5), the following screen appears: no healthy upstream

Materials and Reference

ml-pipeline logs

I1001 10:35:43.871571       7 interceptor.go:37] /api.PipelineService/ListPipelinesV1 handler finished
I1001 10:35:43.962205       7 interceptor.go:29] /api.RunService/ListRunsV1 handler starting
I1001 10:35:43.962262       7 resource_manager.go:1690] Getting user identity
I1001 10:35:43.962283       7 resource_manager.go:1711] User: [email protected], ResourceAttributes: &ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}1beta1,Resource:runs,Subresource:,Name:,}
I1001 10:35:43.962310       7 resource_manager.go:1712] Authorizing request
I1001 10:35:43.969834       7 resource_manager.go:1755] Authorized user '[email protected]': &ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}esource:runs,Subresource:,Name:,}
I1001 10:35:43.973333       7 interceptor.go:37] /api.RunService/ListRunsV1 handler finished
I1001 10:36:08.324576       7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelines handler starting
I1001 10:36:08.327806       7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelines handler finished
I1001 10:36:08.360273       7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelines handler starting
I1001 10:36:08.360340       7 resource_manager.go:1690] Getting user identity
I1001 10:36:08.360361       7 resource_manager.go:1711] User: [email protected], ResourceAttributes: &ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:pipelines,Subresource:,Name:,}1beta1,Resource:pipelines,Subresource:,Name:,}
I1001 10:36:08.360382       7 resource_manager.go:1712] Authorizing request
I1001 10:36:08.363527       7 resource_manager.go:1755] Authorized user '[email protected]': &ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:pipelines,Subresource:,Name:,}esource:pipelines,Subresource:,Name:,}
I1001 10:36:08.366665       7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelines handler finished

ml-pipeline(istio-proxy) logs

[2024-10-01T10:37:34.157Z] "GET /apis/v1beta1/healthz HTTP/1.1" 200 - via_upstream - "-" 0 96 0 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "2bbf19f2-49c9-4360-82b7-7380296b07e4" "10.106.183.67:8888" "192.168.52.177:8888" inbound|8888|| 127.0.0.6:49739 192.168.52.177:8888 192.168.217.73:37648 - default
[2024-10-01T10:35:43.859Z] "- - -" 0 - - - "-" 2700 4440 114505 - "-" "-" "-" "-" "10.3.129.237:6443" outbound|443||kubernetes.default.svc.cluster.local 192.168.52.177:59564 10.96.0.1:443 192.168.52.177:59130 - -
[2024-10-01T10:37:39.154Z] "GET /apis/v1beta1/healthz HTTP/1.1" 200 - via_upstream - "-" 0 96 0 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "40d3f28e-f60f-415d-a83b-0b483418db00" "10.106.183.67:8888" "192.168.52.177:8888" inbound|8888|| 127.0.0.6:41605 192.168.52.177:8888 192.168.217.73:47786 - default
[2024-10-01T10:37:39.154Z] "GET /apis/v1beta1/healthz HTTP/1.1" 200 - via_upstream - "-" 0 96 0 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "c779661f-49c9-46b3-8f37-d8d277873152" "10.106.183.67:8888" "192.168.52.177:8888" inbound|8888|| 127.0.0.6:49739 192.168.52.177:8888 192.168.217.73:37648 - default
[2024-10-01T10:35:43.864Z] "- - -" 0 - - - "-" 5475 37098 120008 - "-" "-" "-" "-" "192.168.52.178:3306" outbound|3306||mysql.kubeflow.svc.cluster.local 192.168.52.177:33288 10.96.191.178:3306 192.168.52.177:39610 - -

ml-pipeline-ui logs

GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /pipeline/
GET /pipeline/static/js/main.b980985e.js
GET /pipeline/static/css/main.e10b3034.css
GET /pipeline/system/project-id
GET /pipeline/apis/v1beta1/healthz
GET /pipeline/apis/v2beta1/pipelines?page_token=&page_size=10&sort_by=created_at%20desc&filter=
Proxied request:  /apis/v2beta1/pipelines?page_token=&page_size=10&sort_by=created_at%20desc&filter=
GET /pipeline/system/cluster-name
GET /pipeline/apis/v2beta1/pipelines?namespace=kubeflow-user-example-com&page_token=&page_size=10&sort_by=created_at%20desc&filter=
Proxied request:  /apis/v2beta1/pipelines?namespace=kubeflow-user-example-com&page_token=&page_size=10&sort_by=created_at%20desc&filter=

/server/node_modules/node-fetch/lib/index.js:1491
                        reject(new FetchError(`request to ${request.url} failed, reason: ${err.message}`, 'system', err));
                               ^
FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
    at ClientRequest.<anonymous> (/server/node_modules/node-fetch/lib/index.js:1491:11)
    at ClientRequest.emit (node:events:517:28)
    at Socket.socketErrorListener (node:_http_client:501:9)
    at Socket.emit (node:events:517:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ENOTFOUND',
  code: 'ENOTFOUND'
}

Node.js v18.18.2

ml-pipeline-ui(istio-proxy) logs

[2024-10-01T10:36:08.021Z] "GET /pipeline/static/css/main.e10b3034.css HTTP/1.1" 200 - via_upstream - "-" 0 14493 5 4 "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "58461348-0b7c-4a85-8fcf-72e0e2c78072" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| 127.0.0.6:44173 192.168.217.73:3000 192.168.64.246:0 - default
[2024-10-01T10:36:08.015Z] "GET /pipeline/static/js/main.b980985e.js HTTP/1.1" 200 - via_upstream - "-" 0 4120994 26 8 "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "85419e0a-4f50-96c3-b4e9-8060a636d949" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| 127.0.0.6:44035 192.168.217.73:3000 192.168.64.246:0 - default
[2024-10-01T10:36:08.327Z] "GET /apis/v1beta1/healthz HTTP/1.1" 200 - via_upstream - "-" 0 96 2 2 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "6cbd9bfd-af63-478d-8ddb-b252884570b1" "10.106.183.67:8888" "192.168.52.177:8888" outbound|8888||ml-pipeline.kubeflow.svc.cluster.local 192.168.217.73:37664 10.106.183.67:8888 192.168.217.73:55642 - default
[2024-10-01T10:36:08.312Z] "GET /pipeline/apis/v1beta1/healthz HTTP/1.1" 200 - via_upstream - "-" 0 291 17 17 "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "95641d97-c788-4b82-80ff-dec317fb7d8a" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| 127.0.0.6:44035 192.168.217.73:3000 192.168.64.246:0 - default
[2024-10-01T10:36:08.327Z] "GET /pipeline/apis/v2beta1/pipelines?page_token=&page_size=10&sort_by=created_at%20desc&filter= HTTP/1.1" 200 - via_upstream - "-" 0 883 6 6 "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "fa1cc37a-6504-4044-be57-f4fc7283abba" "10.106.183.67:8888" "192.168.52.177:8888" outbound|8888||ml-pipeline.kubeflow.svc.cluster.local 192.168.217.73:47786 10.106.183.67:8888 192.168.64.246:0 - default
[2024-10-01T10:36:08.313Z] "GET /pipeline/apis/v2beta1/pipelines?page_token=&page_size=10&sort_by=created_at%20desc&filter= HTTP/1.1" 200 - via_upstream - "-" 0 883 21 20 "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "fa1cc37a-6504-4044-be57-f4fc7283abba" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| 127.0.0.6:44173 192.168.217.73:3000 192.168.64.246:0 - default
[2024-10-01T10:36:08.363Z] "GET /pipeline/apis/v2beta1/pipelines?namespace=kubeflow-user-example-com&page_token=&page_size=10&sort_by=created_at%20desc&filter= HTTP/1.1" 200 - via_upstream - "-" 0 2 9 9 "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "1469974e-9638-4c8c-b485-a25948020125" "10.106.183.67:8888" "192.168.52.177:8888" outbound|8888||ml-pipeline.kubeflow.svc.cluster.local 192.168.217.73:47786 10.106.183.67:8888 192.168.64.246:0 - default
[2024-10-01T10:36:08.312Z] "GET /pipeline/system/project-id HTTP/1.1" 503 UC upstream_reset_before_response_started{connection_termination} - "-" 0 95 69 - "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "5215f428-d320-4cf6-b07f-5156818e6cd6" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| 127.0.0.6:45291 192.168.217.73:3000 192.168.64.246:0 - default
[2024-10-01T10:36:08.313Z] "GET /pipeline/system/cluster-name HTTP/1.1" 503 UC upstream_reset_before_response_started{connection_termination} - "-" 0 95 68 - "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "551d34d9-3a65-456a-92a2-18331db6e9c2" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| 127.0.0.6:49221 192.168.217.73:3000 192.168.64.246:0 - default
[2024-10-01T10:36:08.360Z] "GET /pipeline/apis/v2beta1/pipelines?namespace=kubeflow-user-example-com&page_token=&page_size=10&sort_by=created_at%20desc&filter= HTTP/1.1" 503 UC upstream_reset_before_response_started{connection_termination} - "-" 0 95 21 - "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "1469974e-9638-4c8c-b485-a25948020125" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| 127.0.0.6:44035 192.168.217.73:3000 192.168.64.246:0 - default
[2024-10-01T10:36:08.384Z] "GET /pipeline/system/project-id HTTP/1.1" 503 UF upstream_reset_before_response_started{remote_connection_failure,delayed_connect_error:_111} - "delayed_connect_error:_111" 0 152 0 - "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "5215f428-d320-4cf6-b07f-5156818e6cd6" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| - 192.168.217.73:3000 192.168.64.246:0 - default
[2024-10-01T10:36:08.390Z] "GET /pipeline/system/project-id HTTP/1.1" 503 UF upstream_reset_before_response_started{remote_connection_failure,delayed_connect_error:_111} - "delayed_connect_error:_111" 0 152 0 - "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "5215f428-d320-4cf6-b07f-5156818e6cd6" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| - 192.168.217.73:3000 192.168.64.246:0 - default
[2024-10-01T10:36:08.395Z] "GET /pipeline/system/cluster-name HTTP/1.1" 503 UF upstream_reset_before_response_started{remote_connection_failure,delayed_connect_error:_111} - "delayed_connect_error:_111" 0 152 0 - "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "551d34d9-3a65-456a-92a2-18331db6e9c2" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| - 192.168.217.73:3000 192.168.64.246:0 - default
[2024-10-01T10:36:08.402Z] "GET /pipeline/apis/v2beta1/pipelines?namespace=kubeflow-user-example-com&page_token=&page_size=10&sort_by=created_at%20desc&filter= HTTP/1.1" 503 UF upstream_reset_before_response_started{remote_connection_failure,delayed_connect_error:_111} - "delayed_connect_error:_111" 0 152 0 - "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "1469974e-9638-4c8c-b485-a25948020125" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| - 192.168.217.73:3000 192.168.64.246:0 - default
[2024-10-01T10:36:08.433Z] "GET /pipeline/system/cluster-name HTTP/1.1" 503 UF upstream_reset_before_response_started{remote_connection_failure,delayed_connect_error:_111} - "delayed_connect_error:_111" 0 152 0 - "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "551d34d9-3a65-456a-92a2-18331db6e9c2" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| - 192.168.217.73:3000 192.168.64.246:0 - default
[2024-10-01T10:36:08.441Z] "GET /pipeline/apis/v2beta1/pipelines?namespace=kubeflow-user-example-com&page_token=&page_size=10&sort_by=created_at%20desc&filter= HTTP/1.1" 503 UF upstream_reset_before_response_started{remote_connection_failure,delayed_connect_error:_111} - "delayed_connect_error:_111" 0 152 0 - "192.168.64.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" "1469974e-9638-4c8c-b485-a25948020125" "inl.test:32692" "192.168.217.73:3000" inbound|3000|| - 192.168.217.73:3000 192.168.64.246:0 - default

What I've Tried

kubeflow/kubeflow#5561
#4469
kubeflow/kubeflow#5271

Impacted by this bug? Give it a 👍.

The text was updated successfully, but these errors were encountered:

rimolive · 2024-10-03T19:50:24Z

@ChaeSeungJi We have just released https://github.com/kubeflow/manifests/releases/tag/v1.9.1-rc.2. Can you take a look and see if this release fixes your issue?

utsumi-fj · 2024-10-21T07:32:33Z

I have the same error in v1.9.1-rc.2.

Environment:

k8s: 1.30.5
kustomize: v5.2.1
installation command: kustomize build example | kubectl apply -f -

After running a pipeline from Jupyter Notebook and clicking the link Experiment details. as follows,

the following error occurs.

And, when clicking the menu Pipelines and so on at the sidebar, the following error occurs.

utsumi-fj · 2024-10-22T06:14:37Z

Maybe, this issue is same as #11247. In my environment, #11321 resolved this issue.
For trying this PR, I edited deployment ml-pipeline-ui as follows, and this error disappeared.

kubectl edit deployment ml-pipeline-ui -n kubeflow
# Add environment variable DISABLE_GKE_METADATA and set to true.
#    spec:
#      containers:
#      - env:
#        - name: DISABLE_GKE_METADATA
#          value: "true"

ChaeSeungJi · 2024-10-31T13:06:46Z

@utsumi-fj thank you!!! it works good

…ubeflow#11260 Signed-off-by: Dave Canton <[email protected]>

…11403) Signed-off-by: Dave Canton <[email protected]>

ChaeSeungJi added area/frontend kind/bug labels Oct 1, 2024

ChaeSeungJi closed this as completed Oct 31, 2024

dvcanton added a commit to dvcanton/kubeflow-pipelines that referenced this issue Nov 23, 2024

fix(ui): desabled GKE metadata by default. Fixes kubeflow#11247 and k…

2c494c6

…ubeflow#11260 Signed-off-by: Dave Canton <[email protected]>

This was referenced Nov 23, 2024

fix(ui): Disable GKE Metadata as default Fixes: #11247, fixes #11260 demarna1/pipelines#2

Closed

fix(ui): Disable GKE Metadata as default Fixes: #11247, fixes #11260 #11403

Merged

dvcanton added a commit to dvcanton/kubeflow-pipelines that referenced this issue Nov 26, 2024

fix(ui): desabled GKE metadata by default. Fixes kubeflow#11247 and k…

9efbcec

…ubeflow#11260 Signed-off-by: Dave Canton <[email protected]>

dvcanton added a commit to dvcanton/kubeflow-pipelines that referenced this issue Nov 26, 2024

fix(ui): desabled GKE metadata by default. Fixes kubeflow#11247 and k…

e94d260

…ubeflow#11260 Signed-off-by: Dave Canton <[email protected]>

google-oss-prow bot pushed a commit that referenced this issue Nov 27, 2024

fix(ui): Disable GKE Metadata as default Fixes: #11247, fixes #11260 (#…

23f718d

…11403) Signed-off-by: Dave Canton <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[frontend] Error on Kubeflow Pipelines Dashboard: 'No Healthy Upstream', '503 UC upstream_reset_before_response_started{connection_termination}' #11260

[frontend] Error on Kubeflow Pipelines Dashboard: 'No Healthy Upstream', '503 UC upstream_reset_before_response_started{connection_termination}' #11260

ChaeSeungJi commented Oct 1, 2024

rimolive commented Oct 3, 2024

utsumi-fj commented Oct 21, 2024

utsumi-fj commented Oct 22, 2024 •

edited

Loading

ChaeSeungJi commented Oct 31, 2024

[frontend] Error on Kubeflow Pipelines Dashboard: 'No Healthy Upstream', '503 UC upstream_reset_before_response_started{connection_termination}' #11260

[frontend] Error on Kubeflow Pipelines Dashboard: 'No Healthy Upstream', '503 UC upstream_reset_before_response_started{connection_termination}' #11260

Comments

ChaeSeungJi commented Oct 1, 2024

Environment

Steps to reproduce

Expected result

Materials and Reference

What I've Tried

rimolive commented Oct 3, 2024

utsumi-fj commented Oct 21, 2024

utsumi-fj commented Oct 22, 2024 • edited Loading

ChaeSeungJi commented Oct 31, 2024

utsumi-fj commented Oct 22, 2024 •

edited

Loading