workflow status stuck in Running occasionally (no repro)
#13238
Labels
Comments
agilgur5
changed the title
By "not timely", do you mean it was delayed by 1 min? I don't think that's a very long delay, especially if you have high load. And this error in your logs also suggests you're getting rate-limited (see also You also have an Executor deprecation warning: Which you should really fix by following the Workflow RBAC page. The fallback will be removed in 3.6: #13100
Also, please use text instead of images when listing logs, configurations, or code, as text is much more accessible than images |
agilgur5
added
the
problem/more information needed
|
yeah, but from the image of UI, this step has running 1h10min with pending status. The pod is completed, but the workflow is in pending status. |
|
Is the Workflow stuck in Those are two very different things. I have a feeling you meant the former, but you wrote the latter
Also without a Workflow, nor even a specific version, this issue is not really actionable -- there is nothing to investigate. Please provide the Workflow that is Given that you're using deprecated pod patch permissions though, you have no |
|
yeah, I can not give a workflow to reproduce this bug stably, and today I got this bug again. we can get that the pod of step-ligand-prepare has finished 3h30m ago, but the workflow status is always pending. |
@heidongxianhua could you answer the above questions?
To clarify, by this I meant the output of plain |
|
yeah, this pending status step block the workflow, and the workflow is always in running status, can not reach finished status. wf yamlapiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
annotations:
workflows.argoproj.io/pod-name-format: v2
creationTimestamp: "2024-07-02T02:54:04Z"
generateName: posebuster-bytedock-cpp-eval-8slg-g5a-
generation: 6
labels:
workflows.argoproj.io/creator: system-serviceaccount-argo-argo-server
workflows.argoproj.io/phase: Running
name: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
namespace: bytevs
resourceVersion: "1379931806"
uid: e11ff5c4-d430-449e-87d8-ea40e3139aab
spec:
arguments: {}
entrypoint: posebuster-bytedock-cpp-eval-8slg-g5a-steps
nodeSelector:
node-tag: bytevs
podGC: {}
podPriorityClassName: critical
templates:
- inputs:
artifacts:
- name: flow_bigpar_serverless_input
path: /tmp/inputs/parameters/serverless_input
- name: flow_python_packages
path: /tmp/inputs/artifacts/flow_python_packages
parameters:
- description: '{"type": "<class ''str''>"}'
name: receptor_pdb
value: ""
- description: '{"type": "<class ''str''>"}'
name: receptor_prepared_dir_tos
value: ""
- description: '{"type": "<class ''str''>"}'
name: reference_tos
value: ""
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id": -1,
"gpu_info": {}, "resource_platform": "", "class_name": ""}'
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
- name: flow_key
value: ""
metadata:
annotations:
workflows.argoproj.io/progress: 0/1
name: bytedockpreparereceptor-kjzln-oxn6f
outputs:
parameters:
- description: '{"type": "<class ''str''>"}'
name: receptor_info_tos
valueFrom:
default: ""
path: /tmp/outputs/parameters/receptor_info_tos
- description: '{"type": "<class ''dict''>"}'
name: flow_info
valueFrom:
default: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
-1, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
path: /tmp/outputs/parameters/flow_info
- description: '{"type": "<class ''str''>"}'
name: flow_lock
valueFrom:
default: ""
path: /tmp/outputs/parameters/flow_lock
- globalName: '{{inputs.parameters.flow_key}}'
name: flow_global
value: '{{pod.name}}'
script:
command:
- python3
image: test:6f2c8db77a010c813e0125bf35023d1b
name: ""
resources:
limits:
cpu: "96"
memory: 96Gi
requests:
cpu: "1"
memory: 2Gi
source: ""
- inputs:
artifacts:
- name: flow_bigpar_serverless_input
path: /tmp/inputs/parameters/serverless_input
- name: flow_python_packages
path: /tmp/inputs/artifacts/flow_python_packages
parameters:
- description: '{"type": "<class ''str''>"}'
name: receptor_info_tos
value: ""
- description: '{"type": "<class ''bool''>"}'
name: load_results_if_exist
value: "false"
- description: '{"type": "<class ''bool''>"}'
name: run_vina_local
value: "false"
- description: '{"type": "<class ''int''>"}'
name: vina_cpu
value: "16"
- description: '{"type": "<class ''int''>"}'
name: vina_n_poses
value: "50"
- description: '{"type": "<class ''int''>"}'
name: vina_seed
value: "12345"
- description: '{"type": "<class ''int''>"}'
name: vina_exhaustiveness
value: "32"
- description: '{"type": "<class ''int''>"}'
name: vina_search_depth
value: "0"
- description: '{"type": "<class ''float''>"}'
name: vina_energy_range
value: "1000"
- description: '{"type": "<class ''str''>"}'
name: output_dir_tos
value: ""
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id": -1,
"gpu_info": {}, "resource_platform": "", "class_name": ""}'
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
- name: flow_key
value: ""
metadata:
annotations:
workflows.argoproj.io/progress: 0/1
name: bytedockvinadocking-zsxd5-myk9u
outputs:
artifacts:
- archive:
none: {}
name: flow_bigpar_serverless_output
path: /tmp/outputs/parameters/serverless_output
parameters:
- description: '{"type": "<class ''dict''>"}'
name: flow_info
valueFrom:
default: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
-1, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
path: /tmp/outputs/parameters/flow_info
- description: '{"type": "<class ''str''>"}'
name: flow_lock
valueFrom:
default: ""
path: /tmp/outputs/parameters/flow_lock
- globalName: '{{inputs.parameters.flow_key}}'
name: flow_global
value: '{{pod.name}}'
script:
command:
- python3
image: test:6f2c8db77a010c813e0125bf35023d1b
name: ""
resources:
limits:
cpu: "96"
memory: 96Gi
requests:
cpu: "1"
memory: 2Gi
source: ""
- inputs:
artifacts:
- name: flow_bigpar_serverless_input
path: /tmp/inputs/parameters/serverless_input
- name: flow_python_packages
path: /tmp/inputs/artifacts/flow_python_packages
parameters:
- description: '{"type": "<class ''bool''>"}'
name: load_results_if_exist
value: "false"
- description: '{"type": "<class ''bool''>"}'
name: write_csv_per_row
value: "false"
- description: '{"type": "<class ''str''>"}'
name: output_dir_tos
value: ""
- description: '{"type": "<class ''dict''>"}'
name: kwargs
value: '{}'
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id": -1,
"gpu_info": {}, "resource_platform": "", "class_name": ""}'
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
- name: flow_key
value: ""
metadata:
annotations:
workflows.argoproj.io/progress: 0/1
name: bytedockligandprepare-86wtx-jrgik
outputs:
artifacts:
- archive:
none: {}
name: flow_bigpar_serverless_output
path: /tmp/outputs/parameters/serverless_output
parameters:
- description: '{"type": "<class ''dict''>"}'
name: flow_info
valueFrom:
default: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
-1, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
path: /tmp/outputs/parameters/flow_info
- description: '{"type": "<class ''str''>"}'
name: flow_lock
valueFrom:
default: ""
path: /tmp/outputs/parameters/flow_lock
- globalName: '{{inputs.parameters.flow_key}}'
name: flow_global
value: '{{pod.name}}'
script:
command:
- python3
image: test:6f2c8db77a010c813e0125bf35023d1b
name: ""
resources:
limits:
cpu: "96"
memory: 96Gi
requests:
cpu: "1"
memory: 2Gi
source: ""
- inputs:
artifacts:
- name: flow_bigpar_serverless_input
path: /tmp/inputs/parameters/serverless_input
- name: flow_python_packages
path: /tmp/inputs/artifacts/flow_python_packages
parameters:
- description: '{"type": "<class ''str''>"}'
name: receptor_info_tos
value: ""
- description: '{"type": "<class ''str''>"}'
name: receptor_variable
value: none
- description: '{"type": "<class ''str''>"}'
name: ligand_variable
value: xyz
- description: '{"type": "<class ''bool''>"}'
name: check_rmsd
value: "true"
- description: '{"type": "<class ''bool''>"}'
name: use_map
value: "false"
- description: '{"type": "<class ''float''>"}'
name: granularity
value: "-1"
- description: '{"type": "<class ''float''>"}'
name: penalty_coeff
value: "-100.0"
- description: '{"type": "<class ''str''>"}'
name: maps_tos
value: ""
- description: '{"type": "<class ''str''>"}'
name: reference_tos
value: ""
- description: '{"type": "<class ''str''>"}'
name: output_dir_tos
value: ""
- description: '{"type": "<class ''dict''>"}'
name: kwargs
value: '{}'
- description: '{"type": "<class ''str''>"}'
name: eval_type
value: both
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id": -1,
"gpu_info": {}, "resource_platform": "", "class_name": ""}'
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
- name: flow_key
value: ""
metadata:
annotations:
workflows.argoproj.io/progress: 0/1
name: bytedockminimize-lhgtu-vax30
outputs:
artifacts:
- archive:
none: {}
name: flow_bigpar_serverless_output
path: /tmp/outputs/parameters/serverless_output
parameters:
- description: '{"type": "<class ''dict''>"}'
name: flow_info
valueFrom:
default: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
-1, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
path: /tmp/outputs/parameters/flow_info
- description: '{"type": "<class ''str''>"}'
name: flow_lock
valueFrom:
default: ""
path: /tmp/outputs/parameters/flow_lock
- globalName: '{{inputs.parameters.flow_key}}'
name: flow_global
value: '{{pod.name}}'
script:
command:
- python3
image: test:6f2c8db77a010c813e0125bf35023d1b
name: ""
resources:
limits:
cpu: "96"
memory: 96Gi
requests:
cpu: "1"
memory: 2Gi
source: ""
- inputs: {}
metadata: {}
name: posebuster-bytedock-cpp-eval-8slg-g5a-steps
outputs: {}
steps:
- - arguments:
artifacts:
- name: flow_bigpar_serverless_input
path: /tmp/inputs/parameters/serverless_input
s3:
key: bytevs/workflow/upload/fdf03d88-a9ea-4982-8aed-01ca191de2f3/serverless_input
- name: flow_python_packages
path: /tmp/inputs/artifacts/flow_python_packages
s3:
key: bytevs/workflow/upload/72bc1f56-6e6f-406a-92a1-a8068debb522/tmp8bhkqwnn.tgz
parameters:
- description: '{"type": "<class ''str''>"}'
name: receptor_pdb
value: 8SLG_G5A_protein_prepared_noHETATM
- description: '{"type": "<class ''str''>"}'
name: receptor_prepared_dir_tos
value: bytedock/data_redock/raw/posebuster/prepared_receptor/8SLG_G5A
- description: '{"type": "<class ''str''>"}'
name: reference_tos
value: bytedock/data_redock/raw/posebuster/reference_sdf/8SLG_G5A_ligand_prepared.sdf
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
2186, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
- name: flow_key
value: step-receptor-prepare
continueOn: {}
name: step-receptor-prepare
template: bytedockpreparereceptor-kjzln-oxn6f
- - arguments:
artifacts:
- name: flow_bigpar_serverless_input
path: /tmp/inputs/parameters/serverless_input
s3:
key: bytevs/workflow/upload/5c00e12d-09f4-44b9-819a-1d65c383c381/serverless_input
- name: flow_python_packages
path: /tmp/inputs/artifacts/flow_python_packages
s3:
key: bytevs/workflow/upload/72bc1f56-6e6f-406a-92a1-a8068debb522/tmp8bhkqwnn.tgz
parameters:
- description: '{"type": "<class ''str''>"}'
name: receptor_info_tos
value: '{{=steps[''step-receptor-prepare''].outputs.parameters[''receptor_info_tos'']}}'
- description: '{"type": "<class ''bool''>"}'
name: load_results_if_exist
value: "true"
- description: '{"type": "<class ''bool''>"}'
name: run_vina_local
value: "false"
- description: '{"type": "<class ''int''>"}'
name: vina_cpu
value: "8"
- description: '{"type": "<class ''int''>"}'
name: vina_n_poses
value: "50"
- description: '{"type": "<class ''int''>"}'
name: vina_seed
value: "12345"
- description: '{"type": "<class ''int''>"}'
name: vina_exhaustiveness
value: "32"
- description: '{"type": "<class ''int''>"}'
name: vina_search_depth
value: "0"
- description: '{"type": "<class ''float''>"}'
name: vina_energy_range
value: "1000"
- description: '{"type": "<class ''str''>"}'
name: output_dir_tos
value: bytedock_cpp/data_redock/reference_todock_sdf_seed12345_allfix_fp64/posebuster/8SLG_G5A
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
2186, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
- name: flow_key
value: step-vina-docking
continueOn: {}
name: step-vina-docking
template: bytedockvinadocking-zsxd5-myk9u
- - arguments:
artifacts:
- from: '{{steps.step-vina-docking.outputs.artifacts.flow_bigpar_serverless_output}}'
name: flow_bigpar_serverless_input
path: /tmp/inputs/parameters/serverless_input
- name: flow_python_packages
path: /tmp/inputs/artifacts/flow_python_packages
s3:
key: bytevs/workflow/upload/72bc1f56-6e6f-406a-92a1-a8068debb522/tmp8bhkqwnn.tgz
parameters:
- description: '{"type": "<class ''bool''>"}'
name: load_results_if_exist
value: "true"
- description: '{"type": "<class ''bool''>"}'
name: write_csv_per_row
value: "false"
- description: '{"type": "<class ''str''>"}'
name: output_dir_tos
value: bytedock_cpp/data_redock/reference_todock_sdf_seed12345_allfix_fp64/posebuster/8SLG_G5A
- description: '{"type": "<class ''dict''>"}'
name: kwargs
value: '{"pose_batch_size": 1, "write_to_sdf": true, "select_best_pose":
true}'
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
2186, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
- name: flow_key
value: step-ligand-prepare
continueOn: {}
name: step-ligand-prepare
template: bytedockligandprepare-86wtx-jrgik
- - arguments:
artifacts:
- from: '{{steps.step-ligand-prepare.outputs.artifacts.flow_bigpar_serverless_output}}'
name: flow_bigpar_serverless_input
path: /tmp/inputs/parameters/serverless_input
- name: flow_python_packages
path: /tmp/inputs/artifacts/flow_python_packages
s3:
key: bytevs/workflow/upload/72bc1f56-6e6f-406a-92a1-a8068debb522/tmp8bhkqwnn.tgz
parameters:
- description: '{"type": "<class ''str''>"}'
name: receptor_info_tos
value: '{{=steps[''step-receptor-prepare''].outputs.parameters[''receptor_info_tos'']}}'
- description: '{"type": "<class ''str''>"}'
name: receptor_variable
value: torsion
- description: '{"type": "<class ''str''>"}'
name: ligand_variable
value: xyz
- description: '{"type": "<class ''bool''>"}'
name: check_rmsd
value: "true"
- description: '{"type": "<class ''bool''>"}'
name: use_map
value: "false"
- description: '{"type": "<class ''float''>"}'
name: granularity
value: "-1"
- description: '{"type": "<class ''float''>"}'
name: penalty_coeff
value: "-100.0"
- description: '{"type": "<class ''str''>"}'
name: maps_tos
value: ""
- description: '{"type": "<class ''str''>"}'
name: reference_tos
value: bytedock/data_redock/raw/posebuster/reference_sdf/8SLG_G5A_ligand_prepared.sdf
- description: '{"type": "<class ''str''>"}'
name: output_dir_tos
value: bytedock_cpp/data_redock/reference_todock_sdf_seed12345_allfix_fp64/posebuster/8SLG_G5A
- description: '{"type": "<class ''dict''>"}'
name: kwargs
value: '{"pose_batch_size": 1, "write_to_sdf": true, "select_best_pose":
true}'
- description: '{"type": "<class ''str''>"}'
name: eval_type
value: cpp
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
2186, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
- name: flow_key
value: step-minimize-cpp
continueOn: {}
name: step-minimize-cpp
template: bytedockminimize-lhgtu-vax30
- arguments:
artifacts:
- from: '{{steps.step-ligand-prepare.outputs.artifacts.flow_bigpar_serverless_output}}'
name: flow_bigpar_serverless_input
path: /tmp/inputs/parameters/serverless_input
- name: flow_python_packages
path: /tmp/inputs/artifacts/flow_python_packages
s3:
key: bytevs/workflow/upload/72bc1f56-6e6f-406a-92a1-a8068debb522/tmp8bhkqwnn.tgz
parameters:
- description: '{"type": "<class ''str''>"}'
name: receptor_info_tos
value: '{{=steps[''step-receptor-prepare''].outputs.parameters[''receptor_info_tos'']}}'
- description: '{"type": "<class ''str''>"}'
name: receptor_variable
value: torsion
- description: '{"type": "<class ''str''>"}'
name: ligand_variable
value: xyz
- description: '{"type": "<class ''bool''>"}'
name: check_rmsd
value: "true"
- description: '{"type": "<class ''bool''>"}'
name: use_map
value: "false"
- description: '{"type": "<class ''float''>"}'
name: granularity
value: "-1"
- description: '{"type": "<class ''float''>"}'
name: penalty_coeff
value: "-100.0"
- description: '{"type": "<class ''str''>"}'
name: maps_tos
value: ""
- description: '{"type": "<class ''str''>"}'
name: reference_tos
value: bytedock/data_redock/raw/posebuster/reference_sdf/8SLG_G5A_ligand_prepared.sdf
- description: '{"type": "<class ''str''>"}'
name: output_dir_tos
value: bytedock_cpp/data_redock/reference_todock_sdf_seed12345_allfix_fp64/posebuster/8SLG_G5A
- description: '{"type": "<class ''dict''>"}'
name: kwargs
value: '{"pose_batch_size": 1, "write_to_sdf": true, "select_best_pose":
true}'
- description: '{"type": "<class ''str''>"}'
name: eval_type
value: py
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
2186, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
- name: flow_key
value: step-minimize-py
continueOn: {}
name: step-minimize-py
template: bytedockminimize-lhgtu-vax30
tolerations:
- effect: NoSchedule
key: node-tag
operator: Equal
value: bytevs
ttlStrategy:
secondsAfterCompletion: 3
status:
artifactGCStatus:
notSpecified: true
artifactRepositoryRef:
artifactRepository:
archiveLogs: true
s3:
accessKeySecret:
key: accesskey
name: tos-cred
bucket: ai4s
endpoint: tos-s3-cn-beijing-inner.ivolces.com
insecure: false
keyFormat: argo_logs/{{workflow.name}}/{{pod.name}}
region: cn-beijing
secretKeySecret:
key: secretkey
name: tos-cred
configMap: artifact-repositories
key: default-v1
namespace: bytevs
conditions:
- status: "False"
type: PodRunning
finishedAt: null
nodes:
posebuster-bytedock-cpp-eval-8slg-g5a-gdn68:
children:
- posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-1897784327
displayName: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
finishedAt: null
id: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
name: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
phase: Running
progress: 2/3
startedAt: "2024-07-02T02:54:04Z"
templateName: posebuster-bytedock-cpp-eval-8slg-g5a-steps
templateScope: local/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
type: Steps
posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-891524545:
boundaryID: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
children:
- posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-2269680813
displayName: '[2]'
finishedAt: null
id: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-891524545
name: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68[2]
phase: Running
progress: 0/1
startedAt: "2024-07-02T02:57:01Z"
templateScope: local/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
type: StepGroup
posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-1049446578:
boundaryID: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
children:
- posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-1965041898
displayName: step-receptor-prepare
finishedAt: "2024-07-02T02:54:34Z"
hostNodeName: 10.249.83.145
id: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-1049446578
inputs:
artifacts:
- name: flow_bigpar_serverless_input
path: /tmp/inputs/parameters/serverless_input
s3:
key: bytevs/workflow/upload/fdf03d88-a9ea-4982-8aed-01ca191de2f3/serverless_input
- name: flow_python_packages
path: /tmp/inputs/artifacts/flow_python_packages
s3:
key: bytevs/workflow/upload/72bc1f56-6e6f-406a-92a1-a8068debb522/tmp8bhkqwnn.tgz
parameters:
- description: '{"type": "<class ''str''>"}'
name: receptor_pdb
value: 8SLG_G5A_protein_prepared_noHETATM
- description: '{"type": "<class ''str''>"}'
name: receptor_prepared_dir_tos
value: bytedock/data_redock/raw/posebuster/prepared_receptor/8SLG_G5A
- description: '{"type": "<class ''str''>"}'
name: reference_tos
value: bytedock/data_redock/raw/posebuster/reference_sdf/8SLG_G5A_ligand_prepared.sdf
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
2186, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
- name: flow_key
value: step-receptor-prepare
name: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68[0].step-receptor-prepare
outputs:
artifacts:
- name: main-logs
s3:
key: argo_logs/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-bytedockpreparereceptor-kjzln-oxn6f-1049446578/main.log
exitCode: "0"
parameters:
- description: '{"type": "<class ''str''>"}'
name: receptor_info_tos
value: bytedock/data_redock/raw/posebuster/prepared_receptor/8SLG_G5A/8SLG_G5A_protein_prepared_noHETATM_cpp.yaml
valueFrom:
default: ""
path: /tmp/outputs/parameters/receptor_info_tos
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"cpu_cost": 46, "gpu_cost": 0, "gpu_info": {}, "resource_platform":
"megatron", "class_name": "ByteDockPrepareReceptor", "invt_id": 2186,
"flow_throughput": 1}'
valueFrom:
default: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
-1, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
path: /tmp/outputs/parameters/flow_info
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
valueFrom:
default: ""
path: /tmp/outputs/parameters/flow_lock
- globalName: step-receptor-prepare
name: flow_global
value: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-bytedockpreparereceptor-kjzln-oxn6f-1049446578
phase: Succeeded
progress: 1/1
resourcesDuration:
cpu: 51
memory: 528
startedAt: "2024-07-02T02:54:04Z"
templateName: bytedockpreparereceptor-kjzln-oxn6f
templateScope: local/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
type: Pod
posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-1897784327:
boundaryID: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
children:
- posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-1049446578
displayName: '[0]'
finishedAt: "2024-07-02T02:54:44Z"
id: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-1897784327
name: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68[0]
phase: Succeeded
progress: 2/3
resourcesDuration:
cpu: 304
memory: 3188
startedAt: "2024-07-02T02:54:04Z"
templateScope: local/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
type: StepGroup
posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-1965041898:
boundaryID: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
children:
- posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-2932115425
displayName: '[1]'
finishedAt: "2024-07-02T02:57:01Z"
id: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-1965041898
name: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68[1]
phase: Succeeded
progress: 1/2
resourcesDuration:
cpu: 253
memory: 2660
startedAt: "2024-07-02T02:54:44Z"
templateScope: local/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
type: StepGroup
posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-2269680813:
boundaryID: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
displayName: step-ligand-prepare
finishedAt: null
id: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-2269680813
inputs:
artifacts:
- archive:
none: {}
name: flow_bigpar_serverless_input
path: /tmp/inputs/parameters/serverless_input
s3:
key: argo_logs/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-bytedockvinadocking-zsxd5-myk9u-2932115425/serverless_output
- name: flow_python_packages
path: /tmp/inputs/artifacts/flow_python_packages
s3:
key: bytevs/workflow/upload/72bc1f56-6e6f-406a-92a1-a8068debb522/tmp8bhkqwnn.tgz
parameters:
- description: '{"type": "<class ''bool''>"}'
name: load_results_if_exist
value: "true"
- description: '{"type": "<class ''bool''>"}'
name: write_csv_per_row
value: "false"
- description: '{"type": "<class ''str''>"}'
name: output_dir_tos
value: bytedock_cpp/data_redock/reference_todock_sdf_seed12345_allfix_fp64/posebuster/8SLG_G5A
- description: '{"type": "<class ''dict''>"}'
name: kwargs
value: '{"pose_batch_size": 1, "write_to_sdf": true, "select_best_pose":
true}'
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
2186, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
- name: flow_key
value: step-ligand-prepare
name: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68[2].step-ligand-prepare
phase: Pending
progress: 0/1
startedAt: "2024-07-02T02:57:01Z"
templateName: bytedockligandprepare-86wtx-jrgik
templateScope: local/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
type: Pod
posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-2932115425:
boundaryID: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
children:
- posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-891524545
displayName: step-vina-docking
finishedAt: "2024-07-02T02:56:51Z"
hostNodeName: 10.249.83.197
id: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-2932115425
inputs:
artifacts:
- name: flow_bigpar_serverless_input
path: /tmp/inputs/parameters/serverless_input
s3:
key: bytevs/workflow/upload/5c00e12d-09f4-44b9-819a-1d65c383c381/serverless_input
- name: flow_python_packages
path: /tmp/inputs/artifacts/flow_python_packages
s3:
key: bytevs/workflow/upload/72bc1f56-6e6f-406a-92a1-a8068debb522/tmp8bhkqwnn.tgz
parameters:
- description: '{"type": "<class ''str''>"}'
name: receptor_info_tos
value: bytedock/data_redock/raw/posebuster/prepared_receptor/8SLG_G5A/8SLG_G5A_protein_prepared_noHETATM_cpp.yaml
- description: '{"type": "<class ''bool''>"}'
name: load_results_if_exist
value: "true"
- description: '{"type": "<class ''bool''>"}'
name: run_vina_local
value: "false"
- description: '{"type": "<class ''int''>"}'
name: vina_cpu
value: "8"
- description: '{"type": "<class ''int''>"}'
name: vina_n_poses
value: "50"
- description: '{"type": "<class ''int''>"}'
name: vina_seed
value: "12345"
- description: '{"type": "<class ''int''>"}'
name: vina_exhaustiveness
value: "32"
- description: '{"type": "<class ''int''>"}'
name: vina_search_depth
value: "0"
- description: '{"type": "<class ''float''>"}'
name: vina_energy_range
value: "1000"
- description: '{"type": "<class ''str''>"}'
name: output_dir_tos
value: bytedock_cpp/data_redock/reference_todock_sdf_seed12345_allfix_fp64/posebuster/8SLG_G5A
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
2186, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
- name: flow_key
value: step-vina-docking
name: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68[1].step-vina-docking
outputs:
artifacts:
- archive:
none: {}
name: flow_bigpar_serverless_output
path: /tmp/outputs/parameters/serverless_output
s3:
key: argo_logs/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-bytedockvinadocking-zsxd5-myk9u-2932115425/serverless_output
- name: main-logs
s3:
key: argo_logs/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-bytedockvinadocking-zsxd5-myk9u-2932115425/main.log
exitCode: "0"
parameters:
- description: '{"type": "<class ''dict''>"}'
name: flow_info
value: '{"cpu_cost": 1711, "gpu_cost": 0, "gpu_info": {}, "resource_platform":
"megatron", "class_name": "ByteDockVinaDocking", "invt_id": 2186, "flow_throughput":
1}'
valueFrom:
default: '{"flow_throughput": 0, "cpu_cost": 0, "gpu_cost": 0, "invt_id":
-1, "gpu_info": {}, "resource_platform": "", "class_name": ""}'
path: /tmp/outputs/parameters/flow_info
- description: '{"type": "<class ''str''>"}'
name: flow_lock
value: ""
valueFrom:
default: ""
path: /tmp/outputs/parameters/flow_lock
- globalName: step-vina-docking
name: flow_global
value: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-bytedockvinadocking-zsxd5-myk9u-2932115425
phase: Succeeded
progress: 1/1
resourcesDuration:
cpu: 253
memory: 2660
startedAt: "2024-07-02T02:54:44Z"
templateName: bytedockvinadocking-zsxd5-myk9u
templateScope: local/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
type: Pod
outputs:
parameters:
- name: step-receptor-prepare
value: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-bytedockpreparereceptor-kjzln-oxn6f-1049446578
- name: step-vina-docking
value: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-bytedockvinadocking-zsxd5-myk9u-2932115425
phase: Running
progress: 2/3
resourcesDuration:
cpu: 304
memory: 3188
startedAt: "2024-07-02T02:54:04Z"
|
pod yamlapiVersion: v1
kind: Pod
metadata:
annotations:
kubectl.kubernetes.io/default-container: main
workflows.argoproj.io/node-id: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-2269680813
workflows.argoproj.io/node-name: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68[2].step-ligand-prepare
workflows.argoproj.io/outputs: '{"parameters":[{"name":"flow_info","value":"{\"cpu_cost\":
217, \"gpu_cost\": 0, \"gpu_info\": {}, \"resource_platform\": \"megatron\",
\"class_name\": \"ByteDockLigandPrepare\", \"invt_id\": 2186, \"flow_throughput\":
1}","valueFrom":{"path":"/tmp/outputs/parameters/flow_info","default":"{\"flow_throughput\":
0, \"cpu_cost\": 0, \"gpu_cost\": 0, \"invt_id\": -1, \"gpu_info\": {}, \"resource_platform\":
\"\", \"class_name\": \"\"}"},"description":"{\"type\": \"\u003cclass ''dict''\u003e\"}"},{"name":"flow_lock","value":"","valueFrom":{"path":"/tmp/outputs/parameters/flow_lock","default":""},"description":"{\"type\":
\"\u003cclass ''str''\u003e\"}"},{"name":"flow_global","value":"posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-bytedockligandprepare-86wtx-jrgik-2269680813","globalName":"step-ligand-prepare"}],"artifacts":[{"name":"flow_bigpar_serverless_output","path":"/tmp/outputs/parameters/serverless_output","s3":{"key":"argo_logs/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-bytedockligandprepare-86wtx-jrgik-2269680813/serverless_output"},"archive":{"none":{}}},{"name":"main-logs","s3":{"key":"argo_logs/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68/posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-bytedockligandprepare-86wtx-jrgik-2269680813/main.log"}}]}'
workflows.argoproj.io/progress: 0/1
creationTimestamp: "2024-07-02T02:57:01Z"
labels:
workflows.argoproj.io/completed: "false"
workflows.argoproj.io/workflow: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
name: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-bytedockligandprepare-86wtx-jrgik-2269680813
namespace: bytevs
ownerReferences:
- apiVersion: argoproj.io/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Workflow
name: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
uid: e11ff5c4-d430-449e-87d8-ea40e3139aab
resourceVersion: "1379936854"
uid: fad58c50-bf83-4632-ade6-5204b3068bfc
spec:
containers:
- command:
- argoexec
- wait
- --loglevel
- info
- --log-format
- text
env:
- name: ARGO_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: ARGO_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: GODEBUG
value: x509ignoreCN=0
- name: ARGO_WORKFLOW_NAME
value: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
- name: ARGO_CONTAINER_NAME
value: wait
- name: ARGO_TEMPLATE
value: ''
- name: ARGO_NODE_ID
value: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-2269680813
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: mlp-byted-cn-beijing.cr.volces.com/molecule/argoexec:v3.5.5-ai4s
imagePullPolicy: Always
name: wait
resources:
requests:
cpu: 10m
memory: 64Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /argo/secret/tos-cred
name: tos-cred
readOnly: true
- mountPath: /mainctrfs/tmp/inputs/parameters/serverless_input
name: input-artifacts
subPath: flow_bigpar_serverless_input
- mountPath: /mainctrfs/tmp/inputs/artifacts/flow_python_packages
name: input-artifacts
subPath: flow_python_packages
- mountPath: /mainctrfs/argo/staging
name: argo-staging
- mountPath: /tmp
name: tmp-dir-argo
subPath: "0"
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-r6ndr
readOnly: true
- args:
- /argo/staging/script
command:
- /var/run/argo/argoexec
- emissary
- --loglevel
- info
- --log-format
- text
- --
- python3
env:
- name: ARGO_CONTAINER_NAME
value: main
- name: ARGO_TEMPLATE
value: ''
- name: ARGO_NODE_ID
value: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-2269680813
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: test:6f2c8db77a010c813e0125bf35023d1b
imagePullPolicy: Always
name: main
resources:
limits:
cpu: "96"
memory: 96Gi
requests:
cpu: "1"
memory: 2Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /tmp/inputs/parameters/serverless_input
name: input-artifacts
subPath: flow_bigpar_serverless_input
- mountPath: /tmp/inputs/artifacts/flow_python_packages
name: input-artifacts
subPath: flow_python_packages
- mountPath: /argo/staging
name: argo-staging
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-r6ndr
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
imagePullSecrets:
- name: cr-cr-basic-cn-beijing
- name: cr-mlp-byted-cn-beijing
- name: hub-byted-org
initContainers:
- command:
- argoexec
- init
- --loglevel
- info
- --log-format
- text
env:
- name: ARGO_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: ARGO_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: GODEBUG
value: x509ignoreCN=0
- name: ARGO_WORKFLOW_NAME
value: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68
- name: ARGO_CONTAINER_NAME
value: init
- name: ARGO_TEMPLATE
value: ''
- name: ARGO_NODE_ID
value: posebuster-bytedock-cpp-eval-8slg-g5a-gdn68-2269680813
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: mlp-byted-cn-beijing.cr.volces.com/molecule/argoexec:v3.5.5-ai4s
imagePullPolicy: Always
name: init
resources:
requests:
cpu: 10m
memory: 64Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /argo/secret/tos-cred
name: tos-cred
readOnly: true
- mountPath: /argo/inputs/artifacts
name: input-artifacts
- mountPath: /argo/staging
name: argo-staging
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-r6ndr
readOnly: true
nodeName: 10.249.82.186
nodeSelector:
node-tag: bytevs
preemptionPolicy: PreemptLowerPriority
priority: 400000
priorityClassName: critical
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: node-tag
operator: Equal
value: bytevs
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- emptyDir: {}
name: var-run-argo
- emptyDir: {}
name: tmp-dir-argo
- name: tos-cred
secret:
defaultMode: 420
items:
- key: accesskey
path: accesskey
- key: secretkey
path: secretkey
secretName: tos-cred
- emptyDir: {}
name: input-artifacts
- emptyDir: {}
name: argo-staging
- name: default-token-r6ndr
secret:
defaultMode: 420
secretName: default-token-r6ndr
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-07-02T02:57:03Z"
reason: PodCompleted
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2024-07-02T02:57:29Z"
reason: PodCompleted
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2024-07-02T02:57:29Z"
reason: PodCompleted
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2024-07-02T02:57:01Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://38463f70e13f60b703c3374969c952eb5ea89f09cb7ba51ecd1c3a7afcc35b0f
image: test:6f2c8db77a010c813e0125bf35023d1b
imageID: test@sha256:eedf2748c38bf627ebb7154255baa35f7fffd4221ecee5e1b4cca47b9c36e008
lastState: {}
name: main
ready: false
restartCount: 0
started: false
state:
terminated:
containerID: containerd://38463f70e13f60b703c3374969c952eb5ea89f09cb7ba51ecd1c3a7afcc35b0f
exitCode: 0
finishedAt: "2024-07-02T02:57:29Z"
reason: Completed
startedAt: "2024-07-02T02:57:04Z"
- containerID: containerd://f4a55a4ea01a18d56dcf3dcbae62ffcf2224b792b91497201e0df9d66e4e739b
image: mlp-byted-cn-beijing.cr.volces.com/molecule/argoexec:v3.5.5-ai4s
imageID: mlp-byted-cn-beijing.cr.volces.com/molecule/argoexec@sha256:1d783cadfaf49f348346aa31cb7df0114d56c42206151590bd93135a70a9f34a
lastState: {}
name: wait
ready: false
restartCount: 0
started: false
state:
terminated:
containerID: containerd://f4a55a4ea01a18d56dcf3dcbae62ffcf2224b792b91497201e0df9d66e4e739b
exitCode: 0
finishedAt: "2024-07-02T02:57:29Z"
reason: Completed
startedAt: "2024-07-02T02:57:03Z"
hostIP: 10.249.82.186
initContainerStatuses:
- containerID: containerd://e985f5e4f86fdf3bc3b1b38cac9244ce28982ed63f0e443df908b731a838e52c
image: mlp-byted-cn-beijing.cr.volces.com/molecule/argoexec:v3.5.5-ai4s
imageID: mlp-byted-cn-beijing.cr.volces.com/molecule/argoexec@sha256:1d783cadfaf49f348346aa31cb7df0114d56c42206151590bd93135a70a9f34a
lastState: {}
name: init
ready: true
restartCount: 0
state:
terminated:
containerID: containerd://e985f5e4f86fdf3bc3b1b38cac9244ce28982ed63f0e443df908b731a838e52c
exitCode: 0
finishedAt: "2024-07-02T02:57:02Z"
reason: Completed
startedAt: "2024-07-02T02:57:02Z"
phase: Succeeded
podIP: 192.168.155.104
podIPs:
- ip: 192.168.155.104
qosClass: Burstable
startTime: "2024-07-02T02:57:01Z"
|
|
and until now, the workflow is in running status with the step is pending.
|
This comment was marked as resolved.
This comment was marked as resolved.
github-actions
bot
added
the
problem/stale
agilgur5
changed the title
Running
Ok I've rewritten the title of your issue to reflect this. There are a few other existing issues of this nature as well. I also added syntax highlighting to your I still need to take a look at them, however, they are very large -- a minimal repro would really help. |
agilgur5
removed
problem/stale
agilgur5
changed the title
RunningRunning occasionally (no repro)
|
maybe related to #12103 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Pre-requisites
:latestimage tag (i.e.quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on:latest. If not, I have explained why, in detail, in my description below.What happened/what did you expect to happen?
when I check the workflow, the pod of last step of this workflow has completed, but the wf status is pending.
and the pod status:
and according the pod logs, this pod finished just need 1 min, and there are 1700 running workflow total. And according the logs from controller, it is maybe caused by the max parallelism limit, and the parallelism value is 6000 in configmap.
Version
latest
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
none, cannot reproduce stably.Logs from the workflow controller
Logs from in your workflow's wait container
The text was updated successfully, but these errors were encountered: