MPI arguments for MPI jobs (Fluent)

[ZLPG23]

If you find a similar existing issue, please comment on that issue instead of creating a new one.

If you are submitting a feature request, please start a discussion instead of creating an issue.

Describe the bug

Whe are currently using Slurm on GCP run Ansys Fluent, we tried to used the two parameters to optimize MPI job: "I_MPI_FABRICS =shm:ofa" & "I_MPI_FALLBACK_DEVICE=disable "

The command we ran is like below:

fluent -cnf=host.txt 3ddp -t448 -g -nm -hidden -affinity=off -mpiopt='-genv I_MPI_PIN=1 -genv FI_VERBS_IFACE=eth0' -i batch.jou > logfile.out

The purpose we set up the parameter "-mpipot" was mainly for avoiding scheduler to affect the allocation of CPU, and also to use eth0 as the interface of exchanging data between MPI jobs. However, we saw the followoing error and also the job was stuck there after we ran the job:
"mpi startup() ofa fabric is not available and fallback fabric is not enabled"

However, if we remove the two parameters mentioned at the beginning, then the command works well. Based on this test result, it seems the issue was related to the inexistance of Infiniband device, however, based on our standing even though there is no Infiniband device, it should still work withous any issue, is this understanding correct?

Accordingly, we would like to seek professional help from GCP experts. Would you please kindly help clarify this, and more important is, what is the best configuration for using Slurm on GCP to run Ansys Fluent?

Steps to reproduce

Steps to reproduce the behavior:

Expected behavior

A clear and concise description of what you expected to happen.

Actual behavior

What happened instead.

Version (ghpc --version)

Blueprint

If applicable, attach or paste the blueprint YAML used to produce the bug.

Expanded Blueprint

If applicable, please attach or paste the expanded blueprint. The expanded blueprint can be obtained by running ghpc expand your-blueprint.yaml.

Disregard if the bug occurs when running ghpc expand ... as well.

Output and logs

Screenshots

If applicable, add screenshots to help explain your problem.

Execution environment

• OS: [macOS, ubuntu, ...]
• Shell (To find this, run ps -p $$): [bash, zsh, ...]
• go version:

Additional context

Add any other context about the problem here.

[nick-stroud]

From inspection, this does not appear related to any bug in the HPC Toolkit so I will convert it to a discussion.

On your question about mpiopt flags. I would recommend to try running fluent without any mpiopt flags and adding the -peth fluent flag. Please report if this works for you. If you still feel you have need for the mpiopt flags then please describe the requirements that require these flags.

On your specific question:

based on our understanding even though there is no Infiniband device, it should still work without any issue, is this understanding correct? (Referring to the use of I_MPI_FABRICS =shm:ofa" & "I_MPI_FALLBACK_DEVICE=disable).

I do not have personal knowledge to comment on you understanding of these flags, but as mentioned I do not believe they should be needed.

On your question:

What is the best configuration for using Slurm on GCP to run Ansys Fluent?

It is hard to make a definitive statement about what will work "best" as there are often trade offs based on price, performance, and the specific workload. Some general recommendations on a good configuration that has been shown to work well is:

• Use Intel MPI
• Use c2d-standard-112 compute nodes with SMT disabled (default on Slurm v5)
• Use a BASIC_SSD Filestore as a shared filesystem from which to run the fluent job

# Example filestore (could be /home instead of /shared)
  - id: sharedfs
    source: modules/file-system/filestore
    use: [network1]
    settings:
      local_mount: /shared
      filestore_tier: BASIC_SSD
      size_gb: 2560  # 2.5 * 1024

[cboneti]

Have you seen our documentation on Fluent?