Legion: collective instance freeze on slingshot-11

[syamajala]

I believe using collective instances results in a startup freeze on slingshot-11. I have one commit of S3D that uses them (https://gitlab.com/legion_s3d/legion_s3d/-/commit/e797d71367683580933166a0080a3dbf3f98b978) and freezes at startup and another commit (https://gitlab.com/legion_s3d/legion_s3d/-/commit/5455c8c03e67c32f2fcbee1120d7a40c37486823) where I specifically backed out those changes and it no longer freezes. We will probably need to investigate this with HPE.

[elliottslaughter]

1. What scale is required to hit this freeze? (Nodes, ranks, GPUs/rank?)
2. Can you confirm that the collective instance freeze definitely does not occur on at least one other network (e.g., Infiniband)? Ideally at a similar scale to above. This would help rule out a potential Legion issue.

[syamajala]

The subranks branch starts to freeze at 8 nodes. The tdb branch starts to freeze at 4 nodes, 4 ranks/node, 2 gpus/rank. I can try tdb on sapling. It was working on blaze the last time I tried, but blaze is currently down.

[syamajala]

Actually looks like blaze is back up. Will try it there.

[syamajala]

The network drive on blaze is still down so I couldnt run there, but I built and ran tdb on sapling. Ran on all 4 nodes, 4 ranks/node, 1 gpu/rank. It started up and ran fine. This is probably a slingshot issue.

[elliottslaughter]

Sapling has 4 GPUs per node. Could we run 16 ranks, 4 ranks/node, 1 GPU/rank?

[syamajala]

Sorry I meant 4 ranks/node. Not 1 rank/node.

[lightsighter]

I'm going to assume this is a Slingshot issue unless we can reproduce it on an Infiniband machine.