Fix shared memory permission issue in a shared pod environment

[XiaohanZhangCMU]

Description of changes:

We run into read permission issues when the shared memory is created in a shared computer environment. We relax the retry condition to allow the creation of a new shared memory file under permission error. When creating a shared memory file, we make sure both LOCALS and FILELOCKS do not overlap with existing ones and they have the same prefix ints in the name.

What makes it more complicated is when a program exited non-normally, some of the shared meomory files are cleaned up but some are not. To address that, we need to make sure all files in SHM_TO_CLEAN do not have duplicates.

Issue #, if available:

Merge Checklist:

Put an x without space in the boxes that apply. If you are unsure about any checklist, please don't hesitate to ask. We are here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

• I have read the contributor guidelines
• This is a documentation change or typo fix. If so, skip the rest of this checklist.
• I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the MosaicML team.
• I have updated any necessary documentation, including README and API docs (if appropriate).

Tests

• I ran pre-commit on my change. (check out the pre-commit section of prerequisites)
• I have added tests that prove my fix is effective or that my feature works (if appropriate).
• I ran the tests locally to make sure it pass. (check out testing)
• I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes.

A complete list of testings done for this PR is put in this doc.

[snarayan21]

this seems fine to me, but are you able to reproduce this with an example where this used to fail, and then installing streaming off this branch now succeeds?

[snarayan21]

closing after offline discussion with @XiaohanZhangCMU

[snarayan21]

seems fine to me...i think. would like to see testing

[XiaohanZhangCMU]

@karan6181 @snarayan21 @bigning testings are done. PR is ready for review then merge. PTAL.

[snarayan21]

couple questions, but seems fine to me overall. Deferring to @karan6181 and @bigning on approval

[bigning]

thank you for the fix! Will let @karan6181 stamp!

[karan6181]

LGTM. Thank You!