PR #15577: [PJRT:GPU] Add setting for mocked number of hosts per slice #17711
+181
−40
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![copybara-service[bot]](https://avatars.githubusercontent.com/in/44061?s=60&v=4){kind=small}
copybara-service
bot
force-pushed
the
test_679575608
branch
4 times, most recently
from
aaace2e
to
6d1909f
Compare
Imported from GitHub PR #15577 With the existing `enable_mock_nccl` setting it is impossible to warm up compilation cache when there are multiple processes per node. This is because the cache key includes topology and GPU topology contains information about number of slices and number of hosts per slice. The current mocking of topologies always sets num_hosts_per_slice to 1. However, if you have multiple GPUs on a node and run a process-per-GPU then num_hosts_per_slice must be set to the number of GPUs. This patch allows setting num_hosts_per_slice explicitly when creating the GPU client. Copybara import of the project: -- b88208e by Jaroslav Sevcik <[email protected]>: Add setting for number of hosts per slice -- 0e3200a by Jaroslav Sevcik <[email protected]>: Specify topology as "slice x hosts_per_slice" -- 237308d by Jaroslav Sevcik <[email protected]>: Change topology description to include #devices-per-host -- 813c234 by Jaroslav Sevcik <[email protected]>: Add default value to BuildDistributedDevices new parameter Merging this change closes #15577 FUTURE_COPYBARA_INTEGRATE_REVIEW=#15577 from jaro-sevcik:mock-num-hosts-per-slice 813c234 PiperOrigin-RevId: 679575608
copybara-service
bot
force-pushed
the
test_679575608
branch
from
6d1909f
to
6110410
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR #15577: [PJRT:GPU] Add setting for mocked number of hosts per slice
Imported from GitHub PR #15577
With the existing
enable_mock_ncclsetting it is impossible to warm up compilation cache when there are multiple processes per node. This is because the cache key includes topology and GPU topology contains information about number of slices and number of hosts per slice. The current mocking of topologies always sets num_hosts_per_slice to 1. However, if you have multiple GPUs on a node and run a process-per-GPU then num_hosts_per_slice must be set to the number of GPUs.This patch allows setting num_hosts_per_slice explicitly when creating the GPU client.
Copybara import of the project:
--
b88208e by Jaroslav Sevcik [email protected]:
Add setting for number of hosts per slice
--
0e3200a by Jaroslav Sevcik [email protected]:
Specify topology as "slice x hosts_per_slice"
--
237308d by Jaroslav Sevcik [email protected]:
Change topology description to include #devices-per-host
--
813c234 by Jaroslav Sevcik [email protected]:
Add default value to BuildDistributedDevices new parameter
Merging this change closes #15577
FUTURE_COPYBARA_INTEGRATE_REVIEW=#15577 from jaro-sevcik:mock-num-hosts-per-slice 813c234