Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(feat) Add TransformerEngine build. #76

Merged
merged 55 commits into from
Sep 29, 2024
Merged

(feat) Add TransformerEngine build. #76

merged 55 commits into from
Sep 29, 2024

Conversation

wbrown
Copy link
Collaborator

@wbrown wbrown commented Aug 21, 2024

Adds TransformerEngine to torch-extras.

@wbrown
(feat) Add TransformerEngine build.
5f6ed4c
@wbrown
Fix typo in URI
431fcf1
@wbrown
Compensate for weirs release tagging.
c40675a
@wbrown
Change how we get the branch. Correct version tagging.
bdbf84d
@wbrown
Apparently they chagned the release version tagging scheme.
bd0bb04
@wbrown
Their release tagging uses x.x.x, but the brqnches use x.x.
ff2c28e
@wbrown
Do we need the double build?
1045646
@wbrown
Add WORKDIR back for TransformerEngine build.
42a67b8
@wbrown
We need cudnn9.
2fa7f5d
@wbrown
Forgot the fi
125f56f
@wbrown
Try adding CUDNN path.
f137049
@wbrown
We may need to install CUDNN.
55c9e59
@wbrown
Try again with package name.
9592467
@wbrown
Names are hard.
71e6b2c
@wbrown
Add nsight-systems-cli
f5d0bf1
@wbrown
Remove the --depth argument to the git submodule for `TransformerEn…
fd534ed 
…gine`
@wbrown
Remove --depth in another place.
e0e3160
@wbrown
Add CUDA_* env vars.
d3829e6
@wbrown
Add cuda-nvtx package.
a38df66
@wbrown
Add cuda-nvrtc.
1ad2d97
@wbrown
nvrtc-dev
a13c609
@wbrown
Try TE 1.8
a155b06
@wbrown
Python <3.9 doesn't have cache.
7b5bc5c
@wbrown
Move the sed
07433a8
@wbrown
Disable CUDA 11.8
05610f3
@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/10500333818
Image: ghcr.io/coreweave/ml-containers/torch:wbrown.add-transformerengine-05610f3-base-cuda12.0.1-ubuntu20.04-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/10500333818
Image: ghcr.io/coreweave/ml-containers/torch:wbrown.add-transformerengine-05610f3-base-cuda12.2.2-ubuntu20.04-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/10500333818
Image: ghcr.io/coreweave/ml-containers/torch:wbrown.add-transformerengine-05610f3-base-cuda12.2.2-ubuntu22.04-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/10500333818
Image: ghcr.io/coreweave/ml-containers/torch:wbrown.add-transformerengine-05610f3-base-cuda12.3.2-ubuntu20.04-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/10500333818
Image: ghcr.io/coreweave/ml-containers/torch:wbrown.add-transformerengine-05610f3-base-cuda12.0.1-ubuntu22.04-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976733
Image: ghcr.io/coreweave/ml-containers/torch:wbrown.add-transformerengine-9184858-nccl-cuda12.3.2-ubuntu22.04-nccl2.20.3-1-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976733
Image: ghcr.io/coreweave/ml-containers/torch:wbrown.add-transformerengine-9184858-nccl-cuda12.4.1-ubuntu20.04-nccl2.21.5-1-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976733
Image: ghcr.io/coreweave/ml-containers/torch:wbrown.add-transformerengine-9184858-nccl-cuda12.4.1-ubuntu22.04-nccl2.21.5-1-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch:wbrown.add-transformerengine-9184858-nccl-24092808-cuda12.2.2-ubuntu20.04-nccl2.21.5-1-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch:wbrown.add-transformerengine-9184858-nccl-24092808-cuda12.2.2-ubuntu22.04-nccl2.19.3-1-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch:wbrown.add-transformerengine-9184858-nccl-24092808-cuda12.3.2-ubuntu20.04-nccl2.20.3-1-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch:wbrown.add-transformerengine-9184858-nccl-24092808-cuda12.4.1-ubuntu20.04-nccl2.21.5-1-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch:wbrown.add-transformerengine-9184858-nccl-24092808-cuda12.3.2-ubuntu22.04-nccl2.20.3-1-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch:wbrown.add-transformerengine-9184858-nccl-24092808-cuda12.4.1-ubuntu22.04-nccl2.21.5-1-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch:wbrown.add-transformerengine-9184858-base-24092808-cuda12.3.2-ubuntu20.04-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch:wbrown.add-transformerengine-9184858-base-24092808-cuda12.2.2-ubuntu20.04-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch:wbrown.add-transformerengine-9184858-base-24092808-cuda12.2.2-ubuntu22.04-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch:wbrown.add-transformerengine-9184858-base-24092808-cuda12.4.1-ubuntu20.04-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch:wbrown.add-transformerengine-9184858-base-24092808-cuda12.4.1-ubuntu22.04-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch:wbrown.add-transformerengine-9184858-base-24092808-cuda12.3.2-ubuntu22.04-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976733
Image: ghcr.io/coreweave/ml-containers/torch-extras:wbrown.add-transformerengine-9184858-nccl-cuda12.3.2-ubuntu20.04-nccl2.20.3-1-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976733
Image: ghcr.io/coreweave/ml-containers/torch-extras:wbrown.add-transformerengine-9184858-nccl-cuda12.2.2-ubuntu20.04-nccl2.21.5-1-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976733
Image: ghcr.io/coreweave/ml-containers/torch-extras:wbrown.add-transformerengine-9184858-nccl-cuda12.2.2-ubuntu22.04-nccl2.19.3-1-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976733
Image: ghcr.io/coreweave/ml-containers/torch-extras:wbrown.add-transformerengine-9184858-nccl-cuda12.3.2-ubuntu22.04-nccl2.20.3-1-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976733
Image: ghcr.io/coreweave/ml-containers/torch-extras:wbrown.add-transformerengine-9184858-nccl-cuda12.4.1-ubuntu20.04-nccl2.21.5-1-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976733
Image: ghcr.io/coreweave/ml-containers/torch-extras:wbrown.add-transformerengine-9184858-nccl-cuda12.4.1-ubuntu22.04-nccl2.21.5-1-torch2.4.0-vision0.19.0-audio2.4.0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch-extras:wbrown.add-transformerengine-9184858-nccl-24092808-cuda12.2.2-ubuntu20.04-nccl2.21.5-1-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch-extras:wbrown.add-transformerengine-9184858-nccl-24092808-cuda12.2.2-ubuntu22.04-nccl2.19.3-1-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch-extras:wbrown.add-transformerengine-9184858-nccl-24092808-cuda12.3.2-ubuntu20.04-nccl2.20.3-1-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch-extras:wbrown.add-transformerengine-9184858-nccl-24092808-cuda12.4.1-ubuntu20.04-nccl2.21.5-1-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch-extras:wbrown.add-transformerengine-9184858-nccl-24092808-cuda12.3.2-ubuntu22.04-nccl2.20.3-1-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@github-actions GitHub Actions
Copy link

@wbrown Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11078976729
Image: ghcr.io/coreweave/ml-containers/nightly-torch-extras:wbrown.add-transformerengine-9184858-nccl-24092808-cuda12.4.1-ubuntu22.04-nccl2.21.5-1-torch2.6.0a0-vision0.20.0a0-audio2.5.0a0

@wbrown
Copy link
Collaborator Author

wbrown commented Sep 29, 2024

@Eta0 nightly seems broken but shouldn’t block this.

@wbrown wbrown merged commit bfe03aa into main Sep 29, 2024
92 of 104 checks passed
@wbrown wbrown deleted the wbrown.add-transformerengine branch September 29, 2024 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@wbrown @Eta0