Rebuild cuda118, cuda1120, and protobuf #195

RaulPPelaez · 2023-10-25T16:15:22Z

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

Continuing #191 I added the necessary CUDA arch flags to build_pytorch.sh.

…nda-forge-pinning 2023.10.11.05.05.48

The transition to CUDA 12 SDK includes new packages for all CUDA libraries and build tools. Notably, the cudatoolkit package no longer exists, and packages should depend directly on the specific CUDA libraries (libcublas, libcusolver, etc) as needed. For an in-depth overview of the changes and to report problems [see this issue]( conda-forge/conda-forge.github.io#1963 ). Please feel free to raise any issues encountered there. Thank you! 🙏

…nda-forge-pinning 2023.10.11.19.37.54

…nda-forge-pinning 2023.10.23.08.51.53

conda-forge-webservices · 2023-10-25T16:15:32Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

RaulPPelaez · 2023-10-25T16:16:10Z

@conda-forge-admin, please rerender

…nda-forge-pinning 2023.10.23.08.51.53

RaulPPelaez · 2023-10-26T06:34:40Z

These ones were running, but they were timed out:

  [5220/5707] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/ForeachUnaryOp.cu.o
  [5221/5707] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/IGammaKernel.cu.o
  [5222/5707] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/Im2Col.cu.o
  [5223/5707] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/GridSampler.cu.o
##[error]The operation was canceled.
Finishing: Run docker build

They just take really really long it seems. Not sure what can be done in order to reduce this time.
Any advice here @jakirkham?

Co-authored-by: jakirkham <[email protected]>

hmaarrfk · 2023-10-29T14:20:55Z

I've always been building cuda locally. the fat binaries are just really slow to compile.

hmaarrfk · 2023-10-29T14:23:47Z

I can try to start building this locally and see where it goes. I am not overly enthusiastic to build out 3 cuda versions (11.2, 11.8, and 12.0).....

RaulPPelaez · 2023-10-30T14:07:49Z

I've always been building cuda locally. the fat binaries are just really slow to compile.

I did not know that was an option hehe. I know how to build locally, but I assume the results are not something I can somehow upload here myself?

I can try to start building this locally and see where it goes. I am not overly enthusiastic to build out 3 cuda versions (11.2, 11.8, and 12.0).....

I understand it is a hassle, but torch is a really big package and I believe it is worth to provide these options. It is true though that minor CUDA versions are supposed to be compatible amongst them, but even in that case I would assume CUDA 11.8 provides some kind of performance enhancement with respect 11.2.
At least CUDA 12 should be an option to reduce the chances of dependency hell.

hmaarrfk · 2023-10-30T14:23:06Z

See https://github.com/conda-forge/cfep/blob/main/cfep-03.md

is 11.8 useful if we provide 12.0?

It just really is ALOT of package data (storage) and compilation time.

hmaarrfk · 2023-10-30T20:22:37Z

@RaulPPelaez were you able to get Cuda 12.0 to also compile? if so do you want to add it to this merge request and I can trigger a rebuild.

jakirkham · 2023-10-30T20:41:49Z

the biggest barrier is me starting my script.

Yeah we need to figure out a way to automate builds

…58afc5

RaulPPelaez · 2023-10-31T06:35:39Z

@RaulPPelaez were you able to get Cuda 12.0 to also compile? if so do you want to add it to this merge request and I can trigger a rebuild.

Those also timed out, the PR for CUDA 12 #193 should also be compatible with CUDA 11.8.
Not sure how the pipeline works in this case. I merged #193 here, not sure if that is enough.

RaulPPelaez · 2023-10-31T06:35:49Z

@conda-forge-admin, please rerender

…nda-forge-pinning 2023.10.30.17.31.54

jakirkham · 2023-11-01T19:58:08Z

Think what Mark is asking is if you have tried building locally (and if so whether those builds completed)

Guessing most build errors occur early (like those seen previously: #193 (comment) )

Though there are some errors that might not show up until the linking stage

Another question would be whether we see issues during testing (not sure what testing is done for the CUDA packages)

hmaarrfk · 2023-11-01T23:09:02Z

I have the following done canaries completed.

conda-forge-build-done-linux_64_blas_implgenericc_compiler_version11cuda_compilernvcccuda_compiler_version11.8cxx_compiler_version11numpy1.22python3.10.____cpython
conda-forge-build-done-linux_64_blas_implgenericc_compiler_version11cuda_compilernvcccuda_compiler_version11.8cxx_compiler_version11numpy1.22python3.8.____cpython
conda-forge-build-done-linux_64_blas_implgenericc_compiler_version11cuda_compilernvcccuda_compiler_version11.8cxx_compiler_version11numpy1.22python3.9.____cpython
conda-forge-build-done-linux_64_blas_implgenericc_compiler_version11cuda_compilernvcccuda_compiler_version11.8cxx_compiler_version11numpy1.23python3.11.____cpython
conda-forge-build-done-linux_64_blas_implmklc_compiler_version10cuda_compilernvcccuda_compiler_version11.2cxx_compiler_version10numpy1.22python3.10.____cpython
conda-forge-build-done-linux_64_blas_implmklc_compiler_version10cuda_compilernvcccuda_compiler_version11.2cxx_compiler_version10numpy1.23python3.11.____cpython
conda-forge-build-done-linux_64_blas_implmklc_compiler_version12cuda_compilercuda-nvcccuda_compiler_version12.0cxx_compiler_version12numpy1.22python3.8.____cpython

That said, i'm wondering if for the sake of conda-forge's storage quote, if i shoul dcancel (now that I have one build of 11.2, 11.8 and 12.0 complete) cancel these builds and work to integrate #197

jakirkham · 2023-11-02T02:57:35Z

@h-vetinari do you have a sense of how #197 should factor in here?

h-vetinari · 2023-11-02T03:13:47Z

@h-vetinari do you have a sense of how #197 should factor in here?

If it can be integrated, all the better! If not, it's not the end of the world, but should then go into the next build. With pytorch, the recent protobuf bumps seemed to go pretty smoothly though (from what I can tell, esp. compared to TF).

hmaarrfk · 2023-11-03T01:48:18Z

they work. builds take a full day. restarting them now with protobuf migration

…nda-forge-pinning 2023.11.02.10.11.59

hmaarrfk · 2023-11-04T16:36:25Z

log files

log_files.zip

Tobias-Fischer · 2023-11-06T10:14:23Z

Hi @hmaarrfk - have these builds actually been uploaded to the conda-forge channel? I might be blind seeing only the cpu builds.

Tobias-Fischer · 2023-11-06T10:15:31Z

Background: I’m trying to figure out why conda-forge/pytorch_scatter-feedstock#55 can’t find solve the environment

hmaarrfk · 2023-11-06T12:11:37Z

lol, maybe not.... I might just upload the next ones if that is ok?

jakirkham · 2023-11-06T20:22:50Z

Meaning with the changes from PR ( #199 )?

hmaarrfk · 2023-11-06T21:34:04Z

yes.

I mean, i'm really not excited about them either, but they are complete....

jakirkham · 2023-11-06T21:37:39Z

Yeah that seems better from a user standpoint (recent version + Python 3.12)

hmaarrfk · 2023-11-06T22:46:43Z

Ohhhh. i found the builds:

https://anaconda.org/mark.harfouche/pytorch/files
https://anaconda.org/mark.harfouche/pytorch-gpu/files

Just copied them over.

jakirkham · 2023-11-06T23:37:58Z

Thanks Mark! 🙏

regro-cf-autotick-bot and others added 10 commits October 11, 2023 17:17

Rebuild for CUDA 11.8 w/arch support

3dd3a27

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.27.1, and co…

11d5040

…nda-forge-pinning 2023.10.11.05.05.48

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.27.1, and co…

0b0ebe1

…nda-forge-pinning 2023.10.11.19.37.54

Add archs for CUDA 12

c8a0f99

Add archs for CUDA 11.8

096a663

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.27.1, and co…

a46b0ca

…nda-forge-pinning 2023.10.23.08.51.53

Install cuda-libraries-dev if cuda version is 12

5318120

Fix CUDA 12 detection

70e2f97

Add archs for CUDA 11.8 and CUDA 12.0 to build_pytorch.sh

2b3578f

RaulPPelaez requested review from Tobias-Fischer, beckermr, benjaminrwilson, hmaarrfk and sodre as code owners October 25, 2023 16:15

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.27.1, and co…

f261952

…nda-forge-pinning 2023.10.23.08.51.53

RaulPPelaez and others added 7 commits October 26, 2023 08:45

Add CUDA libraries

0c908ec

Add CUDA libraries to host too

038a2b8

Lint

337c6b8

Remove unused USE_NINJA

dd9a64a

Guard cuda_version

1e6e208

Add cuda-nvml

c67dd74

Update recipe/meta.yaml

06c2d68

Co-authored-by: jakirkham <[email protected]>

Merge branch 'rebuild-cuda120-0-2_hdbc077' into rebuild-cuda118-0-1_h…

1f2a95f

…58afc5

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.27.1, and co…

9ba39b2

…nda-forge-pinning 2023.10.30.17.31.54

Add libprotobuf migration

3541ffc

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.27.1, and co…

cd4bbb4

…nda-forge-pinning 2023.11.02.10.11.59

hmaarrfk changed the title ~~Rebuild cuda118 0 1 h58afc5~~ Rebuild cuda118, cuda1120, and protobuf Nov 3, 2023

hmaarrfk merged commit 8659210 into conda-forge:main Nov 4, 2023
36 of 49 checks passed

This was referenced Nov 6, 2023

PyTorch: Support CUDA 12 #163

Closed

[WIP] Cuda12 rebuild again #182

Closed

up to 2.0.1 #172

Closed

Build for new archs conda-forge/libmagma-feedstock#8

Merged

Rebuild for CUDA 11.8 w/arch support conda-forge/hdk-feedstock#50

Draft

jakirkham mentioned this pull request Nov 28, 2023

Rebuild for CUDA 12 w/arch + Windows support #205

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rebuild cuda118, cuda1120, and protobuf #195

Rebuild cuda118, cuda1120, and protobuf #195

RaulPPelaez commented Oct 25, 2023 •

edited

Loading

conda-forge-webservices bot commented Oct 25, 2023

RaulPPelaez commented Oct 25, 2023

RaulPPelaez commented Oct 26, 2023

hmaarrfk commented Oct 29, 2023

hmaarrfk commented Oct 29, 2023

RaulPPelaez commented Oct 30, 2023

hmaarrfk commented Oct 30, 2023

hmaarrfk commented Oct 30, 2023

jakirkham commented Oct 30, 2023

RaulPPelaez commented Oct 31, 2023

RaulPPelaez commented Oct 31, 2023

jakirkham commented Nov 1, 2023

hmaarrfk commented Nov 1, 2023

jakirkham commented Nov 2, 2023

h-vetinari commented Nov 2, 2023

hmaarrfk commented Nov 3, 2023 •

edited

Loading

hmaarrfk commented Nov 4, 2023

Tobias-Fischer commented Nov 6, 2023

Tobias-Fischer commented Nov 6, 2023

hmaarrfk commented Nov 6, 2023

jakirkham commented Nov 6, 2023

hmaarrfk commented Nov 6, 2023

jakirkham commented Nov 6, 2023

hmaarrfk commented Nov 6, 2023 •

edited

Loading

jakirkham commented Nov 6, 2023

Rebuild cuda118, cuda1120, and protobuf #195

Rebuild cuda118, cuda1120, and protobuf #195

Conversation

RaulPPelaez commented Oct 25, 2023 • edited Loading

conda-forge-webservices bot commented Oct 25, 2023

RaulPPelaez commented Oct 25, 2023

RaulPPelaez commented Oct 26, 2023

hmaarrfk commented Oct 29, 2023

hmaarrfk commented Oct 29, 2023

RaulPPelaez commented Oct 30, 2023

hmaarrfk commented Oct 30, 2023

hmaarrfk commented Oct 30, 2023

jakirkham commented Oct 30, 2023

RaulPPelaez commented Oct 31, 2023

RaulPPelaez commented Oct 31, 2023

jakirkham commented Nov 1, 2023

hmaarrfk commented Nov 1, 2023

jakirkham commented Nov 2, 2023

h-vetinari commented Nov 2, 2023

hmaarrfk commented Nov 3, 2023 • edited Loading

hmaarrfk commented Nov 4, 2023

Tobias-Fischer commented Nov 6, 2023

Tobias-Fischer commented Nov 6, 2023

hmaarrfk commented Nov 6, 2023

jakirkham commented Nov 6, 2023

hmaarrfk commented Nov 6, 2023

jakirkham commented Nov 6, 2023

hmaarrfk commented Nov 6, 2023 • edited Loading

jakirkham commented Nov 6, 2023

RaulPPelaez commented Oct 25, 2023 •

edited

Loading

hmaarrfk commented Nov 3, 2023 •

edited

Loading

hmaarrfk commented Nov 6, 2023 •

edited

Loading