Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compilation failed with torch=1.12.1 #3

Open
tjdgh0715 opened this issue Dec 21, 2023 · 0 comments
Open

Compilation failed with torch=1.12.1 #3

tjdgh0715 opened this issue Dec 21, 2023 · 0 comments

Comments

@tjdgh0715
Copy link

tjdgh0715 commented Dec 21, 2023 •
edited
Loading

Hello! Thank you for the great work.

I am currently trying to use the Mesa with PyTorch 1.12.1 version, but got errors during the system compilation.

I also checked that the internal operations have changed at PyTorch 1.10 as mentioned at #1 .

I think there are major changes at PyTorch 1.12, too.
I used this command to construct the environment.

conda create -n env python=3.8.8
conda activate env
conda install -c anaconda cudatoolkit==11.3.1
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch

And I got this error while compilation.

running install
/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  self.initialize_options()
running bdist_egg
running egg_info
writing Mesa.egg-info/PKG-INFO
writing dependency_links to Mesa.egg-info/dependency_links.txt
writing top-level names to Mesa.egg-info/top_level.txt
reading manifest file 'Mesa.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'Mesa.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying mesa/custom_quant.py -> build/lib.linux-x86_64-cpython-38/mesa
copying mesa/custom_layer_norm.py -> build/lib.linux-x86_64-cpython-38/mesa
copying mesa/custom_relu.py -> build/lib.linux-x86_64-cpython-38/mesa
copying mesa/custom_softmax.py -> build/lib.linux-x86_64-cpython-38/mesa
copying mesa/custom_matmul.py -> build/lib.linux-x86_64-cpython-38/mesa
copying mesa/custom_conv.py -> build/lib.linux-x86_64-cpython-38/mesa
copying mesa/packbit.py -> build/lib.linux-x86_64-cpython-38/mesa
copying mesa/__init__.py -> build/lib.linux-x86_64-cpython-38/mesa
copying mesa/custom_bn.py -> build/lib.linux-x86_64-cpython-38/mesa
copying mesa/policy.py -> build/lib.linux-x86_64-cpython-38/mesa
copying mesa/custom_fc.py -> build/lib.linux-x86_64-cpython-38/mesa
copying mesa/custom_gelu.py -> build/lib.linux-x86_64-cpython-38/mesa
running build_ext
building 'mesa.native' extension
Emitting ninja build file /home/sungho/BackRazor_Neurips22/Mesa/build/temp.linux-x86_64-cpython-38/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/1] c++ -MMD -MF /home/sungho/BackRazor_Neurips22/Mesa/build/temp.linux-x86_64-cpython-38/native.o.d -pthread -B /home/sungho/anaconda3/envs/test/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/include -I/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/include/TH -I/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/include/THC -I/home/sungho/anaconda3/envs/test/include/python3.8 -c -c /home/sungho/BackRazor_Neurips22/Mesa/native.cpp -o /home/sungho/BackRazor_Neurips22/Mesa/build/temp.linux-x86_64-cpython-38/native.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=native -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /home/sungho/BackRazor_Neurips22/Mesa/build/temp.linux-x86_64-cpython-38/native.o 
c++ -MMD -MF /home/sungho/BackRazor_Neurips22/Mesa/build/temp.linux-x86_64-cpython-38/native.o.d -pthread -B /home/sungho/anaconda3/envs/test/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/include -I/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/include/TH -I/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/include/THC -I/home/sungho/anaconda3/envs/test/include/python3.8 -c -c /home/sungho/BackRazor_Neurips22/Mesa/native.cpp -o /home/sungho/BackRazor_Neurips22/Mesa/build/temp.linux-x86_64-cpython-38/native.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=native -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/sungho/BackRazor_Neurips22/Mesa/native.cpp: In function ‘std::tuple<at::Tensor, at::Tensor> conv2d_backward(const at::Tensor&, const at::Tensor&, const at::Tensor&, c10::ArrayRef<long int>, c10::ArrayRef<long int>, c10::ArrayRef<long int>, int64_t, bool, bool, bool, std::array<bool, 2>)’:
/home/sungho/BackRazor_Neurips22/Mesa/native.cpp:26:16: error: ‘cudnn_convolution_backward’ is not a member of ‘at’
     return at::cudnn_convolution_backward(
                ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/sungho/BackRazor_Neurips22/Mesa/native.cpp:26:16: note: suggested alternative: ‘mps_convolution_backward’
     return at::cudnn_convolution_backward(
                ^~~~~~~~~~~~~~~~~~~~~~~~~~
                mps_convolution_backward
/home/sungho/BackRazor_Neurips22/Mesa/native.cpp: In function ‘at::Tensor softmax_backward_cpu(const at::Tensor&, const at::Tensor&, int64_t, const at::Tensor&)’:
/home/sungho/BackRazor_Neurips22/Mesa/native.cpp:200:69: error: cannot convert ‘const at::Tensor’ to ‘c10::ScalarType’ for argument ‘4’ to ‘at::Tensor at::_softmax_backward_data(const at::Tensor&, const at::Tensor&, int64_t, c10::ScalarType)’
     return at::_softmax_backward_data(grad_output, output, dim, self);
                                                                     ^
/home/sungho/BackRazor_Neurips22/Mesa/native.cpp: In function ‘at::Tensor softmax_backward_cuda(const at::Tensor&, const at::Tensor&, int64_t, const at::Tensor&)’:
/home/sungho/BackRazor_Neurips22/Mesa/native.cpp:212:69: error: cannot convert ‘const at::Tensor’ to ‘c10::ScalarType’ for argument ‘4’ to ‘at::Tensor at::_softmax_backward_data(const at::Tensor&, const at::Tensor&, int64_t, c10::ScalarType)’
     return at::_softmax_backward_data(grad_output, output, dim, self);
                                                                     ^
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1808, in _run_ninja_build
    subprocess.run(
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "setup.py", line 15, in <module>
    setup(
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/command/install.py", line 80, in run
    self.do_egg_install()
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/command/install.py", line 129, in do_egg_install
    self.run_command('bdist_egg')
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 164, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 150, in call_command
    self.run_command(cmdname)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/command/install_lib.py", line 111, in build
    self.run_command('build_ext')
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 84, in run
    _build_ext.run(self)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
    self.build_extensions()
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 765, in build_extensions
    build_ext.build_extensions(self)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
    self._build_extensions_serial()
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
    _build_ext.build_extension(self, ext)
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 548, in build_extension
    objects = self.compiler.compile(
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 586, in unix_wrap_ninja_compile
    _write_ninja_file_and_compile_objects(
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1487, in _write_ninja_file_and_compile_objects
    _run_ninja_build(
  File "/home/sungho/anaconda3/envs/test/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1824, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension

Can I get some help to solve this compilation error?
@tjdgh0715 tjdgh0715 changed the title Compilation filed with torch=1.12.1 Compilation failed with torch=1.12.1 Jan 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tjdgh0715