"Could NOT find CUDA" error appeared when building from source

[yoyolicoris]

🐛 Bug

I'm trying to do development installation following the instructions in CONTRIBUTING.md, but the building process cannot find CUDA and failed.

It seems like anaconda has its own way to handle cuda installation, and doesn't put the installation under /usr/local/cuda or something similar.

To Reproduce

Steps to reproduce the behavior:

conda install pytorch cudatoolkit=11.0 -c pytorch-nightly
pip install cmake
git clone https://github.com/pytorch/audio.git
cd audio
git submodule update --init --recursive
conda install -c conda-forge sox
python setup.py develop

Output:

(torchaudio_dev) ycy@ycy-playstation:~/working/audio$ python setup.py develop
-- Building version 0.8.0a0+fa71c5e
running develop
running egg_info
writing torchaudio.egg-info/PKG-INFO
writing dependency_links to torchaudio.egg-info/dependency_links.txt
writing requirements to torchaudio.egg-info/requires.txt
writing top-level names to torchaudio.egg-info/top_level.txt
reading manifest file 'torchaudio.egg-info/SOURCES.txt'
writing manifest file 'torchaudio.egg-info/SOURCES.txt'
running build_ext
CUDA_TOOLKIT_ROOT_DIR not found or specified
-- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) 
CMake Warning at /home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:31 (message):
  Caffe2: CUDA cannot be found.  Depending on whether you are building Caffe2
  or a Caffe2 dependent library, the next warning / error will give you more
  info.
Call Stack (most recent call first):
  /home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
  CMakeLists.txt:55 (find_package)


CMake Error at /home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:90 (message):
  Your installed Caffe2 version uses CUDA but I cannot find the CUDA
  libraries.  Please set the proper CUDA prefixes and / or install CUDA.
Call Stack (most recent call first):
  /home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
  CMakeLists.txt:55 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/ycy/working/audio/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
See also "/home/ycy/working/audio/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "setup.py", line 60, in <module>
    setup(
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/site-packages/setuptools/command/develop.py", line 34, in run
    self.install_for_development()
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/site-packages/setuptools/command/develop.py", line 136, in install_for_development
    self.run_command('build_ext')
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/ycy/working/audio/build_tools/setup_helpers/extension.py", line 52, in run
    super().run()
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/ycy/working/audio/build_tools/setup_helpers/extension.py", line 96, in build_extension
    subprocess.check_call(
  File "/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/home/ycy/working/audio', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_PREFIX_PATH=/home/ycy/miniconda3/envs/torchaudio_dev/lib/python3.8/site-packages/torch/share/cmake', '-DCMAKE_INSTALL_PREFIX=/home/ycy/working/audio/build/lib.linux-x86_64-3.8/torchaudio/', '-DCMAKE_VERBOSE_MAKEFILE=ON', '-DPython_INCLUDE_DIR=/home/ycy/miniconda3/envs/torchaudio_dev/include/python3.8', '-DBUILD_SOX:BOOL=OFF', '-DBUILD_KALDI:BOOL=ON', '-DBUILD_TRANSDUCER:BOOL=OFF', '-DBUILD_TORCHAUDIO_PYTHON_EXTENSION:BOOL=ON', '-DBUILD_LIBTORCHAUDIO:BOOL=OFF', '-GNinja']' returned non-zero exit status 1.

Expected behavior

Environment

Collecting environment information...
PyTorch version: 1.8.0.dev20201112
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.18.4

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 3070
Nvidia driver version: 460.32.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.8.0.dev20201112
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.0.221             h6bb024c_0  
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.2.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.2           py38h54aff64_0  
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] pytorch                   1.8.0.dev20201112 py3.8_cuda11.0.221_cudnn8.0.3_0    pytorch-nightly

Additional context

[imaginary-person]

Setting these environment variables would probably help:

https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions

Be careful about the CUDA version in these commands, though.

[yoyolicoris]

@imaginary-person
Thanks, but the problem is I cannot even find the location where the cuda has installed... I installed it by conda, and cannot find similar folder structure like in the officail nvidia installation.

[mthrok]

I encountered this when I had PyTorch installation with CUDA capability but not having CUDA. Since torchaudio does not need CUDA, you can either try building it with CPU only PyTorch or fix your CUDA installation.

[yoyolicoris]

Temporary solved by using CPU only PyTorch.

[imaginary-person]

I'm also running Ubuntu 20.04.

If you want to leverage the GPU, install CUDA toolkit from the Nvidia website, and then install PyTorch again, and then try building torchaudio. There are many posts on Stackoverflow about conda not installing all packages of the CUDA toolkit, but only the packages it needs.

[mthrok]

Closing as this is not an issue on torchaudio.

[vincentqb]

If the goal is to experiment with the latest features, one can also use the nightly versions, e.g. here.