CUDA build fails to find CUDA

[tomdeakin]

Building with the CUDA PI is still very challenging.

I have gcc 8.1, cuda10.2, cmake 3.18.3.

$ CUDA_BIN_PATH=/cm/shared/apps/cuda10.2/toolkit/10.2.89 CC=gcc CXX=g++ python $DPCPP_HOME/llvm/buildbot/configure.py --cuda
$ CUDA_BIN_PATH=/cm/shared/apps/cuda10.2/toolkit/10.2.89 CC=gcc CXX=g++ python $DPCPP_HOME/llvm/buildbot/compile.py
ninja: error: 'CUDA_CUDA_LIBRARY-NOTFOUND', needed by 'lib/libpi_cuda.so', missing and no known rule to make it

Looking in llvm/build/CMakeCache.txt, I see:

CUDA_CUDA_LIBRARY:FILEPATH=CUDA_CUDA_LIBRARY-NOTFOUND

but I also see all the other CUDA variables defined correctly.

I tried setting it manually on configure, but this gets the same error.

$ CUDA_BIN_PATH=/cm/shared/apps/cuda10.2/toolkit/10.2.89 CC=gcc CXX=g++ python $DPCPP_HOME/llvm/buildbot/configure.py --cuda  --cmake-opt="-DCUDA_CUDA_LIBRARY=/cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64/stubs/libcuda.so" 

If I also set CUDA_TOOLKIT_ROOT_DIR, then I get nonsense errors:

$CUDA_BIN_PATH=/cm/shared/apps/cuda10.2/toolkit/10.2.89 CC=gcc CXX=g++ python $DPCPP_HOME/llvm/buildbot/configure.py --cuda  --cmake-opt="-DCUDA_TOOLKIT_ROOT_DIR=/cm/shared/apps/cuda10.2/toolkit/10.2.89 -DCUDA_CUDA_LIBRARY=/cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64/stubs/libcuda.so"
[...]
-- Including the PI API CUDA backend.
CMake Error at /lustre/projects/bristol/modules/cmake/3.18.3/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
  Could NOT find CUDA (missing: CUDA_CUDART_LIBRARY) (found suitable version
  "10.2", minimum required is "10.1")

What do I need to pass to CMake to guarantee it finds the toolkit?

[alexbatashev]

This is a dirty hack, but it usually helps me in similar situations:
Set CUDA_CUDA_LIBRARY to libcudadrv.so full path and CUDA_INCLUDE_DIRS to full path to CUDA includes. You can do it with something like python configure.py --cuda --no-werror --cmake-opt="-DCUDA_CUDA_LIBRARY=/smth/lib.so" --cmake-opt="-DCUDA_INCLUDE_DIRS=/smth/include".

UPD I just noticed, that you use --cmake-opt incorrectly. You need to pass it multiple times once for each CMake option.

[Naghasan]

From my poor cmake understanding you could set the path using CUDA_PATH or CUDA_LIB_PATH (https://github.com/Kitware/CMake/blob/master/Modules/FindCUDA.cmake#L977)

I think the cmake set-up is using very legacy stuff (hence your trouble I guess),

[alexbatashev]

@Naghasan FindCudaToolkit was introduced in CMake 3.17, and LLVM community has settled with CMake 3.14, so that is probably very close to the right way of linking with CUDA in 2019.

[tomdeakin]

@alexbatashev Thanks for the reply.

There is no libcudadrv.so in the Toolkit. What is the library name?

Please document --cmake-opt. Ideally in the--help output and on the online docs.

[alexbatashev]

@tomdeakin this page says it's called libcuda.so

[tomdeakin]

@alexbatashev - still fails even when setting it manually:

$ CC=gcc CXX=g++ python $DPCPP_HOME/llvm/buildbot/configure.py --cuda  --cmake-opt="-DCUDA_CUDA_LIBRARY=/cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64/stubs/libcuda.so"

$ CC=gcc CXX=g++ python $DPCPP_HOME/llvm/buildbot/compile.py

ninja: error: 'CUDA_CUDA_LIBRARY-NOTFOUND', needed by 'lib/libpi_cuda.so', missing and no known rule to make it

Even specifying it manually doesn't set it in CMakeCache.txt either.

[Ruyk]

Did you try modifying the CMakeCache.txt file directly with the correct path? On my system:

//"cuda" library (older versions only).
CUDA_CUDA_LIBRARY:FILEPATH=/usr/lib/x86_64-linux-gnu/libcuda.so

[tomdeakin]

@Ruyk I did try, but CMake runs again and throws it away when I run the $DPCPP_HOME/llvm/buildbot/compile.py script. 😞

[Ruyk]

compile just re-runs Cmake with --build, seems there is something wrong with the way CMake finds CUDA in your system. Usually there is no need to point to the CUDA library directly. But also the error can be misleading: Sometimes by not finding the header files it makers the entire library as -NOTFOUND, i wonder if you have to manually specify libraries and headers...

[tomdeakin]

@Ruyk Setting CUDA_INCLUDE_DIRS too doesn't help either.

This is the output for grep CUDA llvm/build/CMakeCache.txt. Maybe it shows something weird going on, but from what I can see, everything apart from CUDA_CUDA_LIBRARY, CUDA_SDK_ROOT_DIR and CUDA_cupti_LIBRARY is set to something that exists. The missing ones shouldn't matter given it has everything else filled out.
cuda.txt

[joeatodd]

Have you tried @Naghasan's suggestion to set either the CUDA_PATH or CUDA_LIB_PATH environment variables? Specifically I think you might want to set one of these to the location of libcuda.so, seeing as everything else is being found.

[tomdeakin]

@joeatodd
Setting CUDA_PATH gives me a warning on configure, and the same NOTFOUND error on compile:

CMake Warning:
  Manually-specified variables were not used by the project:

    CUDA_PATH

The same happens for CUDA_LIB_PATH.