[CUDA] CTS usm_api_mem_advise and usm_api_prefetch tests are failed for invalid argument in function cuda_piextUSMEnqueueMemAdvise

[yuxianch]

Describe the bug
CTS usm_api_mem_advise and usm_api_prefetch tests are failed for invalid argument in function cuda_piextUSMEnqueueMemAdvise with CUDA backend on Linux.

To Reproduce

1. Get SYCL with CUDA support (GSG) and SYCL-CTS
2. Build CTS tests

cd SYCL-CTS
mkdir build && cd build
cmake -G "Ninja" -DSYCL_IMPLEMENTATION=Intel_SYCL -DINTEL_SYCL_ROOT=<path to built sycl> -DOpenCL_LIBRARY=<path to built sycl>/lib/libOpenCL.so -DOpenCL_INCLUDE_DIR=<path to built sycl>/include/sycl -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS_RELEASE="-fsycl -fsycl-unnamed-lambda -Wno-deprecated-declarations" -DINTEL_SYCL_FLAGS="-fsycl-device-code-split=per_kernel" -DCMAKE_EXE_LINKER_FLAGS=" -Wl,-no-relax " -DINTEL_SYCL_TRIPLE=nvptx64-nvidia-cuda -DSYCL_CTS_ENABLE_OPENCL_INTEROP_TESTS=Off -DSYCL_CTS_ENABLE_DOUBLE_TESTS=On -DSYCL_CTS_ENABLE_HALF_TESTS=On ..
ninja test_usm

3. Run test

export SYCL_DEVICE_FILTER="CUDA:GPU"
export SYCL_ENABLE_HOST_DEVICE="1"
./bin/test_usm --device "NVIDIA.*CUDA.*NVIDIA"  --test usm_api_mem_advise_handler_no_events

Error message:

--- usm_api_mem_advise_handler_no_events

PI CUDA ERROR:
        Value:           1
        Name:            CUDA_ERROR_INVALID_VALUE
        Description:     invalid argument
        Function:        cuda_piextUSMEnqueueMemAdvise
        Source Location: /<path to built sycl>/sycl/plugins/cuda/pi_cuda.cpp:4722

  . mem_advise using host allocation got an exception for i
  . sycl exception caught
  . what - Native API failed. Native API returns: -30 (CL_INVALID_VALUE) -30 (CL_INVALID_VALUE)
  . line: 26
  . a SYCL exception was caught: Native API failed. Native API returns: -30 (CL_INVALID_VALUE) -30 (CL_INVALID_VALUE)
  - fail

  . Passed 0/1 tests (0%)

For usm_api_prefetch tests, the error is similar and the function with error is changed to cuda_piextUSMEnqueuePrefetch:

--- usm_api_prefetch_handler_no_events

PI CUDA ERROR:
        Value:           1
        Name:            CUDA_ERROR_INVALID_VALUE
        Description:     invalid argument
        Function:        cuda_piextUSMEnqueuePrefetch
        Source Location: /<path to built sycl>/sycl/plugins/cuda/pi_cuda.cpp:4690

  . prefetch using host allocation got an exception for i
  . sycl exception caught
  . what - Native API failed. Native API returns: -30 (CL_INVALID_VALUE) -30 (CL_INVALID_VALUE)
  . line: 26
  . a SYCL exception was caught: Native API failed. Native API returns: -30 (CL_INVALID_VALUE) -30 (CL_INVALID_VALUE)
  - fail

  . Passed 0/1 tests (0%)

Environment

• OS: Linux
• Target device and vendor: Nvidia GPU
• DPC++ version: https://github.com/intel/llvm.git a068b15
• CUDA version:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.29.05    Driver Version: 495.29.05    CUDA Version: 11.5     |
|-------------------------------+----------------------+----------------------+

[yuxianch]

CTS usm_api_mem_advise tests are failed for the same issue on Windows.

[AidanBeltonS]

I have investigated this issue and have been able to reproduce it.
There are two causes to the test failures. The first is that non-managed memory is being passed to mem_advise, this causes CUDA to return an error. The second is that the CUDA PI treats 0 as an unknown function rather than mem_advise reset. This is detailed in the spec.

Acceptable values of advice are device-defined. 
A value of 0 reverts the advice for ptr to the default behavior.

I have proposed two PRs to fix this, #5536 and #5446.

[AerialMantis]

As both #5536 and #5446 are now merged we believe this issue to be resolved and are closing the ticket.