failed to make ompi with CUDA and UCX

[leeQT]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

v4.0.2

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

from git clone

Please describe the system on which you are running

• Operating system/version: Ubuntu 16.04

---

Details of the problem

Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.

Failed to make ompi
commands:

./configure --with-cuda=/data/cuda/cuda-10.0/cuda/ --with-ucx=/home/liqintian/software/ucx/build
make -j 8

output

Makefile:1898: recipe for target 'btl_uct_amo.lo' failed
make[2]: *** [btl_uct_amo.lo] Error 1
make[2]: Leaving directory '/home/liqintian/examples/openmpi-4.0.2/opal/mca/btl/uct'
Makefile:2361: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/liqintian/examples/openmpi-4.0.2/opal'
Makefile:1879: recipe for target 'all-recursive' failed

[brminich]

please add --enable-mca-no-build=btl-uct parameter to your configure

[jsquyres]

@brminich I thought Open MPI v4.0.2 was fine with latest UCX?

There does not appear to be enough information in this bug report to tell exactly what the problem was.

[leeQT]

please add --enable-mca-no-build=btl-uct parameter to your configure

Thanks, this solved problem.

[yosefe]

@leeQT can you pls post the full output of the original build failure? also, which UCX version was used?

[leeQT]

I cloned UCX source code from https://github.com/openucx/ucx, I didn't find the version information but I think it's the latest version.
The full output is here:
make.txt

[leeQT]

please add --enable-mca-no-build=btl-uct parameter to your configure

When I tried to run mpi on multiple nodes by mpirun -H host1,host2 hello_c.out, I got many errors like

ib_device.c:914  UCX  ERROR Invalid gid[0] on mlx5_36:1

And warnings:

--------------------------------------------------------------------------
WARNING: No preset parameters were found for the device that Open MPI
detected:

  Local host:            rollback-119FCEF2-worker-0
  Device name:           mlx5_36
  Device vendor ID:      0x02c9
  Device vendor part ID: 4120

Default device parameters will be used, which may result in lower
performance.  You can edit any of the files specified by the
btl_openib_device_param_files MCA parameter to set values for your
device.

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_no_device_params_found to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           rollback-62B47788-worker-0
  Local device:         mlx5_30
  Local port:           1
  CPCs attempted:       rdmacm, udcm
--------------------------------------------------------------------------
[rollback:10564] 125 more processes have sent help message help-mpi-btl-openib.txt / no device params found
[rollback:10564] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[rollback:10564] 5 more processes have sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port

Is this caused by not building btl-uct?

[yosefe]

Seems there something non-standard with RDMA device configuration on your system
Can you pls upload the output of "ibv_devinfo -vvv"?

[yosefe]

Also, 4.0.2 still doesn't support latest UCX (v1.8.x), that support would be added by #7016 (not merged yet)

[leeQT]

Seems there something non-standard with RDMA device configuration on your system
Can you pls upload the output of "ibv_devinfo -vvv"?

Yes I got it, my ib devices are disabled. Thank you so much for your reply.

[leeQT]

Also, 4.0.2 still doesn't support latest UCX (v1.8.x), that support would be added by #7016 (not merged yet)

So what's the latest UCX version 4.0.2 supports?

[yosefe]

1.6.x