Cannot use rankfile option

[cbordage]

With the master branch of OpenMPI (ba46b35) and I have a problem when I use a rankfile.

shell$ mpirun -np 2 -machinefile mf -rf rf ./test
[miriel044:01383] pmix_mca_base_component_repository_open: unable to open mca_pnet_opa: libpsm2.so.2: cannot open shared object file: No such file or directory (ignored)
[miriel045:135838] pmix_mca_base_component_repository_open: unable to open mca_pnet_opa: libpsm2.so.2: cannot open shared object file: No such file or directory (ignored)
[miriel044:01383] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/././././././././././.][./././././././././././.]
[miriel045:135838] MCW rank 1 bound to socket 0[core 0[hwt 0]]: [B/././././././././././.][./././././././././././.]
[miriel044:01389] pmix_mca_base_component_repository_open: unable to open mca_pnet_opa: libpsm2.so.2: cannot open shared object file: No such file or directory (ignored)
[miriel045:135890] pmix_mca_base_component_repository_open: unable to open mca_pnet_opa: libpsm2.so.2: cannot open shared object file: No such file or directory (ignored)
[miriel045:135890] PMIX ERROR: ERROR STRING NOT FOUND in file ../../../../../../../../../../src/opal/mca/pmix/pmix2x/pmix/src/mca/ptl/tcp/ptl_tcp.c at line 299
*** An error occurred in MPI_Init                      
*** on a NULL communicator 
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,                                       
***    and potentially your MPI job)                   
[miriel045:135890] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
-------------------------------------------------------                                                        
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------                                                        
[miriel044:01389] mca_base_component_repository_open: unable to open mca_mtl_psm2: libpsm2.so.2: cannot open shared object file: No such file or directory (ignored)
--------------------------------------------------------------------------                                     
mpirun detected that one or more processes exited with non-zero status, thus causing                           
the job to be terminated. The first process to do so was:

  Process name: [[21140,1],1]                          
  Exit code:    1          

where mf is

miriel044 slots=24
miriel045 slots=24

and rf is

rank 0=miriel044 slot=0
rank 1=miriel045 slot=0

If I use hostname instead of ./test it works.

Since the commit 48fc339, it does not work (but the error changed, first, it was a not enough slots available problem).

[rhc54]

The problem is that the rank on the remote node isn't getting registered - i.e., the daemon isn't successfully calling opal_pmix.register_client on the proc. This causes the PMIx connection from the process to be rejected because the PMIx server doesn't recognize the rank attempting to connect.

[cbordage]

Okay... So, what do you propose to solve this problem ?

[rhc54]

I propose nothing - I was simply pointing the root cause out to others that have taken over responsibility for that mapper.

[bgoglin]

@rhc54 (cbordage is from my team) So if rankfile mapper isn't maintained anymore, what should we use to manually specify the binding of each rank? They currently use the rankfile for topology-aware process placement after offline analysis of the communication pattern.

[jjhursey]

IBM is planning on picking up support for the rankfile mapper. However, it is going to be another week before we (I) get a chance to look at this. I don't think there is any alternative means of a manual per-rank binding specification outside of the rankfile.

I suppose if you really needed it today you could implement a wrapper program that uses hwloc to bind processes and fork the actual application based upon a file - but at that point you are just re-implementing the rankfile, just outside of Open MPI. Useful, but a fair amount of work. I know of other groups that have implemented such a program to make it easier to portably specify binding without being tied to a specific MPI.

[rhc54]

I'll take a crack at this today to ease in the transition (and since @jjhursey is being kind enough to help track down the pmix race condition)

[rhc54]

Fix is in the oven