mpiexec --cpu-set ORTE error

[GuillaumeMercier]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

Git repo version: v2.x-dev-5428-g17a5416

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

from a git clone (v2.x-dev-5428-g17a5416)

Please describe the system on which you are running

• Operating system/version: 3.10.0-327.36.3.el7.x86_64
• Computer hardware: Xeon E5 V3
• Network type: OPA

---

Details of the problem

Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.

When trying to launch a programm with the --cpu-set option, I've got this:

[miriel016:115300] [[3216,0],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file ../../../../ompi-sources/orte/mca/odls/base/odls_base_default_fns.c at line 565

My command line is:

mpiexec --cpu-set 0,1,2,12,13,14 --bind-to core  -np 4 -host localhost sample_prog   4

[ggouaillardet]

note Git repo version is misleading here (a known git thing ...), and the issue occurs on the (latest) master branch.

can you please give the inline patch a try ?

diff --git a/orte/mca/odls/base/odls_base_default_fns.c b/orte/mca/odls/base/odls_base_default_fns.c
index ff9426b..be20292 100644
--- a/orte/mca/odls/base/odls_base_default_fns.c
+++ b/orte/mca/odls/base/odls_base_default_fns.c
@@ -15,7 +15,7 @@
  *                         All rights reserved.
  * Copyright (c) 2011-2017 Cisco Systems, Inc.  All rights reserved
  * Copyright (c) 2013-2018 Intel, Inc.  All rights reserved.
- * Copyright (c) 2014-2017 Research Organization for Information Science
+ * Copyright (c) 2014-2018 Research Organization for Information Science
  *                         and Technology (RIST). All rights reserved.
  * Copyright (c) 2017      Mellanox Technologies Ltd. All rights reserved.
  * Copyright (c) 2017      IBM Corporation. All rights reserved.
@@ -178,7 +178,11 @@ int orte_odls_base_default_get_add_procs_data(opal_buffer_t *buffer,
     } else {
         nidmap = NULL;
     }
-    opal_dss.pack(buffer, &nidmap, 1, OPAL_STRING);
+    if (ORTE_SUCCESS != (rc = opal_dss.pack(buffer, &nidmap, 1, OPAL_STRING))) {
+        ORTE_ERROR_LOG(rc);
+        free(nidmap);
+        return rc;
+    }
     if (NULL != nidmap) {
         free(nidmap);
     }
@@ -375,11 +379,15 @@ int orte_odls_base_default_get_add_procs_data(opal_buffer_t *buffer,
             ORTE_ERROR_LOG(rc);
             return rc;
         }
-        if (ORTE_SUCCESS != (rc = opal_dss.pack(buffer, &nidmap, 1, OPAL_STRING))) {
-            ORTE_ERROR_LOG(rc);
-            free(nidmap);
-            return rc;
-        }
+    } else {
+        nidmap = NULL;
+    }
+    if (ORTE_SUCCESS != (rc = opal_dss.pack(buffer, &nidmap, 1, OPAL_STRING))) {
+        ORTE_ERROR_LOG(rc);
+        free(nidmap);
+        return rc;
+    }
+    if (NULL != nidmap) {
         free(nidmap);
     }
 

@rhc54 would you mind commenting on it if/when you get a chance ?

that seems good enough, but it could be suboptimal.

[GuillaumeMercier]

It doesn't seems to do the trick.
The program runs, yes, but the processes are not bound:

mpiexec --cpu-set 0,1,2,12,13,14  --tag-output -report-bindings  --oversubscribe  --rank-by core --bind-to core  -np 4  -machinefile ./machinefile ./sample_prog 

outputs:

[miriel074:133390] MCW rank 0 is not bound (or bound to all available processors)
[miriel074:133390] MCW rank 1 is not bound (or bound to all available processors)
[miriel074:133390] MCW rank 2 is not bound (or bound to all available processors)
[miriel074:133390] MCW rank 3 is not bound (or bound to all available processors)

[ggouaillardet]

ok, i will try again on Monday.

meanwhile, can you post the content of your machinefile and the (text) output of lstopo ?

fwiw, I did

mpirun --cpu-set ... --bind-to core grep Cpus_allowed_list /proc/self/status

and as far as I remember, processes were (correctly) bound to distinct cores

[rhc54]

FWIW: I'm pretty sure we won't bind if you set --oversubscribe, regardless of the rest of the command line.

[GuillaumeMercier]

Ok, I removed this option and the result is the same.

[GuillaumeMercier]

@ggouaillardet:
my machinefile contains the two follwing lines:

miriel074
miriel075

and the output of lstopo is:

Machine (128GB)
  Socket L#0 (64GB)
    NUMANode L#0 (P#0 32GB)
      L3 L#0 (15MB)
        L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0)
        L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 (P#2)
        L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 (P#4)
        L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 (P#6)
        L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4 (P#8)
        L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5 (P#10)
      HostBridge L#0
        PCIBridge
          PCI 8086:10fb
            Net L#0 "em1"
          PCI 8086:10fb
            Net L#1 "em2"
        PCI 8086:8d62
          Block L#2 "sda"
        PCIBridge
          PCI 8086:1521
            Net L#3 "em3"
          PCI 8086:1521
            Net L#4 "em4"
        PCIBridge
          PCIBridge
            PCIBridge
              PCIBridge
                PCI 102b:0534
                  GPU L#5 "card0"
                  GPU L#6 "controlD64"
        PCI 8086:8d02
    NUMANode L#1 (P#2 32GB) + L3 L#1 (15MB)
      L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 (P#12)
      L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 (P#14)
      L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8 (P#16)
      L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9 (P#18)
      L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + PU L#10 (P#20)
      L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + PU L#11 (P#22)
  Socket L#1 (64GB)
    NUMANode L#2 (P#1 32GB)
      L3 L#2 (15MB)
        L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + PU L#12 (P#1)
        L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + PU L#13 (P#3)
        L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + PU L#14 (P#5)
        L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + PU L#15 (P#7)
        L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16 + PU L#16 (P#9)
        L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17 + PU L#17 (P#11)
      HostBridge L#7
        PCIBridge
          PCI 1077:7322
            Net L#7 "ib0"
            OpenFabrics L#8 "qib0"
    NUMANode L#3 (P#3 32GB) + L3 L#3 (15MB)
      L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18 + PU L#18 (P#13)
      L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19 + PU L#19 (P#15)
      L2 L#20 (256KB) + L1d L#20 (32KB) + L1i L#20 (32KB) + Core L#20 + PU L#20 (P#17)
      L2 L#21 (256KB) + L1d L#21 (32KB) + L1i L#21 (32KB) + Core L#21 + PU L#21 (P#19)
      L2 L#22 (256KB) + L1d L#22 (32KB) + L1i L#22 (32KB) + Core L#22 + PU L#22 (P#21)
      L2 L#23 (256KB) + L1d L#23 (32KB) + L1i L#23 (32KB) + Core L#23 + PU L#23 (P#23)

[rhc54]

Relates to #5266

[rhc54]

Looks like the --cpu-set option is broken