AVX-based MPI_OP performance regression

[rajachan]

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

v4.1.0

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

From source tarball, default configuration built with GCC 4.8.5.

Please describe the system on which you are running

• Operating system/version: Amazon Linux
• Computer hardware: Intel Skylake
• Network type: EFA

---

Details of the problem

We noticed an OS-specific regression with LAMMPS (in.chute.scaled case) with 4.1.0. Bisecting through the commits, this seems to have been introduced with the AVX-based MPI_OP changes that got backported into this series. Specifically, the commit which moved to the unaligned SSE memory access primitives for reduce OPs seems to be causing it: #7957

This was added to address the Accumulate issue, so it is a necessary correctness fix (#7954)

The actual PR which introduced the SSE-based MPI_OP in the first place was backported from master: #7935

Broadly, allreduce performance seems to have taken a hit in 4.1.0 compared to 4.0.5 in this environment because of these changes. We do not see this with Amazon Linux 2 (which has a 7.x series GCC) or Ubuntu 18, for instance.

Tried with #8322 just in case, that does not help either.

@bosilca does anything obvious stand out to you?

[ggouaillardet]

@rajachan how much performance degradation did you measure?

My understanding is that on recent processors, unaligned load/store are as efficient as aligned load/store when the data is aligned.

Is it fair to say that the performance hit is caused by the combined use of unaligned load/store and gcc 4.8.5?
(e.g. no performance hit when running the very same code on the very same hardware with GCC >= 7)

[shijin-aws]

@ggouaillardet it's a ~10% degradation. The average loop time of lammps (in seconds) jumps from 7.7 to 8.6.

[ggouaillardet]

@shijin-aws thanks,

well, 10% of the user app (e.g. not 10% of the MPI_Reduce() performance) looks like a pretty massive hit.

what if you disable the op/avx component (e.g. mpirun --mca op ^avx ...) ?

[shijin-aws]

@ggouaillardet Using mpirun --mca op ^avx can make the performance back to the level of ompi 4.0.5 (7.7 seconds).

[ggouaillardet]

@shijin-aws Thanks

can you please confirm GCC 4.8.5 is to be blamed here? in that case, should be simply not build the op/avx component with GCC < 7?

[shijin-aws]

@ggouaillardet Sure, @rajachan suggested the same thing. I am trying to build it with gcc7 on my machine to confirm the root cause is gcc 4.8.5.

[bosilca]

you can always use the reduce_local on the test/datatype with the type and op of your liking to see if there is any performance degradation in the AVX part of the reduction operation.

On a skylake machine I see a difference of about 15% between the double SUM local reduction operation compiled with gcc 4.8.5 and gcc 10.2.0. Looking a little deeper into this it seems that gcc 4.8.5 does not understand the -march=skylake-avx512 option, so AVX512 is always turned off.

[ggouaillardet]

@bosilca I am afraid this is a different issue.

The reported issue is op/avx is slower than the default op component on skylake with gcc 4.8.5

I applied the patch below to the reduce_local test (move MPI_Wtime(), add -d <aligments>, typically to reduce_local -d 1 and only use aligned data, implement -r <repeats> to make the test longer.

Then I found op/avx is a bit faster than the default component
(and yes, op/avx is only using AVX2 because gcc 4.8.5 does not know skylake)

diff --git a/test/datatype/reduce_local.c b/test/datatype/reduce_local.c
index 97890f9..9a69b1b 100644
--- a/test/datatype/reduce_local.c
+++ b/test/datatype/reduce_local.c
@@ -115,11 +115,12 @@ do { \
     const TYPE *_p1 = ((TYPE*)(INBUF)), *_p3 = ((TYPE*)(CHECK_BUF)); \
     TYPE *_p2 = ((TYPE*)(INOUT_BUF)); \
     skip_op_type = 0; \
-    for(int _k = 0; _k < min((COUNT), 4); +_k++ ) { \
-        memcpy(_p2, _p3, sizeof(TYPE) * (COUNT)); \
-        tstart = MPI_Wtime(); \
-        MPI_Reduce_local(_p1+_k, _p2+_k, (COUNT)-_k, (MPITYPE), (MPIOP)); \
-        tend = MPI_Wtime(); \
+    tstart = MPI_Wtime(); \
+    for(int _k = 0; _k < min((COUNT), d); +_k++ ) { \
+        for(int _r = 0; _r < repeats; _r++) { \
+            memcpy(_p2, _p3, sizeof(TYPE) * (COUNT)); \
+            MPI_Reduce_local(_p1+_k, _p2+_k, (COUNT)-_k, (MPITYPE), (MPIOP)); \
+        } \
         if( check ) { \
             for( i = 0; i < (COUNT)-_k; i++ ) { \
                 if(((_p2+_k)[i]) == (((_p1+_k)[i]) OPNAME ((_p3+_k)[i]))) \
@@ -131,6 +132,7 @@ do { \
             } \
         } \
     } \
+    tend = MPI_Wtime(); \
     goto check_and_continue; \
 } while (0)
 
@@ -163,15 +165,21 @@ int main(int argc, char **argv)
 {
     static void *in_buf = NULL, *inout_buf = NULL, *inout_check_buf = NULL;
     int count, type_size = 8, rank, size, provided, correctness = 1;
-    int repeats = 1, i, c;
+    int repeats = 1, i, c, d = 4;
     double tstart, tend;
     bool check = true;
     char type[5] = "uifd", *op = "sum", *mpi_type;
     int lower = 1, upper = 1000000, skip_op_type;
     MPI_Op mpi_op;
 
-    while( -1 != (c = getopt(argc, argv, "l:u:t:o:s:n:vfh")) ) {
+    while( -1 != (c = getopt(argc, argv, "d:l:u:t:o:s:n:vr:fh")) ) {
         switch(c) {
+        case 'd':
+            d = atoi(optarg);
+            if( d < 1 ) {
+                fprintf(stderr, "Disalignment must be greater than zero\n");
+                exit(-1);
+            }
         case 'l':
             lower = atoi(optarg);
             if( lower <= 0 ) {

[shijin-aws]

@ggouaillardet @rajachan I rebuilt the application with gcc/g++ 7.2.1 on the same os (alinux1), but the performance does not go back to the level of open mpi 4.0.5.

[rajachan]

Here's what I am seeing with vanilla reduce_local and 32-bit float sums:

		gcc482 (seconds)	gcc721 (seconds)
1	time	0.000017	0.000011
2	time	0	0
4	time	0	0
8	time	0	0
16	time	0	0
32	time	0	0
64	time	0	0
128	time	0	0
256	time	0	0
512	time	0.000001	0
1024	time	0.000001	0.000001
2048	time	0.000002	0.000001
4096	time	0.000003	0.000002
8192	time	0.000007	0.000004
16384	time	0.000013	0.000008
32768	time	0.000025	0.000015
65536	time	0.000052	0.000032
131072	time	0.000109	0.000074
262144	time	0.000221	0.000161
524288	time	0.000457	0.00034

[jsquyres]

So are we coming down to determining that this is a compiler issue? I.e., certain versions of gcc give terrible performance?

If so, is there a way we can detect this in configure and react appropriately?

[rajachan]

That's what it is looking like to me. I'm going to try @shijin-aws's test with the actual application again to make sure he wasn't inadvertently running with the older compiler.

[rajachan]

I've reproduced @shijin-aws's observation. With the LAMMPS application, even with the newer gcc (7.2.1), runs using op/avx perform poorer than the ones without.

$ /shared/ompi/install/bin/mpirun --mca op ^avx -n 1152 -N 36 -hostfile /shared/ompi/hfile /shared/lammps/bin/lmp -in /shared/lammps/bin/in.chute.scaled -var x 90 -var y 90
Loop time of 7.94407 on 1152 procs for 100 steps with 259200000 atoms

$ /shared/ompi/install/bin/mpirun --mca op avx -n 1152 -N 36 -hostfile /shared/ompi/hfile /shared/lammps/bin/lmp -in /shared/lammps/bin/in.chute.scaled -var x 90 -var y 90
Loop time of 8.95102 on 1152 procs for 100 steps with 259200000 atoms

$ gcc --version
gcc (GCC) 7.2.1 20170915 (Red Hat 7.2.1-2)

From OMPI config log:

MCA_BUILD_OP_AVX2_FLAGS='-mavx2'
MCA_BUILD_OP_AVX512_FLAGS='-march=skylake-avx512'
MCA_BUILD_OP_AVX_FLAGS='-mavx'
MCA_BUILD_ompi_op_avx_DSO_FALSE='#'
MCA_BUILD_ompi_op_avx_DSO_TRUE=''
MCA_BUILD_ompi_op_has_avx2_support_FALSE='#'
MCA_BUILD_ompi_op_has_avx2_support_TRUE=''
MCA_BUILD_ompi_op_has_avx512_support_FALSE='#'
MCA_BUILD_ompi_op_has_avx512_support_TRUE=''
MCA_BUILD_ompi_op_has_avx_support_FALSE='#'
MCA_BUILD_ompi_op_has_avx_support_TRUE=''
MCA_ompi_op_ALL_COMPONENTS=' avx'
MCA_ompi_op_ALL_SUBDIRS=' mca/op/avx'
MCA_ompi_op_DSO_COMPONENTS=' avx'
MCA_ompi_op_DSO_SUBDIRS=' mca/op/avx'
#define OMPI_MCA_OP_HAVE_AVX512 1
#define OMPI_MCA_OP_HAVE_AVX2 1
#define OMPI_MCA_OP_HAVE_AVX 1

Looks like there's more to it than the compiler versions and their AVX support.

[ggouaillardet]

@rajachan thanks for confirming there is more that the gcc version.

Would you be able to reproduce this issue with a smaller config ?
(ideally one node and 24 MPI tasks)