Valgrind null pointer dereference in MPI_Allgatherv with count=0

[ompiteam]

I think I have reduced this as far as possible.

$ mpicc -O0 -g3 -Wall -Wextra agv.c
$ mpiexec -n 3 valgrind ./a.out >& ompi.n3.log
$ ompi_info > ompi.info
$ uname -a
Linux kunyang 2.6.32-ARCH #1 SMP PREEMPT Fri Jan 29 09:10:49 CET 2010 x86_64 Intel(R) Core(TM)2 Duo CPU P8700 @ 2.53GHz GenuineIntel GNU/Linux

[ompiteam]

Imported from trac issue 2247. Created by jedbrown on 2010-02-09T18:08:28, last modified: 2012-10-08T14:48:21

• jedbrown attached agv.c on 2010-02-09 18:09:06
• jedbrown attached ompi.n3.log on 2010-02-09 18:09:27
• jedbrown attached ompi.info on 2010-02-09 18:09:42
• jedbrown attached ompi-r22702.log on 2010-02-24 13:19:41

[ompiteam]

Trac comment by jsquyres on 2010-02-09 20:19:48:

George -- this looks like a segv in coll tuned.

[ompiteam]

Trac comment by bosilca on 2010-02-24 09:15:13:

I used the provided test on exactly the same architecture, with ompi 1.4.1 and I cannot reproduce the problem. I tried different process distribution (over 3, 2 and 1 node), forcing the bruck allgather algorithm and still no luck.

[ompiteam]

Trac comment by jedbrown on 2010-02-24 13:21:08:

I have repeated the problem with today's trunk, built --with-debug so the attached stack trace should be useful. Let me know if there is something else I can do to help.

[ompiteam]

Trac comment by bbenton on 2012-02-21 13:40:41:

Milestone Open MPI 1.4.5 deleted

[ompiteam]

Trac comment by bbenton on 2012-10-08 13:16:42:

I, too, have not been able to recreate this. I've tested 1.[467] series. So, I am closing this as "worksforme".

[ompiteam]

Trac comment by jedbrown on 2012-10-08 13:53:49:

Still there for me with openmpi-1.6.2 on Archlinux with valgrind-3.8.1.

==7558== Invalid read of size 2
==7558==    at 0xA5ECEC8: ??? (in /usr/lib/openmpi/openmpi/mca_pml_ob1.so)
==7558==    by 0x4EA4D2D: ompi_request_default_wait_all (in /usr/lib/openmpi/libmpi.so.1.0.3)
==7558==    by 0xBCCDD96: ompi_coll_tuned_sendrecv_actual (in /usr/lib/openmpi/openmpi/mca_coll_tuned.so)
==7558==    by 0xBCD7497: ompi_coll_tuned_allgatherv_intra_bruck (in /usr/lib/openmpi/openmpi/mca_coll_tuned.so)
==7558==    by 0x4EB4A94: PMPI_Allgatherv (in /usr/lib/openmpi/libmpi.so.1.0.3)
==7558==    by 0x400BB6: main (agv.c:18)
==7558==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==7558== 
[batura:07558] *** Process received signal ***
[batura:07556] Signal: Segmentation fault (11)
[batura:07556] Signal code: Address not mapped (1)
[batura:07556] Failing at address: (nil)
[batura:07558] Signal: Segmentation fault (11)
[batura:07558] Signal code: Address not mapped (1)
[batura:07558] Failing at address: (nil)
[batura:07558] [ 0] /usr/lib/libpthread.so.0(+0xf170) [0x566c170]
[batura:07558] [ 1] /usr/lib/openmpi/openmpi/mca_pml_ob1.so(+0x16ec8) [0xa5ecec8]
[batura:07558] [ 2] /usr/lib/openmpi/libmpi.so.1(ompi_request_default_wait_all+0x30e) [0x4ea4d2e]
[batura:07558] [ 3] /usr/lib/openmpi/openmpi/mca_coll_tuned.so(ompi_coll_tuned_sendrecv_actual+0x157) [0xbccdd97]
[batura:07558] [ 4] /usr/lib/openmpi/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_bruck+0x2c8) [0xbcd7498]
[batura:07558] [ 5] /usr/lib/openmpi/libmpi.so.1(MPI_Allgatherv+0x445) [0x4eb4a95]
[batura:07558] [ 6] ./a.out() [0x400bb7]
[batura:07558] [ 7] /usr/lib/libc.so.6(__libc_start_main+0xf5) [0x589a725]
[batura:07558] [ 8] ./a.out() [0x400989]
[batura:07558] *** End of error message ***

[ompiteam]

Trac comment by bbenton on 2012-10-08 14:48:21:

Thanks for the feedback. I'm re-opening and assigning to the 1.6 series.

[clintonstimpson]

Here is a stack trace I'm getting.

==34246== Invalid read of size 2
==34246==    at 0xA415730: mca_pml_ob1_recv_request_free (memchecker.h:115)
==34246==    by 0x6D83854: ompi_request_default_wait_all (request.h:350)
==34246==    by 0xBAEDC26: ompi_coll_tuned_sendrecv_actual (coll_tuned_util.c:54)
==34246==    by 0xBAF7C86: ompi_coll_tuned_allgatherv_intra_bruck (coll_tuned_util.h:57)
==34246==    by 0x6D942B1: PMPI_Allgatherv (pallgatherv.c:139)

In pml_ob1_recvreq.c line 71, we have

        MEMCHECKER(
            memchecker_call(&opal_memchecker_base_mem_defined,
                            recvreq->req_recv.req_base.req_addr,
                            recvreq->req_recv.req_base.req_count,
                            recvreq->req_recv.req_base.req_datatype);
        );

The last parameter, recvreq->req_recv.req_base.req_datatype, is NULL.
This comes from calling MPI_Allgatherv with a count=0 for some ranks.

My OpenMPI configure flags include
--enable-debug --enable-memchecker
--enable-ltdl-convenience
along with --prefix and --with-valgrind.

I'm using OpenMPI 1.6.5 and valgrind 3.10.1.

[bosilca]

Hi,

memchecker_call doesn't take in account when no data is attached to a
communication (either due to the count being 0, or due to the fact that the
datatype has no useful data). Can you please try the following patch:

diff --git a/ompi/include/ompi/memchecker.h b/ompi/include/ompi/memchecker.h
index 6ba22e6..23f4f4a 100644
--- a/ompi/include/ompi/memchecker.h
+++ b/ompi/include/ompi/memchecker.h
@@ -100,6 +100,9 @@ static inline int memchecker_call (int (*f)(void *,
size_t), const void * addr,
         return OMPI_SUCCESS;
     }

+    if( (0 == count) || (0 == datatype->super.size) ) {
+        return OMPI_SUCCESS;
+    }
     if( datatype->super.size == (size_t) (datatype->super.true_ub -
datatype->super.true_lb) ) {
         /*  We have a contiguous type. */
         f( (void*)addr , datatype->super.size * count );

George.

On Wed, Feb 17, 2016 at 2:49 PM, clintonstimpson notifications@github.com
wrote:

Here is a stack trace I'm getting.

==34246== Invalid read of size 2
==34246== at 0xA415730: mca_pml_ob1_recv_request_free (memchecker.h:115)
==34246== by 0x6D83854: ompi_request_default_wait_all (request.h:350)
==34246== by 0xBAEDC26: ompi_coll_tuned_sendrecv_actual
(coll_tuned_util.c:54)
==34246== by 0xBAF7C86: ompi_coll_tuned_allgatherv_intra_bruck
(coll_tuned_util.h:57)
==34246== by 0x6D942B1: PMPI_Allgatherv (pallgatherv.c:139)

In pml_ob1_recvreq.c line 71, we have
MEMCHECKER(
memchecker_call(&opal_memchecker_base_mem_defined,
recvreq->req_recv.req_base.req_addr,
recvreq->req_recv.req_base.req_count,
recvreq->req_recv.req_base.req_datatype);
);

The last parameter, recvreq->req_recv.req_base.req_datatype, is NULL.
This comes from calling MPI_Allgatherv with a count=0 for some ranks.

My OpenMPI configure flags include
--enable-debug --enable-memchecker
--enable-ltdl-convenience
along with --prefix and --with-valgrind.

I'm using OpenMPI 1.6.5 and valgrind 3.10.1.

—
Reply to this email directly or view it on GitHub
#100 (comment).

[clintonstimpson]

The patch fixes my problem. Thanks!