Profile-armhf :: instrprof-gcov-multithread_fork.test fails

[rovka]

Bugzilla Link	46092
Resolution	FIXED
Resolved on	Jun 23, 2020 16:06
Version	unspecified
OS	Linux
Blocks	#44654
CC	@calixteman,@marco-c,@tstellar
Fixed by commit(s)	0da37be 5ccc104

Extended Description

Fails on 32-bit arm for llvm 10.0.1-rc1. I don't think this was failing for 10.0.0, but I can check if needed.

FAIL: Profile-armhf :: instrprof-gcov-multithread_fork.test (56889 of 66740)
******************** TEST 'Profile-armhf :: instrprof-gcov-multithread_fork.test' FAILED ********************
Script:

: 'RUN: at line 3'; mkdir -p /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-tk1_32-build/rc1/Phase3/Release/llvmCore-10.0.1-rc1.obj/projects/compiler-rt/test/profile/Profile-armhf/Output/instrprof-gcov-multithread_fork.test.tmp.d
: 'RUN: at line 4'; cd /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-tk1_32-build/rc1/Phase3/Release/llvmCore-10.0.1-rc1.obj/projects/compiler-rt/test/profile/Profile-armhf/Output/instrprof-gcov-multithread_fork.test.tmp.d
: 'RUN: at line 6'; /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-tk1_32-build/rc1/Phase3/Release/llvmCore-10.0.1-rc1.obj/./bin/clang --driver-mode=g++ -ldl --coverage -lpthread -o /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-tk1_32-build/rc1/Phase3/Release/llvmCore-10.0.1-rc1.obj/projects/compiler-rt/test/profile/Profile-armhf/Output/instrprof-gcov-multithread_fork.test.tmp /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-tk1_32-build/rc1/llvm-project/compiler-rt/test/profile/Inputs/instrprof-gcov-multithread_fork.cpp
: 'RUN: at line 7'; test -f instrprof-gcov-multithread_fork.gcno
: 'RUN: at line 9'; rm -f instrprof-gcov-multithread_fork.gcda
: 'RUN: at line 10'; /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-tk1_32-build/rc1/Phase3/Release/llvmCore-10.0.1-rc1.obj/projects/compiler-rt/test/profile/Profile-armhf/Output/instrprof-gcov-multithread_fork.test.tmp
: 'RUN: at line 11'; llvm-cov gcov instrprof-gcov-multithread_fork.gcda

Exit Code: 134

Command Output (stderr):

terminate called recursively
terminate called recursively
/home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-tk1_32-build/rc1/Phase3/Release/llvmCore-10.0.1-rc1.obj/projects/compiler-rt/test/profile/Profile-armhf/Output/instrprof-gcov-multithread_fork.test.script: line 7: 18084 Aborted /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-tk1_32-build/rc1/Phase3/Release/llvmCore-10.0.1-rc1.obj/projects/compiler-rt/test/profile/Profile-armhf/Output/instrprof-gcov-multithread_fork.test.tmp

--

[rovka]

After a "fun" bisect experience I discovered that the test is actually flaky.

Running it 100 times at commit f082d91, which seems related, I get 'terminate called recursively' about 17 times (and also "Resource temporarily unavailable" a couple other times, which is probably an unrelated kernel issue). So it was actually quite lucky that I got it every single time I tried with 10.0.1. I'm currently doing the same experiment for 10.0.0, if we don't see it at all in 100 runs then it maybe wasn't there, although I'm not sure how much confidence we can have in these results. I also plan to do the same thing for f082d91's parent. I'll report here once I have more info.

[rovka]

I got 100 green runs with llvm 10.0.0 and 100 green runs with baeb500. I also ran again with f082d91 and got 23 'terminate called recursively' out of 100 runs. I therefore think the instability was probably introduced by f082d91.

[marco-c]

The test was introduced in f082d91 and it should consistently fail without that patch :S

[rovka]

Oh, sorry, I wasn't actually running just the test, I was running check-profile :) So, if the test wasn't there before there was nothing to break.

[rovka]

Ping! Anything I can do to help debug this?

[calixteman]

Sorry for the delay.

I fixed a similar issue on CI:
https://reviews.llvm.org/rG0da37bedc2667da371eda30595a06210595881d0

There were too much created threads and the same error.
So I reduced the number of threads.
The current test case launches 6 threads and maybe it's sometimes still too much.
Anyway the goal of my patch was to address a segfault so adding a try/catch around launcher<>(f); should be enough to avoid the intermittent.

[rovka]

Could you please sent a patch with the try-catch, so we can cherry-pick it to the release branch?

[calixteman]

I pushed a patch for review yesterday:
https://reviews.llvm.org/D81650/new/

Please try it.

[rovka]

Oh, sorry, I didn't notice it. I'll give it a try and let you know on Monday. Thanks!

[rovka]

@​tstellar: Hi Tom, it seems this works if we cherry-pick the fix in https://reviews.llvm.org/rG0da37bedc2667da371eda30595a06210595881d0

[calixteman]

@​Diana, thank you to work on that :)
Normally with this patch we've only 6 threads which are created so it should be fine.

[tstellar]

Merged: 5ccc104