[SPARK-23962][SQL][TEST] Fix race in currentExecutionIds().

[squito]

SQLMetricsTestUtils.currentExecutionIds() was racing with the listener
bus, which lead to some flaky tests. We should wait till the listener bus is
empty.

I tested by adding some Thread.sleep()s in SQLAppStatusListener, which
reproduced the exceptions I saw on Jenkins. With this change, they went
away.

[SparkQA]

Test build #89204 has finished for PR 21041 at commit 7108d26.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[squito]

@vanzin @cloud-fan @gatorsmile please take a look, thanks

[cloud-fan]

thanks, merging to master/2.3!

[dongjoon-hyun]

Hi, @squito and @cloud-fan .

Could you take a look at branch-2.3/sbt build?

After this commit on branch-2.3, the following two test cases of ParquetQuerySuite never succeed in branch-2.3/sbt/hadoop-2.6.

• SPARK_15678__not_use_cache_on_overwrite
• SPARK_15678__not_use_cache_on_append

For branch-2.3/sbt/hadoop-2.7, 6 test case of DataFrameSuite does, too. For example, the following.

• https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/315/testReport/junit/org.apache.spark.sql/DataFrameSuite/inputFiles/history/

[squito]

@dongjoon-hyun thanks for reporting this. I think this is the same as https://issues.apache.org/jira/browse/SPARK-23894 . I am nearly certain its not directly caused by this change, but some existing race which has somehow become more common.

I don't totally understand that issue yet -- I'll make a few comments, but would appreciate any help

[squito]

I submitted a PR for SPARK-23894, #21185, please take a look.

[dongjoon-hyun]

Thank you, @squito !