[SPARK-31926][SQL][TEST-HIVE1.2][test-maven] Fix concurrency issue for ThriftCLIService to getPortNumber #28797

yaooqinn · 2020-06-11T03:42:09Z

What changes were proposed in this pull request?

This PR brings 02f32cf back which reverted by 4a25200 because of maven test failure

diffs newly made:

add a missing log4j file to test resources
Call SessionState.detachSession() to clean the thread local one in afterAll.
Not use dedicated JVMs for sbt test runner too

Why are the changes needed?

fix the maven test

Does this PR introduce any user-facing change?

no

How was this patch tested?

add new tests

…ervice to getPortNumber

dongjoon-hyun · 2020-06-11T04:01:21Z

Thank you, @yaooqinn !

SparkQA · 2020-06-11T07:05:02Z

Test build #123816 has finished for PR 28797 at commit dbbd1e0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2020-06-11T07:55:53Z

sql/hive-thriftserver/src/test/resources/log4j.properties

@@ -0,0 +1,65 @@
+#


This file is added because when I run the mvn tests for the hive-thriftserver module, it was reported missing

yaooqinn · 2020-06-11T07:58:48Z

...-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala

      hiveServer2.stop()
    } finally {
      super.afterAll()
+      SessionState.detachSession()


This fix is added to SharedThriftServer not SharedThriftServer for a lower scope in this PR, but I think most of hive and SharedSparkSession related tests that need a dedicated JVM may cause by this.

yaooqinn · 2020-06-11T08:03:03Z

cc @cloud-fan @dongjoon-hyun @gengliangwang @wangyum thanks.

gengliangwang · 2020-06-11T08:15:32Z

project/SparkBuild.scala

    "org.apache.spark.sql.hive.thriftserver.SparkSQLEnvSuite",
    "org.apache.spark.sql.hive.thriftserver.ui.ThriftServerPageSuite",
    "org.apache.spark.sql.hive.thriftserver.ui.HiveThriftServer2ListenerSuite",
-    "org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite",


There are multiple test suites from thriftserver moduel here. Are the other suites OK to be executed in parallel？

I verified it locally with all SharedThriftServer-like tests in a single JVM, which needed to remove org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite here, and passed.

command:

build/sbt "hive-thriftserver/test-only *ThriftServer*" -Phive-thriftserver

Let's make sure we can pass tests when running within a single JVM. We can send a new PR to optimize it by running tests in parallel.

@cloud-fan I meant the other hive-thriftserver suites in testsWhichShouldRunInTheirOwnDedicatedJvm.
From line 479 to 482. I haven't looked into it.

I checked the log again and found that the 3 of SharedThriftServer-like tests in #28797 (comment) were executed sequentially in the single JVM.

https://www.scala-sbt.org/1.x/docs/Testing.html#Forking+tests

By default, sbt runs all tasks in parallel and within the same JVM as sbt itself.

The tests in a single group are run sequentially.

Not quite sure which rule above worked

yaooqinn · 2020-06-11T14:59:45Z

retest this please

SparkQA · 2020-06-11T15:36:19Z

Test build #123834 has finished for PR 28797 at commit c00ac4e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-06-11T15:50:24Z

retest this please

SparkQA · 2020-06-11T15:55:12Z

Test build #123859 has started for PR 28797 at commit c00ac4e.

yaooqinn · 2020-06-11T15:55:16Z

retest this please

SparkQA · 2020-06-11T18:57:14Z

Test build #123855 has finished for PR 28797 at commit c00ac4e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-11T20:32:11Z

Test build #123861 has finished for PR 28797 at commit c00ac4e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2020-06-12T02:00:25Z

retest this please

yaooqinn · 2020-06-12T02:02:46Z

the lastest test failure was not related

- get observable metrics by callback *** FAILED ***
  2 did not equal 1 (DataFrameCallbackSuite.scala:269)

SparkQA · 2020-06-12T07:05:02Z

Test build #123877 has finished for PR 28797 at commit c00ac4e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2020-06-12T07:08:50Z

retest this please

SparkQA · 2020-06-12T07:10:43Z

Test build #123897 has started for PR 28797 at commit c00ac4e.

yaooqinn · 2020-06-12T16:27:53Z

Build timed out (after 500 minutes). Marking the build as aborted.
@cloud-fan any thoughts on increasing the maven build time limit?

yaooqinn · 2020-06-12T16:28:06Z

retest this please

SparkQA · 2020-06-12T23:53:08Z

Test build #123937 has finished for PR 28797 at commit c00ac4e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-06-15T06:10:21Z

thanks, merging to master/3.0!

…r ThriftCLIService to getPortNumber ### What changes were proposed in this pull request? This PR brings 02f32cf back which reverted by 4a25200 because of maven test failure diffs newly made: 1. add a missing log4j file to test resources 2. Call `SessionState.detachSession()` to clean the thread local one in `afterAll`. 3. Not use dedicated JVMs for sbt test runner too ### Why are the changes needed? fix the maven test ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? add new tests Closes #28797 from yaooqinn/SPARK-31926-NEW. Authored-by: Kent Yao <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit a0187cd) Signed-off-by: Wenchen Fan <[email protected]>

dongjoon-hyun · 2020-06-15T14:56:32Z

Hi, All.
The situation is better than before, but this seems to break one test case in the branch-3.0 Jenkins jobs again. Could you take a look at this?

...ThriftServerWithSparkContextInHttpSuite.SPARK-29911: Uncache cached tables when session closed
requirement failed: Failed to bind an actual port for HiveThriftServer2
java.lang.IllegalArgumentException: requirement failed: Failed to bind an actual port for HiveThriftServer2

dongjoon-hyun · 2020-06-15T15:04:29Z

For master branch, it's unclear for now. Some failure on PR builders seems to be relevant.

[SPARK-31957][SQL] Cleanup hive scratch dir for the developer api startWithContext #28784 (comment)
[SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb #28832 (comment)

dongjoon-hyun · 2020-06-15T15:18:44Z

...-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala


  override def afterAll(): Unit = {
    try {
      hiveServer2.stop()


From some Jenkins jobs, NPE is reported here.

sbt.ForkMain$ForkError: java.lang.NullPointerException: null at org.apache.spark.sql.hive.thriftserver.SharedThriftServer.afterAll(SharedThriftServer.scala:53)

dongjoon-hyun · 2020-06-15T15:22:40Z

...-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SharedThriftServer.scala

+    sqlContext.setConf(ConfVars.HIVE_SERVER2_TRANSPORT_MODE.varname, mode.toString)
+
+    try {
+      hiveServer2 = HiveThriftServer2.startWithContext(sqlContext)


Before NPE occurs, this failed.

ThriftServerWithSparkContextInHttpSuite: 05:42:37.405 WARN hive.metastore: Failed to connect to the MetaStore Server... org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused) at org.apache.thrift.transport.TSocket.open(TSocket.java:226) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:480) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:247) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:129) at org.apache.hive.service.cli.CLIService.start(CLIService.java:152) at org.apache.hive.service.CompositeService.start(CompositeService.java:70) at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:105) at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.start(HiveThriftServer2.scala:161) at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.startWithContext(HiveThriftServer2.scala:62) at org.apache.spark.sql.hive.thriftserver.SharedThriftServer.startThriftServer(SharedThriftServer.scala:92)

dongjoon-hyun · 2020-06-15T15:25:04Z

For a record, this fails sometime (not always), but the failures seems to occur frequently on branch-3.0.

yaooqinn · 2020-06-15T15:26:40Z

It seems to hit the derby metastore constraints that fail the connection to metastore

yaooqinn · 2020-06-15T15:30:08Z

The test failures are all by SBT? If so, should I send a followup to use dedicated JVMs for them first？ @dongjoon-hyun

dongjoon-hyun · 2020-06-15T15:31:23Z

The failures occurs on Maven, too. Please see #28797 (comment) .

dongjoon-hyun · 2020-06-15T16:17:25Z

The followings is the failure on master branch in addition to the PR Builder failures.

dongjoon-hyun · 2020-06-16T02:03:31Z

Sorry, @yaooqinn and @cloud-fan . As I mentioned on @yaooqinn's followup PR , it seems that we need to revert this again.

[SPARK-31926][SQL][TESTS][FOLLOWUP][test-hive1.2][test-maven] Fix concurrency issue for ThriftCLIService to getPortNumber #28835 (comment)

[SPARK-31926][SQL][TEST-HIVE1.2] Fix concurrency issue for ThriftCLIS…

dbbd1e0

…ervice to getPortNumber

probot-autolabeler bot added BUILD SQL labels Jun 11, 2020

yaooqinn mentioned this pull request Jun 11, 2020

[SPARK-31926][SQL][test-hive1.2] Fix concurrency issue for ThriftCLIService to getPortNumber #28751

Closed

fix one JVM issue

c00ac4e

yaooqinn commented Jun 11, 2020

View reviewed changes

gengliangwang reviewed Jun 11, 2020

View reviewed changes

yaooqinn changed the title ~~[SPARK-31926][SQL][TEST-HIVE1.2][test-maven] Fix concurrency issue for ThriftCLIService to getPortNumber~~ [SPARK-31926][SQL][TEST-HIVE1.2] Fix concurrency issue for ThriftCLIService to getPortNumber Jun 11, 2020

yaooqinn changed the title ~~[SPARK-31926][SQL][TEST-HIVE1.2] Fix concurrency issue for ThriftCLIService to getPortNumber~~ [SPARK-31926][SQL][TEST-HIVE1.2][test-maven] Fix concurrency issue for ThriftCLIService to getPortNumber Jun 11, 2020

cloud-fan closed this in a0187cd Jun 15, 2020

yaooqinn mentioned this pull request Jun 15, 2020

[SPARK-31957][SQL] Cleanup hive scratch dir for the developer api startWithContext #28784

Closed

dongjoon-hyun mentioned this pull request Jun 15, 2020

[SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb #28832

Closed

dongjoon-hyun reviewed Jun 15, 2020

View reviewed changes

[SPARK-31926][SQL][TEST-HIVE1.2][test-maven] Fix concurrency issue for ThriftCLIService to getPortNumber #28797

[SPARK-31926][SQL][TEST-HIVE1.2][test-maven] Fix concurrency issue for ThriftCLIService to getPortNumber #28797

Uh oh!

Conversation

yaooqinn commented Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Jun 11, 2020

Uh oh!

SparkQA commented Jun 11, 2020

Uh oh!

yaooqinn Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaooqinn Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaooqinn commented Jun 11, 2020

Uh oh!

gengliangwang Jun 11, 2020

Choose a reason for hiding this comment

Uh oh!

yaooqinn Jun 11, 2020

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jun 11, 2020

Choose a reason for hiding this comment

Uh oh!

gengliangwang Jun 11, 2020

Choose a reason for hiding this comment

Uh oh!

yaooqinn Jun 11, 2020

Choose a reason for hiding this comment

Uh oh!

yaooqinn commented Jun 11, 2020

Uh oh!

SparkQA commented Jun 11, 2020

Uh oh!

cloud-fan commented Jun 11, 2020

Uh oh!

SparkQA commented Jun 11, 2020

Uh oh!

yaooqinn commented Jun 11, 2020

Uh oh!

SparkQA commented Jun 11, 2020

Uh oh!

SparkQA commented Jun 11, 2020

Uh oh!

yaooqinn commented Jun 12, 2020

Uh oh!

yaooqinn commented Jun 12, 2020

Uh oh!

SparkQA commented Jun 12, 2020

Uh oh!

yaooqinn commented Jun 12, 2020

Uh oh!

SparkQA commented Jun 12, 2020

Uh oh!

yaooqinn commented Jun 12, 2020

Uh oh!

yaooqinn commented Jun 12, 2020

Uh oh!

SparkQA commented Jun 12, 2020

Uh oh!

cloud-fan commented Jun 15, 2020

Uh oh!

dongjoon-hyun commented Jun 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jun 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun Jun 15, 2020

Choose a reason for hiding this comment

yaooqinn commented Jun 11, 2020 •

edited

Loading

yaooqinn Jun 11, 2020 •

edited

Loading

yaooqinn Jun 11, 2020 •

edited

Loading

dongjoon-hyun commented Jun 15, 2020 •

edited

Loading

dongjoon-hyun commented Jun 15, 2020 •

edited

Loading

dongjoon-hyun commented Jun 15, 2020 •

edited

Loading

dongjoon-hyun commented Jun 15, 2020 •

edited

Loading

dongjoon-hyun commented Jun 16, 2020 •

edited

Loading