[SPARK-13180] Protect against SessionState being null when accessing HiveClientImpl#conf

[tedyu]

See http://search-hadoop.com/m/q3RTtFoTDi2HVCrM1 for related stack trace

java.lang.NullPointerException
at org.apache.spark.sql.hive.client.ClientWrapper.conf(ClientWrapper.scala:205)
at org.apache.spark.sql.hive.HiveContext.hiveconf$lzycompute(HiveContext.scala:552)
at org.apache.spark.sql.hive.HiveContext.hiveconf(HiveContext.scala:551)
at org.apache.spark.sql.hive.HiveContext$$anonfun$configure$1.apply(HiveContext.scala:538)
at org.apache.spark.sql.hive.HiveContext$$anonfun$configure$1.apply(HiveContext.scala:537)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at org.apache.spark.sql.hive.HiveContext.configure(HiveContext.scala:537)
at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:250)
at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:237)
at org.apache.spark.sql.hive.HiveContext$$anon$2.<init>(HiveContext.scala:457)
at org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:457)
at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:456)
at org.apache.spark.sql.hive.HiveContext$$anon$3.<init>(HiveContext.scala:473)
at org.apache.spark.sql.hive.HiveContext.analyzer$lzycompute(HiveContext.scala:473)
at org.apache.spark.sql.hive.HiveContext.analyzer(HiveContext.scala:472)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:34)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:133)
at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52)
at org.apache.spark.sql.SQLContext.baseRelationToDataFrame(SQLContext.scala:442)
at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:146)

[SparkQA]

Test build #50703 has finished for PR 11066 at commit 6ee7d98.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

This is an exact copy of https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala#L171-L182.

Can you refactor this?

[tedyu]

The method in HiveQl is marked private[this].
HiveClientImpl doesn't reference HiveQl.

HiveShim.scala and SparkSQLCLIDriver.scala refer to SessionState but they don't seem to be suitable place to put the common code.

Do you have suggestion on which class to put the common code in ?

Thanks

[hvanhovell]

@tedyu lets create a HiveConfUtil object in the client package, put the method in and use that in both places.

[SparkQA]

Test build #50749 has finished for PR 11066 at commit aff16d6.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #50750 has finished for PR 11066 at commit 438e2a3.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #50751 has finished for PR 11066 at commit 0a43c19.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #50754 has finished for PR 11066 at commit f20a61f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tedyu]

This error surfaced:

[info]   2016-02-04 10:15:21.319 - stderr> Exception in thread "main" java.lang.LinkageError: loader constraint violation: when resolving method "org.apache.spark.sql.hive.      client.HiveConfUtil$.conf()Lorg/apache/hadoop/hive/conf/HiveConf;"
 the class loader (instance of org/apache/spark/sql/hive/client/IsolatedClientLoader$$anon$1) of the current    class, org/apache/spark/sql/hive/client/HiveClientImpl, and the class loader
(instance of sun/misc/Launcher$AppClassLoader) for resolved class, org/apache/spark/sql/hive/client/ HiveConfUtil$, have different Class objects for the type /hadoop/hive/conf/HiveConf; used in the signature

[tedyu]

I wonder if rev 1 should be used:

the utility class only saves less than 10 lines of code
intricacies were introduced w.r.t. class loading

[tedyu]

@hvanhovell
What do you think ?

[SparkQA]

Test build #50827 has finished for PR 11066 at commit 8b9e24c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tedyu]

@davies
Can you take a look ?

[davies]

@tedyu This seems like a workaround. Have you find the root cause? Maybe there is a place that we didn't set the SessionState for current thread.

[tedyu]

From HiveContext.scala , line 552 (the last line below) which appeared in the stack trace:

   * 1. create a new SessionState for each HiveContext
   * 2. when the Hive session is first initialized, params in HiveConf will get picked up by the
   *    SQLConf.  Additionally, any properties set by set() or a SET command inside sql() will be
   *    set in the SQLConf *as well as* in the HiveConf.
   */
  @transient
  protected[hive] lazy val hiveconf: HiveConf = {
    val c = executionHive.conf

If I read the doc correctly, the above call is supposed to create new SessionState which is what the PR is doing.

[davies]

The doc is out-updated, HiveConf is created in HiveClientImp.scla LINE 107 (when created HiveClientImp obj).

[tedyu]

The user was using 1.6.0 where there was no HiveClientImp.scala
In ClientWrapper.scala (1.6.0), HiveConf is created at line 175

[rxin]

Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one. We can also continue the discussion on the JIRA ticket.

Note: Looks like this code path has been rewritten heavily and the problem might not apply anymore.