[SPARK-8306] [SQL] AddJar command needs to set the new class loader to the HiveConf inside executionHive.state.

[yhuai]

https://issues.apache.org/jira/browse/SPARK-8306

I will try to add a test later.

@marmbrus @aarondav

[yhuai]

hmm..Actually, setCurrentSessionState will use the class loader associated with the HiveConf set in the current session state to the context class loader of the current thread.

[yhuai]

Actually, the problem is the class loader associated with the HiveConf inside the ClientWrapper.state. For execution Hive, this classloader will be the one used to load HiveContext. Then, when we call withHiveState, we always call setCurrentSessionState with this state. Inside setCurrentSessionState, we are doing

public static void setCurrentSessionState(SessionState startSs) {
    tss.set(startSs);
    Thread.currentThread().setContextClassLoader(startSs.getConf().getClassLoader());
  }

Basically, we use the classloader of the conf associated with the state to set the context class loader of the current thread.

Inside AddJar command, although we are doing org.apache.hadoop.hive.ql.metadata.Hive.get().getConf().setClassLoader(newClassLoader), this one actually may not set the class loader to the conf associated with executionHive.state because org.apache.hadoop.hive.ql.metadata.Hive.get() returns a thread local variable. So, its conf may not point to the same conf inside the executionHive.state.

Also cc @JoshRosen

[yhuai]

Manually tested. Looks fine. Will try to add a test.

[SparkQA]

Test build #34669 timed out for PR 6758 at commit 5c8b794 after a configured wait of 175m.

[SparkQA]

Test build #34679 has finished for PR 6758 at commit 3d541fd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[aarondav]

This is very likely to work for almost all cases, but we should probably add a TODO or JIRA to clean up some of this stuff. For instance, here we may be setting the executionHive class loader to an entirely different class loader than it originally had (which is likely to work in typical situations, but is not easy to trace).

Additionally, the flow of class loader-setting inside ClientWrapper is very non-obvious, especially as it differs between Hive 12 and Hive 13 -- we should explicitly document this if not make it cleaner.

[SparkQA]

Test build #35055 has finished for PR 6758 at commit 6690a08.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35057 has finished for PR 6758 at commit 1292346.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UnaryPositive(child: Expression) extends UnaryArithmetic
  • // Add jar to isolated hive (metadataHive) class loader.