Skip to content

[SPARK-29015][SQL][test-hadoop3.2]Reset class loader after initializing SessionState for built-in Hive 2.3 #25775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 24 commits into from
Closed

[SPARK-29015][SQL][test-hadoop3.2]Reset class loader after initializing SessionState for built-in Hive 2.3 #25775

Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
c3de557
save change
AngersZhuuuu Sep 10, 2019
2cf3153
Revert "save change"
AngersZhuuuu Sep 10, 2019
e5cd06c
Merge remote-tracking branch 'upstream/master'
AngersZhuuuu Sep 10, 2019
569ab8a
Merge remote-tracking branch 'upstream/master'
AngersZhuuuu Sep 12, 2019
edaf402
fix bug of jdk11
AngersZhuuuu Sep 12, 2019
7ec01e4
Revert "fix bug of jdk11"
AngersZhuuuu Sep 12, 2019
098a432
fix only this point
AngersZhuuuu Sep 12, 2019
07dd32e
still detach SessionState
AngersZhuuuu Sep 13, 2019
35f0a01
remove unused import
AngersZhuuuu Sep 13, 2019
fb39a53
fix in HiveTableScanExec
AngersZhuuuu Sep 13, 2019
a510a23
reset origin classloader
AngersZhuuuu Sep 14, 2019
bc73663
fix scala style
AngersZhuuuu Sep 14, 2019
f772e49
use sharedstate's classloader
AngersZhuuuu Sep 15, 2019
c213b07
fix null SessionState
AngersZhuuuu Sep 15, 2019
c751ce6
save code
AngersZhuuuu Sep 15, 2019
6e70b74
add classloader when doexecute in HiveTableScanExec
AngersZhuuuu Sep 15, 2019
a02d0b6
change back code
AngersZhuuuu Sep 15, 2019
01122e3
remove empty line
AngersZhuuuu Sep 15, 2019
340e0b6
remove space
AngersZhuuuu Sep 15, 2019
35c961f
solve this when construct HiveClientImpl's SessionState.
AngersZhuuuu Sep 16, 2019
f30d001
fix and add comment
AngersZhuuuu Sep 17, 2019
4db3b54
use clientLoader.classLoader
AngersZhuuuu Sep 17, 2019
8e17346
fix hive version wrong in comment
AngersZhuuuu Sep 25, 2019
f37335c
change comment and add Hive Version condition
AngersZhuuuu Sep 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ private[hive] class HiveClientImpl(
// HiveConf is a Hadoop Configuration, which has a field of classLoader and
// the initial value will be the current thread's context class loader
// (i.e. initClassLoader at here).
// We call initialConf.setClassLoader(initClassLoader) at here to make
// We call hiveConf.setClassLoader(initClassLoader) at here to make
// this action explicit.
hiveConf.setClassLoader(initClassLoader)

Expand Down Expand Up @@ -190,6 +190,13 @@ private[hive] class HiveClientImpl(
if (clientLoader.cachedHive != null) {
Hive.set(clientLoader.cachedHive.asInstanceOf[Hive])
}
// Hive 2.3 will set UDFClassLoader to hiveConf when initializing SessionState
// since HIVE-11878, and ADDJarCommand will add jars to clientLoader.classLoader.
// For this reason we cannot load the jars added by ADDJarCommand because of class loader
// got changed. We reset it to clientLoader.ClassLoader here.
if (HiveUtils.isHive23) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not set the classLoader blindly to clientLoader.classLoader? HIVE-11878 got merged into hive 1.3.0 and 2.0.0? This issue can still exist in the production environment when we try to use non-builtin hive libraries.

cc @ulysses-you @cloud-fan @wangyum @AngersZhuuuu

state.getConf.setClassLoader(clientLoader.classLoader)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gatorsmile @srowen @HyukjinKwon @dongjoon-hyun @juliuszsompolski Do you have more comments?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No opinion from me, I know too little context of how these classloaders should be used to review.

SessionState.start(state)
state.out = new PrintStream(outputBuffer, true, UTF_8.name())
state.err = new PrintStream(outputBuffer, true, UTF_8.name())
Expand Down