[SPARK-17710][HOTFIX] Fix ClassCircularityError in ReplSuite tests in Maven build: use 'Class.forName' instead of 'Utils.classForName' #15286
Conversation
… in Maven build: use 'Class.forName' instead of 'Utils.classForName'
|
please change [Follow UP] to [HOTFIX] |
weiqingy
changed the title
|
Also please file a separate jira to investigate more. Link it to https://issues.apache.org/jira/browse/SPARK-15857 and if needed we pull in people more familiar with the Repl classloader stuff. |
|
Yes. The title has been changed. Thanks. @tgravescs |
|
changes look fine, waiting on Jenkins. |
|
Can you change the JIRA number to SPARK-17710, which is the JIRA that I created for this build break? |
weiqingy
changed the title
|
@JoshRosen Yes. The title has been changed. |
|
Test build #66066 has finished for PR 15286 at commit
|
|
committed this to master |
|
Thank you very much. @tgravescs @JoshRosen |
| try { | ||
| val callerContext = Utils.classForName("org.apache.hadoop.ipc.CallerContext") | ||
| val Builder = Utils.classForName("org.apache.hadoop.ipc.CallerContext$Builder") | ||
| // scalastyle:off classforname |
There was a problem hiding this comment.
I think it's important to document why we are not using Utils.classForName here. Otherwise somebody is going to come in the future and get confused or change it.
There was a problem hiding this comment.
Sure. I will submit a PR for this.
5 participants
What changes were proposed in this pull request?
Fix ClassCircularityError in ReplSuite tests when Spark is built by Maven build.
How was this patch tested?
(1)
Then test:
ReplSuite tests passed
(2)
Manual Tests against some Spark applications in Yarn client mode and Yarn cluster mode. Need to check if spark caller contexts are written into HDFS hdfs-audit.log and Yarn RM audit log successfully.