[SPARK-13848][SPARK-5185] Update to Py4J 0.9.2 in order to fix classloading issue #11687
Conversation
|
I suppose we could also backport this change to Spark 1.6.x if there's interest, but for now I'm mostly concerned with fixing this in 2.0.0 because I believe that the bug that this fixes was causing other issues while trying to remove our tests' reliance on the Spark assembly. |
|
Test build #53038 has finished for PR 11687 at commit
|
|
Test build #53042 has finished for PR 11687 at commit
|
| return helperClass.newInstance() | ||
| except Py4JJavaError as e: | ||
| # TODO: use --jar once it also work on driver | ||
| if 'ClassNotFoundException' in str(e.java_exception): |
There was a problem hiding this comment.
Let's still keep this check. For other errors (e.g., the py4j java server is down), we should not call _printErrorMsg as it's confusing.
There was a problem hiding this comment.
I made this change because the call now fails with a different set of exceptions (such as "attempting to call a package") and wanted to err on the side of over-displaying the warning message. Let me try to figure out a narrower exception pattern match.
|
@zsxwing, updated, please take another look. |
|
Test build #53048 has finished for PR 11687 at commit
|
|
I'm so happy this issue is finally resolved on the Py4j side. Thanks @JoshRosen for the update. |
|
Test build #53074 has finished for PR 11687 at commit
|
|
LGTM |
|
Thanks @zsxwing. Merging this into master now. |
…oading issue This patch upgrades Py4J from 0.9.1 to 0.9.2 in order to include a patch which modifies Py4J to use the current thread's ContextClassLoader when performing reflection / class loading. This is necessary in order to fix [SPARK-5185](https://issues.apache.org/jira/browse/SPARK-5185), a longstanding issue affecting the use of `--jars` and `--packages` in PySpark. In order to demonstrate that the fix works, I removed the workarounds which were added as part of [SPARK-6027](https://issues.apache.org/jira/browse/SPARK-6027) / apache#4779 and other patches. Py4J diff: py4j/py4j@0.9.1...0.9.2 /cc zsxwing tdas davies brkyvz Author: Josh Rosen <[email protected]> Closes apache#11687 from JoshRosen/py4j-0.9.2.
…oading issue This patch upgrades Py4J from 0.9.1 to 0.9.2 in order to include a patch which modifies Py4J to use the current thread's ContextClassLoader when performing reflection / class loading. This is necessary in order to fix [SPARK-5185](https://issues.apache.org/jira/browse/SPARK-5185), a longstanding issue affecting the use of `--jars` and `--packages` in PySpark. In order to demonstrate that the fix works, I removed the workarounds which were added as part of [SPARK-6027](https://issues.apache.org/jira/browse/SPARK-6027) / apache#4779 and other patches. Py4J diff: py4j/py4j@0.9.1...0.9.2 /cc zsxwing tdas davies brkyvz Author: Josh Rosen <[email protected]> Closes apache#11687 from JoshRosen/py4j-0.9.2.
4 participants
This patch upgrades Py4J from 0.9.1 to 0.9.2 in order to include a patch which modifies Py4J to use the current thread's ContextClassLoader when performing reflection / class loading. This is necessary in order to fix SPARK-5185, a longstanding issue affecting the use of
--jarsand--packagesin PySpark.In order to demonstrate that the fix works, I removed the workarounds which were added as part of SPARK-6027 / #4779 and other patches.
Py4J diff: py4j/py4j@0.9.1...0.9.2
/cc @zsxwing @tdas @davies @brkyvz