[SPARK-11322][PySpark] Keep full stack trace in captured exception

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-11322

As reported by @JoshRosen in databricks/spark-redshift/issues/89, the exception-masking behavior sometimes makes debugging harder. To deal with this issue, we should keep full stack trace in the captured exception.

[SparkQA]

Test build #44371 has finished for PR 9283 at commit 0ccb4ae.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44395 has finished for PR 9283 at commit 3769f01.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class CapturedException(Exception):\n * class AnalysisException(CapturedException):\n * class IllegalArgumentException(CapturedException):\n

[SparkQA]

Test build #44398 has finished for PR 9283 at commit 18c1695.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class CapturedException(Exception):\n * class AnalysisException(CapturedException):\n * class IllegalArgumentException(CapturedException):\n

[SparkQA]

Test build #44402 has finished for PR 9283 at commit 8f69480.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class CapturedException(Exception):\n * class AnalysisException(CapturedException):\n * class IllegalArgumentException(CapturedException):\n

[viirya]

ping @davies any comments about this? Is it in right direction?

[davies]

@viirya Not sure, could you post an example how the full stack trace looks like?

[viirya]

When the exception is thrown from JVM, we see a simplified description:

>>> df.select(sha2(df.a, 1024)).collect()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hduser/repos/spark/python/pyspark/sql/functions.py", line 943, in sha2
    jc = sc._jvm.functions.sha2(_to_java_column(col), numBits)
  File "/home/hduser/repos/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
  File "/home/hduser/repos/spark/python/pyspark/sql/utils.py", line 52, in deco
    raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.IllegalArgumentException: u'requirement failed: numBits 1024 is not in the permitted values (0, 224, 256, 384, 512)'

If we catch the exception, we can print its full stack trace:

>>> from pyspark.sql.utils import AnalysisException, IllegalArgumentException
>>> try:
...     df.select(sha2(df.a, 1024)).collect()
... except IllegalArgumentException as e:
...     print e.stackTrace
... 
scala.Predef$.require(Predef.scala:233)
org.apache.spark.sql.functions$.sha2(functions.scala:1596)
org.apache.spark.sql.functions.sha2(functions.scala)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:483)
py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
py4j.Gateway.invoke(Gateway.java:259)
py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
py4j.commands.CallCommand.execute(CallCommand.java:79)
py4j.GatewayConnection.run(GatewayConnection.java:209)
java.lang.Thread.run(Thread.java:745)

[davies]

'\n\t at ', make it looks like a Java stacktrace

[SparkQA]

Test build #44513 has finished for PR 9283 at commit 11d6f09.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44514 has finished for PR 9283 at commit f4cae6c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class CapturedException(Exception):\n * class AnalysisException(CapturedException):\n * class IllegalArgumentException(CapturedException):\n

[davies]

another space after at?

[viirya]

yap.

[SparkQA]

Test build #44540 has finished for PR 9283 at commit 8d48e13.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class CapturedException(Exception):\n * class AnalysisException(CapturedException):\n * class IllegalArgumentException(CapturedException):\n

[SparkQA]

Test build #44545 has finished for PR 9283 at commit 8535f06.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class CapturedException(Exception):\n * class AnalysisException(CapturedException):\n * class IllegalArgumentException(CapturedException):\n

[viirya]

@davies Thanks. I think this is ready now.