[SPARK-11188][SQL] Elide stacktraces in bin/spark-sql for AnalysisExceptions #9194
Conversation
|
ok to test |
There was a problem hiding this comment.
I would just say, "Error in query: ${e.message}" or something. It will always be org.apache.spark.sql.AnalysisException and we want to make it clear that is likely the users fault in this case (i.e. not a bug).
There was a problem hiding this comment.
Sure Michael. I will make the change.
|
Test build #44071 has finished for PR 9194 at commit
|
|
You should also be able to add a test in Regarding the Hive Version concern. I believe this is only going to touch the execution version of Hive (as opposed to the Metastore connection version that we load in an isolated classloader). As such it should always be Hive 1.2.1 |
|
Test build #44131 has finished for PR 9194 at commit
|
|
@marmbrus What can we do here ? Please advice .. |
|
retest this please. |
|
@marmbrus |
|
test this please |
|
you need to be on the whitelist for jenkins to respond to you |
|
Test build #44169 has finished for PR 9194 at commit
|
|
Test build #44347 has finished for PR 9194 at commit
|
There was a problem hiding this comment.
This is only testing that you message is there, not that the stack traces are elided. There is still a giant stack trace upon failure:
spark-sql> SELECT * FROM test;
15/10/26 10:34:27 INFO ParseDriver: Parsing command: SELECT * FROM test
15/10/26 10:34:27 INFO ParseDriver: Parse Completed
15/10/26 10:34:27 INFO HiveMetaStore: 0: get_table : db=default tbl=test
15/10/26 10:34:27 INFO audit: ugi=marmbrus ip=unknown-ip-addr cmd=get_table : db=default tbl=test
15/10/26 10:34:27 ERROR SparkSQLDriver: Failed in [SELECT * FROM test]
org.apache.spark.sql.AnalysisException: no such table test; line 1 pos 14
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.getTable(Analyzer.scala:260)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$7.applyOrElse(Analyzer.scala:268)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$7.applyOrElse(Analyzer.scala:264)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:51)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:56)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$1.apply(LogicalPlan.scala:54)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$1.apply(LogicalPlan.scala:54)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:249)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:279)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:54)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:264)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:254)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:83)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:80)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:80)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:72)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:72)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:41)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:41)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:39)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:132)
at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:783)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:61)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:308)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:226)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:680)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Error in query: no such table test; line 1 pos 14
There was a problem hiding this comment.
@marmbrus
Thanks for your quick feedback. Actually this is one of the question i was asking earlier.
- The actual exception is also logged in run method of the SparkSQLDriver. If the
logging is set to CONSOLE then the exception is printed on the screen. I hope this is ok ?
I think by default the logging is enabled to console or stdout. If i change the log4j.properties and direct the logging to a file , then we should not see a stack trace.
Michael, do you want this logging completely suppressed for AnalysisException ?
There was a problem hiding this comment.
@marmbrus
Hi Michael, given that logging to console is default , i have made changes to suppress the logging to console for analysis exceptions. One option is to completely stop logging these kind of exceptions. But i have tried to log them to non-console appenders. Please let me know what you think.
|
Test build #44454 has finished for PR 9194 at commit
|
|
As a side note, I think the target user of |
|
@marmbrus However i like your suggestion of logging it as DEBUG level. Its much simpler and if users have set to log at this level then it is ok for them to see this exception on the console (if they have selected a console appender). I will make the change per your suggestion. |
|
@marmbrus |
|
Test build #44543 has finished for PR 9194 at commit
|
|
This LGTM, can you update the title (remove WIP) and update the description (it will be come the commit message when merging). |
dilipbiswal
changed the title
|
@marmbrus Took out "WIP" and also changed the description. |
|
Thanks for doing this! It would be nice to have this in branch-1.4 and branch-1.5. Would you be able to open up pull requests against those branches as well? This patch does not apply cleanly. |
|
@marmbrus Thanks a LOT for your comments and guidance. I am still little new to the process. So do i have to open two JIRAs one for 1.4 and the other for 1.5 and associate the new pull requests against them ? |
|
I'm fine with using the existing JIRA. I've reopened it.
|
3 participants
Only print the error message to the console for Analysis Exceptions in sql-shell.