[SPARK-2059][SQL] Add analysis checks #1265

rxin · 2014-06-30T00:08:31Z

This replaces #1263 with a test case.

[SPARK-2059][SQL] Add analysis checks

marmbrus · 2014-06-30T00:09:36Z

LGTM

AmplabJenkins · 2014-06-30T00:10:35Z

Merged build triggered.

AmplabJenkins · 2014-06-30T00:10:45Z

Merged build started.

AmplabJenkins · 2014-06-30T02:01:57Z

Merged build finished.

AmplabJenkins · 2014-06-30T02:01:57Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16250/

rxin · 2014-06-30T06:09:49Z

This change actually exposes two other problems ...

-- TOK_ALLCOLREF
explain select * from (select a.key, a.* from (select * from src) a join (select * from src1) b on (a.key = b.key)) t;
-- DOT
explain select * from (select a.key, a.`[k].*` from (select * from src) a join (select * from src1) b on (a.key = b.key)) t;
-- EXPRESSION
explain select * from (select a.key, a.key from (select * from src) a join (select * from src1) b on (a.key = b.key)) t;

[info] - ambiguous_col *** FAILED ***
[info]   Failed to execute query using catalyst:
[info]   Error: Unresolved attributes: *, tree:
[info]   Project [*]
[info]    Subquery t
[info]     Project [key#1302,'a.[k].*]
[info]      Join Inner, Some((key#1302 = key#1311))
[info]       Subquery a
[info]        Project [key#1302,value#1303]
[info]         LowerCaseSchema 
[info]          InMemoryRelation [key#1302,value#1303], false, (HiveTableScan [key#1302,value#1303], (MetastoreRelation default, src, None), None)
[info]       Subquery b
[info]        Project [key#1311,value#1312]
[info]         LowerCaseSchema 
[info]          InMemoryRelation [key#1311,value#1312], false, (HiveTableScan [key#1311,value#1312], (MetastoreRelation default, src1, None), None)
[info]   
[info]   org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved attributes: *, tree:
[info]   Project [*]
[info]    Subquery t
[info]     Project [key#1302,'a.[k].*]
[info]      Join Inner, Some((key#1302 = key#1311))
[info]       Subquery a
[info]        Project [key#1302,value#1303]
[info]         LowerCaseSchema 
[info]          InMemoryRelation [key#1302,value#1303], false, (HiveTableScan [key#1302,value#1303], (MetastoreRelation default, src, None), None)
[info]       Subquery b
[info]        Project [key#1311,value#1312]
[info]         LowerCaseSchema 
[info]          InMemoryRelation [key#1311,value#1312], false, (HiveTableScan [key#1311,value#1312], (MetastoreRelation default, src1, None), None)

-- HIVE-2477 Use name of original expression for name of CAST output
explain select key from (select cast(key as int) from src )t;

--backward
explain select key2 from (select cast(key as int) key2 from src )t;


[info] - alias_casted_column *** FAILED ***
[info]   Failed to execute query using catalyst:
[info]   Error: Unresolved attributes: 'key, tree:
[info]   Project ['key]
[info]    Subquery t
[info]     Project [CAST(key#630, IntegerType) AS c_0#636]
[info]      LowerCaseSchema 
[info]       InMemoryRelation [key#630,value#631], false, (HiveTableScan [key#630,value#631], (MetastoreRelation default, src, None), None)
[info]   
[info]   org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved attributes: 'key, tree:
[info]   Project ['key]
[info]    Subquery t
[info]     Project [CAST(key#630, IntegerType) AS c_0#636]
[info]      LowerCaseSchema 
[info]       InMemoryRelation [key#630,value#631], false, (HiveTableScan [key#630,value#631], (MetastoreRelation default, src, None), None)
[info]

liancheng · 2014-07-03T20:30:07Z

HiveComparisonTest simply ignores all ExplainCommand and don't check their output. We should catch the exception in execution.ExplainCommand and report the exception stacktrace as part of the plan explanation rather than throw it out.

This is a fix for the problem revealed by [PR apache#1265](apache#1265)

liancheng · 2014-07-03T20:53:45Z

PR #1294 should fix this problem, after merger #1294, this one can also be merged safely.

…Command` This is a fix for the problem revealed by PR #1265. Currently `HiveComparisonSuite` ignores output of `ExplainCommand` since Catalyst query plan is quite different from Hive query plan. But exceptions throw from `CheckResolution` still breaks test cases. This PR catches any `TreeNodeException` and reports it as part of the query explanation. After merging this PR, PR #1265 can also be merged safely. For a normal query: ``` scala> hql("explain select key from src").foreach(println) ... [Physical execution plan:] [HiveTableScan [key#9], (MetastoreRelation default, src, None), None] ``` For a wrong query with unresolved attribute(s): ``` scala> hql("explain select kay from src").foreach(println) ... [Error occurred during query planning: ] [Unresolved attributes: 'kay, tree:] [Project ['kay]] [ LowerCaseSchema ] [ MetastoreRelation default, src, None] ``` Author: Cheng Lian <[email protected]> Closes #1294 from liancheng/safe-explain and squashes the following commits: 4318911 [Cheng Lian] Don't throw TreeNodeException in `execution.ExplainCommand` (cherry picked from commit 5448804) Signed-off-by: Reynold Xin <[email protected]>

…Command` This is a fix for the problem revealed by PR #1265. Currently `HiveComparisonSuite` ignores output of `ExplainCommand` since Catalyst query plan is quite different from Hive query plan. But exceptions throw from `CheckResolution` still breaks test cases. This PR catches any `TreeNodeException` and reports it as part of the query explanation. After merging this PR, PR #1265 can also be merged safely. For a normal query: ``` scala> hql("explain select key from src").foreach(println) ... [Physical execution plan:] [HiveTableScan [key#9], (MetastoreRelation default, src, None), None] ``` For a wrong query with unresolved attribute(s): ``` scala> hql("explain select kay from src").foreach(println) ... [Error occurred during query planning: ] [Unresolved attributes: 'kay, tree:] [Project ['kay]] [ LowerCaseSchema ] [ MetastoreRelation default, src, None] ``` Author: Cheng Lian <[email protected]> Closes #1294 from liancheng/safe-explain and squashes the following commits: 4318911 [Cheng Lian] Don't throw TreeNodeException in `execution.ExplainCommand`

…Command` This is a fix for the problem revealed by PR #1265. Currently `HiveComparisonSuite` ignores output of `ExplainCommand` since Catalyst query plan is quite different from Hive query plan. But exceptions throw from `CheckResolution` still breaks test cases. This PR catches any `TreeNodeException` and reports it as part of the query explanation. After merging this PR, PR #1265 can also be merged safely. For a normal query: ``` scala> hql("explain select key from src").foreach(println) ... [Physical execution plan:] [HiveTableScan [key#9], (MetastoreRelation default, src, None), None] ``` For a wrong query with unresolved attribute(s): ``` scala> hql("explain select kay from src").foreach(println) ... [Error occurred during query planning: ] [Unresolved attributes: 'kay, tree:] [Project ['kay]] [ LowerCaseSchema ] [ MetastoreRelation default, src, None] ``` Author: Cheng Lian <[email protected]> Closes #1294 from liancheng/safe-explain and squashes the following commits: 4318911 [Cheng Lian] Don't throw TreeNodeException in `execution.ExplainCommand` (cherry picked from commit 5448804) Signed-off-by: Reynold Xin <[email protected]>

rxin · 2014-07-04T06:43:04Z

Jenkins, retest this please.

AmplabJenkins · 2014-07-04T06:45:51Z

Merged build triggered.

AmplabJenkins · 2014-07-04T06:45:56Z

Merged build started.

rxin · 2014-07-04T07:53:31Z

Ok looks like those two tests are now passing. I'm merging this in master & branch-1.0.

This replaces #1263 with a test case. Author: Reynold Xin <[email protected]> Author: Michael Armbrust <[email protected]> Closes #1265 from rxin/sql-analysis-error and squashes the following commits: a639e01 [Reynold Xin] Added a test case for unresolved attribute analysis. 7371e1b [Reynold Xin] Merge pull request #1263 from marmbrus/analysisChecks 448c088 [Michael Armbrust] Add analysis checks (cherry picked from commit b3e768e) Signed-off-by: Reynold Xin <[email protected]>

AmplabJenkins · 2014-07-04T08:23:53Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-07-04T08:23:53Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16340/

…Command` This is a fix for the problem revealed by PR apache#1265. Currently `HiveComparisonSuite` ignores output of `ExplainCommand` since Catalyst query plan is quite different from Hive query plan. But exceptions throw from `CheckResolution` still breaks test cases. This PR catches any `TreeNodeException` and reports it as part of the query explanation. After merging this PR, PR apache#1265 can also be merged safely. For a normal query: ``` scala> hql("explain select key from src").foreach(println) ... [Physical execution plan:] [HiveTableScan [key#9], (MetastoreRelation default, src, None), None] ``` For a wrong query with unresolved attribute(s): ``` scala> hql("explain select kay from src").foreach(println) ... [Error occurred during query planning: ] [Unresolved attributes: 'kay, tree:] [Project ['kay]] [ LowerCaseSchema ] [ MetastoreRelation default, src, None] ``` Author: Cheng Lian <[email protected]> Closes apache#1294 from liancheng/safe-explain and squashes the following commits: 4318911 [Cheng Lian] Don't throw TreeNodeException in `execution.ExplainCommand`

This replaces apache#1263 with a test case. Author: Reynold Xin <[email protected]> Author: Michael Armbrust <[email protected]> Closes apache#1265 from rxin/sql-analysis-error and squashes the following commits: a639e01 [Reynold Xin] Added a test case for unresolved attribute analysis. 7371e1b [Reynold Xin] Merge pull request apache#1263 from marmbrus/analysisChecks 448c088 [Michael Armbrust] Add analysis checks

…1265) * [CARMEL-6568] Analyze join operator and support data expansion check * persist db config

marmbrus and others added 3 commits June 29, 2014 11:43

Add analysis checks

448c088

Merge pull request apache#1263 from marmbrus/analysisChecks

7371e1b

[SPARK-2059][SQL] Add analysis checks

Added a test case for unresolved attribute analysis.

a639e01

liancheng added a commit to liancheng/spark that referenced this pull request Jul 3, 2014

Don't throw TreeNodeException in execution.ExplainCommand

4318911

This is a fix for the problem revealed by [PR apache#1265](apache#1265)

liancheng mentioned this pull request Jul 3, 2014

[SPARK-2059][SQL] Don't throw TreeNodeException in execution.ExplainCommand #1294

Closed

asfgit closed this in b3e768e Jul 4, 2014

rxin deleted the sql-analysis-error branch July 4, 2014 07:54

wangyum pushed a commit that referenced this pull request May 26, 2023

[CARMEL-6568] Analyze join operator and support data expansion check (#…

10a2794

…1265) * [CARMEL-6568] Analyze join operator and support data expansion check * persist db config

[SPARK-2059][SQL] Add analysis checks #1265

[SPARK-2059][SQL] Add analysis checks #1265

Uh oh!

Conversation

rxin commented Jun 30, 2014

Uh oh!

marmbrus commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

rxin commented Jun 30, 2014

Uh oh!

liancheng commented Jul 3, 2014

Uh oh!

liancheng commented Jul 3, 2014

Uh oh!

rxin commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

rxin commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants