Skip to content

[SPARK-16368] [SQL] Fix Strange Errors When Creating View With Unmatched Column Num #14047

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

[SPARK-16368] [SQL] Fix Strange Errors When Creating View With Unmatched Column Num #14047

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,11 @@ case class CreateViewCommand(
qe.assertAnalyzed()
val analyzedPlan = qe.analyzed

require(tableDesc.schema == Nil || tableDesc.schema.length == analyzedPlan.output.length)
if (tableDesc.schema != Nil && tableDesc.schema.length != analyzedPlan.output.length) {
throw new AnalysisException(s"The number of columns produced by the SELECT clause " +
s"(num: `${analyzedPlan.output.length}`) does not match the number of column names " +
s"specified by CREATE VIEW (num: `${tableDesc.schema.length}`).")
}
val sessionState = sparkSession.sessionState

if (isTemporary) {
Expand Down
23 changes: 23 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1314,6 +1314,29 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
}
}

test("create temporary view with mismatched schema") {
withTable("tab1") {
spark.range(10).write.saveAsTable("tab1")
withView("view1") {
val e = intercept[AnalysisException] {
sql("CREATE TEMPORARY VIEW view1 (col1, col3) AS SELECT * FROM tab1")
}.getMessage
assert(e.contains("the SELECT clause (num: `1`) does not match")
&& e.contains("CREATE VIEW (num: `2`)"))
}
}
}

test("create temporary view with specified schema") {
withView("view1") {
sql("CREATE TEMPORARY VIEW view1 (col1, col2) AS SELECT 1, 2")
checkAnswer(
sql("SELECT * FROM view1"),
Row(1, 2) :: Nil
)
}
}

test("truncate table - external table, temporary table, view (not allowed)") {
import testImplicits._
val path = Utils.createTempDir().getAbsolutePath
Expand Down
23 changes: 23 additions & 0 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -391,6 +391,29 @@ class HiveDDLSuite
}
}

test("create view with mismatched schema") {
Copy link
Contributor

@jaceklaskowski jaceklaskowski Jul 4, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you copy the tests? Could you file a JIRA issue to get rid of the duplication at the very least? Thanks.

Copy link
Member Author

@gatorsmile gatorsmile Jul 4, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is for CREATE VIEW Another is for CREATE TEMP VIEW. They are testing different code paths.

Copy link
Member Author

@gatorsmile gatorsmile Jul 4, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without enabling Hive supports, we are unable to CREATE VIEW and then SELECT VIEW.

Copy link
Contributor

@jaceklaskowski jaceklaskowski Jul 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. You're right. I must've overlooked this no-so-little difference. Is temporary the only difference between the tests? Could we have a template test method to generate proper versions for the case? (even if it were too much for this change, it'd pave the way for more tests like this in the future). We could also have a follow-up change after this one is committed. WDYT @srowen @rxin? Is this worth the effort? I'm concerned with this duplication.

Copy link
Member Author

@gatorsmile gatorsmile Jul 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eventually, we will remove the functionality gap. At that moment, we need to clean the test suites and combine all the duplicate test cases.

withTable("tab1") {
spark.range(10).write.saveAsTable("tab1")
withView("view1") {
val e = intercept[AnalysisException] {
sql("CREATE VIEW view1 (col1, col3) AS SELECT * FROM tab1")
}.getMessage
assert(e.contains("the SELECT clause (num: `1`) does not match")
&& e.contains("CREATE VIEW (num: `2`)"))
}
}
}

test("create view with specified schema") {
withView("view1") {
sql("CREATE VIEW view1 (col1, col2) AS SELECT 1, 2")
checkAnswer(
sql("SELECT * FROM view1"),
Row(1, 2) :: Nil
)
}
}

test("desc table for Hive table") {
withTable("tab1") {
val tabName = "tab1"
Expand Down