Skip to content

[SPARK-30151][SQL] Issue better error message when user-specified schema mismatched #26781

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 12 commits into from
Closed

[SPARK-30151][SQL] Issue better error message when user-specified schema mismatched #26781

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
468660f
mismatch schema
Ngone51 Dec 6, 2019
9847847
update
Ngone51 Dec 6, 2019
aab2391
update again
Ngone51 Dec 6, 2019
a4f3122
improve error message
Ngone51 Dec 6, 2019
dd45804
updated tests
Ngone51 Dec 6, 2019
de036b6
reword error msg
Ngone51 Dec 9, 2019
8fac1fa
revert tests
Ngone51 Dec 9, 2019
6700dee
update tests
Ngone51 Dec 9, 2019
f54dea9
remove TestJdbcRelationProvider
Ngone51 Dec 9, 2019
ffdff16
revert import
Ngone51 Dec 9, 2019
c2b5eea
remove unneeded string interpolator
Ngone51 Dec 9, 2019
814821a
address comment
Ngone51 Dec 10, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -343,7 +343,12 @@ case class DataSource(
val baseRelation =
dataSource.createRelation(sparkSession.sqlContext, caseInsensitiveOptions)
if (baseRelation.schema != schema) {
throw new AnalysisException(s"$className does not allow user-specified schemas.")
throw new AnalysisException(
"The user-specified schema doesn't match the actual schema: " +
s"user-specified: ${schema.toDDL}, actual: ${baseRelation.schema.toDDL}. If " +
"you're using DataFrameReader.schema API or creating a table, please do not " +
"specify the schema. Or if you're scanning an existed table, please drop " +
"it and re-create it.")
}
baseRelation

Expand Down
5 changes: 3 additions & 2 deletions sql/core/src/test/scala/org/apache/spark/sql/sources/TableScanSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,7 @@ class TableScanSuite extends DataSourceTest with SharedSparkSession {
// Make sure we do throw correct exception when users use a relation provider that
// only implements the RelationProvider or the SchemaRelationProvider.
Seq("TEMPORARY VIEW", "TABLE").foreach { tableType =>
val schemaNotAllowed = intercept[Exception] {
val schemaNotMatch = intercept[Exception] {
Copy link
Member Author

@Ngone51 Ngone51 Dec 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the variable name to make it be more readable according to current error message.

sql(
s"""
|CREATE $tableType relationProvierWithSchema (i int)
Expand All @@ -369,7 +369,8 @@ class TableScanSuite extends DataSourceTest with SharedSparkSession {
|)
""".stripMargin)
}
assert(schemaNotAllowed.getMessage.contains("does not allow user-specified schemas"))
assert(schemaNotMatch.getMessage.contains(
"The user-specified schema doesn't match the actual schema"))

val schemaNeeded = intercept[Exception] {
sql(
Expand Down
5 changes: 2 additions & 3 deletions sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -489,11 +489,10 @@ class DataFrameReaderWriterSuite extends QueryTest with SharedSparkSession with
// when users do not specify the schema
checkAnswer(dfReader.load(), spark.range(1, 11).toDF())

// when users specify the schema
// when users specify a wrong schema
val inputSchema = new StructType().add("s", IntegerType, nullable = false)
val e = intercept[AnalysisException] { dfReader.schema(inputSchema).load() }
assert(e.getMessage.contains(
"org.apache.spark.sql.sources.SimpleScanSource does not allow user-specified schemas"))
assert(e.getMessage.contains("The user-specified schema doesn't match the actual schema"))
}

test("read a data source that does not extend RelationProvider") {
Expand Down