[SPARK-30151][SQL] Issue better error message when user-specified schema mismatched #26781

Ngone51 · 2019-12-06T15:27:22Z

What changes were proposed in this pull request?

Issue better error message when user-specified schema and not match relation schema

Why are the changes needed?

Inspired by #25248 (comment), user could get a weird error message when type mapping behavior change between Spark schema and datasource schema(e.g. JDBC). Instead of saying "SomeProvider does not allow user-specified schemas.", we'd better tell user what is really happening here to make user be more clearly about the error.

Does this PR introduce any user-facing change?

Yes, user will see error message changes.

How was this patch tested?

Updated existed tests.

Ngone51 · 2019-12-06T15:52:18Z

cc @cloud-fan @gatorsmile

SparkQA · 2019-12-06T16:05:41Z

Test build #114950 has finished for PR 26781 at commit dd45804.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Ngone51 · 2019-12-06T16:26:22Z

The failed test ThriftServerWithSparkContextSuite.SPARK-29911: Uncache cached tables when session closed looks unrelated and it passed locally.

Ngone51 · 2019-12-06T16:26:35Z

Jenkins, retest this please.

SparkQA · 2019-12-06T20:38:34Z

Test build #114955 has finished for PR 26781 at commit dd45804.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-12-07T01:36:14Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala

+            .unzip
+          if (persistentFields.nonEmpty) {
+            val errorMsg =
+              s"Mismatched fields detected between persistent schema and user specified schema: " +


nit: seems like we can remove ss.

Do you mean: fields -> filed ?

Nope, I meant the s for string interpolation (s"...)

HyukjinKwon · 2019-12-07T01:49:18Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala

+        val persistentSize = persistentSchema.size
+        val specifiedSize = specifiedSchema.size
+        if (persistentSize == specifiedSize) {
+          val (persistentFields, specifiedFields) = persistentSchema.zip(specifiedSchema)


If we're going to improve such error message case across the codebase, we might also think about having a common method (maybe something called assertEquality in StructType?) that checks each type recursively and shows a better message. Can we at least have a private method here for this case in the future?

Not sure whether we'd require this similar functionality in some cases in the future. But, maybe, we could still give it a try.

Yeah, I think it wont handle nested cases. There are other external data sources that support nested schema and the current code tells only root columns.

Also, there are many cases to show better error messages like this. E.g., StructType.merge or _merge_type in Python's schema inference (https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L1097-L1111)

see #19792 or #18521 as an example.

Hi @HyukjinKwon , after discussing with wenchen offline, we decide not to make it too complicated here. If schemas are detected not match, we simply show the whole schema to user rather than those mismatched fields as previously did. Please see de036b6.

Ngone51 · 2019-12-09T14:05:42Z

sql/core/src/test/scala/org/apache/spark/sql/sources/TableScanSuite.scala

    // only implements the RelationProvider or the SchemaRelationProvider.
    Seq("TEMPORARY VIEW", "TABLE").foreach { tableType =>
-      val schemaNotAllowed = intercept[Exception] {
+      val schemaNotMatch = intercept[Exception] {


Updated the variable name to make it be more readable according to current error message.

SparkQA · 2019-12-09T14:12:27Z

Test build #115037 has finished for PR 26781 at commit f54dea9.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class JdbcUtilsSuite extends SparkFunSuite

SparkQA · 2019-12-09T18:08:58Z

Test build #115040 has finished for PR 26781 at commit c2b5eea.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2019-12-10T03:34:32Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala

+            "you're using DataFrameReader.schema API or creating a table, please do not " +
+            "specify the schema. Or if you're scanning an existed table, please drop " +
+            "it and re-create it."
+          throw new AnalysisException(errorMsg)


nit: format like this?

throw new AnalysisException("The user-specified schema doesn't match the actual schema: " + s"user-specified: ${schema.toDDL}, actual: ${baseRelation.schema.toDDL}. If " + "you're using DataFrameReader.schema API or creating a table, please do not " + "specify the schema. Or if you're scanning an existed table, please drop " + "it and re-create it.")

updated, thanks!

SparkQA · 2019-12-10T08:05:02Z

Test build #115083 has finished for PR 26781 at commit 814821a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Ngone51 · 2019-12-10T08:06:41Z

Jenkins, retest this please.

SparkQA · 2019-12-10T12:08:29Z

Test build #115093 has finished for PR 26781 at commit 814821a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-12-10T12:56:30Z

thanks, merging to master!

Ngone51 · 2019-12-10T14:02:25Z

thanks @cloud-fan @HyukjinKwon @maropu

Ngone51 added 5 commits December 6, 2019 15:23

mismatch schema

468660f

update

9847847

update again

aab2391

improve error message

a4f3122

updated tests

dd45804

dongjoon-hyun added the SQL label Dec 6, 2019

HyukjinKwon reviewed Dec 7, 2019

View reviewed changes

Ngone51 added 4 commits December 9, 2019 21:33

reword error msg

de036b6

revert tests

8fac1fa

update tests

6700dee

remove TestJdbcRelationProvider

f54dea9

Ngone51 commented Dec 9, 2019

View reviewed changes

Ngone51 added 2 commits December 9, 2019 22:06

revert import

ffdff16

remove unneeded string interpolator

c2b5eea

cloud-fan approved these changes Dec 9, 2019

View reviewed changes

maropu reviewed Dec 10, 2019

View reviewed changes

maropu approved these changes Dec 10, 2019

View reviewed changes

HyukjinKwon approved these changes Dec 10, 2019

View reviewed changes

address comment

814821a

cloud-fan closed this in aa9da93 Dec 10, 2019

[SPARK-30151][SQL] Issue better error message when user-specified schema mismatched #26781

[SPARK-30151][SQL] Issue better error message when user-specified schema mismatched #26781

Uh oh!

Conversation

Ngone51 commented Dec 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Ngone51 commented Dec 6, 2019

Uh oh!

SparkQA commented Dec 6, 2019

Uh oh!

Ngone51 commented Dec 6, 2019

Uh oh!

Ngone51 commented Dec 6, 2019

Uh oh!

SparkQA commented Dec 6, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Dec 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Dec 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 9, 2019

Uh oh!

SparkQA commented Dec 9, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 10, 2019

Uh oh!

Ngone51 commented Dec 10, 2019

Uh oh!

SparkQA commented Dec 10, 2019

Uh oh!

cloud-fan commented Dec 10, 2019

Uh oh!

Ngone51 commented Dec 10, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Ngone51 commented Dec 6, 2019 •

edited

Loading

HyukjinKwon Dec 7, 2019 •

edited

Loading

HyukjinKwon Dec 8, 2019 •

edited

Loading