Skip to content

[SPARK-12271][SQL] Improve error message when Dataset.as[ ] has incompatible schemas. #10260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

[SPARK-12271][SQL] Improve error message when Dataset.as[ ] has incompatible schemas. #10260

wants to merge 3 commits into from

Conversation

@nongli
Copy link
Contributor

@nongli nongli commented Dec 11, 2015

No description provided.

@nongli nongli changed the title [SPARK-12271][SQL] Improve error message when Dataset.as has incompatible schemas. [SPARK-12271][SQL] Improve error message when Dataset.as[ ] has incompatible schemas. Dec 11, 2015
@SparkQA
Copy link

SparkQA commented Dec 11, 2015

Test build #47560 has finished for PR 10260 at commit ceed59b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed Dec 11, 2015
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala Outdated
Copy link
Contributor

@cloud-fan cloud-fan Dec 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the simpleString of attribute includes expr id, which I think is noisy to users, should we generate the string manually or use StructType.fromAttributes to turn schema into StructType and call StrcutType.simpleString?

Copy link
Contributor Author

@nongli nongli Dec 14, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@cloud-fan cloud-fan mentioned this pull request Dec 11, 2015
Closed
@holdenk
Copy link
Contributor

holdenk commented Dec 14, 2015

Could you maybe update the PR description to include the description from the JIRA?

@SparkQA
Copy link

SparkQA commented Dec 14, 2015

Test build #47680 has finished for PR 10260 at commit f177d55.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Dec 15, 2015
@cloud-fan @marmbrus
[SPARK-12274][SQL] WrapOption should not have type constraint for child
9ea1a8e 
I think it was a mistake, and we have not catched it so far until #10260 which begin to check if the `fromRowExpression` is resolved.

Author: Wenchen Fan <[email protected]>

Closes #10263 from cloud-fan/encoder.
marmbrus
marmbrus reviewed Dec 15, 2015
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala Outdated
Copy link
Contributor

@marmbrus marmbrus Dec 15, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to not just put this in CheckAnalysis and make it more general (i.e. not just Project)? I realize its only used here now, but if we ever do use it elsewhere, its invalid to remain after analysis.

Copy link
Contributor Author

@nongli nongli Dec 15, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

much better!

@marmbrus
Copy link
Contributor

marmbrus commented Dec 15, 2015

This looks great now!

LGTM pending tests / conflict resolution

@SparkQA
Copy link

SparkQA commented Dec 15, 2015

Test build #47747 has finished for PR 10260 at commit 359b3e2.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds the following public classes (experimental):\n * case class WrapOption(child: Expression, optType: DataType)\n

@SparkQA
Copy link

SparkQA commented Dec 15, 2015

Test build #47754 has finished for PR 10260 at commit b5007d7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * case class WrapOption(child: Expression, optType: DataType)\n

@cloud-fan
Copy link
Contributor

cloud-fan commented Dec 16, 2015

LGTM

@marmbrus
Copy link
Contributor

marmbrus commented Dec 16, 2015

Thanks, merging to master.

@asfgit asfgit closed this in 86ea64d Dec 16, 2015
@nongli nongli deleted the spark-11271 branch December 16, 2015 05:04
marmbrus pushed a commit to marmbrus/spark that referenced this pull request Jan 7, 2016
@cloud-fan @marmbrus
[SPARK-12274][SQL] WrapOption should not have type constraint for child
c938a43 
I think it was a mistake, and we have not catched it so far until apache#10260 which begin to check if the `fromRowExpression` is resolved.

Author: Wenchen Fan <[email protected]>

Closes apache#10263 from cloud-fan/encoder.
marmbrus pushed a commit to marmbrus/spark that referenced this pull request Jan 7, 2016
@nongli @marmbrus
[SPARK-12271][SQL] Improve error message when Dataset.as[ ] has incom…
862cb12 
…patible schemas.

Author: Nong Li <[email protected]>

Closes apache#10260 from nongli/spark-11271.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@nongli @SparkQA @holdenk @marmbrus @cloud-fan