Skip to content

[SPARK-11453][SQL] append data to partitioned table will messes up the result #9408

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 7 commits into from
Closed

[SPARK-11453][SQL] append data to partitioned table will messes up the result #9408

wants to merge 7 commits into from

Conversation

@cloud-fan
Copy link
Contributor

@cloud-fan cloud-fan commented Nov 2, 2015

The reason is that:

  1. For partitioned hive table, we will move the partitioned columns after data columns. (e.g. <a: Int, b: Int> partition by a will become <b: Int, a: Int>)
  2. When append data to table, we use position to figure out how to match input columns to table's columns.

So when we append data to partitioned table, we will match wrong columns between input and table. A solution is reordering the input columns before match by position, like what we did for InsertIntoHadoopFsRelation

@cloud-fan
Copy link
Contributor Author

cloud-fan commented Nov 2, 2015

cc @yhuai

@SparkQA
Copy link

SparkQA commented Nov 2, 2015

Test build #44804 has finished for PR 9408 at commit b1512b0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * case class Corr(\n * case class Corr(left: Expression, right: Expression)\n * case class RepartitionByExpression(\n * logInfo(s\"Hive class not found $e\")\n * logDebug(\"Hive class not found\", e)\n

@cloud-fan
Copy link
Contributor Author

cloud-fan commented Nov 3, 2015

cc @liancheng

liancheng
liancheng reviewed Nov 4, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala Outdated
Copy link
Contributor

@liancheng liancheng Nov 4, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to consider case sensitivity for partition column names here?

@liancheng
Copy link
Contributor

liancheng commented Nov 4, 2015

Overall looks good, left some minor comments.

@SparkQA
Copy link

SparkQA commented Nov 4, 2015

Test build #45015 has finished for PR 9408 at commit 06b96ed.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

yhuai
yhuai reviewed Nov 4, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala Outdated
Copy link
Contributor

@yhuai yhuai Nov 4, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only need to print column names, right?

@SparkQA
Copy link

SparkQA commented Nov 5, 2015

Test build #45084 has finished for PR 9408 at commit e682c86.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

yhuai
yhuai reviewed Nov 5, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
Copy link
Contributor

@yhuai yhuai Nov 5, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually, can we have a check to make sure that partition columns do appear at the end of the column list?

Copy link
Contributor Author

@cloud-fan cloud-fan Nov 6, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmm, I think normalizedParCols already did this check?

@cloud-fan
Copy link
Contributor Author

cloud-fan commented Nov 8, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Nov 8, 2015

Test build #2013 has finished for PR 9408 at commit 186d281.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 8, 2015

Test build #2015 has finished for PR 9408 at commit 186d281.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 8, 2015

Test build #45305 has finished for PR 9408 at commit 4a61037.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai
Copy link
Contributor

yhuai commented Nov 9, 2015

LGTM. Merging to master and branch 1.6.

asfgit pushed a commit that referenced this pull request Nov 9, 2015
@cloud-fan @yhuai
[SPARK-11453][SQL] append data to partitioned table will messes up th…
7eaf48e 
…e result

The reason is that:

1. For partitioned hive table, we will move the partitioned columns after data columns. (e.g. `<a: Int, b: Int>` partition by `a` will become `<b: Int, a: Int>`)
2. When append data to table, we use position to figure out how to match input columns to table's columns.

So when we append data to partitioned table, we will match wrong columns between input and table. A solution is reordering the input columns before match by position, like what we did for [`InsertIntoHadoopFsRelation`](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelation.scala#L101-L105)

Author: Wenchen Fan <[email protected]>

Closes #9408 from cloud-fan/append.

(cherry picked from commit d8b50f7)
Signed-off-by: Yin Huai <[email protected]>
@asfgit asfgit closed this in d8b50f7 Nov 9, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@cloud-fan @SparkQA @liancheng @yhuai