[SPARK-15911][SQL] Remove the additional Project to be consistent with SQL #13631
Conversation
|
Test build #60380 has finished for PR 13631 at commit
|
|
@cloud-fan There is a test ("Detect table partitioning with correct partition order") in |
|
Test build #60403 has finished for PR 13631 at commit
|
| } | ||
| } | ||
|
|
||
| test("Detect table partitioning with correct partition order") { |
There was a problem hiding this comment.
can you link to the PR that added this test?
|
Test build #60481 has finished for PR 13631 at commit
|
| inputPartCols.find(_.name == name).getOrElse( | ||
| throw new AnalysisException(s"Cannot find partition column $name")) | ||
| tablePartitionNames.filterNot { name => | ||
| child.output.exists(_.name == name) |
There was a problem hiding this comment.
do we really need this check?
There was a problem hiding this comment.
hmm. indeed. As we use ordering not name, this check is not needed anymore. I will remove it.
|
Test build #60546 has finished for PR 13631 at commit
|
|
@cloud-fan any other thoughts? |
|
hi @viirya , we are auditing the insertion behaviour of spark sql, and will have an agreement this week. How about we revisit this PR after that? |
|
@cloud-fan no problem at all. |
|
please track the process of https://issues.apache.org/jira/browse/SPARK-16032 and update your PR, thanks! |
|
@cloud-fan Looks like a part of this PR is done by some PRs in that JIRA. I will update this. |
Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertIntoHiveTableSuite.scala
|
@cloud-fan Updated. Please take a look. Thanks! |
|
Test build #60795 has finished for PR 13631 at commit
|
|
ping @cloud-fan |
|
@cloud-fan Looks like the change in this PR is done by #13766. Let me close it now. |
3 participants
What changes were proposed in this pull request?
Currently In
DataFrameWriter'sinsertIntoandResolveRelationsofAnalyzer, we add additional Project to adjust column ordering. However, it should be using ordering not name for this resolution. This is how Hive does for dynamic partition.How was this patch tested?
Existing tests.