[SPARK-7289] [SPARK-9949] [SQL] Backport SPARK-7289 and SPARK-9949 to branch 1.4 #8252
Conversation
make the `TakeOrdered` strategy and operator more general, such that it can optionally handle a projection when necessary Author: Wenchen Fan <[email protected]> Closes #6780 from cloud-fan/limit and squashes the following commits: 34aa07b [Wenchen Fan] revert 07d5456 [Wenchen Fan] clean closure 20821ec [Wenchen Fan] fix 3676a82 [Wenchen Fan] address comments b558549 [Wenchen Fan] address comments 214842b [Wenchen Fan] fix style 2d8be83 [Wenchen Fan] add LimitPushDown 948f740 [Wenchen Fan] fix existing Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala
https://issues.apache.org/jira/browse/SPARK-9949 Author: Yin Huai <[email protected]> Closes #8179 from yhuai/SPARK-9949. (cherry picked from commit 932b24f) Signed-off-by: Reynold Xin <[email protected]> Conflicts: sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala
|
@cloud-fan Can you take a look and make sure I am not missing any thing? btw, I am intentionally not adding changes of |
|
Test build #41052 has finished for PR 8252 at commit
|
|
LGTM. The change to optimizer is just refactor, so it's not necessary to backport. |
|
Thanks. I am merging it to branch 1.4. |
… branch 1.4 The bug fixed by SPARK-7289 is a pretty serious one (Spark SQL generates wrong results). We should backport the fix to branch 1.4 (#6780). Also, we need to backport the fix of `TakeOrderedAndProject` as well (#8179). Author: Wenchen Fan <[email protected]> Author: Yin Huai <[email protected]> Closes #8252 from yhuai/backport7289And9949.
|
OK. Merged. I am closing this pr. |
|
This backport also includes the refactor of TakeOrdered to TakeOrderedAndProject, which breaks code that refers to TakeOrdered. In a minor version update that is perhaps not expected - specifically, the Cassandra connector refers to this by name and no longer works. I'll submit an issue on this to Datastax as well, but you may want to consider keeping the TakeOrdered name for this branch unless the semantics have changed too much. |
|
@frodeso Thank you for bringing this up. Can you open a JIRA for it and attach the specific use-case at there (it will be good to also attach a link to the place where Cassandra connector uses it)? |
|
@yhuai I've reported the issue (https://issues.apache.org/jira/browse/SPARK-10377) . It has also been reported to Datastax: https://datastax-oss.atlassian.net/browse/SPARKC-238 . |
… TakeOrdered. https://issues.apache.org/jira/browse/SPARK-10377 #8252 (comment) renames `TakeOrdered` to `TakeOrderedAndProject` (this is an internal API), which breaks third-party code using Spark 1.4 branch (e.g. Cassandra connector https://github.com/datastax/spark-cassandra-connector/blob/v1.4.0-M3/spark-cassandra-connector/src/main/scala/org/apache/spark/sql/cassandra/CassandraSQLContext.scala#L90). This PR renames this class back to `TakeOrdered`. Author: Yin Huai <[email protected]> Closes #8545 from yhuai/SPARK-10377.
4 participants
The bug fixed by SPARK-7289 is a pretty serious one (Spark SQL generates wrong results). We should backport the fix to branch 1.4 (#6780). Also, we need to backport the fix of
TakeOrderedAndProjectas well (#8179).