[SPARK-7184] [SQL] enable codegen by default #6726

davies · 2015-06-09T17:35:23Z

In order to have better performance out of box, this PR turn on codegen by default, then codegen can be tested by sql/test and hive/test.

This PR also fix some corner cases for codegen.

Before 1.5 release, we should re-visit this, turn it off if it's not stable or causing regressions.

cc @rxin @JoshRosen

SparkQA · 2015-06-09T18:25:41Z

Test build #34518 has finished for PR 6726 at commit a8618c9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- logInfo(s"Using user defined output committer class $
- logInfo(s"Using output committer class $

rxin · 2015-06-09T22:38:14Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala

can we call a function here? define some BinaryUtils to put everything for binary there. That makes it much easier to change.

Code generated version can just generate the code to call that function.

SparkQA · 2015-06-10T00:31:13Z

Test build #34544 has finished for PR 6726 at commit c8e7cd2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-10T00:59:33Z

Test build #34545 has finished for PR 6726 at commit f3886fa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2015-06-10T06:45:25Z

...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala

also remove the case for other cases.

rxin · 2015-06-10T07:07:12Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala

so code gen is completely off if I use any thread safe == false function?

We can improve the coverage later, make all the hive tests passed first.

SparkQA · 2015-06-10T07:10:20Z

Test build #34574 has finished for PR 6726 at commit 207e339.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-10T07:49:28Z

Test build #34566 has finished for PR 6726 at commit 44573a3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class IsNull(child: Expression) extends UnaryExpression with Predicate
- case class IsNotNull(child: Expression) extends UnaryExpression with Predicate

…codegen Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala

SparkQA · 2015-06-10T17:54:53Z

Test build #34623 has finished for PR 6726 at commit 99fc139.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/BaseRow.java sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateProjection.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateUtilsSuite.scala

SparkQA · 2015-06-12T00:33:38Z

Test build #34727 has finished for PR 6726 at commit ff5b75a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullFunctions.scala sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala sql/core/src/main/scala/org/apache/spark/sql/execution/expressions/MonotonicallyIncreasingID.scala sql/core/src/main/scala/org/apache/spark/sql/sources/commands.scala sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala

SparkQA · 2015-06-13T22:25:26Z

Test build #34844 has finished for PR 6726 at commit a7d75da.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class IsNull(child: Expression) extends UnaryExpression with Predicate
- case class IsNotNull(child: Expression) extends UnaryExpression with Predicate

Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullFunctions.scala

SparkQA · 2015-06-14T02:08:24Z

Test build #34859 has finished for PR 6726 at commit 3017a47.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class IsNull(child: Expression) extends UnaryExpression with Predicate
- case class IsNotNull(child: Expression) extends UnaryExpression with Predicate

SparkQA · 2015-06-14T04:44:38Z

Test build #34865 has finished for PR 6726 at commit 73750ea.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class JoinedRow extends InternalRow
- class JoinedRow2 extends InternalRow
- class JoinedRow3 extends InternalRow
- class JoinedRow4 extends InternalRow
- class JoinedRow5 extends InternalRow
- class JoinedRow6 extends InternalRow
- class BaseOrdering extends Ordering[InternalRow]
- case class IsNull(child: Expression) extends UnaryExpression with Predicate
- case class IsNotNull(child: Expression) extends UnaryExpression with Predicate

davies · 2015-06-14T06:18:03Z

@marmbrus @rxin I think this is ready to review, thanks!

SparkQA · 2015-06-14T08:47:21Z

Test build #34877 has finished for PR 6726 at commit f3b25a5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class IsNull(child: Expression) extends UnaryExpression with Predicate
- case class IsNotNull(child: Expression) extends UnaryExpression with Predicate

rxin · 2015-06-16T06:03:05Z

Going to merge this one.

In order to have better performance out of box, this PR turn on codegen by default, then codegen can be tested by sql/test and hive/test. This PR also fix some corner cases for codegen. Before 1.5 release, we should re-visit this, turn it off if it's not stable or causing regressions. cc rxin JoshRosen Author: Davies Liu <[email protected]> Closes apache#6726 from davies/enable_codegen and squashes the following commits: f3b25a5 [Davies Liu] fix warning 73750ea [Davies Liu] fix long overflow when compare 3017a47 [Davies Liu] Merge branch 'master' of github.com:apache/spark into enable_codegen a7d75da [Davies Liu] Merge branch 'master' of github.com:apache/spark into enable_codegen ff5b75a [Davies Liu] Merge branch 'master' of github.com:apache/spark into enable_codegen f4cf2c2 [Davies Liu] fix style 99fc139 [Davies Liu] Merge branch 'enable_codegen' of github.com:davies/spark into enable_codegen 91fc7a2 [Davies Liu] disable codegen for ScalaUDF 207e339 [Davies Liu] Update CodeGenerator.scala 44573a3 [Davies Liu] check thread safety of expression f3886fa [Davies Liu] don't inline primitiveTerm for null literal c8e7cd2 [Davies Liu] address comment a8618c9 [Davies Liu] enable codegen by default

enable codegen by default

a8618c9

rxin reviewed Jun 9, 2015
View reviewed changes

Davies Liu added 2 commits June 9, 2015 16:39

address comment

c8e7cd2

don't inline primitiveTerm for null literal

f3886fa

check thread safety of expression

44573a3

rxin reviewed Jun 10, 2015
View reviewed changes

Update CodeGenerator.scala

207e339

rxin reviewed Jun 10, 2015
View reviewed changes

Davies Liu added 2 commits June 10, 2015 10:48

disable codegen for ScalaUDF

91fc7a2

Merge branch 'enable_codegen' of github.com:davies/spark into enable_…

99fc139

…codegen Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala

fix style

f4cf2c2

davies changed the title ~~[SPARK-7184] [SQL] enable codegen by default~~ [SPARK-7184] [SPARK-8190] [SQL] enable codegen by default Jun 10, 2015

davies changed the title ~~[SPARK-7184] [SPARK-8190] [SQL] enable codegen by default~~ [SPARK-7184] [SQL] enable codegen by default Jun 11, 2015

davies force-pushed the enable_codegen branch from 279e264 to ff5b75a Compare June 11, 2015 23:41

Merge branch 'master' of github.com:apache/spark into enable_codegen

3017a47

Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullFunctions.scala

fix long overflow when compare

73750ea

fix warning

f3b25a5

davies mentioned this pull request Jun 16, 2015

[SPARK-8371][SQL] improve unit test for MaxOf and MinOf and fix bugs #6825

Closed

asfgit closed this in bc76a0f Jun 16, 2015

[SPARK-7184] [SQL] enable codegen by default #6726

[SPARK-7184] [SQL] enable codegen by default #6726

Uh oh!

Conversation

davies commented Jun 9, 2015

Uh oh!

SparkQA commented Jun 9, 2015

Uh oh!

rxin Jun 9, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 10, 2015

Uh oh!

SparkQA commented Jun 10, 2015

Uh oh!

rxin Jun 10, 2015

Choose a reason for hiding this comment

Uh oh!

rxin Jun 10, 2015

Choose a reason for hiding this comment

Uh oh!

rxin Jun 10, 2015

Choose a reason for hiding this comment

Uh oh!

davies Jun 10, 2015

Choose a reason for hiding this comment

Uh oh!

davies Jun 10, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 10, 2015

Uh oh!

SparkQA commented Jun 10, 2015

Uh oh!

SparkQA commented Jun 10, 2015

Uh oh!

SparkQA commented Jun 12, 2015

Uh oh!

SparkQA commented Jun 13, 2015

Uh oh!

SparkQA commented Jun 14, 2015

Uh oh!

SparkQA commented Jun 14, 2015

Uh oh!

davies commented Jun 14, 2015

Uh oh!

SparkQA commented Jun 14, 2015

Uh oh!

rxin commented Jun 16, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants