Skip to content

Conversation

@clockfly
Copy link
Contributor

@clockfly clockfly commented Aug 4, 2016

What changes were proposed in this pull request?

This PR adds argument type information for typed logical plan like MapElements, TypedFilter, and AppendColumn, so that we can use these info in customized optimizer rule.

How was this patch tested?

Existing test.

@SparkQA
Copy link

SparkQA commented Aug 4, 2016

Test build #63229 has finished for PR 14494 at commit c15b46c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class TypedSumDouble[IN](val f: IN => Double) extends Aggregator[IN, Double, Double]
    • class TypedSumLong[IN](val f: IN => Long) extends Aggregator[IN, Long, Long]
    • class TypedCount[IN](val f: IN => Any) extends Aggregator[IN, Long, Long]
    • class TypedAverage[IN](val f: IN => Double) extends Aggregator[IN, (Double, Long), Double]



class TypedSumDouble[IN](f: IN => Double) extends Aggregator[IN, Double, Double] {
class TypedSumDouble[IN](val f: IN => Double) extends Aggregator[IN, Double, Double] {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why expose the f?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TypedFilter and MapElements also exposes the function closure? With this, we can apply the possible optimization describe in https://issues.apache.org/jira/browse/SPARK-16898?

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@asfgit asfgit closed this in bca43cd Aug 9, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants