Skip to content

SPARK-6548 Adding stddev to DataFrame functions #6297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 56 commits into from
Closed

SPARK-6548 Adding stddev to DataFrame functions #6297

wants to merge 56 commits into from

Conversation

@JihongMA
Copy link
Contributor

@JihongMA JihongMA commented May 20, 2015

Adding STDDEV support for DataFrame using 1-pass online /parallel algorithm to compute variance. Please review the code change.

JihongMA and others added 29 commits May 5, 2015 21:17
@JihongMA
SPARK-7265 Improving documentation for Spark SQL Hive support
a399aa6
@JihongMA
SPARK-7265 address review comment
9c84695
@JihongMA
Merge remote-tracking branch 'upstream/master'
7b842e6
@JihongMA
Merge remote-tracking branch 'upstream/master'
d5bf3f5
@JihongMA
Merge remote-tracking branch 'upstream/master'
9ea0832
@JihongMA
SPARK-7265 take out PySpark on YARN limitation
ace454d
@JihongMA
SPARK-7265 minor fix of the content
dfaa971
@JihongMA
Merge remote-tracking branch 'upstream/master'
a38e9f2
@JihongMA
SPARK-6548 add stddev for datafram api
125f3ae
@JihongMA
SPARK-6548 Add stddev for DataFrame
634d6a7
@JihongMA
Merge remote-tracking branch 'upstream/master'
213c60e
@JihongMA
SPARK-6548 minor fix
3e7d889
@JihongMA
SPARK-6548 minor fix
c40701a
@JihongMA
Merge remote-tracking branch 'upstream/master'
9d9bdcf
@JihongMA
Merge remote branch 'upstream/master'
7adc7a3
@JihongMA
Revert "SPARK-6548 minor fix"
c712568 
This reverts commit c40701a.
@JihongMA
Revert "SPARK-6548 minor fix"
665ce35 
This reverts commit 3e7d889.
@JihongMA
Revert "SPARK-6548 Add stddev for DataFrame"
319bd23 
This reverts commit 634d6a7.
@JihongMA
Revert "SPARK-6548 add stddev for datafram api"
f784358 
This reverts commit 125f3ae.
@JihongMA
Revert "SPARK-7265 minor fix of the content"
e7d2c0d 
This reverts commit dfaa971.
@JihongMA
Revert "SPARK-7265 take out PySpark on YARN limitation"
2ed3921 
This reverts commit ace454d.
@JihongMA
Revert "SPARK-7265 address review comment"
430bcfc 
This reverts commit 9c84695.

Conflicts:

	docs/running-on-yarn.md
@JihongMA
Revert "SPARK-7265 Improving documentation for Spark SQL Hive support"
3790855 
This reverts commit a399aa6.

Conflicts:

	docs/running-on-yarn.md
@JihongMA
Merge remote branch 'upstream/master'
e11ea50
@JihongMA
fix running-on-yarn.md
1c871ff
@JihongMA
merging runnin-on-yarn.md
3ee3c03
@JihongMA
Merge remote branch 'upstream/master'
8204410
@JihongMA
SPARK-6548 Add Stddev support as Catalyst expression in DataFrame
a73f041
Merge remote-tracking branch 'upstream/master'
df34292
marmbrus
marmbrus reviewed Jun 23, 2015
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala Outdated
Copy link
Contributor

@marmbrus marmbrus Jun 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have changed how these plug in. You'll need to change the FunctionRegistry now.

@JihongMA
Copy link
Contributor Author

JihongMA commented Jul 24, 2015

Please don't test it yet, need to make change to accomodate API change introduced by other JIRA.

@SparkQA
Copy link

SparkQA commented Jul 24, 2015

Test build #38399 has finished for PR 6297 at commit 87fd2dc.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • abstract class InternalRow extends Serializable
    • case class Stddev(child: Expression) extends PartialAggregate with trees.UnaryNode[Expression]
    • case class ComputePartialStd(child: Expression) extends AggregateExpression
    • case class CombinePartialStd(child: Expression) extends AggregateExpression
    • case class ComputePartialStdFunction (
    • case class CombinePartialStdFunction(
    • case class StddevFunction(
    • class GenericRow(protected[sql] val values: Array[Any]) extends Row
    • class GenericInternalRow(protected[sql] val values: Array[Any]) extends InternalRow
    • class GenericInternalRowWithSchema(values: Array[Any], val schema: StructType)
    • class GenericMutableRow(val values: Array[Any]) extends MutableRow

@yhuai
Copy link
Contributor

yhuai commented Jul 29, 2015

@JihongMA Will you get time to implement the function based on the new API? It will be good if we can merge it before the 1.5 deadline for new features (end of this month).

@SparkQA
Copy link

SparkQA commented Aug 28, 2015

Test build #41730 has finished for PR 6297 at commit 25425ac.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class Stddev(child: Expression) extends StddevAgg(child)
    • case class StddevPop(child: Expression) extends StddevAgg(child)
    • case class StddevSamp(child: Expression) extends StddevAgg(child)
    • abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
    • abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
    • case class Stddev(child: Expression) extends StddevAgg1(child)
    • case class StddevPop(child: Expression) extends StddevAgg1(child)
    • case class StddevSamp(child: Expression) extends StddevAgg1(child)
    • case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
    • case class ComputePartialStdFunction (
    • case class MergePartialStd(child: Expression, isSample: Boolean) extends UnaryExpression with AggregateExpression1
    • case class MergePartialStdFunction(
    • case class StddevFunction(

@SparkQA
Copy link

SparkQA commented Aug 28, 2015

Test build #41732 has finished for PR 6297 at commit f4c725c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class Stddev(child: Expression) extends StddevAgg(child)
    • case class StddevPop(child: Expression) extends StddevAgg(child)
    • case class StddevSamp(child: Expression) extends StddevAgg(child)
    • abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
    • abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
    • case class Stddev(child: Expression) extends StddevAgg1(child)
    • case class StddevPop(child: Expression) extends StddevAgg1(child)
    • case class StddevSamp(child: Expression) extends StddevAgg1(child)
    • case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
    • case class ComputePartialStdFunction (
    • case class MergePartialStd(
    • case class MergePartialStdFunction(
    • case class StddevFunction(

@SparkQA
Copy link

SparkQA commented Aug 28, 2015

Test build #41748 has finished for PR 6297 at commit 0902ceb.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class Stddev(child: Expression) extends StddevAgg(child)
    • case class StddevPop(child: Expression) extends StddevAgg(child)
    • case class StddevSamp(child: Expression) extends StddevAgg(child)
    • abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
    • abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
    • case class Stddev(child: Expression) extends StddevAgg1(child)
    • case class StddevPop(child: Expression) extends StddevAgg1(child)
    • case class StddevSamp(child: Expression) extends StddevAgg1(child)
    • case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
    • case class ComputePartialStdFunction (
    • case class MergePartialStd(
    • case class MergePartialStdFunction(
    • case class StddevFunction(

@SparkQA
Copy link

SparkQA commented Sep 4, 2015

Test build #42006 has finished for PR 6297 at commit a81d0fc.

  • This patch fails R style tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class Stddev(child: Expression) extends StddevAgg(child)
    • case class StddevPop(child: Expression) extends StddevAgg(child)
    • case class StddevSamp(child: Expression) extends StddevAgg(child)
    • abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
    • abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
    • case class Stddev(child: Expression) extends StddevAgg1(child)
    • case class StddevPop(child: Expression) extends StddevAgg1(child)
    • case class StddevSamp(child: Expression) extends StddevAgg1(child)
    • case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
    • case class ComputePartialStdFunction (
    • case class MergePartialStd(
    • case class MergePartialStdFunction(
    • case class StddevFunction(

@JihongMA
Copy link
Contributor Author

JihongMA commented Sep 4, 2015

R style check failure is caused by commit of SPARK-8951

@SparkQA
Copy link

SparkQA commented Sep 6, 2015

Test build #42062 has finished for PR 6297 at commit 6035648.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class Stddev(child: Expression) extends StddevAgg(child)
    • case class StddevPop(child: Expression) extends StddevAgg(child)
    • case class StddevSamp(child: Expression) extends StddevAgg(child)
    • abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
    • abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
    • case class Stddev(child: Expression) extends StddevAgg1(child)
    • case class StddevPop(child: Expression) extends StddevAgg1(child)
    • case class StddevSamp(child: Expression) extends StddevAgg1(child)
    • case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
    • case class ComputePartialStdFunction (
    • case class MergePartialStd(
    • case class MergePartialStdFunction(
    • case class StddevFunction(

davies
davies reviewed Sep 10, 2015
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala Outdated
Copy link
Contributor

@davies davies Sep 10, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should always return Double, because Sqrt() only works with Double, also other databases just return Double/float.

@SparkQA
Copy link

SparkQA commented Sep 12, 2015

Test build #42366 has finished for PR 6297 at commit 6351fc8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class Stddev(child: Expression) extends StddevAgg(child)
    • case class StddevPop(child: Expression) extends StddevAgg(child)
    • case class StddevSamp(child: Expression) extends StddevAgg(child)
    • abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
    • abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
    • case class Stddev(child: Expression) extends StddevAgg1(child)
    • case class StddevPop(child: Expression) extends StddevAgg1(child)
    • case class StddevSamp(child: Expression) extends StddevAgg1(child)
    • case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
    • case class ComputePartialStdFunction (
    • case class MergePartialStd(
    • case class MergePartialStdFunction(
    • case class StddevFunction(

@davies
Copy link
Contributor

davies commented Sep 12, 2015

LGTM, merging this into master, thanks!

@asfgit asfgit closed this in f4a2280 Sep 12, 2015
@davies davies mentioned this pull request Sep 29, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@JihongMA @marmbrus @yhuai @SparkQA @davies @sujkh85