[SPARK-8279][SQL]Add math function round

[yjshen]

JIRA: https://issues.apache.org/jira/browse/SPARK-8279

[rxin]

Jenkins, add to whitelist.

[marmbrus]

ok to test

[SparkQA]

Test build #35467 has finished for PR 6938 at commit 0495feb.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Round(children: Seq[Expression]) extends Expression
  • trait BigDecimalConverter[T]

[SparkQA]

Test build #35469 has finished for PR 6938 at commit c5ce169.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Round(children: Seq[Expression]) extends Expression
  • trait BigDecimalConverter[T]

[chenghao-intel]

duplicated with #6836?

[yjshen]

@chenghao-intel, Yep, not aware there are two JIRAs for Round. Mind looking at this as well?

[chenghao-intel]

Actually we support multiple constructors now in expression, see https://github.com/apache/spark/pull/6806/files#diff-d788f93e29b4d25cdd7d60328587678bR229

[chenghao-intel]

As most of issues that I raised is solved in #6836, do you mind jump there and give some comments?
BTW, as the https://issues.apache.org/jira/browse/SPARK-8279 suggested, perhaps we'd better focus on the test case udf_round_3 test, which is not covered by #6836, what do you think?

[yjshen]

@chenghao-intel, I think the main difference between this and #6836 is whether to make Round act as GenericUDFRound, if we want to support udf_round and udf_round_3.
I suppose following Hive's semantic is a good choice.

[SparkQA]

Test build #35521 has finished for PR 6938 at commit be02d5b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[chenghao-intel]

Yes, #6836 follows the Hive's GenericUDFRound, but we should keep the output / input to o.a.s.s.types.Decimal, not BigDecimal (if it's a DecimalType), otherwise it will causes problem when interact with other Spark SQL expression. See https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala#L291

[yjshen]

Oh, I think there exists misunderstood of the round method and BigDecimalConverter, BDC just as as a implicit type parameter of round to infer the first argument's type, for example Byte, call the corresponding constructor of BigDecimal and use it's setScale method to round, at last, round is using fromBigDecimal to convert the dataType back, i.e. to Byte in our example.

Therefore, I preserve the dataType of children(0), the only exceptions are String and Binary which would be regarded as Double in Hive.

[chenghao-intel]

Oh, sorry, you did use the Decimal, for Round.eval, ignore my previous comment.

[SparkQA]

Test build #35539 has finished for PR 6938 at commit 40f4a99.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Round(child: Expression, scale: Expression) extends Expression
  • trait BigDecimalConverter[T]

[chenghao-intel]

depends on child.nullable || scala.nullable?

[yjshen]

probably more than that, Hive support String ,Double.NaN, Double.Infinity as input, all of these would result in null result.

[chenghao-intel]

OK, let's say if the both children are literals e.g. Literal(123.0, FloatType) and Literal(1. IntegerType), still be nullable?

[rxin]

it is not the end of the world to have nullable be more conservative, since it is technically correct to be nullable. however, if there is a way to do a more accurate way to determine nullability, we should do that.

[chenghao-intel]

The nullable will be great useful in the expression optimization, we'd better handle it properly.

[yjshen]

I mean nullable is sometimes determined at runtime, a not null string, double.NaN is not null themselves, but would eval to null in Round.

[SparkQA]

Test build #35555 has finished for PR 6938 at commit 479fa9b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Round(child: Expression, scale: Expression) extends Expression
  • trait BigDecimalConverter[T]

[yjshen]

@rxin @marmbrus, mind reviewing this?

[SparkQA]

Test build #35660 has finished for PR 6938 at commit c14f64d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Round(child: Expression, scale: Expression) extends Expression
  • trait BigDecimalConverter[T]

[yjshen]

@chenghao-intel , refactored eval and moved type specific branching out.

[SparkQA]

Test build #37221 has finished for PR 6938 at commit 392b65b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Round(child: Expression, scale: Expression)

[rxin]

i don't think you need this

[rxin]

i -> scale

[yjshen]

I was also using i for array index here?

[rxin]

ok - can you at least move bdResults closer to this loop?

[yjshen]

OK

[rxin]

OK I'm going to merge this. Please submit a patch to fix the minor comments.

[rxin]

you can remove this, since the expression is already named Round

[SparkQA]

Test build #37326 has finished for PR 6938 at commit 07a124c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Round(child: Expression, scale: Expression)