[SPARK-26950][SQL][TEST] Make RandomDataGenerator use Float.NaN or Double.NaN for all NaN values

[dongjoon-hyun]

What changes were proposed in this pull request?

Apache Spark uses the predefined Float.NaN and Double.NaN for NaN values, but there exists more NaN values with different binary presentations.

scala> java.nio.ByteBuffer.allocate(4).putFloat(Float.NaN).array
res1: Array[Byte] = Array(127, -64, 0, 0)

scala> val x = java.lang.Float.intBitsToFloat(-6966608)
x: Float = NaN

scala> java.nio.ByteBuffer.allocate(4).putFloat(x).array
res2: Array[Byte] = Array(-1, -107, -78, -80)

Since users can have these values, RandomDataGenerator generates these NaN values. However, this causes checkEvaluationWithUnsafeProjection failures due to the difference between UnsafeRow binary presentation. The following is the UT failure instance. This PR aims to fix this UT flakiness.

• https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/102528/testReport/

How was this patch tested?

Pass the Jenkins with the newly added test cases.

[dongjoon-hyun]

cc @dbtsai , @cloud-fan , @gatorsmile , @HyukjinKwon

[SparkQA]

Test build #102576 has finished for PR 23851 at commit 5444a62.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Retest this please.

[dongjoon-hyun]

Retest this please.

[cloud-fan]

it seems the better fix is to wrap expressions with NormalizeNaNAndZero in checkEvaluationWithUnsafeProjection.

[SparkQA]

Test build #102579 has finished for PR 23851 at commit 5444a62.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Huh! I didn't realize there were many representations of NaN.
https://en.wikipedia.org/wiki/IEEE_754-1985#NaN

This seems OK.

However this code caught my attention; it seems to be trying to generate floats 'uniformly' with intBitsToFloat(r.nextInt()) . That's not uniform; small values are way more likely. That may be desirable or not matter. If you feel like it, maybe change randomNumeric's uniformRand argument to not say 'uniform'.

[dongjoon-hyun]

Thank you for review, @cloud-fan and @srowen .

To @cloud-fan .
checkEvaluationWithUnsafeProjection should handle more complex expressions like from_avro(to_avro([NaN]), {"type":"record","name":"topLevelRecord","fields":[{"name":"col_1","type":["float","null"]}]}) described in the PR description. However,

1. NormalizeNaNAndZero expects and handles Float and Double instances only.
2. NormalizeFloatingNumbers expects Plan.

We can wrap (transform) the first argument expression, but the second argument expected is Any type.

  protected def checkEvaluationWithUnsafeProjection(
      expression: Expression,
      expected: Any,
      inputRow: InternalRow = EmptyRow): Unit = {

[dongjoon-hyun]

Hi, @srowen . For uniformRand argument, it looks like randomNumeric's initial aspiration instead of its requirement. If we can provide some uniform function instead of intBitsToFloat, that would be better. For now, I'd like to keep the argument name~

[srowen]

That's fine, it's not a big deal. I think the intent is to choose from all possible float values with equal probability, which isn't uniform over its range, but, 'uniform' over all possible values of a float.

[SparkQA]

Test build #102586 has finished for PR 23851 at commit 5444a62.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master/2.4!

[srowen]

Oh ha I also tried to merge it just now and got weird errors. That's why.
I have no idea why it shows I pushed to my fork?
srowen@ffef3d4

[dongjoon-hyun]

Thank you, @cloud-fan , @srowen , @maropu .

[dongjoon-hyun]

To prevent flakiness, I merged this to branch-2.3, too.