[SPARK-20548][FLAKY-TEST] share one REPL instance among REPL test cases

[cloud-fan]

What changes were proposed in this pull request?

ReplSuite.newProductSeqEncoder with REPL defined class was flaky and throws OOM exception frequently. By analyzing the heap dump, we found the reason is that, in each test case of ReplSuite, we create a REPL instance, which creates a classloader and loads a lot of classes related to SparkContext. More details please see #17833 (comment).

In this PR, we create a new test suite, SingletonReplSuite, which shares one REPL instances among all the test cases. Then we move most of the tests from ReplSuite to SingletonReplSuite, to avoid creating a lot of REPL instances and reduce memory footprint.

How was this patch tested?

test only change

[cloud-fan]

cc @zsxwing @sameeragarwal

[SparkQA]

Test build #76423 has finished for PR 17844 at commit 1565a34.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sameeragarwal]

test this please

[SparkQA]

Test build #76461 has finished for PR 17844 at commit 1565a34.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #76471 has finished for PR 17844 at commit 9248a5e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

Looks pretty good. Just some nits.

[zsxwing]

nit: it's better to use some more special string. E.g.,

val timestamp = System.currentTimeMillis()
in.write((input + s"\nval _result_$timestamp = 1\n").getBytes)
in.flush()
val stopMessage = s"_result_$timestamp: Int = 1"

[zsxwing]

Could you add a comment mentioning that the codes in input should not throw an exception?

[zsxwing]

You can use eventually

    eventually(timeout(50.seconds), interval(500.millis)) {
      assert(cond(), "current output: " + out.toString)
    }

[zsxwing]

Why you don't move this test?

[cloud-fan]

this test only works in local mode...

[zsxwing]

nit: I think it's safe to just use local-cluster[2,1,1024].

[zsxwing]

nit: please call Thread.setDaemon(true) in case that something is wrong and it prevents the process from exiting.

[SparkQA]

Test build #76569 has finished for PR 17844 at commit fd5b43b.

• This patch fails from timeout after a configured wait of `250m`.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

retest this please

[SparkQA]

Test build #76577 has finished for PR 17844 at commit fd5b43b.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

retest this please

[SparkQA]

Test build #76582 has finished for PR 17844 at commit fd5b43b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

retest this please

[SparkQA]

Test build #76600 has finished for PR 17844 at commit fd5b43b.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #76618 has started for PR 17844 at commit 7dde745.

[gatorsmile]

retest this please

[SparkQA]

Test build #76649 has finished for PR 17844 at commit 7dde745.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #76656 has finished for PR 17844 at commit 7dde745.

• This patch fails from timeout after a configured wait of `250m`.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

retest this please

[SparkQA]

Test build #76675 has finished for PR 17844 at commit 7dde745.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks for the review, merging to maser/2.2!