[SPARK-25144][SQL][TEST] Free aggregate map when task ends

[dongjoon-hyun]

What changes were proposed in this pull request?

SPARK-25144 reports memory leaks on Apache Spark 2.0.2 ~ 2.3.2-RC5. The bug is already fixed via #21738 as a part of SPARK-21743. This PR only adds a test case to prevent any future regression.

scala> case class Foo(bar: Option[String])
scala> val ds = List(Foo(Some("bar"))).toDS
scala> val result = ds.flatMap(_.bar).distinct
scala> result.rdd.isEmpty
18/08/19 23:01:54 WARN Executor: Managed memory leak detected; size = 8650752 bytes, TID = 125
res0: Boolean = false

How was this patch tested?

Pass the Jenkins with a new added test case.

[dongjoon-hyun]

cc @HyukjinKwon

[dongjoon-hyun]

Thank you for review and approval, @HyukjinKwon .

[SparkQA]

Test build #94962 has finished for PR 22155 at commit 27497f5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class Foo(bar: Option[String])

[HyukjinKwon]

Merged to master.

[dongjoon-hyun]

Thank you!