[SPARK-21993][SQL] Close sessionState when finish

[jinxing64]

What changes were proposed in this pull request?

In current code, SessionState in SparkSQLCLIDriver is not guaranteed to be closed. As a result, some tmp files/dirs are not removed when job finished. We can close SessionState in shutdown hook.

[srowen]

I don't know if it matters, but keep in mind this will hold a reference to sessionState then until the JVM exits

[SparkQA]

Test build #81713 has finished for PR 19219 at commit 2a06c0c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jinxing64]

@gatorsmile @HyukjinKwon
Could you please give some advice on this?

[jinxing64]

It seems there are still some other places where session state is not guaranteed to be closed. I will update this pr soon.

[SparkQA]

Test build #81838 has finished for PR 19219 at commit 1187dac.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[cenyuhai]

newSession is to isolate SparkSession in Spark ThriftServer, You must judge whether is thriftserver or not.

[SparkQA]

Test build #81874 has finished for PR 19219 at commit d7a484d.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #81997 has finished for PR 19219 at commit df49a19.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #82017 has finished for PR 19219 at commit e421113.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

cc @liufengdb

[liufengdb]

The major issue this PR tries to cover has been fixed by #20029, so I think we are good if there are no calls to HiveClientImpl.newSession. We can close this PR with no-fix.

[SparkQA]

Test build #86911 has finished for PR 19219 at commit e421113.

• This patch fails PySpark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.