[SPARK-23992][CORE] ShuffleDependency does not need to be deserialized every time

[10110346]

What changes were proposed in this pull request?

In the same stage, 'ShuffleDependency' is not necessary to be deserialized each time.

How was this patch tested?

Existing unit test

[SparkQA]

Test build #89394 has finished for PR 21079 at commit 2239e78.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[Ngone51]

Do we need to clear this cache at the end of a app ?

[10110346]

we don't need to clear, this cache can recovery itself

[SparkQA]

Test build #89433 has finished for PR 21079 at commit c647f0a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

Have you seen any performance improvement with this change?

[HyukjinKwon]

ping @10110346 for ^

[10110346]

I'm sorry to be so late reply to you!
I have tested in my production environment, it has a bit of performance improvement. @jiangxb1987 @HyukjinKwon

[HyukjinKwon]

Mind if I ask to share the number and post it in the PR description?

[SparkQA]

Test build #94505 has finished for PR 21079 at commit c647f0a.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.