[SPARK-30199][DSTREAM] Recover spark.(ui|blockManager).port from checkpoint

[dongjoon-hyun]

What changes were proposed in this pull request?

This is a backport of #26827.

This PR aims to recover spark.ui.port and spark.blockManager.port from checkpoint like spark.driver.port.

Why are the changes needed?

When the user configures these values, we can respect them.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the Jenkins with the newly added test cases.

[dongjoon-hyun]

This is requested by 2.4.6 release manager as SPARK-31544.
But, for traceability, I keep SPARK-30199 as a PR title. Please update both JIRAs when merging.

cc @holdenk since she is 2.4.6 release manager.
cc @gatorsmile since he asked a one-by-one review for backporting.
cc @aaruna since he is the original author of the patch. In master/3.0 and this PR, the authorship is @aaruna always.

Although SPARK-30199 is categorized as an improvement originally, I'm also supporting @holdenk 's decision on SPARK-31544 as a bug fix.

cc @dbtsai

[holdenk]

I think this looks good to me, pending Jenkins. If folks disagree about it being a bugfix I'm ok with us skipping it (although I think it's more bugfix than improvement).

[SparkQA]

Test build #121699 has finished for PR 28320 at commit 0c14b27.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dbtsai]

LGTM. Thanks!

[aaruna]

LGTM, thanks for the backport!

[dongjoon-hyun]

Thank you, @holdenk , @dbtsai , @aaruna .
Merged to branch-2.4 for Apache Spark 2.4.6 release.