SPARK-5492. Thread statistics can break with older Hadoop versions

[sryza]

No description provided.

[SparkQA]

Test build #26495 has started for PR 4305 at commit b7d4497.

• This patch merges cleanly.

[pwendell]

LGTM pending tests. Thanks, Sandy

[SparkQA]

Test build #26495 has finished for PR 4305 at commit b7d4497.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26495/
Test PASSed.