[SPARK-23807][FOLLOW-UP][BUILD][test-hadoop3.1] Add test-hadoop3.1 phrase

[wangyum]

What changes were proposed in this pull request?

Add test-hadoop3.1 phrase to test Spark against Spark’s Hadoop 3.1 profile.

How was this patch tested?

Tested on jenkins. This is output:

[info] Using build tool sbt with Hadoop profile hadoop3.1 under environment amplab_jenkins
...
[info] Building Spark (w/Hive 1.2.1) using SBT with these arguments:  -Phadoop-3.1 -Pkubernetes -Phive-thriftserver -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos test:package streaming-kinesis-asl-assembly/assembly

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103282/console

[felixcheung]

doesn't riselab jenkins need to be configured to set test-hadoop3.1?

[SparkQA]

Test build #103282 has finished for PR 24045 at commit 081f559.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

It does to build regularly like spark-branch-2.4-test-sbt-hadoop-2.7. Looks this targets to extract profile from PR title, which can be also used in PR builder. Hadoop 3 profile tests cannot be passed for now.

[HyukjinKwon]

Merged to master, to unblock @wangyum's work.

[HyukjinKwon]

@wangyum, BTW, let's file a separate JIRA next time when the fixed versions are expected to be different.

[wangyum]

Got it. Thanks @HyukjinKwon
This PR very similar to SPARK-12842. we still need update release-build.sh and building-spark.md once we officially support Hadoop-3.1.