[WIP][SPARK-29250][BUILD][test-hadoop3.2][test-maven] Upgrade to Hadoop 3.2.1

[HyukjinKwon]

What changes were proposed in this pull request?

This PR aims to upgrade Hadoop version from 3.2.0 to 3.2.1 in hadoop-3.2 profile. This is #25932.

Why are the changes needed?

Hadoop 3.2.1 has 493 patches including client bug fixes and improvements.

• https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-common/release/3.2.1/CHANGELOG.3.2.1.html

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the Jenkins with the existing tests.

For the dependency, this PR is tested on both JDK8/JDK11. There is no difference based on JDK versions.

[SparkQA]

Test build #115788 has finished for PR 27009 at commit 7bcb8d4.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I think you'll have to vary the guava version that we override with, to 27.0-jre IIRC, in the Hadoop 3.2 profile

[HyukjinKwon]

Sure, thanks Sean. Let me take a look

[SparkQA]

Test build #115835 has finished for PR 27009 at commit 7ac9e3e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Hm, seems it's difficult to fix:

  java.lang.IllegalAccessError: tried to access method com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator; from class org.apache.hadoop.hive.ql.exec.FetchOperator
  at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:108)
  at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
  at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
  at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
  at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$runHive$1(HiveClientImpl.scala:820)
  at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:321)
  at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:255)
  ...

Hive 2.3.6 itself looks needing a fix but seems there's no plan to upgrade. now I got #25932 (comment) :-) ..

[SparkQA]

Test build #115855 has finished for PR 27009 at commit 3224e90.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Hm, I can't see the easy way to work around too. Closing for now.