[SPARK-29981][BUILD] Add hive-1.2/2.3 profiles

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims the followings.

• Add two profiles, hive-1.2 and hive-2.3 (default)
• Validate if we keep the existing combination at least. (Hadoop-2.7 + Hive 1.2 / Hadoop-3.2 + Hive 2.3).

For now, we assumes that hive-1.2 is explicitly used with hadoop-2.7 and hive-2.3 with hadoop-3.2. The followings are beyond the scope of this PR.

• SPARK-29988 Adjust Jenkins jobs for hive-1.2/2.3 combination
• SPARK-29989 Update release-script for hive-1.2/2.3 combination
• SPARK-29991 Support hive-1.2/2.3 in PR Builder

Why are the changes needed?

This will help to switch our dependencies to update the exposed dependencies.

Does this PR introduce any user-facing change?

This is a dev-only change that the build profile combinations are changed.

• -Phadoop-2.7 => -Phadoop-2.7 -Phive-1.2
• -Phadoop-3.2 => -Phadoop-3.2 -Phive-2.3

How was this patch tested?

Pass the Jenkins with the dependency check and tests to make it sure we don't change anything for now.

• Jenkins (-Phadoop-2.7 -Phive-1.2)
• Jenkins (-Phadoop-3.2 -Phive-2.3)

Also, from now, GitHub Action validates the following combinations.

[dongjoon-hyun]

Hi, @srowen , @liancheng , @wangyum .
Could you review this approach? This PR switched the default and adds hive-1.2 and hive-2.3 profiles

[srowen]

That looks correct so far. Does the release script need to accommodate the new build combinations we've been talking about? We need at least the Hadoop 2 / Hive 1, and Hadoop 3 / Hive 2 flavors. I think I lost track of whether there was an argument for Hadoop 3 / Hive 1 ? if that even works.

[dongjoon-hyun]

Retest this please.

[dongjoon-hyun]

Thank you for review.

Up to now, there was no request on Hadoop 3 / Hive 1 combination~ And, it will not work. So, the following four?

• -Phadoop-2.7 -Phive-1.2
• -Phadoop-2.7 -Phive-2.3 (This is a new one and we don't have this combination in Jenkins yet)
• -Phadoop-3.2 -Phive-2.3
• -Phadoop-provided

[dongjoon-hyun]

@srowen , @liancheng , @wangyum , @gatorsmile .
One thing might look weird for now. We have v2.3.5 directory in sql/core. It means we need hive-2.3 for sql/core compilation in order to bring the extra dependency. If we make this as a top dependency, we need to change dev/deps. So, I didn't change it.

$ find . -name v2.3.5
./sql/core/v2.3.5
./sql/hive-thriftserver/v2.3.5

Previously, it's -Phadoop-3.2. So, it doesn't look weird much, but now it's -Phive-2.3 without -Phive.

    <profile>
       <id>hive-2.3</id>
       <!-- Default hive profile. Uses global properties. -->
       <dependencies>
         <!-- Both Hive and ORC need hive-storage-api, but it is excluded by orc-mapreduce -->
         <dependency>
          <groupId>org.apache.hive</groupId>
          <artifactId>hive-storage-api</artifactId>
          <version>2.6.0</version>
        </dependency>
      </dependencies>
    </profile>

[HyukjinKwon]

Hm, @dongjoon-hyun, can you create JIRAs to follow up? Seems we will change -Phive-2.3 as a default profile as discussed in the mailing list.

1. Verify *Hadoop 2.7 and Hive 2.3 combination in both JDK 8 and JDK 11 (WIP) as it will be the default
2. Setting a Jenkins job for -Phadoop-2.7 -Phive-2.3.
3. We will need to be able to configure Hive version and also Hadoop version in the PR builder.
4. Change the default profile to Hive 2.3.
5. [SPARK-29981][BUILD] Add hive-1.2/2.3 profiles #26619 (comment)
6. Release script updates.
7. ... (other more .. ?)

*Hadoop 2 will be default at this moment. it's being discussed in the mailing list

Many things are going on so it looks very easy to lose the track.

[dongjoon-hyun]

Sure!

[dongjoon-hyun]

BTW, the thread is independent from JDK11 or Hadoop 3. For the following, only JDK8 is the target of the follow-up of this PR. And, for JDK11, I believe @wangyum will handle it at his on-going work.

Verify *Hadoop 2.7 and Hive 2.3 combination in both JDK 8 and JDK 11 (WIP) as it will be the default

For (4), Hive 2.3 is already default for Hadoop-3. And, this PR makes Hive 2.3 as a default in the pom files. So, I guess the remains of (4) is equal to (2) and (3).

[dongjoon-hyun]

The follow-up issues are created and mentioned in the PR description, @HyukjinKwon .

[HyukjinKwon]

Ah, sure, thanks for clarification. @wangyum, can you verify the new JDK 11 combination and take a following action?

[wangyum]

@HyukjinKwon I will do it later.

[SparkQA]

Test build #114192 has finished for PR 26619 at commit 1e0f23d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #114196 has finished for PR 26619 at commit 1e0f23d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #114204 has finished for PR 26619 at commit f93742a.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, All.

I updated the PR description. This PR successfully verified the existing profiles and this PR also updates PR builder and dependency check script in the same way like before.

To update Jenkins jobs with new way, we need to merge this PR first because we need to use these profile. Also, I noticed that there is some redundancy in GitHub Action Cache. I'll handle them in another PR to avoid conflicts. The master branch already updated GitHub Action Cache yesterday.

[dongjoon-hyun]

Hi, @shaneknapp .
We are changing the build combination which will affects the Jenkins jobs in master branch. Basically, this PR makes the same situation in the existing jobs.

We need to rename first. (SBT/Maven)

• spark-master-test-sbt-hadoop-2.7 -> spark-master-test-sbt-hadoop-2.7-hive-1.2
• spark-master-test-sbt-hadoop-3.2 -> spark-master-test-sbt-hadoop-3.2-hive-2.3

After that, we need to add new combination. (SBT/Maven)

• spark-master-test-sbt-hadoop-2.7-hive-2.3

I'll file a JIRA for this after when this PR is merged.

[liancheng]

BTW, any reason why we needed the patch version in the directory names initially? I don't think we need different shims for Hive patch versions. If that's true, I'd suggest renaming the sql/hive-thriftserver/{v1.2.1,v2.3.5} folders to just v1.2 and v2.3. Of course, this can be done in a follow-up PR.

[dongjoon-hyun]

Sure. We can use the short versions (v1.2 and v2.3). I'll do in the next follow-up PR. It will include renaming files mostly.

[dongjoon-hyun]

Since we are already in Apache Hive 2.3.6, I don't expect many changes at 2.3.7+.

[HyukjinKwon]

Looks fine so far to me but let me defer to other people here.

[dongjoon-hyun]

Thank you, @HyukjinKwon !

[dongjoon-hyun]

Thank you all. I'll merge this to the master and continue to work. This keeps every existing features. I'm sure it's safe until now.

[gatorsmile]

-Phive-2.3 -Phive What is the result if we specify both at the same time?

[dongjoon-hyun]

-Phive enables modules like hive and hive-thriftserver.
-Phive-1.2 and -Phive-2.3 chooses the Hive version (and it's classes).
In principal, we should use both of them(-Phive -Phive-1.2 or -Phive -Phive-2.3). However, one of -Phive-2.3 and -Phive-2.3 is the default and we can ignore that.

The default of AS-IS master branch is -Phive-2.3.

[dongjoon-hyun]

One additional stuff is sql/core module.
Previously, hadoop-3.2 replaces the previous ORC code with Hive code.
When -Phadoop-3.2 is given, Apache Hive storage classes are used.
Now, it depends on -Phive-1.2 or -Phive-2.3. In other words, -Phive-1.2 and -Phive-2.3 is used in sql/core module without -Phive.

[liancheng]

Hey @dongjoon-hyun, was investigating some dependency related issues, and found that we are using the nohive classifier for orc-core under profile hive-1.2 but not hive-2.3.

I'm not quite familiar with the orc-core part. Seems that the nohive classifier is mostly for shading hive-storage-api and protobuf. Why we want different artifacts for hive-1.2 and hive-2.3 profiles?

[dongjoon-hyun]

Yes. It's correct.

The removal of nohive classifier in Hive 2.3 code path is @wangyum 's change.
While updating for Hadoop 3.2 and Hive 2.3, @wangyum 's updated Hive-related Spark code and switched them to use the vanilla jars for Hive 2.3 code path.

For hive-1.2, we still need a shaded nohive classifier jars to avoid the conflicts.

[dongjoon-hyun]

Historically I was against the change because sql/core is designed to use ORC without -Phive. And, Apache ORC nohive classifier jar is designed to support that in Apache Spark. Anyway, the current status of Apache Spark is supporting both.

[liancheng]

Thank you very much for the context, @dongjoon-hyun!

[yhuai]

@dongjoon-hyun i found hive storage api and hive common have the following common class files

org/apache/hadoop/hive/common/ValidReadTxnList.class
org/apache/hadoop/hive/common/ValidTxnList.class
org/apache/hadoop/hive/common/ValidTxnList$RangeResponse.class

For example, https://github.com/apache/hive/blob/rel/storage-release-2.6.0/storage-api/src/java/org/apache/hadoop/hive/common/ValidReadTxnList.java (pulled in by orc 1.5.8) and https://github.com/apache/hive/blob/rel/release-2.3.6/common/src/java/org/apache/hadoop/hive/common/ValidReadTxnList.java (from hive-common 2.3.6) both are in the classpath and they are different. I am worried that users may hit issues due to classloading order. I think it will be safe to still use nohive classifier for orc, which shades storage api.

I think there is no downside of using orc nohive, right? Why not just keep using that. So, we will not need to worry about having class conflicts.

[yhuai]

i created https://issues.apache.org/jira/browse/SPARK-30784 for switching back to orc-nohive.

[dongjoon-hyun]

Thank you, @yhuai . During weekend, I missed your comment here.

[dongjoon-hyun]

BTW, @yhuai . As I mentioned before to @liancheng , the transition from orc-nohive to hive-storage-api is not made by me. That is irrelevant to this PR. Technically, I also was against it initially like you.

I think there is no downside of using orc nohive, right?

Let me you ping on the right PR.

• [SPARK-27176][SQL] Upgrade hadoop-3's built-in Hive maven dependencies to 2.3.4 #23788

[suryanshagnihotri]

@dongjoon-hyun have we tried to change hive version to 3.x in spark in order to make it work with hive metastore 3.x.

[dongjoon-hyun]

Did you try spark.sql.hive.metastore.version, @suryanshagnihotri ?

• https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore

For some known issues, we are upgrading the built-in hive module to 2.3.9 here.

• [SPARK-34512][BUILD][SQL] Upgrade built-in Hive to 2.3.9 #32750

[suryanshagnihotri]

@dongjoon-hyun yes have tried it but it fails with class not found exceptions like

java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.plan.LoadTableDesc$LoadFileType
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:255)
	at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:244)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.apache.spark.sql.hive.client.Shim_v3_0.clazzLoadFileType$lzycompute(HiveShim.scala:1272)
	at org.apache.spark.sql.hive.client.Shim_v3_0.clazzLoadFileType(HiveShim.scala:1271)
	at org.apache.spark.sql.hive.client.Shim_v3_0.loadTable(HiveShim.scala:1352)
	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$loadTable$1(HiveClientImpl.scala:894)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:295)
	at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:228)
	at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:227)
	at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:277)
	at org.apache.spark.sql.hive.client.HiveClientImpl.loadTable(HiveClientImpl.scala:889)
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$loadTable$1(HiveExternalCatalog.scala:884)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:103)
	at org.apache.spark.sql.hive.HiveExternalCatalog.loadTable(HiveExternalCatalog.scala:878)
	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.loadTable(ExternalCatalogWithListener.scala:167)
	at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:327)
	at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:102)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:120)
	at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:229)

and I checked this LoadTableDesc$LoadFileType (enum) is added in hive 3.x

[suryanshagnihotri]

@dongjoon-hyun I get this exception when i try to insert data into table. I checked that spark_home/jars contain only hive 2.3.7 jars hence we are getting this exception.

query which does not work

CREATE TABLE students (id int, name string, marks int);
INSERT INTO students VALUES
(1, 'Jerry', 50);

I also noticed that when i make table as acid then it works.

CREATE TABLE students (id int, name string, marks int)
    STORED AS ORC TBLPROPERTIES ('transactional' = 'true');
INSERT INTO students VALUES
(1, 'Jerry', 50);