HIVE-28191: Upgrade Hadoop and Hbase Client version #5481

java.io.IOException: Shutting down
	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:257)
	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:110)
	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1185)
	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1232)
	at org.apache.hadoop.hive.hbase.HBaseTestSetup.setUpFixtures(HBaseTestSetup.java:112)
	at org.apache.hadoop.hive.hbase.HBaseTestSetup.preTest(HBaseTestSetup.java:78)
	at org.apache.hadoop.hive.ql.QTestMiniClusters.setup(QTestMiniClusters.java:240)
	at org.apache.hadoop.hive.ql.QTestUtil.<init>(QTestUtil.java:222)
	at org.apache.hadoop.hive.hbase.HBaseQTestUtil.<init>(HBaseQTestUtil.java:44)
	at org.apache.hadoop.hive.cli.control.CoreHBaseNegativeCliDriver.beforeClass(CoreHBaseNegativeCliDriver.java:49)
	at org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
Caused by: java.lang.RuntimeException: Master not initialized after 200000ms
	at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:221)
	at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:190)
	at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:407)
	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:250)
	... 21 more

But the info is not very clear. I think you can refer to this ticket https://issues.apache.org/jira/browse/HBASE-28061 to explore the real failed exception details. You can write a simple java test to test the MiniHbaseCluster on Hadoop 3.4.0

linghengqian · 2024-10-05T14:52:05Z

But the info is not very clear. I think you can refer to this ticket https://issues.apache.org/jira/browse/HBASE-28061 to explore the real failed exception details. You can write a simple java test to test the MiniHbaseCluster on Hadoop 3.4.0

@zhangbutao

I've opened https://issues.apache.org/jira/browse/HIVE-28559 and I can't even start any relevant unit tests locally, which prevents me from even debugging. Unit tests that failed to execute include org.apache.hadoop.hive.metastore.TestRemoteHiveMetastoreWithHttpJwt, org.apache.hive.service.auth.jwt.TestHttpJwtAuthentication and org.apache.hadoop.hive.cli.TestHBaseCliDriver.
I'm not sure what happened to the master branch, it doesn't seem like something documented at https://cwiki.apache.org/confluence/display/Hive/HowToContribute . Either Hive's master branch can't be compiled with JDK8, or Hive's master branch can't be compiled with Maven 3.9.9. And Hive hasn't used Maven Wrapper to unify the environment for contributors yet.

zhangbutao · 2024-10-09T07:29:21Z

Failed Hbase Handler tests are related to HBASE-28908
Failed tests TestRemoteHiveMetastoreWithHttpJwt#testValidJWT and TestHttpJwtAuthentication#testAuthorizedUser is related Hadoop3.4.0 change HADOOP-17009: Embrace Immutability of Java Collections hadoop#1974
https://github.com/apache/hadoop/pull/1974/files#diff-372a0d25bcccd88b409a8149949628abd7d3472a1798bebe813e7617b0ef73c7L918-L920

Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.ProcessEnvironment$Variable
	at java.lang.ProcessEnvironment$StringEnvironment.toEnvironmentBlock(ProcessEnvironment.java:273) ~[?:1.8.0_221]
	at java.lang.ProcessEnvironment.toEnvironmentBlock(ProcessEnvironment.java:298) ~[?:1.8.0_221]
	at java.lang.ProcessImpl.start(ProcessImpl.java:86) ~[?:1.8.0_221]
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_221]
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:998) ~[hadoop-common-3.4.0.jar:?]
	at org.apache.hadoop.util.Shell.run(Shell.java:959) ~[hadoop-common-3.4.0.jar:?]
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1282) ~[hadoop-common-3.4.0.jar:?]
	at org.apache.hadoop.util.Shell.execCommand(Shell.java:1377) ~[hadoop-common-3.4.0.jar:?]
	at org.apache.hadoop.util.Shell.execCommand(Shell.java:1359) ~[hadoop-common-3.4.0.jar:?]
	at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1535) ~[hadoop-common-3.4.0.jar:?]
	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNonNativeIO(RawLocalFileSystem.java:1000) ~[hadoop-common-3.4.0.jar:?]
	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:991) ~[hadoop-common-3.4.0.jar:?]
	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:952) ~[hadoop-common-3.4.0.jar:?]
	at org.apache.hadoop.hive.ql.exec.Utilities.ensurePathIsWritable(Utilities.java:4954) ~[classes/:?]
	at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:843) ~[classes/:?]
	at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:784) ~[classes/:?]
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:708) ~[classes/:?]
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:669) ~[classes/:?]
	at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:182) ~[classes/:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_221]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_221]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_221]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_221]
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[classes/:?]
	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[classes/:?]
	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[classes/:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_221]
	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_221]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) ~[hadoop-common-3.4.0.jar:?]
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[classes/:?]
	at com.sun.proxy.$Proxy58.open(Unknown Source) ~[?:?]
	at org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:485) ~[classes/:?]
	... 40 more

slfan1989 · 2024-10-10T09:11:36Z

@linghengqian @zhangbutao Thank you for your attention to this JIRA! I will close my PR(#5187) and let’s work together to resolve the issue in this PR.

linghengqian · 2024-10-10T10:27:41Z

@linghengqian @zhangbutao Thank you for your attention to this JIRA! I will close my PR(#5187) and let’s work together to resolve the issue in this PR.

Sorry, my outlook mailbox mistakenly classified the emails from the ASF JIRA issue into spam. I just saw the launch method for executing integration tests, I am still investigating on my local machine.

zhangbutao · 2024-10-10T13:38:38Z

Superseded by #5500

linghengqian · 2024-10-10T13:42:46Z

Superseded by #5500

No problem, I'll close the current PR.

slfan1989 and others added 5 commits April 10, 2024 08:20

HIVE-28191. Upgrade Hadoop Version to 3.4.0.

28c59ab

Merge branch 'apache:master' into HIVE-28191

1fb7cab

Merge branch 'apache:master' into HIVE-28191

a4332f8

HIVE-28191. Upgrade Hadoop Version to 3.4.0.

62581c8

HIVE-28191. Fix CheckStyle.

971461b

github-actions bot requested a review from miklosgergely October 3, 2024 05:29

asf-ci-hive added tests pending tests unstable and removed tests pending labels Oct 3, 2024

Merge remote-tracking branch 'apache/master' into HIVE-28191

c8d3333

asf-ci-hive added tests pending and removed tests unstable labels Oct 3, 2024

HIVE-28191: Upgrade Hadoop and Hbase Client version

1175b1c

asf-ci-hive added tests failed tests pending and removed tests pending tests failed labels Oct 3, 2024

asf-ci-hive added tests unstable and removed tests pending labels Oct 3, 2024

linghengqian commented Oct 4, 2024

View reviewed changes

slfan1989 mentioned this pull request Oct 10, 2024

HIVE-28191. Upgrade Hadoop Version to 3.4.0. #5187

Closed

linghengqian closed this Oct 10, 2024

linghengqian deleted the HIVE-28191 branch October 10, 2024 14:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

HIVE-28191: Upgrade Hadoop and Hbase Client version #5481

HIVE-28191: Upgrade Hadoop and Hbase Client version #5481

Uh oh!

linghengqian commented Oct 3, 2024 •

edited

Loading

Uh oh!

sonarqubecloud bot commented Oct 3, 2024

Uh oh!

linghengqian left a comment

Uh oh!

zhangbutao commented Oct 4, 2024 •

edited

Loading

Uh oh!

linghengqian commented Oct 5, 2024

Uh oh!

zhangbutao commented Oct 9, 2024 •

edited

Loading

Uh oh!

slfan1989 commented Oct 10, 2024 •

edited

Loading

Uh oh!

linghengqian commented Oct 10, 2024

Uh oh!

zhangbutao commented Oct 10, 2024

Uh oh!

linghengqian commented Oct 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

HIVE-28191: Upgrade Hadoop and Hbase Client version #5481

HIVE-28191: Upgrade Hadoop and Hbase Client version #5481

Uh oh!

Conversation

linghengqian commented Oct 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

Is the change a dependency upgrade?

How was this patch tested?

Uh oh!

sonarqubecloud bot commented Oct 3, 2024

Quality Gate passed

Uh oh!

linghengqian left a comment

Choose a reason for hiding this comment

Uh oh!

zhangbutao commented Oct 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linghengqian commented Oct 5, 2024

Uh oh!

zhangbutao commented Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slfan1989 commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linghengqian commented Oct 10, 2024

Uh oh!

zhangbutao commented Oct 10, 2024

Uh oh!

linghengqian commented Oct 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

linghengqian commented Oct 3, 2024 •

edited

Loading

zhangbutao commented Oct 4, 2024 •

edited

Loading

zhangbutao commented Oct 9, 2024 •

edited

Loading

slfan1989 commented Oct 10, 2024 •

edited

Loading