HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) #616

virajjasani · 2019-09-12T12:48:03Z

No description provided.

hbase-common/pom.xml

hbase-server/src/main/java/org/apache/hadoop/hbase/http/jmx/JMXJsonServlet.java

hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheUtil.java

hbase-shaded/hbase-shaded-server/pom.xml

hbase-shaded/hbase-shaded-testing-util-tester/pom.xml

hbase-shaded/hbase-shaded-testing-util/pom.xml

hbase-testing-util/pom.xml

hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java

busbey · 2019-09-12T14:34:05Z

Is this a backport of HBASE-20587 or all new?

virajjasani · 2019-09-12T18:25:00Z

Except for supplemental-models.xml and adding hbase-shaded-miscellaneous dependency, mostly this is backport of HBASE-20587 for relevant files(a couple of files on branch-1 don't have Jackson2 and hence, unchanged in this PR)

hbase-testing-util/pom.xml

hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheUtil.java

busbey · 2019-09-14T11:47:09Z

This is looking great! I still need to try using it with a downstream project and after that I think it'll be good to go.

When we commit this it needs to go to all branches-1.

virajjasani · 2019-09-14T14:13:56Z

This is looking great! I still need to try using it with a downstream project and after that I think it'll be good to go.

When we commit this it needs to go to all branches-1.

Sure, in the meanwhile let me work on preparing backport patch for branch-1.4 and 1.3 if they have significant changes involved?

Update: It seems backport to branch-1.4 doesn't require extra effort, however branch-1.3 does

busbey · 2019-09-17T04:50:25Z

ugh. Can't believe I didn't think of jdk requirements. Just built this on branch-1 with -P release so I'd have shaded artifacts to test downstream

[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (hadoop-profile-min-maven-min-java-banned-xerces) @ hbase-common ---
[INFO] Restricted to JDK 1.7 yet org.apache.hbase.thirdparty:hbase-shaded-miscellaneous:jar:2.2.1:compile contains org/apache/hbase/thirdparty/com/google/common/annotations/Beta.class targeted to JDK 1.8
[WARNING] Rule 4: org.apache.maven.plugins.enforcer.EnforceBytecodeVersion failed with message:
HBase has unsupported dependencies.
  HBase requires that all dependencies be compiled with version 1.7 or earlier
  of the JDK to properly build from source.  You appear to be using a newer dependency. You can use
  either "mvn -version" or "mvn enforcer:display-info" to verify what version is active.
  Non-release builds can temporarily build with a newer JDK version by setting the
  'compileSource' property (eg. mvn -DcompileSource=1.8 clean package).
Found Banned Dependency: org.apache.hbase.thirdparty:hbase-shaded-miscellaneous:jar:2.2.1
Use 'mvn dependency:tree' to locate the source of the banned dependencies.

Let me think about what's going to be faster here. We could exclude the stuff other than gson or we could see if updating the dependencies of hbase-thirdparty to be jdk7 compatible. I know guava still works for jdk7 if we use the "android" variant instead of "jre".

busbey · 2019-09-20T02:19:32Z

I have some ideas on moving this forward. discussion on HBASE-23052

hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java

busbey · 2019-09-25T17:10:08Z

Okay! a new version of the hbase-thirdparty library is out now and includes a jdk7 compatible relocated GSON.

I just published it on the ASF nexus, so it might take a bit for it to show up in maven central. When it does you should be able to update this PR to use the following dependency:

<dependency>
  <groupId>org.apache.hbase.thirdparty</groupId>
  <artifactId>hbase-shaded-gson</artifactId>
  <version>3.0.0</version>
</dependency>

virajjasani · 2019-09-25T18:17:10Z

That's great! Thanks @busbey
Sure I will be on it, soon as my local build passes, will commit it

virajjasani · 2019-09-25T18:27:55Z

It is available: https://repository.apache.org/content/repositories/releases/org/apache/hbase/thirdparty/hbase-shaded-gson/3.0.0/

hbase-testing-util/pom.xml

busbey · 2019-09-25T19:59:33Z

Building with -P release fails while checking the shaded jars for correctness.

Need to update hbase-shaded/hbase-shaded-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh to allow for packages in org/apache/hbase since that's what the hbase-thirdparty stuff uses (as compared to org/apache/hadoop/hbase for the rest of our stuff). you can either cherry-pick b3f353c or just make the change yourself.

busbey

with the addition of the shaded jar check, I can successfully build with the release profile and verify that the jars we produce are usable downstream (checked by building YCSB with some local changes to allow use of the shaded testing util).

Also the module is visible in Maven Central now:

https://search.maven.org/artifact/org.apache.hbase.thirdparty/hbase-shaded-gson/3.0.0/jar

busbey · 2019-09-25T20:33:01Z

I plan to merge this tonight and get it back-ported to branch-1.4 as well, unless someone else has more feedback.

Apache-HBase · 2019-09-26T01:34:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
💙	reexec	1m 28s	Docker mode activated.
		_ Prechecks _
💚	dupname	0m 1s	No case conflicting files found.
💙	shelldocs	0m 0s	Shelldocs was not available.
💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
💚	@author	0m 0s	The patch does not contain any @author tags.
💚	test4tests	0m 0s	The patch appears to include 4 new or modified test files.
		_ branch-1 Compile Tests _
💙	mvndep	6m 21s	Maven dependency ordering for branch
💚	mvninstall	3m 42s	branch-1 passed
💚	compile	11m 28s	branch-1 passed with JDK v1.8.0_222
💔	compile	0m 17s	root in branch-1 failed with JDK v1.7.0_232.
💚	checkstyle	9m 56s	branch-1 passed
💚	shadedjars	3m 11s	branch has no errors when building our shaded downstream artifacts.
💚	javadoc	4m 27s	branch-1 passed with JDK v1.8.0_222
💚	javadoc	6m 45s	branch-1 passed with JDK v1.7.0_232
💙	spotbugs	11m 25s	Used deprecated FindBugs config; considering switching to SpotBugs.
💙	findbugs	0m 22s	branch/hbase-resource-bundle no findbugs output file (findbugsXml.xml)
💙	findbugs	0m 20s	branch/hbase-testing-util no findbugs output file (findbugsXml.xml)
💙	findbugs	0m 19s	branch/hbase-shaded/hbase-shaded-server no findbugs output file (findbugsXml.xml)
💙	findbugs	0m 20s	branch/hbase-shaded/hbase-shaded-testing-util no findbugs output file (findbugsXml.xml)
💙	findbugs	0m 12s	branch/hbase-shaded/hbase-shaded-testing-util-tester no findbugs output file (findbugsXml.xml)
💙	findbugs	0m 18s	branch/hbase-shaded/hbase-shaded-check-invariants no findbugs output file (findbugsXml.xml)
		_ Patch Compile Tests _
💙	mvndep	0m 27s	Maven dependency ordering for patch
💚	mvninstall	2m 6s	the patch passed
💚	compile	11m 6s	the patch passed with JDK v1.8.0_222
💚	javac	11m 6s	the patch passed
💔	compile	0m 17s	root in the patch failed with JDK v1.7.0_232.
💔	javac	0m 17s	root in the patch failed with JDK v1.7.0_232.
💚	checkstyle	0m 12s	The patch passed checkstyle in hbase-resource-bundle
💚	checkstyle	0m 30s	The patch passed checkstyle in hbase-common
💚	checkstyle	0m 37s	The patch passed checkstyle in hbase-client
💚	checkstyle	1m 48s	hbase-server: The patch generated 0 new + 256 unchanged - 25 fixed = 256 total (was 281)
💚	checkstyle	0m 12s	The patch passed checkstyle in hbase-testing-util
💚	checkstyle	0m 9s	The patch passed checkstyle in hbase-shaded-server
💚	checkstyle	0m 13s	The patch passed checkstyle in hbase-shaded-testing-util
💚	checkstyle	0m 12s	The patch passed checkstyle in hbase-shaded-testing-util-tester
💚	checkstyle	0m 13s	The patch passed checkstyle in hbase-shaded-check-invariants
💚	checkstyle	6m 1s	root: The patch generated 0 new + 256 unchanged - 25 fixed = 256 total (was 281)
💚	shellcheck	0m 0s	There were no new shellcheck issues.
💚	whitespace	0m 0s	The patch has no whitespace issues.
💔	xml	0m 1s	The patch has 9 ill-formed XML file(s).
💚	shadedjars	3m 4s	patch has no errors when building our shaded downstream artifacts.
💚	hadoopcheck	5m 12s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
💚	javadoc	4m 23s	the patch passed with JDK v1.8.0_222
💚	javadoc	6m 50s	the patch passed with JDK v1.7.0_232
💙	findbugs	0m 13s	hbase-resource-bundle has no data from findbugs
💙	findbugs	0m 20s	hbase-testing-util has no data from findbugs
💙	findbugs	0m 19s	hbase-shaded/hbase-shaded-server has no data from findbugs
💙	findbugs	0m 21s	hbase-shaded/hbase-shaded-testing-util has no data from findbugs
💙	findbugs	0m 13s	hbase-shaded/hbase-shaded-testing-util-tester has no data from findbugs
💙	findbugs	0m 17s	hbase-shaded/hbase-shaded-check-invariants has no data from findbugs
		_ Other Tests _
💔	unit	162m 49s	root in the patch failed.
💚	asflicense	4m 27s	The patch does not generate ASF License warnings.
		301m 43s

Reason	Tests
XML	Parsing Error(s):
	hbase-client/pom.xml
	hbase-common/pom.xml
	hbase-resource-bundle/src/main/resources/supplemental-models.xml
	hbase-server/pom.xml
	hbase-shaded/hbase-shaded-server/pom.xml
	hbase-shaded/hbase-shaded-testing-util-tester/pom.xml
	hbase-shaded/hbase-shaded-testing-util/pom.xml
	hbase-testing-util/pom.xml
	pom.xml
Failed junit tests	hadoop.hbase.replication.TestReplicationSmallTests

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-616/6/artifact/out/Dockerfile
GITHUB PR	#616
Optional Tests	dupname asflicense javac javadoc unit shadedjars hadoopcheck xml compile spotbugs findbugs hbaseanti checkstyle shellcheck shelldocs
uname	Linux a3c1d4838deb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-616/out/precommit/personality/provided.sh
git revision	branch-1 / `771e184`
Default Java	1.7.0_232
Multi-JDK versions	/usr/lib/jvm/zulu-8-amd64:1.8.0_222 /usr/lib/jvm/zulu-7-amd64:1.7.0_232
compile	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-616/6/artifact/out/branch-compile-root-jdk1.7.0_232.txt
compile	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-616/6/artifact/out/patch-compile-root-jdk1.7.0_232.txt
javac	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-616/6/artifact/out/patch-compile-root-jdk1.7.0_232.txt
xml	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-616/6/artifact/out/xml.txt
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-616/6/artifact/out/patch-unit-root.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-616/6/testReport/
Max. process+thread count	3924 (vs. ulimit of 10000)
modules	C: hbase-resource-bundle hbase-common hbase-client hbase-server hbase-testing-util hbase-shaded/hbase-shaded-server hbase-shaded/hbase-shaded-testing-util hbase-shaded/hbase-shaded-testing-util-tester hbase-shaded/hbase-shaded-check-invariants . U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-616/6/console
versions	git=1.9.1 maven=3.0.5 shellcheck=0.7.0 findbugs=3.0.1
Powered by	Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

busbey · 2019-09-26T02:46:43Z

compile | 0m 17s | root in branch-1 failed with JDK v1.7.0_232.

This is concerning, but unrelated. I'll chase it down later.

xml | 0m 1s | The patch has 9 ill-formed XML file(s).

this is HBASE-22732

Failed junit tests | hadoop.hbase.replication.TestReplicationSmallTests

Not related AFAICT.

* Backport HBASE-20587 * moving to shaded gson with jdk7 compatibility * Include jackson-mapper-asl for testing-util because of Hadoop * Update shaded jar check to allow hbase-thirdparty libs Signed-off-by: Sean Busbey <[email protected]> Co-authored-by: Duo Zhang <[email protected]>

HBASE-23015 : Moving from Jackson2 to shaded Gson

46ebfe9

This comment has been minimized.

Sign in to view

busbey requested changes Sep 12, 2019

View reviewed changes

virajjasani changed the title ~~HBASE-23015 : Moving from Jackson2 to shaded Gson~~ HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) Sep 12, 2019

addressing comments

0242e26

xcangCRM force-pushed the branch-1 branch from e5fdef5 to de9b1d4 Compare September 12, 2019 20:42

busbey requested changes Sep 12, 2019

View reviewed changes

hbase-testing-util/pom.xml Show resolved Hide resolved

hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheUtil.java Show resolved Hide resolved

This comment has been minimized.

Sign in to view

addendum

a0687c9

This comment has been minimized.

Sign in to view

busbey reviewed Sep 20, 2019

View reviewed changes

hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java Outdated Show resolved Hide resolved

import change

a7e298e

This comment has been minimized.

Sign in to view

moving to shaded gson with jdk7 compatibility

f392001

busbey reviewed Sep 25, 2019

View reviewed changes

hbase-testing-util/pom.xml Show resolved Hide resolved

adding comment for jackson-mapper-asl dependency

c67f90b

addendum shaded lib check

bc7d35b

busbey approved these changes Sep 25, 2019

View reviewed changes

busbey merged commit f77c14d into apache:branch-1 Sep 26, 2019

HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) #616

HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) #616

Uh oh!

Conversation

virajjasani commented Sep 12, 2019

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

busbey commented Sep 12, 2019

Uh oh!

virajjasani commented Sep 12, 2019

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

busbey commented Sep 14, 2019

Uh oh!

virajjasani commented Sep 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

busbey commented Sep 17, 2019

Uh oh!

busbey commented Sep 20, 2019

Uh oh!

Uh oh!

This comment has been minimized.

busbey commented Sep 25, 2019

Uh oh!

virajjasani commented Sep 25, 2019

Uh oh!

virajjasani commented Sep 25, 2019

Uh oh!

Uh oh!

busbey commented Sep 25, 2019

Uh oh!

busbey left a comment

Choose a reason for hiding this comment

Uh oh!

busbey commented Sep 25, 2019

Uh oh!

Apache-HBase commented Sep 26, 2019

Uh oh!

busbey commented Sep 26, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

virajjasani commented Sep 14, 2019 •

edited

Loading