HBASE-25754 StripeCompactionPolicy should support compacting cold reg… #3152

sunhelly · 2021-04-12T11:32:25Z

…ions

Apache-HBase · 2021-04-12T12:30:33Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 27s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 48s	master passed
+1 💚	compile	3m 37s	master passed
+1 💚	checkstyle	1m 21s	master passed
+1 💚	spotbugs	2m 44s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 25s	the patch passed
+1 💚	compile	3m 37s	the patch passed
+1 💚	javac	3m 37s	the patch passed
+1 💚	checkstyle	1m 15s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	22m 48s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 21s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		57m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3152
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 3cdb0f1dd169 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `f9e928e`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…ions

Apache-HBase · 2021-04-14T03:52:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 29s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 3s	master passed
+1 💚	compile	3m 17s	master passed
+1 💚	checkstyle	1m 6s	master passed
+1 💚	spotbugs	2m 12s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 33s	the patch passed
+1 💚	compile	3m 21s	the patch passed
-0 ⚠️	javac	3m 21s	hbase-server generated 1 new + 192 unchanged - 1 fixed = 193 total (was 193)
+1 💚	checkstyle	1m 3s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	20m 30s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	3m 6s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		52m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3152
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux c42c3465c295 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c5b0989`
Default Java	AdoptOpenJDK-1.8.0_282-b08
javac	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/2/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-04-14T05:50:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 30s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	5m 45s	master passed
+1 💚	compile	1m 14s	master passed
+1 💚	shadedjars	8m 5s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 44s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 19s	the patch passed
+1 💚	compile	1m 14s	the patch passed
+1 💚	javac	1m 14s	the patch passed
+1 💚	shadedjars	7m 49s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	the patch passed
		_ Other Tests _
-1 ❌	unit	137m 50s	hbase-server in the patch failed.
		170m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3152
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 945ac0664cc5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c5b0989`
Default Java	AdoptOpenJDK-11.0.10+9
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/2/testReport/
Max. process+thread count	4342 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-04-14T07:59:01Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 6s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	6m 0s	master passed
+1 💚	compile	1m 26s	master passed
+1 💚	shadedjars	10m 5s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 54s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 47s	the patch passed
+1 💚	compile	1m 34s	the patch passed
+1 💚	javac	1m 34s	the patch passed
+1 💚	shadedjars	10m 37s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 50s	the patch passed
		_ Other Tests _
-1 ❌	unit	256m 37s	hbase-server in the patch failed.
		298m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3152
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux f52feef47ed4 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c5b0989`
Default Java	AdoptOpenJDK-1.8.0_282-b08
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/2/testReport/
Max. process+thread count	2542 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3152/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

sunhelly · 2021-04-14T16:57:54Z

Reported the test failure to https://issues.apache.org/jira/browse/HBASE-25774

Apache9 · 2021-04-21T12:31:41Z

You use StripeCompactionPolicy? Good, mind sharing some more information? Thanks.

sunhelly · 2021-04-21T15:47:10Z

Hi, @Apache9 , it's my pleasure to share these information.
We use StripeCompactionPolicy in almost all of our production clusters. And we let recently data in memstore flush to L0, limit stripe size to about 10G. Most of our regions are large than 50G, there even exist regions larger than 2T... StripeCompactionPolicy has no major compactions, and it can limit compactions in only L0 and one stripe files, and it can perform cells deletion in one stripe just like the major-compaction. The pressure of compactions is broken down. Though the total files count in a region maybe a little larger than normal compactions, because the files are organized as in mini-regions, it works well for most read requests.
And we also implemented a fast split and compact method based on StripeCompactionPolicy, practiced in all our production clusters, results show that split is very light weight and no need to perform read+write files compactions right after split. Details are in HBASE-25302, hope you have interest...
Thanks.

Apache9 · 2021-04-23T06:52:57Z

Thanks. If you do not mind, I plan to send the above comment to mailing list.

…ions (#3152) Signed-off-by: Duo Zhang <[email protected]>

sunhelly · 2021-04-23T08:57:15Z

Of course, I'm glad to share these information. Thanks, @Apache9.

HBASE-25754 StripeCompactionPolicy should support compacting cold reg…

4e02917

…ions

sunhelly force-pushed the HBASE-25754 branch from fa1cad9 to 4e02917 Compare April 14, 2021 02:59

Apache9 approved these changes Apr 21, 2021

View reviewed changes

Apache9 merged commit 996862c into apache:master Apr 23, 2021

asfgit pushed a commit that referenced this pull request Apr 23, 2021

HBASE-25754 StripeCompactionPolicy should support compacting cold reg…

e6cb7f2

…ions (#3152) Signed-off-by: Duo Zhang <[email protected]>

asfgit pushed a commit that referenced this pull request Apr 23, 2021

HBASE-25754 StripeCompactionPolicy should support compacting cold reg…

1dc881a

…ions (#3152) Signed-off-by: Duo Zhang <[email protected]>

asfgit pushed a commit that referenced this pull request Apr 23, 2021

HBASE-25754 StripeCompactionPolicy should support compacting cold reg…

fd7b2c6

…ions (#3152) Signed-off-by: Duo Zhang <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-25754 StripeCompactionPolicy should support compacting cold reg… #3152

HBASE-25754 StripeCompactionPolicy should support compacting cold reg… #3152

Uh oh!

sunhelly commented Apr 12, 2021

Uh oh!

Apache-HBase commented Apr 12, 2021

Uh oh!

Apache-HBase commented Apr 14, 2021

Uh oh!

Apache-HBase commented Apr 14, 2021

Uh oh!

Apache-HBase commented Apr 14, 2021

Uh oh!

sunhelly commented Apr 14, 2021

Uh oh!

Apache9 commented Apr 21, 2021

Uh oh!

sunhelly commented Apr 21, 2021

Uh oh!

Apache9 commented Apr 23, 2021

Uh oh!

sunhelly commented Apr 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HBASE-25754 StripeCompactionPolicy should support compacting cold reg… #3152

HBASE-25754 StripeCompactionPolicy should support compacting cold reg… #3152

Uh oh!

Conversation

sunhelly commented Apr 12, 2021

Uh oh!

Apache-HBase commented Apr 12, 2021

Uh oh!

Apache-HBase commented Apr 14, 2021

Uh oh!

Apache-HBase commented Apr 14, 2021

Uh oh!

Apache-HBase commented Apr 14, 2021

Uh oh!

sunhelly commented Apr 14, 2021

Uh oh!

Apache9 commented Apr 21, 2021

Uh oh!

sunhelly commented Apr 21, 2021

Uh oh!

Apache9 commented Apr 23, 2021

Uh oh!

sunhelly commented Apr 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants