HBASE-27229 BucketCache statistics should not count evictions by hfile #4639

bbeaudreault · 2022-07-21T00:38:18Z

This change mirrors how it works in LruBlockCache and the other implementations. A evictedByEvictionProcess boolean is added to doEvictBlock and blockEvicted, which wraps the cacheStats.evicted calls. Then, all calls to those methods are set to pass in false except the calls originating from the freeSpace(String why) method.

There are various edge cases which can result in a cache eviction. It's hard to test all of those, but I added a test which covers the 3 main cases.

Apache-HBase · 2022-07-21T01:32:27Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 40s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 21s	master passed
+1 💚	compile	7m 19s	master passed
+1 💚	checkstyle	0m 40s	master passed
+1 💚	spotless	0m 56s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 32s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 41s	the patch passed
+1 💚	compile	2m 37s	the patch passed
+1 💚	javac	2m 37s	the patch passed
+1 💚	checkstyle	0m 30s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	15m 31s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotless	1m 9s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 20s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		47m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4639
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 95969b20207a 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `da27a67`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-21T11:23:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 16s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 3s	master passed
+1 💚	compile	2m 50s	master passed
+1 💚	checkstyle	0m 34s	master passed
+1 💚	spotless	0m 49s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 36s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 51s	the patch passed
+1 💚	compile	2m 38s	the patch passed
+1 💚	javac	2m 38s	the patch passed
+1 💚	checkstyle	0m 36s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	13m 24s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotless	0m 47s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 36s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		37m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4639
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux abb3f991cd1d 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `31fc97e`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-21T14:35:15Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 10s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 35s	master passed
+1 💚	compile	0m 40s	master passed
+1 💚	shadedjars	3m 47s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 17s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	3m 46s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	211m 56s	hbase-server in the patch passed.
		229m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4639
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 9b4ce8c9d477 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `31fc97e`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/testReport/
Max. process+thread count	2453 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-21T15:08:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 7s	Docker mode activated.
-0 ⚠️	yetus	0m 5s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 54s	master passed
+1 💚	compile	0m 46s	master passed
+1 💚	shadedjars	3m 45s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 29s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 39s	the patch passed
+1 💚	compile	0m 47s	the patch passed
+1 💚	javac	0m 47s	the patch passed
+1 💚	shadedjars	3m 47s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	the patch passed
		_ Other Tests _
-1 ❌	unit	244m 28s	hbase-server in the patch failed.
		262m 49s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4639
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b58d740b4ea4 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `31fc97e`
Default Java	AdoptOpenJDK-11.0.10+9
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/testReport/
Max. process+thread count	2789 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-25T02:11:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 9s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 15s	master passed
+1 💚	compile	2m 14s	master passed
+1 💚	checkstyle	0m 31s	master passed
+1 💚	spotless	0m 44s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 18s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 15s	the patch passed
+1 💚	compile	2m 13s	the patch passed
+1 💚	javac	2m 13s	the patch passed
+1 💚	checkstyle	0m 30s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	11m 28s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 24s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 9s	The patch does not generate ASF License warnings.
		31m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4639
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 28daa80393db 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0ae42dd`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-25T05:29:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 10s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 17s	master passed
+1 💚	compile	0m 41s	master passed
+1 💚	shadedjars	3m 45s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 21s	the patch passed
+1 💚	compile	0m 39s	the patch passed
+1 💚	javac	0m 39s	the patch passed
+1 💚	shadedjars	3m 47s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	212m 57s	hbase-server in the patch passed.
		229m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4639
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 2104842518a3 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0ae42dd`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/testReport/
Max. process+thread count	2592 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-25T07:10:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 49s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 17s	master passed
+1 💚	compile	1m 9s	master passed
+1 💚	shadedjars	4m 55s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 33s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 43s	the patch passed
+1 💚	compile	1m 6s	the patch passed
+1 💚	javac	1m 6s	the patch passed
+1 💚	shadedjars	4m 40s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 29s	the patch passed
		_ Other Tests _
-1 ❌	unit	305m 36s	hbase-server in the patch failed.
		330m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4639
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 1754aff4332a 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0ae42dd`
Default Java	AdoptOpenJDK-11.0.10+9
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/testReport/
Max. process+thread count	2673 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

#4639) Signed-off-by: Andrew Purtell <[email protected]>

… evictions by hfile (apache#4639) Signed-off-by: Andrew Purtell <[email protected]>

apache#4639) Signed-off-by: Andrew Purtell <[email protected]> (cherry picked from commit 25136c3) Change-Id: I44ade79a5d6011e86c8c3f7dff9ecef72dbad3bb

HBASE-27229 BucketCache statistics should not count evictions by hfile

a8d72e9

bbeaudreault force-pushed the HBASE-27229 branch from 81e150d to a8d72e9 Compare July 21, 2022 00:49

apurtell approved these changes Jul 26, 2022

View reviewed changes

bbeaudreault merged commit 277058e into apache:master Jul 26, 2022

bbeaudreault deleted the HBASE-27229 branch July 26, 2022 21:00

bbeaudreault added a commit that referenced this pull request Jul 26, 2022

HBASE-27229 BucketCache statistics should not count evictions by hfile (

827c1ab

#4639) Signed-off-by: Andrew Purtell <[email protected]>

bbeaudreault added a commit that referenced this pull request Jul 26, 2022

HBASE-27229 BucketCache statistics should not count evictions by hfile (

25136c3

#4639) Signed-off-by: Andrew Purtell <[email protected]>

bbeaudreault added a commit to HubSpot/hbase that referenced this pull request Jul 28, 2022

HubSpot Backport: HBASE-27229 BucketCache statistics should not count…

c6e14f9

… evictions by hfile (apache#4639) Signed-off-by: Andrew Purtell <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-27229 BucketCache statistics should not count evictions by hfile #4639

HBASE-27229 BucketCache statistics should not count evictions by hfile #4639

Uh oh!

bbeaudreault commented Jul 21, 2022 •

edited

Loading

Uh oh!

Apache-HBase commented Jul 21, 2022

Uh oh!

Apache-HBase commented Jul 21, 2022

Uh oh!

Apache-HBase commented Jul 21, 2022

Uh oh!

Apache-HBase commented Jul 21, 2022

Uh oh!

Apache-HBase commented Jul 25, 2022

Uh oh!

Apache-HBase commented Jul 25, 2022

Uh oh!

Apache-HBase commented Jul 25, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HBASE-27229 BucketCache statistics should not count evictions by hfile #4639

HBASE-27229 BucketCache statistics should not count evictions by hfile #4639

Uh oh!

Conversation

bbeaudreault commented Jul 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Apache-HBase commented Jul 21, 2022

Uh oh!

Apache-HBase commented Jul 21, 2022

Uh oh!

Apache-HBase commented Jul 21, 2022

Uh oh!

Apache-HBase commented Jul 21, 2022

Uh oh!

Apache-HBase commented Jul 25, 2022

Uh oh!

Apache-HBase commented Jul 25, 2022

Uh oh!

Apache-HBase commented Jul 25, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bbeaudreault commented Jul 21, 2022 •

edited

Loading