Skip to content

HBASE-22114 Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 #251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

HBASE-22114 Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 #251

wants to merge 1 commit into from

Conversation

@apurtell
Copy link
Contributor

@apurtell apurtell commented May 23, 2019

HBASE-15560 W-TinyLFU based BlockCache (Ben Manes)

@apurtell apurtell requested review from busbey and lhofhansl May 23, 2019 18:39
@apurtell
Copy link
Contributor Author

apurtell commented May 23, 2019

Just opening a PR for this backport. I think we are blocked on issues with precommit handling of the JDK version specific conditional build stuff that could permanently break precommit, but not any issue with this patch itself.

@Apache-HBase

This comment has been minimized.

Sign in to view
busbey
busbey requested changes May 24, 2019
View reviewed changes
Copy link
Contributor

@busbey busbey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I can make time this holiday weekend to work out the qabot things.

will do a more thorough review as a part of that as well.

@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from c928fc6 to ffc4f3e Compare June 2, 2019 18:16
@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from ffc4f3e to a8a5b16 Compare June 3, 2019 18:59
@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from a8a5b16 to f4a176d Compare June 10, 2019 19:31
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from f4a176d to 29613fc Compare June 12, 2019 19:57
@Apache-HBase

This comment has been minimized.

Sign in to view
busbey
busbey approved these changes Jun 13, 2019
View reviewed changes
Copy link
Contributor

@busbey busbey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

neat, pushing my updates worked.

the changes to hbase-personality will need to go to all branches. this should fix most of the complaints from qa bot. it won't fix the xml thing (it's a long standing issue on jdk7 branches), and I haven't dug into what's up with the unit tests that failed.

Apache-HBase
Apache-HBase reviewed Jun 13, 2019
View reviewed changes
dev-support/hbase-personality.sh Outdated
# on branches-1, the hbase-tinylfu-blockcache only works for jdk8. so if it changes
# run whatever test is requested at the top level so that maven can handle only including
# the module when the correct jdk is around.
if [[ "${MODULES[*]}" =~ hbase-tinylfu-blockcache ]]; then
Copy link

@Apache-HBase Apache-HBase Jun 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shellcheck:25: warning: Did you forget to close this double quoted string? [SC1078]

@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from e27842f to a57a936 Compare June 18, 2019 17:04
@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from a57a936 to fb4513b Compare June 26, 2019 01:30
@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from fb4513b to 9abf0b8 Compare July 11, 2019 01:32
@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from 9abf0b8 to b7d6af8 Compare July 13, 2019 01:26
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache9
Copy link
Contributor

Apache9 commented Jul 25, 2019

Let's finish this? @apurtell

@busbey
Copy link
Contributor

busbey commented Jul 25, 2019

Was this waiting on me and handling the jdk7/jdk8 bit? If so, sorry been swamped. I can block out some time this week to give it another go.

@saintstack
Copy link
Contributor

saintstack commented Sep 10, 2020

@apurtell close?

@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from 785829d to e5c0921 Compare September 23, 2020 01:01
@Apache-HBase

This comment has been minimized.

Sign in to view
saintstack
saintstack reviewed Oct 6, 2020
View reviewed changes
Copy link
Contributor

@saintstack saintstack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we close out this PR since it has been hanging out a while @apurtell ? Thanks.

@saintstack
Copy link
Contributor

saintstack commented Oct 6, 2020

Oh, had a push 13 days ago so not so old....

@Apache-HBase

This comment has been minimized.

Sign in to view
@busbey
Copy link
Contributor

busbey commented Oct 16, 2020

I would really like to not close this out. Iirc it's just waiting on an update to our CI tooling to properly handle jdk dependent modules. That's debt that has a hard time finding help, to be sure.

@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from e5c0921 to a263f0b Compare November 3, 2020 20:15
@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from a263f0b to 4daa600 Compare November 25, 2020 22:35
@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from 4daa600 to 06b3055 Compare December 4, 2020 19:12
@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from 06b3055 to b70c26b Compare April 8, 2021 17:51
@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from b70c26b to 2028e56 Compare May 2, 2021 21:32
@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from 2028e56 to aa17212 Compare June 20, 2021 17:40
@Apache-HBase

This comment has been minimized.

Sign in to view
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from aa17212 to 4be8fd0 Compare June 22, 2021 01:04
@Apache-HBase

This comment has been minimized.

Sign in to view
@busbey busbey added the backport This PR is a back port of some issue or issues already committed to master label Jun 24, 2021
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from 4be8fd0 to c1a597b Compare November 3, 2021 19:06
@Apache-HBase
Copy link

Apache-HBase commented Nov 3, 2021

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 4m 58s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
-1 ❌ hbaseanti 0m 0s The patch appears use Hadoop classification instead of HBase.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ branch-1 Compile Tests _
+0 🆗 mvndep 2m 54s Maven dependency ordering for branch
+1 💚 mvninstall 8m 12s branch-1 passed
+1 💚 compile 2m 11s branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 compile 1m 57s branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 checkstyle 7m 42s branch-1 passed
+0 🆗 refguide 3m 55s branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 shadedjars 3m 13s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 3m 37s branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 javadoc 4m 16s branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+0 🆗 spotbugs 2m 50s Used deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗 findbugs 0m 27s branch/hbase-resource-bundle no findbugs output file (findbugsXml.xml)
_ Patch Compile Tests _
+0 🆗 mvndep 0m 26s Maven dependency ordering for patch
+1 💚 mvninstall 2m 9s the patch passed
-1 ❌ compile 0m 10s root in the patch failed with JDK Azul Systems, Inc.-1.8.0_262-b19.
-1 ❌ javac 0m 10s root in the patch failed with JDK Azul Systems, Inc.-1.8.0_262-b19.
+1 💚 compile 2m 0s the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 javac 2m 0s the patch passed
+1 💚 checkstyle 4m 59s root: The patch generated 0 new + 83 unchanged - 11 fixed = 83 total (was 94)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
-1 ❌ xml 0m 1s The patch has 5 ill-formed XML file(s).
+0 🆗 refguide 2m 59s patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 shadedjars 3m 2s patch has no errors when building our shaded downstream artifacts.
+1 💚 hadoopcheck 5m 12s Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
-1 ❌ javadoc 0m 10s root in the patch failed with JDK Azul Systems, Inc.-1.8.0_262-b19.
+1 💚 javadoc 2m 44s the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+0 🆗 findbugs 0m 14s hbase-resource-bundle has no data from findbugs
-1 ❌ findbugs 0m 10s hbase-tinylfu-blockcache in the patch failed.
_ Other Tests _
-1 ❌ unit 147m 55s root in the patch failed.
+1 💚 asflicense 0m 39s The patch does not generate ASF License warnings.
249m 43s
Reason Tests
XML Parsing Error(s):
hbase-common/src/main/resources/hbase-default.xml
hbase-it/pom.xml
hbase-resource-bundle/src/main/resources/supplemental-models.xml
hbase-tinylfu-blockcache/pom.xml
pom.xml
Failed junit tests hadoop.hbase.replication.TestReplicationSource
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/13/artifact/out/Dockerfile
GITHUB PR #251
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide xml
uname Linux 0ff5ffaccee3 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-home/workspace/HBase-PreCommit-GitHub-PR_PR-251/out/precommit/personality/provided.sh
git revision branch-1 / 40b4cb1
Default Java Azul Systems, Inc.-1.7.0_272-b10
Multi-JDK versions /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10
refguide https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/13/artifact/out/branch-site/book.html
compile https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/13/artifact/out/patch-compile-root-jdkAzulSystems,Inc.-1.8.0_262-b19.txt
javac https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/13/artifact/out/patch-compile-root-jdkAzulSystems,Inc.-1.8.0_262-b19.txt
xml https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/13/artifact/out/xml.txt
refguide https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/13/artifact/out/patch-site/book.html
javadoc https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/13/artifact/out/patch-javadoc-root-jdkAzulSystems,Inc.-1.8.0_262-b19.txt
findbugs https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/13/artifact/out/patch-findbugs-hbase-tinylfu-blockcache.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/13/artifact/out/patch-unit-root.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/13/testReport/
Max. process+thread count 4453 (vs. ulimit of 10000)
modules C: hbase-common hbase-resource-bundle hbase-server hbase-it . hbase-tinylfu-blockcache U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/13/console
versions git=2.17.1 maven=3.6.0 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@busbey
Copy link
Contributor

busbey commented Nov 4, 2021

FYI stack I'll be taking a look at getting this PR merged over the weekend.

@apurtell
HBASE-22114 Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
0a0ff9f 
HBASE-15560 W-TinyLFU based BlockCache (Ben Manes)
@apurtell apurtell force-pushed the HBASE-22114-branch-1 branch from c1a597b to 0a0ff9f Compare December 7, 2021 00:05
@Apache-HBase
Copy link

Apache-HBase commented Dec 7, 2021

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 5m 3s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
-1 ❌ hbaseanti 0m 0s The patch appears use Hadoop classification instead of HBase.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ branch-1 Compile Tests _
+0 🆗 mvndep 2m 55s Maven dependency ordering for branch
+1 💚 mvninstall 8m 30s branch-1 passed
+1 💚 compile 2m 14s branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 compile 2m 0s branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 checkstyle 7m 48s branch-1 passed
+0 🆗 refguide 4m 11s branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 shadedjars 4m 19s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 4m 26s branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 javadoc 5m 20s branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+0 🆗 spotbugs 3m 41s Used deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗 findbugs 0m 29s branch/hbase-resource-bundle no findbugs output file (findbugsXml.xml)
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 mvninstall 2m 49s the patch passed
-1 ❌ compile 0m 11s root in the patch failed with JDK Azul Systems, Inc.-1.8.0_262-b19.
-1 ❌ javac 0m 11s root in the patch failed with JDK Azul Systems, Inc.-1.8.0_262-b19.
+1 💚 compile 2m 48s the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 javac 2m 48s the patch passed
+1 💚 checkstyle 6m 54s root: The patch generated 0 new + 83 unchanged - 11 fixed = 83 total (was 94)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
-1 ❌ xml 0m 1s The patch has 5 ill-formed XML file(s).
+0 🆗 refguide 4m 11s patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 shadedjars 4m 8s patch has no errors when building our shaded downstream artifacts.
+1 💚 hadoopcheck 7m 1s Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
-1 ❌ javadoc 0m 10s root in the patch failed with JDK Azul Systems, Inc.-1.8.0_262-b19.
+1 💚 javadoc 3m 26s the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+0 🆗 findbugs 0m 16s hbase-resource-bundle has no data from findbugs
-1 ❌ findbugs 0m 10s hbase-tinylfu-blockcache in the patch failed.
_ Other Tests _
-1 ❌ unit 156m 35s root in the patch failed.
+1 💚 asflicense 0m 39s The patch does not generate ASF License warnings.
281m 29s
Reason Tests
XML Parsing Error(s):
hbase-common/src/main/resources/hbase-default.xml
hbase-it/pom.xml
hbase-resource-bundle/src/main/resources/supplemental-models.xml
hbase-tinylfu-blockcache/pom.xml
pom.xml
Failed junit tests hadoop.hbase.replication.TestReplicationSource
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/14/artifact/out/Dockerfile
GITHUB PR #251
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide xml
uname Linux 1cda19d72a10 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-home/workspace/HBase-PreCommit-GitHub-PR_PR-251/out/precommit/personality/provided.sh
git revision branch-1 / 8d96fc3
Default Java Azul Systems, Inc.-1.7.0_272-b10
Multi-JDK versions /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10
refguide https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/14/artifact/out/branch-site/book.html
compile https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/14/artifact/out/patch-compile-root-jdkAzulSystems,Inc.-1.8.0_262-b19.txt
javac https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/14/artifact/out/patch-compile-root-jdkAzulSystems,Inc.-1.8.0_262-b19.txt
xml https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/14/artifact/out/xml.txt
refguide https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/14/artifact/out/patch-site/book.html
javadoc https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/14/artifact/out/patch-javadoc-root-jdkAzulSystems,Inc.-1.8.0_262-b19.txt
findbugs https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/14/artifact/out/patch-findbugs-hbase-tinylfu-blockcache.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/14/artifact/out/patch-unit-root.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/14/testReport/
Max. process+thread count 5029 (vs. ulimit of 10000)
modules C: hbase-common hbase-resource-bundle hbase-server hbase-it . hbase-tinylfu-blockcache U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/14/console
versions git=2.17.1 maven=3.6.0 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9
Copy link
Contributor

Apache9 commented Jan 17, 2022

Any progress here?

@Apache9 Apache9 closed this Aug 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@busbey busbey busbey approved these changes

@saintstack saintstack saintstack left review comments

@virajjasani virajjasani virajjasani approved these changes

@lhofhansl lhofhansl Awaiting requested review from lhofhansl

+1 more reviewer

@Apache-HBase Apache-HBase Apache-HBase left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

backport This PR is a back port of some issue or issues already committed to master

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@apurtell @Apache-HBase @Apache9 @busbey @ndimiduk @saintstack @Reidddddd @virajjasani