Skip to content

Forward-port "HBASE-28065 Corrupt HFile data is mishandled in several cases" to branch-2 #5416

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 21, 2023
Merged

Forward-port "HBASE-28065 Corrupt HFile data is mishandled in several cases" to branch-2 #5416

merged 1 commit into from
Sep 21, 2023

Conversation

@ndimiduk
Copy link
Member

  • when no block size is provided and there's not a preread headerBuf, treat the value with caution.
  • verify HBase checksums before making use of the block header.
  • inline verifyOnDiskSizeMatchesHeader to keep throw/return logic in the method body.
  • separate validation of onDiskSizeWithHeader as input parameter from as read from block header
  • simplify branching around fetching and populating onDiskSizeWithHeader.
  • inline retrieving nextOnDiskBlockSize ; add basic validation.
  • whenever a read is determined to be corrupt and fallback to HDFS checksum is necessary, also invalidate the cached value of headerBuf.
  • build out a test suite covering various forms of block header corruption, for blocks in first and second positions.

@ndimiduk
HBASE-28065 Corrupt HFile data is mishandled in several cases
1f0db97 
* when no block size is provided and there's not a preread headerBuf, treat the value with
  caution.
* verify HBase checksums before making use of the block header.
* inline verifyOnDiskSizeMatchesHeader to keep throw/return logic in the method body.
* separate validation of onDiskSizeWithHeader as input parameter from as read from block header
* simplify branching around fetching and populating onDiskSizeWithHeader.
* inline retrieving nextOnDiskBlockSize ; add basic validation.
* whenever a read is determined to be corrupt and fallback to HDFS checksum is necessary, also
  invalidate the cached value of headerBuf.
* build out a test suite covering various forms of block header corruption, for blocks in first
  and second positions.

Signed-off-by: Bryan Beaudreault <[email protected]>
@ndimiduk ndimiduk added the backport This PR is a back port of some issue or issues already committed to master label Sep 20, 2023
@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 44s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ branch-2 Compile Tests _
+1 💚 mvninstall 3m 32s branch-2 passed
+1 💚 compile 3m 6s branch-2 passed
+1 💚 checkstyle 0m 42s branch-2 passed
-1 ❌ spotless 0m 59s branch has 1 errors when running spotless:check, run spotless:apply to fix.
+1 💚 spotbugs 1m 43s branch-2 passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 9s the patch passed
+1 💚 compile 3m 11s the patch passed
+1 💚 javac 3m 11s the patch passed
-0 ⚠️ checkstyle 0m 41s hbase-server: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 16m 54s Patch does not cause any errors with Hadoop 2.10.2 or 3.2.4 3.3.6.
+1 💚 spotless 0m 47s patch has no errors when running spotless:check.
+1 💚 spotbugs 2m 6s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 11s The patch does not generate ASF License warnings.
39m 36s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5416/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #5416
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 364b6013a339 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / 4bb7409
Default Java Eclipse Adoptium-11.0.17+8
spotless https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5416/1/artifact/yetus-general-check/output/branch-spotless.txt
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5416/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count 76 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5416/1/console
versions git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 42s Docker mode activated.
-0 ⚠️ yetus 0m 4s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚 mvninstall 2m 49s branch-2 passed
+1 💚 compile 0m 41s branch-2 passed
+1 💚 shadedjars 4m 57s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 23s branch-2 passed
_ Patch Compile Tests _
+1 💚 mvninstall 2m 28s the patch passed
+1 💚 compile 0m 41s the patch passed
+1 💚 javac 0m 41s the patch passed
+1 💚 shadedjars 4m 54s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 23s the patch passed
_ Other Tests _
+1 💚 unit 208m 50s hbase-server in the patch passed.
231m 1s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5416/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #5416
Optional Tests javac javadoc unit shadedjars compile
uname Linux bfbb75bd21eb 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / 4bb7409
Default Java Eclipse Adoptium-11.0.17+8
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5416/1/testReport/
Max. process+thread count 4999 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5416/1/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 24s Docker mode activated.
-0 ⚠️ yetus 0m 4s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚 mvninstall 2m 48s branch-2 passed
+1 💚 compile 0m 46s branch-2 passed
+1 💚 shadedjars 5m 1s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 26s branch-2 passed
_ Patch Compile Tests _
+1 💚 mvninstall 2m 38s the patch passed
+1 💚 compile 0m 47s the patch passed
+1 💚 javac 0m 47s the patch passed
+1 💚 shadedjars 4m 55s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 28s the patch passed
_ Other Tests _
-1 ❌ unit 273m 5s hbase-server in the patch failed.
296m 26s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5416/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR #5416
Optional Tests javac javadoc unit shadedjars compile
uname Linux 7307d3ab4a24 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / 4bb7409
Default Java Temurin-1.8.0_352-b08
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5416/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5416/1/testReport/
Max. process+thread count 4601 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5416/1/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@ndimiduk ndimiduk merged commit 3ddadcf into apache:branch-2 Sep 21, 2023
@ndimiduk ndimiduk deleted the 28065-hfile-corrupt-reads-branch-2 branch September 21, 2023 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backport This PR is a back port of some issue or issues already committed to master

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ndimiduk @Apache-HBase