HBASE-20902 when WAL sync failed, we should bypass the failed DN that previously used #205

chenxu14 · 2019-04-30T07:39:20Z

during Log rolling, we should bypass the slow DN that the previously Log used.

sync master branch

previously used

Apache-HBase · 2019-04-30T10:50:43Z

...ver/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputHelper.java

+      }
+      if (LOG.isDebugEnabled()) {
+        StringBuilder sb = new StringBuilder("create new output because old wal sync failed, old path is: ");
+    	sb.append(oldPathStr).append(", newPath excludesNodes are :");


whitespace:tabs in line

Apache-HBase · 2019-04-30T10:50:44Z

...ver/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputHelper.java

+      if (LOG.isDebugEnabled()) {
+        StringBuilder sb = new StringBuilder("create new output because old wal sync failed, old path is: ");
+    	sb.append(oldPathStr).append(", newPath excludesNodes are :");
+    	for(DatanodeInfo info : excludesNodes) {


whitespace:tabs in line

Apache-HBase · 2019-04-30T10:50:45Z

...ver/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputHelper.java

+        StringBuilder sb = new StringBuilder("create new output because old wal sync failed, old path is: ");
+    	sb.append(oldPathStr).append(", newPath excludesNodes are :");
+    	for(DatanodeInfo info : excludesNodes) {
+    	  sb.append(info.getInfoAddr()).append(";");


whitespace:tabs in line

Apache-HBase · 2019-04-30T10:50:46Z

...ver/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputHelper.java

+    	sb.append(oldPathStr).append(", newPath excludesNodes are :");
+    	for(DatanodeInfo info : excludesNodes) {
+    	  sb.append(info.getInfoAddr()).append(";");
+    	}


whitespace:tabs in line

Apache-HBase · 2019-04-30T10:50:47Z

...ver/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputHelper.java

+    	for(DatanodeInfo info : excludesNodes) {
+    	  sb.append(info.getInfoAddr()).append(";");
+    	}
+    	LOG.debug(sb.toString());


whitespace:tabs in line

Apache-HBase · 2019-04-30T10:50:49Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	23	Docker mode activated.
		_ Prechecks _
+1	hbaseanti	0	Patch does not have any anti-patterns.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 23 new or modified test files.
		_ master Compile Tests _
0	mvndep	26	Maven dependency ordering for branch
+1	mvninstall	240	master passed
+1	compile	83	master passed
+1	checkstyle	92	master passed
+1	shadedjars	265	branch has no errors when building our shaded downstream artifacts.
+1	findbugs	264	master passed
+1	javadoc	64	master passed
		_ Patch Compile Tests _
0	mvndep	16	Maven dependency ordering for patch
+1	mvninstall	234	the patch passed
+1	compile	86	the patch passed
+1	javac	86	the patch passed
-1	checkstyle	71	hbase-server: The patch generated 14 new + 157 unchanged - 0 fixed = 171 total (was 157)
-1	whitespace	0	The patch 5 line(s) with tabs.
+1	shadedjars	257	patch has no errors when building our shaded downstream artifacts.
+1	hadoopcheck	509	Patch does not cause any errors with Hadoop 2.7.4 or 3.0.0.
+1	findbugs	279	the patch passed
+1	javadoc	63	the patch passed
		_ Other Tests _
+1	unit	27	hbase-hadoop-compat in the patch passed.
+1	unit	36	hbase-hadoop2-compat in the patch passed.
-1	unit	8582	hbase-server in the patch failed.
+1	asflicense	62	The patch does not generate ASF License warnings.
		11424

Reason	Tests
Failed junit tests	hadoop.hbase.replication.TestSerialSyncReplication
	hadoop.hbase.replication.TestSyncReplicationMoreLogsInLocalCopyToRemote
	hadoop.hbase.TestFullLogReconstruction
	hadoop.hbase.replication.TestSyncReplicationActive

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-205/1/artifact/out/Dockerfile
GITHUB PR	#205
Optional Tests	dupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux 1b4b38d5e6a3 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build tool	maven
Personality	/testptch/patchprocess/precommit/personality/provided.sh
git revision	master / `3f40df8`
maven	version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java	1.8.0_181
findbugs	v3.1.11
checkstyle	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-205/1/artifact/out/diff-checkstyle-hbase-server.txt
whitespace	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-205/1/artifact/out/whitespace-tabs.txt
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-205/1/artifact/out/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-205/1/testReport/
Max. process+thread count	4812 (vs. ulimit of 10000)
modules	C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-205/1/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

Apache9 · 2019-05-01T23:56:07Z

Mind explaining the implementation a bit? The patch is so big, and across lots of files, not easy to follow up...

chenxu14 · 2019-05-06T08:37:01Z

Mind explaining the implementation a bit? The patch is so big, and across lots of files, not easy to follow up...

modified WALActionsListener#logRollRequested(boolean tooFewReplicas, boolean syncFailed)
add syncFailed param to identify whether logRoll request was caused by a sync failure,
If so, LogRoller thread will call WAL#rollWriter(boolean force, boolean syncFailed), and pass syncFailed param to it
modified WAL#rollWriter(boolean force, boolean syncFailed) throws FailedLogCloseException, IOException;
If syncFailed is true, pass current WAL’s path to AbstractFSWAL#createWriterInstance(newPath, oldPath) as oldPath,
so FanOutOneBlockAsyncDFSOutputHelper can determine which DN is slow based on the oldPath, and add it to excludesNodes

carp84

Please also fix the checkstyle warnings reported by @Apache-HBase robot.

carp84 · 2019-05-10T05:28:53Z

...ver/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputHelper.java

+      for(LocatedBlock block : namenode.getBlockLocations(oldPathStr, Math.max(0, len - 1), len)
+          .getLocatedBlocks()) {
+        for(DatanodeInfo dn : block.getLocations()) {
+          excludesNodes = ArrayUtils.add(excludesNodes, dn);


Here it always adding nodes into exclude list but never check and remove even after the DN recovers, right? So it's possible that one day all DN nodes are excluded and the OutputStream will fail due to could only be replicated to 0 nodes?

thank you for your review @carp84, there seems to be some difference opinions with HBASE-22301, i will fix the checkstyle first.

So it's possible that one day all DN nodes are excluded

excludesNodes are not a global variable, each FanOutOneBlockAsyncDFSOutput will use different instance, when new FanOutOneBlockAsyncDFSOutput created, it's Initial excludesNodes will be an empty array(code in FanOutOneBlockAsyncDFSOutputHelper#createOutput)

sync master branch

merge master branch

Apache-HBase · 2019-05-14T16:06:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	7	#205 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem	Report/Notes
GITHUB PR	#205
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-205/3/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2019-05-15T12:36:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	6	#205 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem	Report/Notes
GITHUB PR	#205
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-205/4/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

saintstack · 2019-06-24T18:33:13Z

Any updates @chenxu14 ? Thanks.

Apache9 · 2019-07-23T14:19:50Z

Close since the PR is stale. Feel free to reopen.

This reverts commit 70f6120.

chenxu14 and others added 2 commits April 30, 2019 11:32

Merge pull request #1 from apache/master

6ad0a40

sync master branch

HBASE-20902 when WAL sync failed, we should bypass the failed DN that

a39ff69

previously used

chenxu14 changed the title ~~when WAL sync failed, we should bypass the failed DN that previously used~~ HBASE-20902 when WAL sync failed, we should bypass the failed DN that previously used Apr 30, 2019

Apache-HBase reviewed Apr 30, 2019

View reviewed changes

carp84 requested changes May 10, 2019

View reviewed changes

chenxu14 and others added 4 commits May 13, 2019 15:50

Merge pull request #2 from apache/master

a27093a

sync master branch

resolved conflicts with HBASE-22301

35b6876

Merge branch 'HBASE-20902' into master

d055d11

Merge pull request #3 from chenxu14/master

55afae6

merge master branch

do not enable this feature by default

934edcf

Apache9 closed this Jul 23, 2019

ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Oct 14, 2025

Revert "use offstack buildpack for modules (apache#200)" (apache#205)

db85d71

This reverts commit 70f6120.

Uh oh!

HBASE-20902 when WAL sync failed, we should bypass the failed DN that previously used #205

HBASE-20902 when WAL sync failed, we should bypass the failed DN that previously used #205

Uh oh!

Conversation

chenxu14 commented Apr 30, 2019

Uh oh!

Apache-HBase Apr 30, 2019

Choose a reason for hiding this comment

Uh oh!

Apache-HBase Apr 30, 2019

Choose a reason for hiding this comment

Uh oh!

Apache-HBase Apr 30, 2019

Choose a reason for hiding this comment

Uh oh!

Apache-HBase Apr 30, 2019

Choose a reason for hiding this comment

Uh oh!

Apache-HBase Apr 30, 2019

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Apr 30, 2019

Uh oh!

Apache9 commented May 1, 2019

Uh oh!

chenxu14 commented May 6, 2019

Uh oh!

carp84 left a comment

Choose a reason for hiding this comment

Uh oh!

carp84 May 10, 2019

Choose a reason for hiding this comment

Uh oh!

chenxu14 May 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented May 14, 2019

Uh oh!

Apache-HBase commented May 15, 2019

Uh oh!

saintstack commented Jun 24, 2019

Uh oh!

Apache9 commented Jul 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

chenxu14 May 10, 2019 •

edited

Loading