Skip to content

Conversation

@hanishakoneru
Copy link
Contributor

JsonUtils#toJsonStringWithDefaultPrettyPrinter() does not validate the Json String before serializing it which could result in Json Injection.
This method is mostly used along with JsonUtils#toJsonString() by first converting to String and then back to object. This can be avoided by directly writing the object through PrettyPrinter.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 75 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 22 Maven dependency ordering for branch
-1 mvninstall 28 hadoop-ozone in trunk failed.
-1 compile 19 hadoop-ozone in trunk failed.
+1 checkstyle 48 trunk passed
+1 mvnsite 0 trunk passed
-1 shadedclient 91 branch has errors when building and testing our client artifacts.
+1 javadoc 150 trunk passed
0 spotbugs 172 Used deprecated FindBugs config; considering switching to SpotBugs.
-1 findbugs 21 hadoop-ozone in trunk failed.
_ Patch Compile Tests _
0 mvndep 21 Maven dependency ordering for patch
-1 mvninstall 29 hadoop-ozone in the patch failed.
-1 compile 19 hadoop-ozone in the patch failed.
-1 javac 19 hadoop-ozone in the patch failed.
-0 checkstyle 23 hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
-1 shadedclient 28 patch has errors when building and testing our client artifacts.
+1 javadoc 149 the patch passed
-1 findbugs 21 hadoop-ozone in the patch failed.
_ Other Tests _
+1 unit 265 hadoop-hdds in the patch passed.
-1 unit 22 hadoop-ozone in the patch failed.
+1 asflicense 24 The patch does not generate ASF License warnings.
1807
Subsystem Report/Notes
Docker Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/Dockerfile
GITHUB PR #1486
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 414a6d781e05 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 3f223be
Default Java 1.8.0_222
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/branch-compile-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/patch-compile-hadoop-ozone.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/diff-checkstyle-hadoop-hdds.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/patch-findbugs-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/testReport/
Max. process+thread count 483 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/tools hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 38 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 70 Maven dependency ordering for branch
-1 mvninstall 32 hadoop-ozone in trunk failed.
-1 compile 22 hadoop-ozone in trunk failed.
+1 checkstyle 64 trunk passed
+1 mvnsite 0 trunk passed
-1 shadedclient 114 branch has errors when building and testing our client artifacts.
-1 javadoc 46 hadoop-ozone in trunk failed.
0 spotbugs 157 Used deprecated FindBugs config; considering switching to SpotBugs.
-1 findbugs 24 hadoop-ozone in trunk failed.
_ Patch Compile Tests _
0 mvndep 25 Maven dependency ordering for patch
-1 mvninstall 30 hadoop-ozone in the patch failed.
-1 compile 23 hadoop-ozone in the patch failed.
-1 javac 23 hadoop-ozone in the patch failed.
-0 checkstyle 24 hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
-1 shadedclient 30 patch has errors when building and testing our client artifacts.
-1 javadoc 48 hadoop-ozone in the patch failed.
-1 findbugs 23 hadoop-ozone in the patch failed.
_ Other Tests _
+1 unit 211 hadoop-hdds in the patch passed.
-1 unit 27 hadoop-ozone in the patch failed.
+1 asflicense 28 The patch does not generate ASF License warnings.
1745
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/Dockerfile
GITHUB PR #1486
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 17bb1cefdfb0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / d7d6ec8
Default Java 1.8.0_222
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/branch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/patch-compile-hadoop-ozone.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/diff-checkstyle-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/patch-findbugs-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/testReport/
Max. process+thread count 578 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/tools hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 73 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 42 Maven dependency ordering for branch
-1 mvninstall 35 hadoop-hdds in trunk failed.
-1 mvninstall 24 hadoop-ozone in trunk failed.
-1 compile 18 hadoop-hdds in trunk failed.
-1 compile 13 hadoop-ozone in trunk failed.
+1 checkstyle 52 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 993 branch has no errors when building and testing our client artifacts.
-1 javadoc 22 hadoop-hdds in trunk failed.
-1 javadoc 20 hadoop-ozone in trunk failed.
0 spotbugs 1088 Used deprecated FindBugs config; considering switching to SpotBugs.
-1 findbugs 31 hadoop-hdds in trunk failed.
-1 findbugs 16 hadoop-ozone in trunk failed.
_ Patch Compile Tests _
0 mvndep 26 Maven dependency ordering for patch
-1 mvninstall 33 hadoop-hdds in the patch failed.
-1 mvninstall 25 hadoop-ozone in the patch failed.
-1 compile 20 hadoop-hdds in the patch failed.
-1 compile 15 hadoop-ozone in the patch failed.
-1 javac 20 hadoop-hdds in the patch failed.
-1 javac 15 hadoop-ozone in the patch failed.
+1 checkstyle 57 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 785 patch has no errors when building and testing our client artifacts.
-1 javadoc 18 hadoop-hdds in the patch failed.
-1 javadoc 15 hadoop-ozone in the patch failed.
-1 findbugs 28 hadoop-hdds in the patch failed.
-1 findbugs 16 hadoop-ozone in the patch failed.
_ Other Tests _
-1 unit 24 hadoop-hdds in the patch failed.
-1 unit 18 hadoop-ozone in the patch failed.
+1 asflicense 29 The patch does not generate ASF License warnings.
2570
Subsystem Report/Notes
Docker Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/Dockerfile
GITHUB PR #1486
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 885b88adb895 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 66400c1
Default Java 1.8.0_222
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/branch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/branch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/patch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/patch-compile-hadoop-hdds.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/patch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/patch-findbugs-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/testReport/
Max. process+thread count 306 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/tools hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/2/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

System.out.printf("%s%n", JsonUtils.toJsonStringWithDefaultPrettyPrinter(
JsonUtils.toJsonString("Acl set successfully: " + result)));
System.out.printf("%s%n", "Acl set successfully: " +
JsonUtils.toJsonStringWithDefaultPrettyPrinter(result));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here the result is true/false, we can directly print. Do we need toJsonStringWithDefaultPrettyPrinter here? Previously this was called with Acl set successfully: + result. But now just result, so is it okay if we directly use result to print?

Same comment for all AclHandler classes.

Copy link
Contributor

@bharatviswa504 bharatviswa504 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One comment posted inline.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 11 #1486 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
Subsystem Report/Notes
GITHUB PR #1486
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/3/console
versions git=2.17.1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be fixed?

Copy link
Contributor

@bharatviswa504 bharatviswa504 Oct 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be fine, as we are not converting to String and printing. (Same for others too)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't that what toJsonStringWithDefaultPrettyPrinter does - i.e. convert to string. Then we print it out.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, toJsonStringWithDefaultPrettyPrinter converts to string but in Json format (with proper indentations and new line for each element).
Previously, we were converting the object to be printed to string (using JsonUtils.toJsonString(result)), then converting it back into object (inside old toJsonStringWithDefaultPrettyPrinter()) and then again converting it into string.

//Old code
public static String toJsonStringWithDefaultPrettyPrinter(String jsonString)
      throws IOException {
    Object json = READER.readValue(jsonString);
    return WRITTER.writeValueAsString(json);
  }

The Json Injection issue happens when converting the jsonString back to Object without validation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

False means Acl passed is already existing, in this case we return false. So message ACL not set is meaningful here or do we need to reword this?

Copy link
Contributor

@bharatviswa504 bharatviswa504 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. I have one minor comment posted in place.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RemoveAcl returns false, when passedAcl is not existing, then it cannot be removed. Do we need to reword this to convey this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reverted to previous print messages (fixing add/set/remove part only).
But I think that statement also does not convey the message properly. If we are trying to add an already existing ACL, shouldn't we return true? And same for remove also.
We can open a new Jira to work on that if you agree.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my understanding, addAcl behavior is if acl is added successfully returns true, it will return false when acl trying to be added already exists.

If we are trying to add an already existing ACL, shouldn't we return true?

I think returning true is not right behavior, as it will not be clear whether add is successful or not. We should have returned with clear message to end user, what is differenece between true/false.

But I think that statement also does not convey the message properly.

Agreed this was existing behavior, if you want to fix in a new Jira I am okay with that.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 11 #1486 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
Subsystem Report/Notes
GITHUB PR #1486
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/4/console
versions git=2.17.1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@arp7 arp7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@bharatviswa504, look fine to you?

Copy link
Contributor

@bharatviswa504 bharatviswa504 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM.
We can open a new Jira for the discussion.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 10 #1486 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
Subsystem Report/Notes
GITHUB PR #1486
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/5/console
versions git=2.17.1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hanishakoneru
Copy link
Contributor Author

Thanks @bharatviswa504 and @arp7 for the reviews.
Checkstyle issues were related. Fixed that.
Opened HDDS-2255 to for improving the messages.
Will commit the patch after CI run.

@hanishakoneru
Copy link
Contributor Author

Remaining failures are not related to this patch.
Committing it.

@hanishakoneru hanishakoneru merged commit 8de4374 into apache:trunk Oct 4, 2019
amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019
RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants