Skip to content

HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars #2411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Nov 10, 2020
Merged

HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars #2411

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
640b4a0
HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
sunchao Oct 23, 2020
8f2a490
Add tests
sunchao Oct 26, 2020
e5d64e7
Include dependencies needed by MiniYARNCluster
sunchao Nov 3, 2020
7149aa1
Address comments
sunchao Nov 9, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions hadoop-client-modules/hadoop-client-api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,8 @@
<exclude>org/w3c/dom/**/*</exclude>
<exclude>org/xml/sax/*</exclude>
<exclude>org/xml/sax/**/*</exclude>
<exclude>org/bouncycastle/*</exclude>
<exclude>org/bouncycastle/**/*</exclude>
</excludes>
</relocation>
<relocation>
Expand Down
4 changes: 4 additions & 0 deletions ...doop-client-check-test-invariants/src/test/resources/ensure-jars-have-correct-contents.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ allowed_expr+="|^org/apache/hadoop/"
allowed_expr+="|^META-INF/"
# * whatever under the "webapps" directory; for things shipped by yarn
allowed_expr+="|^webapps/"
# * Resources files used by Hadoop YARN mini cluster
allowed_expr+="|^TERMINAL/"
# * Hadoop's default configuration files, which have the form
# "_module_-default.xml"
allowed_expr+="|^[^-]*-default.xml$"
Expand All @@ -54,6 +56,8 @@ allowed_expr+="|^org.apache.hadoop.application-classloader.properties$"
# * Used by JavaSandboxLinuxContainerRuntime as a default, loaded
# from root, so can't relocate. :(
allowed_expr+="|^java.policy$"
# * Used by javax.annotation
allowed_expr+="|^jndi.properties$"
# * allowing native libraries from rocksdb. Leaving native libraries as it is.
allowed_expr+="|^librocksdbjni-linux32.so"
allowed_expr+="|^librocksdbjni-linux64.so"
Expand Down
21 changes: 21 additions & 0 deletions hadoop-client-modules/hadoop-client-integration-tests/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,27 @@
<artifactId>hadoop-client-minicluster</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk15on</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcpkix-jdk15on</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax.activation</groupId>
<artifactId>activation</artifactId>
<version>1.1.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
Expand Down
10 changes: 10 additions & 0 deletions ...op-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

import org.apache.hadoop.conf.Configuration;

Expand All @@ -43,6 +44,7 @@

import org.apache.hadoop.hdfs.web.WebHdfsTestUtil;
import org.apache.hadoop.hdfs.web.WebHdfsConstants;
import org.apache.hadoop.yarn.server.MiniYARNCluster;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should go straight after line 45 - no gap

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will fix. I was trying to follow the existing style though which separates imports based on package.


/**
* Ensure that we can perform operations against the shaded minicluster
Expand All @@ -54,6 +56,7 @@ public class ITUseMiniCluster {
LoggerFactory.getLogger(ITUseMiniCluster.class);

private MiniDFSCluster cluster;
private MiniYARNCluster yarnCluster;

private static final String TEST_PATH = "/foo/bar/cats/dee";
private static final String FILENAME = "test.file";
Expand All @@ -73,13 +76,20 @@ public void clusterUp() throws IOException {
.numDataNodes(3)
.build();
cluster.waitActive();

conf.set("yarn.scheduler.capacity.root.queues", "default");
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another issue: we currently don't bundle YARN test resources like capacity-scheduler.xml in hadoop-client-minicluster so this is a temporary workaround. I'll open another JIRA for this later.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we leave out all of test/resources to stop log4j files, site configs etc getting onto the classpath of apps downstream -so making it impossible for them to choose their own options

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should include them for test-only purpose? I think hadoop-client-minicluster should always be test scope right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it may be test only, but the other test suite may have its own configs that it wants. If we put them in the test JAR, it becomes near impossible to change. People will hate us (more)

conf.setInt("yarn.scheduler.capacity.root.default.capacity", 100);
yarnCluster = new MiniYARNCluster(getClass().getName(), 1, 1, 1, 1);
yarnCluster.init(conf);
yarnCluster.start();
}

@After
public void clusterDown() {
if (cluster != null) {
cluster.close();
}
IOUtils.cleanupWithLogger(LOG, yarnCluster);
}

@Test
Expand Down
29 changes: 18 additions & 11 deletions hadoop-client-modules/hadoop-client-minicluster/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -678,10 +678,8 @@
<exclude>junit:junit</exclude>
<exclude>com.google.code.findbugs:jsr305</exclude>
<exclude>log4j:log4j</exclude>
<exclude>org.eclipse.jetty.websocket:*</exclude>
<exclude>javax.websocket:javax.websocket-api</exclude>
<exclude>javax.annotation:javax.annotation-api</exclude>
<exclude>org.eclipse.jetty:jetty-jndi</exclude>
<exclude>org.eclipse.jetty.websocket:websocket-common</exclude>
<exclude>org.eclipse.jetty.websocket:websocket-api</exclude>
<!-- We need a filter that matches just those things that are included in the above artiacts -->
<!-- Leave bouncycastle unshaded because it's signed with a special Oracle certificate so it can be a custom JCE security provider -->
<exclude>org.bouncycastle:*</exclude>
Expand Down Expand Up @@ -731,13 +729,6 @@
<exclude>testdata/*</exclude>
</excludes>
</filter>
<!-- Skip terminal html and javascript -->
<filter>
<artifact>org.apache.hadoop:hadoop-yarn-server-nodemanager:*</artifact>
<excludes>
<exclude>TERMINAL/**/*</exclude>
</excludes>
</filter>

<!-- Mockito tries to include its own unrelocated copy of hamcrest. :( -->
<filter>
Expand Down Expand Up @@ -784,6 +775,13 @@
<exclude>about.html</exclude>
</excludes>
</filter>
<filter>
<!-- skip jetty license info already incorporated into LICENSE/NOTICE -->
<artifact>org.eclipse.jetty.websocket:*</artifact>
<excludes>
<exclude>about.html</exclude>
</excludes>
</filter>
<filter>
<artifact>org.apache.hadoop:*</artifact>
<excludes>
Expand Down Expand Up @@ -881,6 +879,8 @@
<exclude>org/w3c/dom/**/*</exclude>
<exclude>org/xml/sax/*</exclude>
<exclude>org/xml/sax/**/*</exclude>
<exclude>org/bouncycastle/*</exclude>
<exclude>org/bouncycastle/**/*</exclude>
</excludes>
</relocation>
<relocation>
Expand Down Expand Up @@ -977,6 +977,13 @@
<exclude>**/pom.xml</exclude>
</excludes>
</relocation>
<relocation>
<pattern>javax/annotation/</pattern>
<shadedPattern>${shaded.dependency.prefix}.javax.websocket.</shadedPattern>
<excludes>
<exclude>**/pom.xml</exclude>
</excludes>
</relocation>
<relocation>
<pattern>jersey/</pattern>
<shadedPattern>${shaded.dependency.prefix}.jersey.</shadedPattern>
Expand Down
2 changes: 2 additions & 0 deletions hadoop-client-modules/hadoop-client-runtime/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,8 @@
<exclude>org/w3c/dom/**/*</exclude>
<exclude>org/xml/sax/*</exclude>
<exclude>org/xml/sax/**/*</exclude>
<exclude>org/bouncycastle/*</exclude>
<exclude>org/bouncycastle/**/*</exclude>
</excludes>
</relocation>
<relocation>
Expand Down