HDDS-1510. Classpath files are deployed to the maven repository as pom/jar files

[elek]

1. Classpath files are plain text files which are generatede for each ozone projects. Classpath files are used to defined the classpath of a module (om, scm, etc) based on the maven classpath.

Example classpath file:

{code}
classpath=$HDDS_LIB_JARS_DIR/kerb-simplekdc-1.0.1.jar:$HDDS_LIB_JARS_DIR/hk2-utils-2.5.0.jar:$HDDS_LIB_JARS_DIR/jackson-core-2.9.5.jar:$HDDS_LIB_JARS_DIR/ratis-netty-0.4.0-fe2b15d-SNAPSHOT.jar:$HDDS_LIB_JARS_DIR/protobuf-java-2.5.0.jar:...
{code}

Classpath files are maven artifacts and copied to share/ozone/classpath in the distribution

2. 0.4.0 was the first release when we deployed the artifacts to the apache nexus. [~ajayydv] reported the problem that the staging repository can't be closed: INFRA-18344

It turned out that the classpath files are uploaded with jar extension to the repository. We deleted all the classpath files manually and the repository became closable.

To avoid similar issues we need to fix this problem and make sure that the classpath files are not uploaded to the repository during a 'mvn deploy' or uploaded but with a good extension.

ps: I don't know the exact solution yet, but I can imagine that bumping the version of maven deploy plugin can help. Seems to be a bug in the plugin.

ps2: This is blocker as we need to fix it before the next release

See: https://issues.apache.org/jira/browse/HDDS-1510

[elek]

Every file can be an "artifact" in the maven word if they are marked as artifact. We use custom classpath descriptors to generate the exact classpath and they are installed to the local maven repo + deployed to the maven repository during release.

This is a problem in the build-classpath goal of the maven-depedency-plugin which is used to generate the classpath file. With using the <attach>true</attach> configuration we ask the plugin to mark the file as an uploadable artifact.

Unfortunately the plugin doesn't set the type of the artifact and the default is a jar. As a result the classpath file (text) is uploaded as a jar file, and nexus validation is failed because these fake jars.

See the output of a normal mvn install

[INFO] --- maven-install-plugin:2.5.1:install (default-install) @ hadoop-hdds-common ---
...
[INFO] Installing /home/elek/projects/hadoop-review/hadoop-hdds/common/target/classpath to /home/elek/.m2/repository/org/apache/hadoop/hadoop-hdds-common/0.5.0-SNAPSHOT/hadoop-hdds-common-0.5.0-SNAPSHOT-classpath.jar

But the classpath file is a text file:

head  /home/elek/projects/hadoop-review/hadoop-hdds/common/target/classpath to /home/elek/.m2/repository/org/apache/hadoop/hadoop-hdds-common/0.5.0-SNAPSHOT/hadoop-hdds-common-0.5.0-SNAPSHOT-classpath.jar
classpath=$HDDS_LIB_JARS_DIR/kerb-simplekdc-1.0.1.jar:$HDDS_LIB_JARS_DIR/jackson-core-2.9.5.jar:$HDDS_LIB_JARS_DIR/ratis-netty-0.4.0-fe2b15d-SNAPSHOT.jar:$HDDS_LIB_JARS_DIR/protobuf-java-2.5.0.jar:$HDDS_LIB_JARS_DIR/bcpkix-jdk15on-1.60.jar
...

One easy workaround is to mark the classpath file as an uploadable artifact with a separated plugin which has more flexibility (maven-build-helper-plugin).

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	41	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	23	Maven dependency ordering for branch
+1	mvninstall	593	trunk passed
+1	compile	320	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	1798	branch has no errors when building and testing our client artifacts.
+1	javadoc	167	trunk passed
		_ Patch Compile Tests _
0	mvndep	30	Maven dependency ordering for patch
+1	mvninstall	484	the patch passed
+1	compile	288	the patch passed
+1	javac	288	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	3	The patch has no ill-formed XML file.
+1	shadedclient	750	patch has no errors when building and testing our client artifacts.
+1	javadoc	184	the patch passed
		_ Other Tests _
-1	unit	271	hadoop-hdds in the patch failed.
-1	unit	1406	hadoop-ozone in the patch failed.
+1	asflicense	48	The patch does not generate ASF License warnings.
		5674

Reason	Tests
Failed junit tests	hadoop.hdds.scm.block.TestBlockManager
	hadoop.ozone.client.rpc.TestBCSID
	hadoop.ozone.client.rpc.TestWatchForCommit
	hadoop.ozone.client.rpc.TestCommitWatcher

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-900/1/artifact/out/Dockerfile
GITHUB PR	#900
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux 3ddbc142a27d 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 7991159
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-900/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-900/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-900/1/testReport/
Max. process+thread count	4840 (vs. ulimit of 5500)
modules	C: hadoop-hdds hadoop-ozone hadoop-ozone/dist U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-900/1/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[anuengineer]

+1. LGTM.