HADOOP-18590. Publish SBOM artifacts #5555

dongjoon-hyun · 2023-04-13T20:44:12Z

Description of PR

This is a second try of #5281 with new cyclonedx plugin 2.7.6 and tested with Maven 3.9.1.

This PR aims to publish SBOM artifacts.

https://cwiki.apache.org/confluence/display/COMDEV/SBOM

Here is an article to give some context.

https://www.activestate.com/blog/why-the-us-government-is-mandating-software-bill-of-materials-sbom/

Software Bill of Materials (SBOM) are additional artifacts containing the aggregate of all direct and transitive dependencies of a project. The US Government (based on NIST recommendations) currently accepts only the three most popular SBOM standards as valid, namely: CycloneDX, Software Identification (SWID) tag, Software Package Data Exchange® (SPDX).

This PR uses CycloneDX maven plugin v2.7.6, a lightweight software bill of materials (SBOM) standard designed for use in application security contexts and supply chain component analysis. cyclonedx-maven-plugin 2.7.6 is verified in Apache ORC/Parquet/Avro/Arrow/Spark community as of today.

How was this patch tested?

Manually. For example, hadoop-auth-3.4.0-SNAPSHOT.jar will have hadoop-auth-3.4.0-SNAPSHOT-cyclonedx.xml and hadoop-auth-3.4.0-SNAPSHOT-cyclonedx.json SBOM files additionally.

$ mvn --version
Apache Maven 3.9.1 (2e178502fcdbffc201671fb2537d0cb4b4cc58f8)
Maven home: /opt/homebrew/Cellar/maven/3.9.1/libexec
Java version: 11.0.18, vendor: Apple Inc., runtime: /Library/Java/JavaVirtualMachines/applejdk-11.0.18.10.1.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.3", arch: "aarch64", family: "mac"

$ ls -l ~/.m2/repository/org/apache/hadoop/hadoop-auth/3.4.0-SNAPSHOT
total 1008
-rw-r--r--  1 dongjoon  staff     373 Apr 13 13:43 _remote.repositories
-rw-r--r--  1 dongjoon  staff  100849 Apr 13 13:42 hadoop-auth-3.4.0-SNAPSHOT-cyclonedx.json
-rw-r--r--  1 dongjoon  staff   85972 Apr 13 13:42 hadoop-auth-3.4.0-SNAPSHOT-cyclonedx.xml
-rw-r--r--  1 dongjoon  staff   84455 Apr 13 13:42 hadoop-auth-3.4.0-SNAPSHOT-sources.jar
-rw-r--r--  1 dongjoon  staff  113598 Apr 13 13:42 hadoop-auth-3.4.0-SNAPSHOT-tests.jar
-rw-r--r--  1 dongjoon  staff  106123 Apr 13 13:42 hadoop-auth-3.4.0-SNAPSHOT.jar
-rw-r--r--  1 dongjoon  staff    8246 Apr 13 13:38 hadoop-auth-3.4.0-SNAPSHOT.pom
-rw-r--r--  1 dongjoon  staff    1537 Apr 13 13:43 maven-metadata-local.xml

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

dongjoon-hyun · 2023-04-13T20:45:18Z

cc @steveloughran and @cnauroth

ayushtkn

Should we do this only when -Pdist is active, during normal dev work, no need for this?, we can save time there...

dongjoon-hyun · 2023-04-13T21:03:07Z

Do you have a regular test pipeline with -Pdist, @ayushtkn ?

dongjoon-hyun · 2023-04-13T21:18:15Z

Anyway, I moved it to under dist profile according to your advice, @ayushtkn .

ayushtkn · 2023-04-13T21:35:18Z

Thanx, We don't have a regular test with -Pdist, I thought this was required only during release time, we moved this to -Pdist in hive as well, recently in favour of saving time during normal dev work.

If you want this to be executed for every PR, we can explore adding -Pdist as well during execution, it will increase our cost for every build, may be have github actions for just Pdist or something of that sort

dongjoon-hyun · 2023-04-13T21:56:37Z

Ya, I agree with your decision, @ayushtkn .

dongjoon-hyun · 2023-04-13T21:57:23Z

BTW, do we have a regular SNAPSHOT publishing job? Apache ORC and Apache Spark has a daily SNAPSHOT publishing jobs and SBOM is tested there independently.

Apache ORC Snapsnot: https://github.com/apache/orc/actions/workflows/publish_snapshot.yml
Apache Spark Snapshot: https://github.com/apache/spark/actions/workflows/publish_snapshot.yml

dongjoon-hyun · 2023-04-13T22:18:57Z

Also, cc @sunchao

ayushtkn · 2023-04-13T22:20:20Z

We have one here: https://ci-hadoop.apache.org/view/Hadoop/job/Hadoop-trunk-Commit/

I can add -Pdist here if you say, but we don't check the results of this build :)

We have a post commit github action as well to deploy trunk website, can add -Pdist over there if you want

https://github.com/apache/hadoop/blob/trunk/.github/workflows/website.yml#L47

dongjoon-hyun · 2023-04-13T22:27:12Z

Got it. Thank you for the pointers.

cnauroth

+1 pending CI. @ayushtkn , good idea putting it behind the dist profile. @dongjoon-hyun , thank you for the patch.

dongjoon-hyun · 2023-04-14T00:38:08Z

Thank you, @ayushtkn and @cnauroth !

hadoop-yetus · 2023-04-14T14:31:22Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 7s		trunk passed
+1 💚	compile	23m 7s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	20m 34s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	mvnsite	25m 27s		trunk passed
+1 💚	javadoc	8m 4s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	7m 19s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	shadedclient	143m 37s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	30m 42s		the patch passed
+1 💚	compile	22m 45s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	22m 45s		the patch passed
+1 💚	compile	20m 38s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	javac	20m 38s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	20m 6s		the patch passed
+1 💚	javadoc	8m 12s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	7m 18s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	shadedclient	61m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	781m 8s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 35s		The patch does not generate ASF License warnings.
		1065m 50s

Reason	Tests
Failed junit tests	hadoop.mapreduce.v2.TestMRJobsWithProfiler
	hadoop.mapreduce.v2.TestMRJobs
	hadoop.mapreduce.v2.TestUberAM

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5555/1/artifact/out/Dockerfile
GITHUB PR	#5555
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 7bf65afbc649 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `acae311`
Default Java	Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5555/1/testReport/
Max. process+thread count	3424 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5555/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-04-14T15:19:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 1s		trunk passed
+1 💚	compile	25m 20s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	21m 54s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	mvnsite	26m 31s		trunk passed
+1 💚	javadoc	8m 35s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	7m 26s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	shadedclient	156m 51s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	24m 54s		the patch passed
+1 💚	compile	24m 47s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	24m 47s		the patch passed
+1 💚	compile	21m 48s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	javac	21m 48s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	21m 8s		the patch passed
+1 💚	javadoc	8m 13s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	7m 24s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	shadedclient	60m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	786m 22s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 26s		The patch does not generate ASF License warnings.
		1080m 16s

Reason	Tests
Failed junit tests	hadoop.mapreduce.v2.TestUberAM
	hadoop.mapreduce.v2.TestMRJobsWithProfiler
	hadoop.mapreduce.v2.TestMRJobs
	hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2
	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5555/2/artifact/out/Dockerfile
GITHUB PR	#5555
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux e22c8a3f5e09 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `dbd7022`
Default Java	Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5555/2/testReport/
Max. process+thread count	2616 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5555/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

sunchao

LGTM too

… Hyun.

…ngjoon Hyun.

HADOOP-18590. Publish SBOM artifacts

acae311

ayushtkn reviewed Apr 13, 2023

View reviewed changes

Move to profile dist

dbd7022

cnauroth approved these changes Apr 13, 2023

View reviewed changes

sunchao approved these changes Apr 14, 2023

View reviewed changes

ayushtkn merged commit 0d1b4a3 into apache:trunk Apr 15, 2023

asfgit pushed a commit that referenced this pull request Apr 15, 2023

HADOOP-18590. Publish SBOM artifacts (#5555). Contributed by Dongjoon…

854eddb

… Hyun.

dongjoon-hyun deleted the HADOOP-18590-2 branch May 4, 2023 03:26

ferdelyi pushed a commit to ferdelyi/hadoop that referenced this pull request May 26, 2023

HADOOP-18590. Publish SBOM artifacts (apache#5555). Contributed by Do…

9fef2d6

…ngjoon Hyun.

HADOOP-18590. Publish SBOM artifacts #5555

HADOOP-18590. Publish SBOM artifacts #5555

Uh oh!

Conversation

dongjoon-hyun commented Apr 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

dongjoon-hyun commented Apr 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Apr 13, 2023

Uh oh!

dongjoon-hyun commented Apr 13, 2023

Uh oh!

ayushtkn commented Apr 13, 2023

Uh oh!

dongjoon-hyun commented Apr 13, 2023

Uh oh!

dongjoon-hyun commented Apr 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Apr 13, 2023

Uh oh!

ayushtkn commented Apr 13, 2023

Uh oh!

dongjoon-hyun commented Apr 13, 2023

Uh oh!

cnauroth left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Apr 14, 2023

Uh oh!

hadoop-yetus commented Apr 14, 2023

Uh oh!

hadoop-yetus commented Apr 14, 2023

Uh oh!

sunchao left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dongjoon-hyun commented Apr 13, 2023 •

edited

Loading

dongjoon-hyun commented Apr 13, 2023 •

edited

Loading

dongjoon-hyun commented Apr 13, 2023 •

edited

Loading