HADOOP-19038. Improve create-release RUN script. #6448

slfan1989 · 2024-01-13T13:16:40Z

Description of PR

JIRA: HADOOP-19038. Improve create-release RUN script.

Using create-release will create a docker image locally, but three of the RUN scripts may fail to execute.

RUN groupadd --non-unique -g 0 root

=> ERROR [16/20] RUN groupadd --non-unique -g 0 root                                                            0.2s
------
 > [16/20] RUN groupadd --non-unique -g 0 root:
0.154 groupadd: group 'root' already exists
------
Dockerfile:100
--------------------
  98 |
  99 |     LABEL org.apache.hadoop.create-release="cr-19697"
 100 | >>> RUN groupadd --non-unique -g 0 root
 101 |     RUN useradd -g 0 -u 0 -m root
 102 |     RUN chown -R root /home/root

RUN useradd -g 0 -u 0 -m root

 > [17/20] RUN useradd -g 0 -u 0 -m root:
0.165 useradd: user 'root' already exists
------
Dockerfile:101
--------------------
  99 |     LABEL org.apache.hadoop.create-release="cr-12068"
 100 |     RUN groupadd --non-unique -g 0 root; exit 0;
 101 | >>> RUN useradd -g 0 -u 0 -m root
 102 |     RUN chown -R root /home/root
 103 |     ENV HOME /home/root

RUN chown -R root /home/root

 > [18/20] RUN chown -R root /home/root:
0.168 chown: cannot access '/home/root': No such file or directory
------
Dockerfile:102
--------------------
 100 |     RUN groupadd --non-unique -g 0 root; exit 0;
 101 |     RUN useradd -g 0 -u 0 -m root; exit 0;
 102 | >>> RUN chown -R root /home/root
 103 |     ENV HOME /home/root
 104 |     RUN mkdir -p /maven
--------------------

Even if these three scripts fail, subsequent steps can continue to be executed, so I added exit 0 after the script.

After the modification is completed, the script can be executed normally:

> [internal] load .dockerignore                                                                                                                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                            0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                       0.0s
 => => transferring dockerfile: 3.26kB                                                                                                                                                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/ubuntu:focal                                                                                                                                                                                                            1.5s
 => [ 1/20] FROM docker.io/library/ubuntu:focal@sha256:f2034e7195f61334e6caff6ecf2e965f92d11e888309065da85ff50c617732b8                                                                                                                                                    0.0s
 => [internal] load build context                                                                                                                                                                                                                                          0.0s
 => => transferring context: 997B                                                                                                                                                                                                                                          0.0s
 => CACHED [ 2/20] WORKDIR /root                                                                                                                                                                                                                                           0.0s
 => CACHED [ 3/20] RUN echo APT::Install-Recommends "0"; > /etc/apt/apt.conf.d/10disableextras                                                                                                                                                                             0.0s
 => CACHED [ 4/20] RUN echo APT::Install-Suggests "0"; >>  /etc/apt/apt.conf.d/10disableextras                                                                                                                                                                             0.0s
 => CACHED [ 5/20] COPY pkg-resolver pkg-resolver                                                                                                                                                                                                                          0.0s
 => CACHED [ 6/20] RUN chmod a+x pkg-resolver/*.sh pkg-resolver/*.py     && chmod a+r pkg-resolver/*.json                                                                                                                                                                  0.0s
 => CACHED [ 7/20] RUN apt-get -q update     && apt-get -q install -y --no-install-recommends python3     && apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py ubuntu:focal)     && apt-get clean     && rm -rf /var/lib/apt/lists/*                 0.0s
 => CACHED [ 8/20] RUN locale-gen en_US.UTF-8                                                                                                                                                                                                                              0.0s
 => CACHED [ 9/20] RUN pkg-resolver/install-common-pkgs.sh                                                                                                                                                                                                                 0.0s
 => CACHED [10/20] RUN pkg-resolver/install-maven.sh ubuntu:focal                                                                                                                                                                                                          0.0s
 => CACHED [11/20] RUN pkg-resolver/install-spotbugs.sh ubuntu:focal                                                                                                                                                                                                       0.0s
 => CACHED [12/20] RUN pkg-resolver/install-boost.sh ubuntu:focal                                                                                                                                                                                                          0.0s
 => CACHED [13/20] RUN pkg-resolver/install-protobuf.sh ubuntu:focal                                                                                                                                                                                                       0.0s
 => CACHED [14/20] RUN pkg-resolver/install-hadolint.sh ubuntu:focal                                                                                                                                                                                                       0.0s
 => CACHED [15/20] RUN pkg-resolver/install-intel-isa-l.sh ubuntu:focal                                                                                                                                                                                                    0.0s
 => CACHED [16/20] RUN groupadd --non-unique -g 0 root; exit 0;                                                                                                                                                                                                            0.0s
 => CACHED [17/20] RUN useradd -g 0 -u 0 -m root; exit 0;                                                                                                                                                                                                                  0.0s
 => [18/20] RUN chown -R root /home/root; exit 0;                                                                                                                                                                                                                          0.2s
 => [19/20] RUN mkdir -p /maven                                                                                                                                                                                                                                            0.3s
 => [20/20] RUN chown -R root /maven

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2024-01-13T15:10:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	18m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 3s		Maven dependency ordering for branch
+1 💚	shadedclient	38m 22s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	shellcheck	0m 1s		No new issues.
+1 💚	shadedclient	36m 40s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		112m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6448/1/artifact/out/Dockerfile
GITHUB PR	#6448
Optional Tests	dupname asflicense codespell detsecrets shellcheck shelldocs
uname	Linux f57beaab6d95 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e8632c4`
Max. process+thread count	540 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6448/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao

Leave one nit comment inline. Thanks.

Hexiaoqiao · 2024-01-14T10:22:37Z

dev-support/bin/create-release

-    echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"
-    echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}"
-    echo "RUN chown -R ${user_name} /home/${user_name}"
+    echo "RUN groupadd --non-unique -g ${group_id} ${user_name}; exit 0;"


Do we need to check if this command execute successfully or not here?

Thank you for reviewing this PR! Once an error occurs in the RUN script, the image building process is terminated. The error I encountered does not block the overall compilation process, so I added exit 0. I hope that even if there is an error, the subsequent compilation process can continue.

steveloughran

i understand: existence of the root user in the docker image must not be a blocker

slfan1989 · 2024-01-18T06:25:03Z

@Hexiaoqiao Can you help review this PR again? Thank you very much!

Hexiaoqiao

LGTM. +1

slfan1989 · 2024-01-18T11:12:56Z

@steveloughran @Hexiaoqiao Thanks for reviewing the code!

…by Shilun Fan. Reviewed-by: Steve Loughran <[email protected]> Reviewed-by: He Xiaoqiao <[email protected]> Signed-off-by: Shilun Fan <[email protected]>

…buted by Shilun Fan. Reviewed-by: Steve Loughran <[email protected]> Reviewed-by: He Xiaoqiao <[email protected]> Signed-off-by: Shilun Fan <[email protected]>

HADOOP-19038. Improve create-release RUN script.

e8632c4

github-actions bot added trunk Infra labels Jan 13, 2024

Hexiaoqiao reviewed Jan 14, 2024

View reviewed changes

steveloughran approved these changes Jan 17, 2024

View reviewed changes

slfan1989 requested a review from Hexiaoqiao January 18, 2024 06:24

Hexiaoqiao approved these changes Jan 18, 2024

View reviewed changes

slfan1989 merged commit 4c3d4e6 into apache:trunk Jan 18, 2024

slfan1989 mentioned this pull request Feb 3, 2024

HADOOP-19062. [thirdparty] Improve create-release RUN script. apache/hadoop-thirdparty#29

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

HADOOP-19038. Improve create-release RUN script. #6448

HADOOP-19038. Improve create-release RUN script. #6448

Uh oh!

slfan1989 commented Jan 13, 2024 •

edited

Loading

Uh oh!

hadoop-yetus commented Jan 13, 2024

Uh oh!

Hexiaoqiao left a comment

Uh oh!

Hexiaoqiao Jan 14, 2024

Uh oh!

slfan1989 Jan 16, 2024 •

edited

Loading

Uh oh!

steveloughran left a comment

Uh oh!

slfan1989 commented Jan 18, 2024

Uh oh!

Hexiaoqiao left a comment

Uh oh!

slfan1989 commented Jan 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

HADOOP-19038. Improve create-release RUN script. #6448

HADOOP-19038. Improve create-release RUN script. #6448

Uh oh!

Conversation

slfan1989 commented Jan 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Jan 13, 2024

Uh oh!

Hexiaoqiao left a comment

Choose a reason for hiding this comment

Uh oh!

Hexiaoqiao Jan 14, 2024

Choose a reason for hiding this comment

Uh oh!

slfan1989 Jan 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

slfan1989 commented Jan 18, 2024

Uh oh!

Hexiaoqiao left a comment

Choose a reason for hiding this comment

Uh oh!

slfan1989 commented Jan 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

slfan1989 commented Jan 13, 2024 •

edited

Loading

slfan1989 Jan 16, 2024 •

edited

Loading