HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation #3991

virajjasani · 2022-02-14T09:47:13Z

Description of PR

As part of RC preparation, we need to identify all git commits that landed on release branch, however their corresponding Jira is either not resolved yet or does not contain expected fixVersions. Only when we have git commits and corresponding Jiras with expected fixVersion resolved, we get all such Jiras included in auto-generated CHANGES.md as per Yetus changelog generator.

Proposal of this Jira is to provide such script that can be useful for all upcoming RC preparations and list down all Jiras where we need manual intervention. This utility script should use Jira API to retrieve individual fields and use git log to loop through commit history.

The script should identify these issues:

commit is reverted as per commit message
commit does not contain Jira number format (e.g. HADOOP-XXXX / HDFS-XXXX etc) in message
Jira does not have expected fixVersion
Jira has expected fixVersion, but it is not yet resolved
Jira has release corresponding fixVersion and is resolved, but no corresponding commit yet found

It can take inputs as:

First commit hash to start excluding commits from history
Fix Version
JIRA Project Name
Path of project's working dir
Jira server url

How was this patch tested?

Locally.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?

…pancies for RC preparation

hadoop-yetus · 2022-02-14T11:14:45Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 30s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 32s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	22m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-1 ❌	blanks	0m 0s	/blanks-tabs.txt	The patch 17 line(s) with tabs.
+1 💚	mvnsite	0m 0s		the patch passed
-1 ❌	pylint	0m 6s	/results-pylint.txt	The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0)
+1 💚	shadedclient	22m 58s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		86m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3991/1/artifact/out/Dockerfile
GITHUB PR	#3991
Optional Tests	dupname asflicense codespell pylint mvnsite markdownlint
uname	Linux ad8bb6491e4a 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 3edf1d3c7395d457c1024f9c3be5c1749a9214d4
Max. process+thread count	593 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3991/1/console
versions	git=2.25.1 maven=3.6.3 pylint=2.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-02-14T12:15:22Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 21s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 51s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	22m 42s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 23s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 3 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-1 ❌	blanks	0m 0s	/blanks-tabs.txt	The patch 17 line(s) with tabs.
+1 💚	mvnsite	0m 0s		the patch passed
-1 ❌	pylint	0m 3s	/results-pylint.txt	The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0)
+1 💚	shadedclient	22m 8s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		86m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3991/2/artifact/out/Dockerfile
GITHUB PR	#3991
Optional Tests	dupname asflicense codespell pylint mvnsite markdownlint
uname	Linux fa64ffdfbeba 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ddaf8e0`
Max. process+thread count	523 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3991/2/console
versions	git=2.25.1 maven=3.6.3 pylint=2.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-02-14T15:51:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	2m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 26s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 55s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	22m 59s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-tabs.txt	The patch 17 line(s) with tabs.
+1 💚	mvnsite	0m 0s		the patch passed
-1 ❌	pylint	0m 4s	/results-pylint.txt	The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	22m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		90m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3991/3/artifact/out/Dockerfile
GITHUB PR	#3991
Optional Tests	dupname asflicense codespell pylint mvnsite markdownlint
uname	Linux a5431b43a778 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a85d41d`
Max. process+thread count	600 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3991/3/console
versions	git=2.25.1 maven=3.6.3 pylint=2.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-02-14T15:51:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 27s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 20s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	22m 39s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-tabs.txt	The patch 17 line(s) with tabs.
+1 💚	mvnsite	0m 0s		the patch passed
-1 ❌	pylint	0m 3s	/results-pylint.txt	The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	22m 27s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		88m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3991/4/artifact/out/Dockerfile
GITHUB PR	#3991
Optional Tests	dupname asflicense codespell pylint mvnsite markdownlint
uname	Linux 024281419d01 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a85d41d`
Max. process+thread count	595 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3991/4/console
versions	git=2.25.1 maven=3.6.3 pylint=2.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

virajjasani · 2022-02-21T07:20:46Z

Thanks for the review @jojochuang

…pancies for RC preparation (#3991) Signed-off-by: Wei-Chiu Chuang <[email protected]> (cherry picked from commit 697e5d4) (cherry picked from commit d763c99)

…pancies for RC preparation (#3991) Signed-off-by: Wei-Chiu Chuang <[email protected]> (cherry picked from commit 697e5d4) (cherry picked from commit d763c99) (cherry picked from commit 9d618bc)

…pancies for RC preparation (#3991) Signed-off-by: Wei-Chiu Chuang <[email protected]> (cherry picked from commit 697e5d4)

…pancies for RC preparation (#3991) Signed-off-by: Wei-Chiu Chuang <[email protected]> (cherry picked from commit 697e5d4) (cherry picked from commit d763c99) (cherry picked from commit 9d618bc)

…pancies for RC preparation (apache#3991) Signed-off-by: Wei-Chiu Chuang <[email protected]> (cherry picked from commit 697e5d4)

…pancies for RC preparation (#3991) Signed-off-by: Wei-Chiu Chuang <[email protected]> (cherry picked from commit 697e5d4)

…pancies for RC preparation (apache#3991) Signed-off-by: Wei-Chiu Chuang <[email protected]>

HADOOP-18125. Utility to identify git commit / Jira fixVersion discre…

ddaf8e0

…pancies for RC preparation

virajjasani force-pushed the HADOOP-18125-trunk branch from 3edf1d3 to ddaf8e0 Compare February 14, 2022 10:47

virajjasani added 2 commits February 14, 2022 19:48

resolve pylint warnings

f90f824

resolve blank eol

a85d41d

jojochuang approved these changes Feb 16, 2022

View reviewed changes

jojochuang merged commit 697e5d4 into apache:trunk Feb 22, 2022

asfgit pushed a commit that referenced this pull request Feb 22, 2022

HADOOP-18125. Utility to identify git commit / Jira fixVersion discre…

d763c99

…pancies for RC preparation (#3991) Signed-off-by: Wei-Chiu Chuang <[email protected]> (cherry picked from commit 697e5d4)

asfgit pushed a commit that referenced this pull request Apr 13, 2022

HADOOP-18125. Utility to identify git commit / Jira fixVersion discre…

53ea32d

…pancies for RC preparation (#3991) Signed-off-by: Wei-Chiu Chuang <[email protected]> (cherry picked from commit 697e5d4)

asfgit pushed a commit that referenced this pull request Apr 14, 2022

HADOOP-18125. Utility to identify git commit / Jira fixVersion discre…

c6b9fcf

…pancies for RC preparation (#3991) Signed-off-by: Wei-Chiu Chuang <[email protected]> (cherry picked from commit 697e5d4)

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HADOOP-18125. Utility to identify git commit / Jira fixVersion discre…

90c69a9

…pancies for RC preparation (apache#3991) Signed-off-by: Wei-Chiu Chuang <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation #3991

HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation #3991

Uh oh!

virajjasani commented Feb 14, 2022

Uh oh!

hadoop-yetus commented Feb 14, 2022

Uh oh!

hadoop-yetus commented Feb 14, 2022

Uh oh!

hadoop-yetus commented Feb 14, 2022

Uh oh!

hadoop-yetus commented Feb 14, 2022

Uh oh!

virajjasani commented Feb 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation #3991

HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation #3991

Uh oh!

Conversation

virajjasani commented Feb 14, 2022

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Feb 14, 2022

Uh oh!

hadoop-yetus commented Feb 14, 2022

Uh oh!

hadoop-yetus commented Feb 14, 2022

Uh oh!

hadoop-yetus commented Feb 14, 2022

Uh oh!

virajjasani commented Feb 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants