HBASE-23339 Release scripts should use forwarded gpg-agent #1620

busbey · 2020-05-01T03:19:56Z

This gets through full builds now both on a local mac with docker desktop and on a remote GCE debian box.

put together a docker container that can use host gpg-agent forwarded over ssh.
use gpg-agent forwarding container on OS X and directly forward the agent on Linux
clean up the release container on exit
use docker mounts instead of the deprecated volume syntax
use image names within our project namespace
update README for gce builds to include forwarding gpg-agent to VM such that it can be forwarded to docker on the VM.

saintstack

Man. The amount of pain involved getting to here I can only imagine. Masochist!

saintstack · 2020-05-01T17:22:31Z

dev-support/create-release/do-release-docker.sh

+
 cd "$WORKDIR"
 rm -rf "$WORKDIR/output"
+rm -rf "${WORKDIR}/gpg-proxy.ssh.pid" "${WORKDIR}/gpg-proxy.cid" "${WORKDIR}/release.cid"


Move to a trap/signal handler? Follow-on.

All these clean ups should be in a cleanup handler... Not your issue.

Hows this relate to the below nice cleanup function?

so the clean up function should take care of them, but something can go wrong such that they don't get handled.

since docker will fail loudly if the passed container id file path exists, this ensures we have a smooth path back to running again even if something goes wrong.

also the latest version of the WIP has an option to purposefully leave those files in place at the end of execution in case someone needs to debug what the containers are doing.

dev-support/create-release/do-release-docker.sh

dev-support/create-release/release-build.sh

busbey · 2020-05-18T00:12:20Z

sorry for the lag here. missed that I had comments.

busbey · 2020-05-18T00:19:52Z

shellcheck and whitespace complaints should be easy enough to clean up. will get to it after I get PRs up for all the commits on the branch that are not this particular jira.

busbey · 2020-05-23T17:06:55Z

I think this is ready now.

mattf-apache · 2020-06-14T19:59:10Z

If you've set up your gpg agent and unlocked your private key for it then you do not need to give the passphrase during the build. the README goes over increasing the timeout to a day instead of 2 hours.

Nice, I missed that on my earlier read-thru. Thanks.

yes your gpg agent has to be running on your local machine before hand. ...
I could add an example of forcing a clean start of the gpg-agent, e.g. gpgconf --kill all && gpg-connect-agent /bye
The test signature at container launch should ensure the key is unlocked prior to us building artifacts.

I recommend adding that example in the README. I think most people do not have gpg-agent running by default, may not know how to launch it (I was vague on it), and while it is true that the test signature at container launch will catch a missing agent timely, it won't tell the user how to fix the problem. -- So, maybe also have that test sig command catch failure and give an instructive error message. Please?

ndimiduk · 2020-06-15T16:48:15Z

I was able to use this patch over the weekend to build an rc as dry run. all the gpg passthrough worked well for me.

ndimiduk · 2020-06-15T23:14:00Z

@busbey @mattf-apache I'm trying to use this to create an RC for real, using -r .git. Does this mean anything to you?

$ cat ~/build-rc0/output/tag.log
...
21:57:58 [INFO] ------------------------------------------------------------------------
21:57:58 [INFO] BUILD SUCCESS
21:57:58 [INFO] ------------------------------------------------------------------------
21:57:58 [INFO] Total time:  5.609 s
21:57:58 [INFO] Finished at: 2020-06-15T21:57:58Z
21:57:58 [INFO] ------------------------------------------------------------------------
ESC[0m
+ git commit -a -m 'Preparing development version 2.3.1-SNAPSHOT'
error: object directory /home/vagrant/repos/hbase/.git/objects does not exist; check .git/objects/info/alternates.
error: object directory /home/vagrant/repos/hbase/.git/objects does not exist; check .git/objects/info/alternates.

ndimiduk · 2020-06-15T23:44:04Z

Using it without -r doesn't work either.

Cloning into 'hbase'...
fatal: unable to access 'https://ndimiduk:[email protected]/repos/asf/hbase.git/': Port number ended with 'h'

mattf-apache · 2020-06-16T19:31:19Z

@ndimiduk @busbey , both the creation of the -r option and the use of shared objects while cloning local repo, were part of commit a9fefd7f533c of PR #1725 (HBASE-24297 release scripts should be able to use an existing project clone). I didn't test with it, sorry.

Parenthetically, I am confused by the -r .git idiom. If I understand the GIT_REPO usage properly, it needs to point at an actual repository, while .git is the usual name of the git account metadata directory. Why would I point -r at it? The processing code doesn't shed any light for me.

mattf-apache · 2020-06-16T19:46:22Z

Similarly, regarding the failure without -r, I don't have insights. The code in release-utils.sh:git_clone_overwrite() is supposed to url-encode the ASF_PASSWORD, and it was inherited from prior code that worked in-situ before PR #1725 . See https://github.pie.apple.com/IPR/apache-hbase/blob/8ff8e70edfbb50967cf123e6afdd9cc5dcfa4878/dev-support/create-release/release-build.sh#L116-L119

ndimiduk · 2020-06-16T20:26:36Z

Similarly, regarding the failure without -r, I don't have insights. The code in release-utils.sh:git_clone_overwrite() is supposed to url-encode the ASF_PASSWORD, and it was inherited from prior code that worked in-situ before PR #1725 . See https://github.pie.apple.com/IPR/apache-hbase/blob/8ff8e70edfbb50967cf123e6afdd9cc5dcfa4878/dev-support/create-release/release-build.sh#L116-L119

I tracked this down to a character that the library helpfully does not encode by default. See PR#1907.

mattf-apache · 2020-06-16T21:45:17Z

Righto. Nice catch.

busbey · 2020-06-17T03:28:30Z

error: object directory /home/vagrant/repos/hbase/.git/objects does not exist; check .git/objects/info/alternates.

the object directory messages are a side effect of using the shared objects. It shouldn't be listed as an error since git then immediately checks the alternates we provide and finds what it needs.

Parenthetically, I am confused by the -r .git idiom. If I understand the GIT_REPO usage properly, it needs to point at an actual repository, while .git is the usual name of the git account metadata directory. Why would I point -r at it? The processing code doesn't shed any light for me.

the .git metadata directory is itself a repository, albeit one only on the local system. using it for a release candidate means the tag generated will be in that local repository. If the RM wants the tag published somewhere else then they need to push it to the other remote.

mattf-apache · 2020-06-20T21:34:22Z

Thanks for the explanation.

ndimiduk · 2020-06-22T18:53:24Z

the object directory messages are a side effect of using the shared objects. It shouldn't be listed as an error since git then immediately checks the alternates we provide and finds what it needs.

I think the build failed for me due to this error. Will try it again for RC1.

… container. * put together a docker container that can use host gpg-agent forwarded over ssh. * use gpg-agent forwarding container on OS X and directly forward the agent on Linux * clean up the release container on exit * use docker mounts instead of the deprecated volume syntax * use image names within our project namespace

…h gpg-agent forwarding

…elease info

busbey · 2020-06-24T05:36:42Z

force push was just me rebasing to the current HEAD with the changes as they were. the next commit is a minor issue I found while doing non-docker builds.

…g gpg-agent.

busbey · 2020-06-24T07:07:24Z

yes your gpg agent has to be running on your local machine before hand. ...
I could add an example of forcing a clean start of the gpg-agent, e.g. gpgconf --kill all && gpg-connect-agent /bye
The test signature at container launch should ensure the key is unlocked prior to us building artifacts.

I recommend adding that example in the README. I think most people do not have gpg-agent running by default, may not know how to launch it (I was vague on it), and while it is true that the test signature at container launch will catch a missing agent timely, it won't tell the user how to fix the problem. -- So, maybe also have that test sig command catch failure and give an instructive error message. Please?

@mattf-apache latest commit adds in an example in the README and some error handling to point to it.

busbey · 2020-06-24T07:08:41Z

these two additional commits are both minor tweaks. @ndimiduk and @mattf-apache please shout if you'd like to give further feedback. barring that I'll go ahead with merging after the qabot comes back and I get time to squash+merge.

Apache-HBase · 2020-06-24T07:12:55Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for patch
		_ Other Tests _
		1m 46s

Subsystem	Report/Notes
Docker	Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/10/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#1620
Optional Tests
uname	Linux 74f5e59a6074 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b556343`
Max. process+thread count	45 (vs. ulimit of 12500)
modules	C: U:
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/10/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-06-24T07:13:23Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	shelldocs	0m 0s	Shelldocs was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 22s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for patch
+1 💚	hadolint	0m 3s	There were no new hadolint issues.
+1 💚	shellcheck	0m 3s	There were no new shellcheck issues.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
		_ Other Tests _
+0 🆗	asflicense	0m 0s	ASF License check generated no output?
		2m 13s

Subsystem	Report/Notes
Docker	Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/10/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1620
Optional Tests	dupname asflicense shellcheck shelldocs hadolint
uname	Linux 612a9920d620 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b556343`
Max. process+thread count	50 (vs. ulimit of 12500)
modules	C: U:
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/10/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 hadolint=1.17.5-0-g443423c
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-06-24T07:14:43Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 16s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for patch
		_ Other Tests _
		3m 25s

Subsystem	Report/Notes
Docker	Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/10/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#1620
Optional Tests
uname	Linux 24983de37e6a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b556343`
Max. process+thread count	48 (vs. ulimit of 12500)
modules	C: U:
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/10/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

This comment has been minimized.

Sign in to view

saintstack approved these changes May 1, 2020

View reviewed changes

busbey force-pushed the HBASE-23339 branch from 71c1a97 to fef3d92 Compare May 7, 2020 19:03

This comment has been minimized.

Sign in to view

busbey force-pushed the HBASE-23339 branch from fb531d1 to 8fa07be Compare May 17, 2020 00:51

This comment has been minimized.

Sign in to view

busbey force-pushed the HBASE-23339 branch from 8fa07be to 8dbd6e9 Compare May 17, 2020 21:07

This comment has been minimized.

Sign in to view

busbey force-pushed the HBASE-23339 branch from 8dbd6e9 to 6386d52 Compare May 23, 2020 04:27

This comment has been minimized.

Sign in to view

busbey changed the title ~~HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge~~ HBASE-23339 Release scripts should use forwarded gpg-agent May 23, 2020

ndimiduk mentioned this pull request Jun 16, 2020

HBASE-24567 Create release should url-encode all characters when building git uri #1907

Merged

busbey and others added 3 commits June 24, 2020 00:35

HBASE-23339 addendum fix whitespace and shellcheck complaints.

c326dd3

HBASE-23339 addendum update README to walk through running on GCE wit…

4fb05d5

…h gpg-agent forwarding

busbey force-pushed the HBASE-23339 branch from c1ef03d to 4fb05d5 Compare June 24, 2020 05:35

HBASE-23339 wait to set the local user until we have gotten it from r…

335c682

…elease info

This comment has been minimized.

Sign in to view

HBASE-23339 include help when gpg test fails and example for launchin…

566d847

…g gpg-agent.

ndimiduk approved these changes Jun 24, 2020

View reviewed changes

asfgit closed this in 497862b Jun 25, 2020

busbey deleted the HBASE-23339 branch June 25, 2020 07:48

HBASE-23339 Release scripts should use forwarded gpg-agent #1620

HBASE-23339 Release scripts should use forwarded gpg-agent #1620

Uh oh!

Conversation

busbey commented May 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

saintstack left a comment

Choose a reason for hiding this comment

Uh oh!

saintstack May 1, 2020

Choose a reason for hiding this comment

Uh oh!

saintstack May 1, 2020

Choose a reason for hiding this comment

Uh oh!

saintstack May 1, 2020

Choose a reason for hiding this comment

Uh oh!

busbey May 18, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

busbey commented May 18, 2020

Uh oh!

busbey commented May 18, 2020

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

busbey commented May 23, 2020

Uh oh!

mattf-apache commented Jun 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ndimiduk commented Jun 15, 2020

Uh oh!

ndimiduk commented Jun 15, 2020

Uh oh!

ndimiduk commented Jun 15, 2020

Uh oh!

mattf-apache commented Jun 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattf-apache commented Jun 16, 2020

Uh oh!

ndimiduk commented Jun 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattf-apache commented Jun 16, 2020

Uh oh!

busbey commented Jun 17, 2020

Uh oh!

mattf-apache commented Jun 20, 2020

Uh oh!

ndimiduk commented Jun 22, 2020

Uh oh!

busbey commented Jun 24, 2020

Uh oh!

This comment has been minimized.

This comment has been minimized.

busbey commented May 1, 2020 •

edited

Loading

mattf-apache commented Jun 14, 2020 •

edited

Loading

mattf-apache commented Jun 16, 2020 •

edited

Loading

ndimiduk commented Jun 16, 2020 •

edited

Loading