Skip to content

HBASE-23339 Release scripts should use forwarded gpg-agent #1620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

HBASE-23339 Release scripts should use forwarded gpg-agent #1620

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
efc5b8b
HBASE-23339 release scripts should use host gpg-agent when running in…
busbey Apr 18, 2020
c326dd3
HBASE-23339 addendum fix whitespace and shellcheck complaints.
busbey May 23, 2020
4fb05d5
HBASE-23339 addendum update README to walk through running on GCE wit…
busbey May 23, 2020
335c682
HBASE-23339 wait to set the local user until we have gotten it from r…
busbey Jun 6, 2020
566d847
HBASE-23339 include help when gpg test fails and example for launchin…
busbey Jun 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 69 additions & 13 deletions dev-support/create-release/README.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,29 @@ anomalies are explained up in JIRA.

See http://hbase.apache.org/book.html#maven.release

Regardless of where your release build will run (locally, locally in docker, on a remote machine,
etc) you will need a local gpg-agent with access to your secret keys. A quick way to tell gpg
to clear out state and start a gpg-agent is via the following command phrase:

$ gpgconf --kill all && gpg-connect-agent /bye

Before starting an RC build, make sure your local gpg-agent has configs
to properly handle your credentials, especially if you want to avoid
typing the passphrase to your secret key.

e.g. if you are going to run and step away, best to increase the TTL
on caching the unlocked secret via ~/.gnupg/gpg-agent.conf
# in seconds, e.g. a day
default-cache-ttl 86400
max-cache-ttl 86400

Running a build on GCE is easy enough. Here are some notes if of use.
Create an instance. 4CPU/15G/10G disk seems to work well enough.
Once up, run the below to make your machine fit for RC building:

# Presuming debian-compatible OS
$ sudo apt-get install -y git openjdk-8-jdk maven gnupg gnupg-agent
Copy link
Member

@mattf-apache mattf-apache Jun 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice minimization. The JDK is indeed unneeded on the host, unless the user for some reason chooses to use the -j option. And maven is not needed.

# Install docker
# Presuming debian-compatible OS, do these steps on the VM
# your VM username should be your ASF id, because it will show up in build artifacts.
# Follow the docker install guide: https://docs.docker.com/engine/install/debian/
Copy link
Contributor

@apurtell apurtell Jun 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to release without installing/requiring docker. Is that possible?

Copy link
Contributor Author

@busbey busbey Jun 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, since HBASE-24318 / #1643 you can run do-release.sh instead of do-release-docker.sh.

I believe it's not recommended because of needed build set up. I don't know if we document the steps for doing a non-release build. if we don't I'd rather do that in a different jira.

Copy link
Contributor Author

@busbey busbey Jun 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, I meant a non-docker release build. long day :p

Copy link
Contributor Author

@busbey busbey Jun 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am reasonably certain we provide useful error messages if you are missing build requirements when doing a non-docker release build. I expect probably we only give one error at a time though.

$ sudo apt-get install -y \
apt-transport-https \
ca-certificates \
Expand All @@ -37,15 +53,55 @@ $ sudo add-apt-repository -y \
stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
$ sudo usermod -a -G docker $USERID
# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/
$ sudo usermod -aG docker $USER
# LOGOUT and then LOGIN again so $USERID shows as part of docker group
# Copy up private key for $USERID export from laptop and import on gce.
$ gpg --import stack.duboce.net.asc
$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798
$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
$ export PROJECT="${PROJECT:-hbase}"
$ git clone https://github.com/apache/${PROJECT}.git
$ cd "${PROJECT}"
# Test here by running docker's hello world as your build user
$ docker run hello-world

# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM
# https://wiki.gnupg.org/AgentForwarding
# On the VM find out the location of the gpg agent socket and extra socket
$ gpgconf --list-dir agent-socket
/run/user/1000/gnupg/S.gpg-agent
$ gpgconf --list-dir agent-extra-socket
/run/user/1000/gnupg/S.gpg-agent.extra
# On the VM configure sshd to remove stale sockets
$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'
$ sudo systemctl restart ssh
# logout of the VM

# Do these steps on your local machine.
# make sure gpg-agent is running
$ gpg-connect-agent /bye
# Export your public key and copy it to the VM.
# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP)
$ gpg --export [email protected] > ~/gpg.example.apache.pub
Copy link
Member

@ndimiduk ndimiduk Jun 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

really helpful docs here, thank you.

Copy link
Member

@mattf-apache mattf-apache Jun 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, brilliant! (The added documentation, that is.)

$ scp ~/gpg.example.apache.pub example.gce.host:
# ssh into the VM while forwarding the remote gpg socket locations found above to your local
# gpg-agent's extra socket (this will restrict what commands the remote node is allowed to have
# your agent handle. Note that the gpg guide above can help you set this up in your ssh config
# rather than typing it in ssh like this every time.
$ ssh -i ~/.ssh/my_id \
-R "/run/user/1000/gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \
-R "/run/user/1000/gnupg/S.gpg-agent.extra:$(gpgconf --list-dir agent-extra-socket)" \
example.gce.host
Copy link
Member

@mattf-apache mattf-apache Jun 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@busbey , I'm not sure this is an issue, I'm just going by the docs. However:
Does this work correctly if gpg-agent is not already running on my local machine, given that the remote invocation of gpg on line 84 uses --no-autostart?
In other words, should the instructions include something at about line 72 that will guarantee the local gpg-agent is running, such as signing a foo file locally too?

This also would have the advantage of pre-loading the passphrase into the local host's gpg-agent cache, so maybe the RM doesn't have to be alert at the keyboard when signing time comes around.

Copy link
Contributor Author

@busbey busbey Jun 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes your gpg agent has to be running on your local machine before hand. The remote invocations of gpg have to use --no-autostart because if the gpg agent starts on that machine it'll overwrite the socket that we forwarded.

I could add an example of forcing a clean start of the gpg-agent, e.g. gpgconf --kill all && gpg-connect-agent /bye

The test signature at container launch should ensure the key is unlocked prior to us building artifacts.


# now in an SSH session on the VM with the socket forwarding
# import your public key and test signing with the forwarding to your local agent.
$ gpg --no-autostart --import gpg.example.apache.pub
$ echo "foo" > foo.txt
$ gpg --no-autostart --detach --armor --sign foo.txt
$ gpg --no-autostart --verify foo.txt.asc

# install git and clone the main project on the build machine
$ sudo apt-get install -y git
$ git clone https://github.com/apache/hbase.git
# finally set up an output folder and launch a dry run.
$ mkdir ~/build
$ ./dev-resources/create-release/do-release-docker.sh -d ~/build
# etc.
$ cd hbase
$ ./dev-support/create-release/do-release-docker.sh -d ~/build

# for building the main repo specifically you can save an extra download by pointing the build
# to the local clone you just made
$ ./dev-support/create-release/do-release-docker.sh -d ~/build -r .git
Copy link
Member

@ndimiduk ndimiduk Jun 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice! should this be the default ?

Copy link
Contributor Author

@busbey busbey Jun 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dunno. I didn't want to force anyone into using my workflow so the goal was to have the default after these changes stay the same for as much as possible. FWIW I can't think of a reason not to take this approach when releasing the main project repo.

142 changes: 123 additions & 19 deletions dev-support/create-release/do-release-docker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ Options:
-s [step] runs a single step of the process; valid steps are: tag|publish-dist|publish-release.
If none specified, runs tag, then publish-dist, and then publish-release.
'publish-snapshot' is also an allowed, less used, option.
-x debug. do less clean up. (env file, gpg forwarding on mac)
EOF
exit 1
}
Expand All @@ -85,7 +86,7 @@ IMGTAG=latest
JAVA=
RELEASE_STEP=
GIT_REPO=
while getopts "d:fhj:p:r:s:t:" opt; do
while getopts "d:fhj:p:r:s:t:x" opt; do
case $opt in
d) WORKDIR="$OPTARG" ;;
f) DRY_RUN=0 ;;
Expand All @@ -94,6 +95,7 @@ while getopts "d:fhj:p:r:s:t:" opt; do
p) PROJECT="$OPTARG" ;;
r) GIT_REPO="$OPTARG" ;;
s) RELEASE_STEP="$OPTARG" ;;
x) DEBUG=1 ;;
h) usage ;;
?) error "Invalid option. Run with -h for help." ;;
esac
Expand All @@ -102,6 +104,7 @@ shift $((OPTIND-1))
if (( $# > 0 )); then
error "Arguments can only be provided with option flags, invalid args: $*"
fi
export DEBUG

if [ -z "$WORKDIR" ] || [ ! -d "$WORKDIR" ]; then
error "Work directory (-d) must be defined and exist. Run with -h for help."
Expand All @@ -114,12 +117,26 @@ if [ -d "$WORKDIR/output" ]; then
fi
fi

if [ -f "${WORKDIR}/gpg-proxy.ssh.pid" ] || \
[ -f "${WORKDIR}/gpg-proxy.cid" ] || \
[ -f "${WORKDIR}/release.cid" ]; then
read -r -p "container/pid files from prior run exists. Overwrite and continue? [y/n] " ANSWER
if [ "$ANSWER" != "y" ]; then
error "Exiting."
fi
fi

cd "$WORKDIR"
rm -rf "$WORKDIR/output"
rm -rf "${WORKDIR}/gpg-proxy.ssh.pid" "${WORKDIR}/gpg-proxy.cid" "${WORKDIR}/release.cid"
Copy link
Contributor

@saintstack saintstack May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move to a trap/signal handler? Follow-on.

Copy link
Contributor

@saintstack saintstack May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All these clean ups should be in a cleanup handler... Not your issue.

Copy link
Contributor

@saintstack saintstack May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hows this relate to the below nice cleanup function?

Copy link
Contributor Author

@busbey busbey May 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the clean up function should take care of them, but something can go wrong such that they don't get handled.

since docker will fail loudly if the passed container id file path exists, this ensures we have a smooth path back to running again even if something goes wrong.

also the latest version of the WIP has an option to purposefully leave those files in place at the end of execution in case someone needs to debug what the containers are doing.

mkdir "$WORKDIR/output"

banner "Gathering release details."
HOST_OS="$(get_host_os)"
get_release_info

banner "Setup"

# Place all RM scripts and necessary data in a local directory that must be defined in the command
# line. This directory is mounted into the image. Its WORKDIR, the arg passed with -d.
for f in "$SELF"/*; do
Expand All @@ -128,25 +145,65 @@ for f in "$SELF"/*; do
fi
done

GPG_KEY_FILE="$WORKDIR/gpg.key"
# We need to import that public key in the container in order to use the private key via the agent.
GPG_KEY_FILE="$WORKDIR/gpg.key.public"
echo "Exporting public key for ${GPG_KEY}"
fcreate_secure "$GPG_KEY_FILE"
$GPG --passphrase "$GPG_PASSPHRASE" --export-secret-key --armor "$GPG_KEY" > "$GPG_KEY_FILE"
$GPG "${GPG_ARGS[@]}" --export "${GPG_KEY}" > "${GPG_KEY_FILE}"

function cleanup {
local id
banner "Release Cleanup"
if is_debug; then
echo "skipping due to debug run"
return 0
fi
echo "details in cleanup.log"
if [ -f "${ENVFILE}" ]; then
rm -f "$ENVFILE"
fi
rm -f "$GPG_KEY_FILE"
if [ -f "${WORKDIR}/gpg-proxy.ssh.pid" ]; then
id=$(cat "${WORKDIR}/gpg-proxy.ssh.pid")
echo "Stopping ssh tunnel for gpg-agent at PID ${id}" | tee -a cleanup.log
kill -9 "${id}" >>cleanup.log 2>&1 || true
rm -f "${WORKDIR}/gpg-proxy.ssh.pid" >>cleanup.log 2>&1
fi
if [ -f "${WORKDIR}/gpg-proxy.cid" ]; then
id=$(cat "${WORKDIR}/gpg-proxy.cid")
echo "Stopping gpg-proxy container with ID ${id}" | tee -a cleanup.log
docker kill "${id}" >>cleanup.log 2>&1 || true
rm -f "${WORKDIR}/gpg-proxy.cid" >>cleanup.log 2>&1
# TODO we should remove the gpgagent volume?
fi
if [ -f "${WORKDIR}/release.cid" ]; then
id=$(cat "${WORKDIR}/release.cid")
echo "Stopping release container with ID ${id}" | tee -a cleanup.log
docker kill "${id}" >>cleanup.log 2>&1 || true
rm -f "${WORKDIR}/release.cid" >>cleanup.log 2>&1
fi
}

trap cleanup EXIT

echo "Host OS: ${HOST_OS}"
if [ "${HOST_OS}" == "DARWIN" ]; then
run_silent "Building gpg-agent-proxy image with tag ${IMGTAG}..." "docker-proxy-build.log" \
docker build --build-arg "UID=${UID}" --build-arg "RM_USER=${USER}" \
--tag "org.apache.hbase/gpg-agent-proxy:${IMGTAG}" "${SELF}/mac-sshd-gpg-agent"
fi

run_silent "Building hbase-rm image with tag $IMGTAG..." "docker-build.log" \
docker build -t "hbase-rm:$IMGTAG" --build-arg UID=$UID "$SELF/hbase-rm"
docker build --tag "org.apache.hbase/hbase-rm:$IMGTAG" --build-arg "UID=$UID" \
--build-arg "RM_USER=${USER}" "$SELF/hbase-rm"

banner "Final prep for container launch."
echo "Writing out environment for container."
# Write the release information to a file with environment variables to be used when running the
# image.
ENVFILE="$WORKDIR/env.list"
fcreate_secure "$ENVFILE"

function cleanup {
rm -f "$ENVFILE"
rm -f "$GPG_KEY_FILE"
}

trap cleanup EXIT

cat > "$ENVFILE" <<EOF
PROJECT=$PROJECT
DRY_RUN=$DRY_RUN
Expand All @@ -162,15 +219,15 @@ GIT_NAME=$GIT_NAME
GIT_EMAIL=$GIT_EMAIL
GPG_KEY=$GPG_KEY
ASF_PASSWORD=$ASF_PASSWORD
GPG_PASSPHRASE=$GPG_PASSPHRASE
RELEASE_STEP=$RELEASE_STEP
API_DIFF_TAG=$API_DIFF_TAG
HOST_OS=$HOST_OS
EOF

JAVA_VOL=()
JAVA_MOUNT=()
if [ -n "$JAVA" ]; then
echo "JAVA_HOME=/opt/hbase-java" >> "$ENVFILE"
JAVA_VOL=(--volume "$JAVA:/opt/hbase-java")
JAVA_MOUNT=(--mount "type=bind,src=${JAVA},dst=/opt/hbase-java,readonly")
Copy link
Member

@ndimiduk ndimiduk Jun 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we actually need java in the host environment? why not install it in the docker image?

How is it the mounted java version is compatible with the docker environment? If you're running with Mac as the host and the docker image is linux...

Copy link
Contributor Author

@busbey busbey Jun 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do install it in the docker image, that's the default. I have only done my java builds via the docker image java. I just updated this to use the preferred cli options instead of the legacy ones. I presume it was important for someone.

fi

#TODO some debug output would be good here
Expand Down Expand Up @@ -226,14 +283,61 @@ if [ -n "${GIT_REPO}" ]; then
echo "GIT_REPO=${GIT_REPO}" >> "${ENVFILE}"
fi

echo "Building $RELEASE_TAG; output will be at $WORKDIR/output"
GPG_PROXY_MOUNT=()
if [ "${HOST_OS}" == "DARWIN" ]; then
GPG_PROXY_MOUNT=(--mount "type=volume,src=gpgagent,dst=/home/${USER}/.gnupg/")
echo "Setting up GPG agent proxy container needed on OS X."
echo " we should clean this up for you. If that fails the container ID is below and in " \
"gpg-proxy.cid"
#TODO the key pair used should be configurable
docker run --rm -p 62222:22 \
--detach --cidfile "${WORKDIR}/gpg-proxy.cid" \
--mount \
"type=bind,src=${HOME}/.ssh/id_rsa.pub,dst=/home/${USER}/.ssh/authorized_keys,readonly" \
"${GPG_PROXY_MOUNT[@]}" \
"org.apache.hbase/gpg-agent-proxy:${IMGTAG}"
# gotta trust the container host
ssh-keyscan -p 62222 localhost 2>/dev/null | sort > "${WORKDIR}/gpg-agent-proxy.ssh-keyscan"
sort "${HOME}/.ssh/known_hosts" | comm -1 -3 - "${WORKDIR}/gpg-agent-proxy.ssh-keyscan" \
> "${WORKDIR}/gpg-agent-proxy.known_hosts"
if [ -s "${WORKDIR}/gpg-agent-proxy.known_hosts" ]; then
echo "Your ssh known_hosts does not include the entries for the gpg-agent proxy container."
echo "The following entry(ies) arre missing:"
sed -e 's/^/ /' "${WORKDIR}/gpg-agent-proxy.known_hosts"
read -r -p "Okay to add these entries to ${HOME}/.ssh/known_hosts? [y/n] " ANSWER
if [ "$ANSWER" != "y" ]; then
error "Exiting."
fi
cat "${WORKDIR}/gpg-agent-proxy.known_hosts" >> "${HOME}/.ssh/known_hosts"
fi
echo "Launching ssh reverse tunnel from the container to gpg agent."
echo " we should clean this up for you. If that fails the PID is in gpg-proxy.ssh.pid"
ssh -p 62222 -R "/home/${USER}/.gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \
-i "${HOME}/.ssh/id_rsa" -N -n localhost >gpg-proxy.ssh.log 2>&1 &
echo $! > "${WORKDIR}/gpg-proxy.ssh.pid"
else
# Note that on linux we always directly mount the gpg agent's extra socket to limit what the
# container can ask the gpg-agent to do.
# When working on a remote linux machine you should be sure to forward both the remote machine's
# agent socket and agent extra socket to your local gpg-agent's extra socket. See the README.txt
# for an example.
GPG_PROXY_MOUNT=(--mount \
"type=bind,src=$(gpgconf --list-dir agent-extra-socket),dst=/home/${USER}/.gnupg/S.gpg-agent")
fi

banner "Building $RELEASE_TAG; output will be at $WORKDIR/output"
echo "We should clean the container up when we are done. If that fails then the container ID " \
"is in release.cid"
echo
# Where possible we specifcy "consistency=delegated" when we do not need host access during the
# build run. On Mac OS X specifically this gets us a big perf improvement.
cmd=(docker run -ti \
cmd=(docker run --rm -ti \
--env-file "$ENVFILE" \
--mount "type=bind,src=${WORKDIR},dst=/opt/hbase-rm,consistency=delegated" \
"${JAVA_VOL[@]}" \
--cidfile "${WORKDIR}/release.cid" \
--mount "type=bind,src=${WORKDIR},dst=/home/${USER}/hbase-rm,consistency=delegated" \
"${JAVA_MOUNT[@]}" \
"${GIT_REPO_MOUNT[@]}" \
"hbase-rm:$IMGTAG")
"${GPG_PROXY_MOUNT[@]}" \
"org.apache.hbase/hbase-rm:$IMGTAG")
echo "${cmd[*]}"
"${cmd[@]}"
Loading